57870 LESSONS FROM A REVIEW OF INTERVENTIONS TO REDUCE CHILD MALNUTRITION IN DEVELOPING COUNTRIES The World Bank Group WORKING FOR A WORLD FREE OF POVERTY T he World Bank Group consists of five institutions-- the International Bank for Reconstruction and De- velopment (IBRD), the International Finance Corporation (IFC), the International Development Association (IDA), the Multilateral Investment Guarantee Agency (MIGA), and the International Centre for the Settlement of Invest- ment Disputes (ICSID). Its mission is to fight poverty for lasting results and to help people help themselves and their environment by providing resources, sharing knowl- edge, building capacity, and forging partnerships in the public and private sectors. The Independent Evaluation Group IMPROVING DEVELOPMENT RESULTS THROUGH EXCELLENCE IN EVALUATION T he Independent Evaluation Group (IEG) is an indepen- dent, three-part unit within the World Bank Group. IEG-World Bank is charged with evaluating the activities of the IBRD (The World Bank) and IDA, IEG-IFC focuses on assessment of IFC's work toward private sector develop- ment, and IEG-MIGA evaluates the contributions of MIGA guarantee projects and services. IEG reports directly to the Bank's Board of Directors through the Director-General, Evaluation. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank Group's work, and to provide accountability in the achievement of its objectives. It also improves Bank Group work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. LESSONS FROM A REVIEW OF INTERVENTIONS TO REDUCE CHILD MALNUTRITION IN DEVELOPING COUNTRIES W h a t Ca n We Le a r n f ro m Nutrition Impact Evaluations? 2010 The World Bank Washington, D.C. Executive Summary | i Copyright © 2010 The International Bank for Reconstruction and Development/The World Bank 1818 H Street, N.W. Washington, D.C. 20433 Telephone: 202-473-1000 Internet: www.worldbank.org E-mail: feedback@worldbank.org All rights reserved 1 2 3 4 13 12 11 10 This volume is a product of the staff of the International Bank for Reconstruction and Development/The World Bank. The findings, interpretations, and conclusions expressed in this volume do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. This volume does not support any general inferences beyond the scope of the evaluation, including any inferences about the World Bank Group's past, current, or prospective overall performance. The World Bank Group does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgement on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this publication is copyrighted. Copying and/or transmitting portions or all of this work without permission may be a violation of applicable law. The International Bank for Reconstruction and Development/The World Bank encourages dissemination of its work and will normally grant permission to reproduce portions of the work promptly. For permission to photocopy or reprint any part of this work, please send a request with complete information to the Copyright Clearance Center Inc., 222 Rosewood Drive, Danvers, MA 01923, USA; telephone: 978-750-8400; fax: 978-750- 4470; Internet: www.copyright.com. All other queries on rights and licenses, including subsidiary rights, should be addressed to the Office of the Publisher, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2422; e-mail: pubrights@worldbank.org. Cover: Girl being weighed during a routine checkup at the Santa Rosa de Lima clinic in Nueva Esperanza, Honduras, which specializes in health care for children. Photo by Alfred Srur, courtesy of the World Bank Photo Library. ISBN-13: 978-0-8213-8406-0 e-ISBN-13: 978-0-8213-8407-7 DOI: 10.1596/978-0-8213-8406-0 Library of Congress Cataloging-in-Publication Data have been applied for. World Bank InfoShop Independent Evaluation Group E-mail: pic@worldbank.org Communication, Strategy, and Learning Telephone: 202-458-5454 E-mail: eline@worldbank.org Facsimile: 202-522-1500 Telephone: 202-458-4497 Printed on Recycled Paper Facsimile: 202-522-3125 Printed on Recycled Paper Table of Contents Abbreviations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . v Acknowledgments . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .vi Executive Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . vii 1. Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 The Heavy Toll of Malnutrition in Developing Countries . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 The World Bank Is Ramping Up Its Nutrition Response . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 What Do We Know about Reducing Malnutrition? . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4 Objectives of this Study . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8 2. Findings from Recent Nutrition Impact Evaluations . . . . . . . . . . . . . . . . . . . . . . 9 Methodology . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 Programmatic Impacts on Anthropometric Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14 Heterogeneity in Impacts . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Understanding the Causal Chain . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 Program Costs and Cost-Effectiveness . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Accounting for the Variability in Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 3. Evaluations of World Bank Nutrition Support . . . . . . . . . . . . . . . . . . . . . . . . . . . 27 The Programs Evaluated . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 The Design and Implementation of the Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 32 Findings . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 The Impact of the Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 40 Lessons . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 44 4. Conclusions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 45 Appendixes A: The Impact Evaluations Reviewed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 48 B: Impact Evaluations of Height, Height for Age, and Stunting. . . . . . . . . . . . . . . . . . . . . . 53 C: Impact Evaluations of Weight, Weight for Age, and Underweight . . . . . . . . . . . . . . . . 63 D: Impact Evaluations of Weight for Height and Wasting . . . . . . . . . . . . . . . . . . . . . . . . . . . 71 E: Impact Evaluations of Birthweight and Low Birthweight . . . . . . . . . . . . . . . . . . . . . . . . . 76 F: Impact Evaluation Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 78 Endnotes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .80 Bibliography . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .84 Executive Summary | iii Boxes 2.1 The Impact of School-Based Feeding Interventions in Burkina Faso on School-Age and Preschool Children . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .18 3.1 Measuring the Impact of Additional Exposure to a Community Nutrition Program Using Program Data in Madagascar . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .34 Figures 1.1 Stunting among Children under Five by Developing Region and Socioeconomic Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2 1.2 Pathways from Public Policy to Child Nutrition Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 2.1 Number of Evaluations Reporting Each of 10 Anthropometric Outcome Indicators . . . . .14 2.2 Child Anthropometry Findings of Three Evaluations of the BINP . . . . . . . . . . . . . . . . . . . . . . . .26 Tables 2.1 Interventions, Components, Countries, Evaluation Method, and Outcomes Analyzed . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .11 2.2 Definition and Interpretation of Anthropometric Indicators Used by the Nutrition Impact Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .13 2.3 Share of Evaluations with Positive Impacts on Height, HAZ, or Stunting, by Indicator and Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .15 2.4 Share of Evaluations with Positive Impacts on WAZ, Underweight, or Weight, by Indicator and Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .17 2.5 Share of Evaluations with Positive Impacts on WHZ or Wasting, by Indicator and Program . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .19 2.6 Share of Evaluations that Found Impacts on Measures of Birthweight . . . . . . . . . . . . . . . . . . .20 3.1 Impact Evaluations of Programs and Interventions Supported by World Bank Projects . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .29 3.2 Sources of Funding for Evaluations of the Impact of World Bank­Supported Programs on Nutrition Outcomes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .32 3.3 Nutrition Impact Evaluations and the Results Chain for World Bank Projects . . . . . . . . . . . . .36 3.4 Summary of the Impact of the Nutrition Impact Evaluations . . . . . . . . . . . . . . . . . . . . . . . . . . . .41 iv | What Can We Learn from Nutrition Impact Evaluations? Abbreviations BINP Bangladesh Integrated Nutrition Project BDH Bono de Desarrollo Humano CAC Componente de Atención a Crisis CCT Conditional cash transfer CENP Community Empowerment and Nutrition Program, Vietnam CNP Community nutrition promoter CNW Community nutrition worker DALY Disability-adjusted life-year DHS Demographic and Health Survey DID Difference-in-difference DIME Development Impact Evaluation Initiative FA Familias en Acción HAZ Height-for-age z-score HIV Human immunodeficiency virus HNP Health, nutrition, and population ICDS Integrated Child Development Services ICR Implementation Completion and Results Report IEG Independent Evaluation Group IMCI Integrated Management of Childhood Illness ITT Intent-to-treat IV Instrumental variables LBW Low birthweight MDG Millennium Development Goal NGO Nongovernmental organization PAD Project Appraisal Document PIDI Proyecto Integral de Desarrollo Infantil (Integrated Child Development Project), Bolivia PLW Pregnant or lactating women PRN Programme de Renforcement de la Nutrition (Nutrition Enhancement Program), Senegal PROGRESA Programa Nacional de Educación, Salud y Alimentación (National Program for Education, Health and Nutrition), Mexico (now Oportunidades) PSM Propensity score matching RCT Randomized controlled trial RPS Red de Protección Social SCF Save the Children Federation SD Standard deviation SEECALINE Projet de Surveillance et Education des Écoles et des Communautés en Matière d'Alimentation et de Nutrition Élargi (Expanded Project for Monitoring and Education of Schools and Communities in Food and Nutrition), Madagascar THR Take-home rations UCT Unconditional cash transfer UNICEF United Nations Children's Fund WAZ Weight-for-age z-score WHZ Weight-for-height z-score Abbreviations | v Acknowledgments This report was prepared by Martha Ainsworth (Task Man- We also thank the following researchers and task managers ager) and Alemayehu Ambel, with inputs from Ximena del of World Bank projects with nutrition impact evaluations Carpio, Gayle Martin, and Shampa Sinha, under the super- and developing country policy makers interviewed for vision of Monika Huppi. The report was edited by William chapter 3: Tahmeed Ahmed, Harold Alderman, Orazio Hurlbut and Linda Harteker, and administrative support Attanasio, Jere Behrman, Carla Bertoncino, Lynn Brown, was provided by Marie-Jeanne Ndiaye. The external re- Donald Bundy, Manuel Fernando Castro, Khadidiatou viewers for the report were Alessandra Marini and Agnes Dieng, Daniel Dulitzky, Emanuela Galasso, Marito Garcia, Quisumbing. Michele Gragnolati, Nelson Gutierrez, Theresa Ho, Polly Jones, Elizabeth King, Kees Kostermans, John Newman, The authors wish to thank Jean-Pierre Habicht, Yi-Kyoung Rekha Menon, Menno Mulder-Sibanda, Rahman Mustafiz, Lee, and Meera Shekar for their feedback and comments at John Mutumba, Peter Okwero, Norotiana Rakotomalala, various stages of preparation of the report, and Anupa Laura Rawlings, Claudia Rokx, Sandra Rosenhouse, Norbert Bhaumik of the Research Committee staff for facilitating Schady, Maryanne Sharp, Meera Shekar, and Howard access to the research proposals for several of the impact White. evaluations. Comments from the following individuals on previous drafts are much appreciated: Javier Baez, The opinions expressed in this report are those of the au- Arup Banerji, Hans-Martin Boehmer, Lynn Brown, Nils thors, who are responsible for the interpretations and any Fostvedt, Emauela Galasso, Ali Khadr, Nestor Ntungwa- omissions. nayo, Meera Shekar, J.P. Singh, Mark Sundberg, Marvin Taylor-Dormond, Denise Vaillancourt, Christine Wallich, and Andrew Warner. Director-General, Evaluation: Vinod Thomas Director, Independent Evaluation Group (IEG)­World Bank: Cheryl Gray Manager, IEG Sector Evaluation: Monika Huppi Task Manager: Martha Ainsworth vi | What Can We Learn from Nutrition Impact Evaluations? Executive Summary High levels of child malnutrition in developing countries contribute to mortality and have long-term consequences for children's cognitive development and earnings in adulthood. Recent impact evaluations show that many interventions have had an impact on children's anthropometric outcomes (height, weight, and birthweight), but there is no simple answer to the question "what works?" to address the problem. Similar interventions have widely dif- fering results in various settings, owing to local context, the causes and severity of malnutri- tion, and the capacity for program implementation. Impact evaluations of World Bank­supported programs, which are generally large-scale, complex interventions in low-capacity settings, show equally variable results. The findings confirm that it should not be assumed that an intervention found effective in a randomized medical setting will have the same effects when implemented under field conditions. How- ever, there are robust experimental and quasi-experimental methods for assessing impact under the difficult circumstances often found in field settings. The relevance and impact of nutrition impact evaluations could be enhanced by collecting data on service delivery, demand-side behavioral outcomes, and implementation processes to better understand the causal chain and what part of the chain is weak. It is also important to better understand the distribution of impacts, particularly among the poor, and to better document the costs and effectiveness of interventions. High levels of child malnutrition in developing countries The World Bank has recently taken steps to expand its sup- contribute to mortality and present long-term consequences port for nutrition in response to the underlying need and for the survivors. An estimated 178 million children under the increased urgency added by the crises. age five in developing countries are stunted (low height for age) and 55 million are wasted (low weight for height). Mal- What Do We Know about Reducing nutrition makes children more susceptible to illness and Malnutrition? strongly affects child mortality. Beyond the mortality risk in the short run, the developmental delays caused by under- The increased interest and resources focused on the prob- nutrition affect children's cognitive outcomes and productive lem of high and potentially increasing rates of undernutri- potential as adults. Micronutrient deficiencies--of vitamin tion raise a critical question: what do we know about the A, iron, zinc, and iodine, for example--are also common causes of malnutrition and the interventions most likely to and have significant consequences. reduce it? The medical literature points to the need to intervene during Progress in reducing childhood malnutrition in developing gestation and the first two years of life to prevent child mal- countries has been slow. More than half of these countries nutrition and its consequences. It suggests that investments are not on track to achieve the Millennium Development in interventions during this window of opportunity among Goal of halving the share of children who are malnour- children under two are likely to have the greatest benefits. ished (low weight for age) by 2015. The food-price and fi- nancial crises are making achievement of this goal even Recently published meta-analyses of the impact evaluation more elusive. literature point to several interventions found effective for Executive Summary | vii reducing undernutrition in specific settings. But there is a food aid, integrated health and nutrition services, and limit to how much these findings can be generalized, par- de-worming. ticularly in the context of large-scale government programs All the evaluations used research designs that compared the most likely to be supported by the World Bank. The meta- outcomes among those affected by the project with the analyses tend to disproportionately draw on the findings of counterfactual--that is, what would have happened to a smaller, controlled experiments. There are few examples of similar group of people in the absence of the intervention. evaluations of large-scale programs, over which there is less About half of the evaluations used randomized assignment control in implementation. The meta-analyses also tend to to create treatment and control groups; the remainder used focus on average impacts and generally do not explain the matching and various econometric techniques to construct magnitude or variability of impacts across or within studies. a counterfactual. Very few of the evaluations reviewed address the program- matic reasons why some interventions work or don't work; Among the 46 evaluations, 12 assessed the impact of World moreover, few assess the cost-effectiveness of interventions. Bank­supported programs on nutrition outcomes in eight countries. The broader review relies on the analysis of the published impact evaluations as the main source of data, Objectives of the Review but for these 12 evaluations, project documents and research This paper reviews recent impact evaluations of interven- outputs were reviewed and World Bank staff, country offi- tions and programs to improve child anthropometric out- cials, and the evaluators and researchers who conducted comes--height, weight, and birthweight--with an emphasis the studies were interviewed. on both the findings and the limitations of the literature and on understanding what might happen in a nonresearch Findings setting. It further reviews the experience and lessons from evaluations of the impact of World Bank­supported pro- A wide range of interventions had a positive impact on grams on nutrition outcomes. indicators related to height, weight, wasting, and low birthweight. Specifically, the review addresses the following four questions: There were a total of 10 different outcome indicators for the four main anthropometric outcomes. A little more 1. What can be said about the impact of different interven- than half of the evaluations addressing a height-related tions on children's anthropometric outcomes? indicator found program impacts on at least one group of 2. How do these findings vary across settings and within children, and this was true for about the same share of target groups, and what accounts for this variability? interventions aimed at improving weight-related and wasting-related (low weight for height) indicators. About 3. What is the evidence of the cost-effectiveness of these three-quarters of the 11 evaluations of interventions that interventions? aimed at improving birthweight indicators registered an 4. What have been the lessons from implementing impact impact in at least one specification, including five of evaluations of Bank-supported programs with anthro- seven micronutrient interventions. pometric impacts? There was no clear pattern of impacts across interven- Although many different dimensions of child nutrition tions--in every intervention group there were exam- could be explored, this report focuses on child anthropo- ples of programs that did and did not have an impact metric outcomes--weight, height, and birthweight. These on a given indicator, and with varying magnitudes. are the most common nutrition outcome indicators in the Evaluations of the nutritional impact of programs sup- literature and the ones most frequently monitored by na- ported by the World Bank, which are generally large scale, tional nutrition programs supported by the World Bank. complex, and implemented in low-capacity settings, show Low weight for age (underweight) is also the indicator for equally variable results. Even controlling for the specific one of the Millennium Development Goals. outcome indicator, studies often targeted children of dif- ferent age groups that might be more or less susceptible to Methodology and Scope the interventions. It is thus difficult to point to interven- tions that are systematically more effective than others The Independent Evaluation Group systematically reviewed in reducing malnutrition across diverse settings and age 46 nutrition impact evaluations published since 2000. These groups. evaluations assessed the impact of diverse interventions-- community nutrition programs, conditional and uncondi- Differences in local context, variation in the age of tional cash transfers, early child development programs, the children studied, the length of exposure to the viii | What Can We Learn from Nutrition Impact Evaluations? intervention, and differing methodologies of the stud- Bank researchers. Most used quasi-experimental evaluation ies accounted for much of the variability in results. designs, and two-thirds assessed impact after--at most-- three years of program implementation. Only half of the Context includes factors such as the level and local deter- evaluations documented the distribution of impacts, and minants of malnutrition, differences in the characteristics only a third presented information on the costs of the inter- of beneficiaries (including age), the availability of service vention (falling short of cost-effectiveness analysis). In two of infrastructure, and the implementation capacity of govern- the countries (Colombia and the Philippines) the evaluations ment. Outside a research setting, in the context of a large likely had an impact on government policy or programs. government program, many things can go wrong in service delivery or demand response that can compromise impact. Beyond this, social factors, such as the status of women or Lessons the presence of civil unrest, can affect outcomes. A number of lessons for development practitioners and These findings underscore the conclusion that it should not evaluators arose from the review of impact evaluations of be assumed that an intervention found effective in a ran- World Bank nutrition support. domized controlled trial in a research setting will have the For task managers: same effects when implemented under field conditions in a · Impact evaluations of interventions that are clearly be- different setting. The findings also point to the need to under- yond the means of the government to sustain are of lim- stand the prevailing underlying causes of malnutrition in a ited relevance. The complexity, costs, and fiscal sustain- given setting and the age groups most likely to benefit when ability of the intervention should figure into the decision selecting an intervention. Further, to improve performance, as to whether an impact evaluation is warranted. impact evaluations need to supplement data measuring impact with data on service delivery and demand-side be- Photo by Julio Pantoja, courtesy of the World Bank Photo Library. havioral outcomes to demonstrate the plausibility of the findings, to understand what part of a program works, and to address weak links in the results chain. Evidence on the distribution of nutrition impacts-- who is benefiting and who is not--and on the cost- effectiveness of interventions is scant. Just because malnutrition is more common among the poor does not mean that children living in poverty will dispro- portionately benefit from an intervention, particularly if acting on new knowledge or different incentives relies on access to education or quality services. Fewer than half of the 46 evaluations measured the distribution of impacts by gender, mother's education, poverty status, or availability of complementary health services. Only nine evaluations assessed the impacts on nutritional outcomes of the poor compared with the nonpoor. Among the evaluations that did examine variation in results, several found that the · Impact evaluations are often launched to evaluate com- children of better-educated mothers or children living in pletely new programs, but they may be equally or even better-off communities are benefiting the most. more useful in improving the effectiveness of ongoing Bank-supported cash transfers, community nutrition, and programs. early child development programs in six of eight countries · There are methods for obtaining reliable impact evalua- had some impact on child anthropometric outcomes. tion results when randomized assignment of interven- Of the 12 impact evaluations of Bank support, 11 were of tions is not possible for political, ethical, or practical large-scale government programs with multiple interven- reasons. tions and a long results chain. Three-quarters of the evalua- For evaluators: tions found a positive impact on anthropometric outcomes of children in at least one age group, although the magnitude · In light of the challenges of evaluating large-scale pro- of the impact was in some cases not large or applied to a nar- grams with a long results chain, it is well worth the effort row age group. Most of the impact evaluations involved as- to assess the risks to disruption of the impact evaluation sessment of completely new programs and involved World ahead of time and identify mitigation measures. Executive Summary | ix · The design and analysis of nutrition impact evaluations · The relevance of impact evaluations for policy makers need to take into account the likely sensitivity of children would be greatly enhanced if they documented both the of different ages to the intervention. effects and costs of nutrition programs and interventions. · For the purposes of correctly gauging impact, it is impor- In sum, in approaching the impact evaluation literature and tant to know exactly when delivery of an intervention the conduct of nutrition impact evaluations, we should not took place in the field (as opposed to the official start of be asking simply, "What works?" but rather "Under what the program). conditions does it work, for whom, what part of the inter- vention works, and for how much?" These are important · Evaluations need to be designed to provide evidence for questions that managers should be asking in reviewing the timely decision making, but with sufficient elapsed time literature; addressing them will also improve the relevance for a plausible impact to have occurred. and impact of nutrition impact evaluations. x | What Can We Learn from Nutrition Impact Evaluations? Chapter 1 EVALUATION HIGHLIGHTS · Malnutrition is widespread among children in developing countries, raising morbidity and mortality. · Impact evaluations can provide insights about effective interventions to reduce malnutrition, though the findings are variable. · The World Bank is ramping up its nutrition response and its impact evaluation efforts. · This report reviews the findings of recent nutrition impact evaluations, the experience of evaluations of the nutrition impact of Bank support, and the use of the evaluation results to improve outcomes. Photo by Jamie Martin, courtesy of the World Bank Photo Library. Introduction This report reviews recent impact evaluations of interventions and programs that seek to reduce child malnutrition as measured by low anthropometric outcomes. The objec- tive is to distill lessons on effective approaches and to improve the relevance of nutri- tion impact evaluations of World Bank­sponsored programs. The Heavy Toll of Malnutrition in wasting make children more susceptible to death from com- Developing Countries mon infectious diseases that do not affect better-nourished children (Caulfield and others 2006). Beyond the mortality High levels of child malnutrition in developing countries risk, the developmental delays caused by undernutrition contribute to high mortality and have long-term conse- affect children's cognitive development and productive po- quences for the survivors. An estimated 178 million chil- tential as adults. Maternal and child undernutrition are esti- dren under the age of five in developing countries (32 mated to be the underlying cause of 3.5 million deaths percent) are stunted (low height for age), and 55 million annually (Black and others 2008, p. 243). (10 percent) are wasted (low weight for height) (Black and others 2008).1 Within countries, undernutrition--in terms One-third of the children under five are of stunting, wasting, and underweight--is far worse among stunted and one child in ten is wasted--the the poor than among the nonpoor (figure 1.1). Increasing poor are most affected. levels of underweight (low weight for age), stunting, or FIGURE 1.1 Stuntinga among Children under Five by Developing Region and Socioeconomic Status 70 60 60 50 47 Percent stunted 40 39 38 34 33 30 27 20 16 12 10 9 0 South Asia Sub-Saharan Latin America Eastern Europe/ Middle East/ (n = 4) Africa and the Caribbean Central Asia North Africa (n = 26) (n = 9) (n = 5) (n = 2) Regionb Lowest quintile Highest quintile Source: Van de Poel and others 2008, based on the most recent Demographic and Health Survey data for 47 countries. a. The percentage of children less than ­2 standard deviations below the median height of children of the same age in the World Health Organization reference population. b. Regional medians for South Asia, Europe and Central Asia, and the Middle East and North Africa are calculated by the Independent Evalu- ation Group, based on table 2 of Van de Poel and others 2008. East Asia is not presented because there was only one country (Cambodia) from that Region. The levels of undernutrition by quintile in the two North African countries (Egypt and Morocco) were remarkably similar. 2 | What Can We Learn from Nutrition Impact Evaluations? Micronutrient deficiencies are also common among chil- 2015 (World Bank 2009a, Annex, MDG 1, figure 4). None dren in developing countries and have significant conse- of the Sub-Saharan African countries with available data is quences (Caulfield and others 2006, p. 552­54). Vitamin A on track to reduce the under-five mortality rate by two- deficiency, estimated to affect from 1 percent to 40 percent of children under five, is a preventable cause of blindness The food and financial crises have set back and raises the severity and mortality risk of infectious dis- efforts to reduce malnutrition. eases such as measles, diarrhea, and malaria. Iron deficiency anemia, which affects 22 percent­76 percent of children thirds--a goal that is heavily influenced by high malnutri- under five, can cause neurological impairment and a reduc- tion (World Bank 2009a, Annex, MDG 4, figure 2). The tion in immune function. Zinc deficiency affects 7 percent­ food price and financial crises will push many more people into poverty, exacerbating malnutrition and making the Malnutrition affects cognitive development MDGs even more difficult to attain. The Global Monitoring and long-run productive potential and Report 2009 estimates that 1 billion people suffer from hun- raises a child's risk of dying. ger, 2 billion are undernourished and 44 million more will suffer the lasting effects of childhood malnutrition in 2008 79 percent of children. It retards growth and increases sus- because of these crises, with implications for health, cogni- ceptibility to infection. Iodine deficiency can lead to mental tive development, and, eventually, earnings (World Bank retardation and impaired physical growth, reducing the 2009a). Achieving the MDG for malnutrition will affect the earnings of affected children when they reach adulthood. ability to achieve the goals of reducing child and maternal mortality and of boosting schooling. Although the overwhelming focus of public policy for child malnutrition in developing countries has been on The World Bank Is Ramping Up Its undernutrition, childhood obesity is a growing problem Nutrition Response and carries different health risks. Average overweight (high weight for height) among preschool children in de- Following a decade of low and declining lending for nu- veloping countries is on the order of 3 percent, but is sub- trition, the World Bank has taken steps to expand its sup- stantially higher in some regions and subregions.2 The port. Over the decade 1997­2006, the share of World Bank United Nations Children's Fund (UNICEF) has identified lend-ing for nutrition objectives declined, from 12 percent 20 countries in which more than 5 percent of preschool to 7 percent of approved projects managed by the health, children are overweight, a prevalence that often exceeds the nutrition, and population (HNP) sector (IEG 2009, p. 18).3 share of children who are wasted (UNICEF 2007). Child- However, Repositioning Nutrition as Central to Development hood obesity is associated with high blood pressure, diabe- in 2006 (World Bank 2006a ) and the 2007 strategy for HNP tes, and respiratory illness in childhood. To the extent that (World Bank 2007a) renewed the commitment to reduce obese children become obese adults, they are at increased malnutrition and to pilot innovations in service delivery risk of chronic diseases such as diabetes, hypertension, and in Latin America and the Caribbean (World Bank 2009, cardiovascular disease (De Onis and Blössner 2000). p. 22).4 More than 20 impact evaluations of interventions to reduce undernutrition are under way as part of the Devel- More than half of countries are not on opment Impact Evaluation Initiative (DIME) coordinated track to halve the share of children who are by the Research Department of the World Bank (World Bank 2009c).5 underweight by 2015. Beyond this, in May 2008 the Bank's Board provided Slow progress in reducing undernutrition has been set $1.2 billion in rapid financing through the Global Food back by the global food and financial crises. According to Price Crisis Response Program, offering access under fast- the Global Monitoring Report 2009, more than half of the track procedures to International Development Asso- countries with available data are not on track to achieve the ciation (IDA)/International Bank for Reconstruction and Millennium Development Goal (MDG) of halving the Development (IBRD) grants, credits, and loans and an ad- share of children who are malnourished (underweight) by ditional $200 million in grants for the poorest and most Introduction | 3 vulnerable countries. These emergency funds had financed Many causal pathways lead to nutrition projects in 30 countries as of mid-March 2009 for targeted outcomes. safety nets, food-for-work programs, emergency food aid Children and their mothers become undernourished distribution, and school feeding programs, among other through many causal pathways. Figure 1.2 highlights both interventions. the main pathways and the channels through which public policy can affect them. It also underscores the critical role What Do We Know about Reducing of household and individual behavior in ensuring the suc- cess of any intervention. Malnutrition? In the lower half of the figure, the immediate, proximate The increased interest and resources focused on the prob- factors affecting child undernutrition and LBW have to do lem of high and potentially increasing malnutrition raises with the quality and quantity of food intake, childcare prac- the immediate question, "What do we know about the tices (such as the duration of breastfeeding and the timing causes of malnutrition and the interventions most likely to of introduction of solid foods), the number and spacing of reduce it?" Many factors determine nutrition outcomes, the mother's pregnancies and her own nutritional status, and the pathway connecting public policy, private behavior, personal hygiene and sanitation facilities (including hy- and better nutrition is complex. The medical literature giene behaviors and water treatment), and the use of pre- points to the need to intervene in the first two years of life ventive and curative health care. The figure also highlights to prevent child malnutrition and its consequences. Recent the central point that child nutritional status and health published reviews of the literature point to promising inter- status are strongly related: low nutritional status makes ventions, but the generalizability of the findings of such children more vulnerable to illness and at higher risk of studies is limited, particularly for national nutrition pro- death if they become ill, and many illnesses--particularly grams with multiple activities and long results chains, as diarrheal disease--can contribute to acute or chronic mal- implemented in field settings. nutrition. Further, malnutrition and infection are affected Intervening early in life is key. through many of the same channels. The first two years of life are the window of opportunity As shown in the upper half of the figure, public policy can to prevent malnutrition and its consequences. At birth, have an impact through government finance and regulation children in developing countries are remarkably similar to of many types of services--from preventive and curative children in well-nourished populations in their weight and health or nutrition services to safety net programs, edu- length, but growth begins to falter immediately and pre- cation, agricultural information and extension, and safe cipitously after birth, continuing to decline for up to three water. In the background, all the actors and outcomes can years (Shrimpton and others 2001). Children's weight, given be affected by exogenous factors beyond their control, such their height, begins to decline at age three months, but it as climate (for example, drought or floods), geography, eventually recovers to levels only slightly lower than those macroeconomic variables (global food or fuel prices or labor market conditions, for example), or social context Children are particularly vulnerable to (for example, the status of women, institutions, and civil malnutrition in the first years of life. unrest). seen in well-nourished populations. However, the mean The pathway connecting public policy to levels of stunting of young children generally do not re- nutrition outcomes is long and complex. cover; the children grow at the same rate as the reference population, but are much shorter for their age. Gestation These complex pathways and the numerous actors in- and the first year of life are critical periods of human brain volved in implementing interventions point to a few im- development; it is thus not surprising that there is a correla- portant considerations in reviewing the literature on tion between low birthweight (LBW) and stunting early in what works in reducing malnutrition. Because of the dif- life and later cognitive deficits (McGregor and others 2007; ferent local contexts in which interventions are imple- Walker and others 2007). This points to the importance of mented, the role of service providers and households in intervening early to prevent stunting and its long-run con- determining outcomes, and the lengthy results chain, the sequences. It also suggests that the potential for interven- results of government nutrition programs as implemented tions to prevent malnutrition is greatest during pregnancy in the field conditions of developing countries are likely to and the first 24 months of life (Bhutta and others 2008; be quite different from results of randomized trials of dis- Shrimpton and others 2001; World Bank 2006a). crete interventions in a controlled setting. 4 | What Can We Learn from Nutrition Impact Evaluations? FIGURE 1.2 Pathways from Public Policy to Child Nutrition Outcomes Government Cultural/Social context Exogenous factors ! !!!!! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! ! Policies! and !programs, ! Women's status, Weather, geography, prices Regulatory framework institutions, political technology structure, civil unrest ! Public/private health Government welfare Other public services system programs Agriculture, education, Access to health care, Cash transfers, !! food subsidies, water nutrients, quality of care, food transfers supply, infrastructure information !! Households and individuals !! Income, assets and savings, household composition, !! human capital, time !! Child care Dietary Number Hygiene Use of practices intake of and spacing and preventive mothers and of children sanitation and curative children health care Nutrition status Low birthweight, Health status stunting, underweight, wasting, micronutrient deficiency Sources: Authors' construction, adapted from Black and others 2008, Ruel and Hoddinott 2008, Smith and Haddad 2002, and UNICEF 1990. First, many factors affect nutrition; we might not expect root causes of the problem in that setting, and the extent to similar results across settings for a given intervention, which other significant causes are working in parallel (Allen even if it could be implemented in exactly the same way and Gillespie 2001). An intervention is also likely to have in each case. Access to nutrients can be important in some differential impacts on nutritional status of different groups contexts, but there are populations with access to adequate of people within countries, depending on context. food who nonetheless suffer from undernutrition because of poor feeding practices or diarrheal disease linked to poor The impact of public policy on nutrition hygiene and unsafe water. Mothers' knowledge of childcare outcomes depends on local context . . . practices may improve, but low access to health or nutrition services may prevent them from realizing the benefits of Second, the effectiveness of any intervention is likely to that knowledge.6 The impact of an intervention will also de- depend on the behavior of two groups of people--service pend on baseline levels of malnutrition, with a greater im- providers and households. The quality of service delivery pact likely among those in greatest need. Thus, the measured involves incentives and decisions by health workers, be they impact of a given intervention may differ widely across set- in government, the private sector, or a nongovernmental or- tings, depending on the baseline levels of malnutrition, the ganization (NGO). Are they trained? Will they come to Introduction | 5 work? Will their advice be good? Will they have the drugs in the public health literature--encompass information on they need, the fuel for transport, and other complementary the entire causal chain of intermediate outputs and outcomes. inputs? To what extent, in effect, will the intervention be Without this information, it is difficult to know how to in- implemented as designed? terpret the differences in outcomes between program re- cipients or nonrecipients--whether the interventions were . . . and on the behavior of service implemented as planned, whether households participated providers and households. and their behavior changed, who benefited, and which parts of the program worked or did not work and why (Heckman Household and individual behaviors also affect impact. and Smith 1995; Ravallion 2009a). Will households participate in the program? If so, which Recent meta-analyses provide limited households, and which household members? Will they guidance for what works in the context of change their behavior? It is rare to find a public program or large-scale nutrition programs. intervention that does not substantially involve behavioral The most recent comprehensive meta-analysis of the im- aspects on both the supply and demand side.7 But in most pact of nutrition interventions appeared in The Lancet in instances the effectiveness of public programs in reducing early 2008 (Bhutta and others 2008). The review included malnutrition hinges to some extent on the ability of provid- not only rigorous impact evaluations but also other types of ers to deliver services effectively and on the extent to which published and unpublished program evaluations. The au- the intervention enables households and individuals to thors grouped their findings according to who was affected make better choices. Thus, in trying to understand whether (mothers, newborn babies, and infants and young chil- an intervention works and why or why not, it would be im- dren), the intervention, and the strength of the evidence. portant to understand whether both provider and household behaviors have changed in a way that is compatible with the Understanding "what works" in large-scale intervention (Victora, Habicht, and Bryce 2004). nutrition programs requires information Third, the causal chain between public policy and nutri- from the entire causal chain. tion outcomes is a long one. Randomized clinical trials of specific nutrition interventions in controlled experiments-- This follows on an earlier review of the efficacy and effec- referred to in the public health literature as efficacy stud- tiveness of nutrition interventions in low-income Asian ies--generally have a short, direct link between the inter- and Pacific countries (Allen and Gillespie 2001). These two vention and the outcome (Victora, Habicht, and Bryce meta-analyses found a number of consistent results, par- Photo by Mararu Goto, courtesy of the World Bank Photo Library. 2004). This type of evaluation can establish the technical ticularly with respect to micronutrient supplementation. efficacy of an intervention in controlled conditions. In con- Among the main findings from the 2008 review: trast, the results chain for large-scale programs is longer · Promoting breastfeeding has been shown to have a large and more complex, often including multiple interventions impact on child survival but little effect on stunting. and implemented by government workers or contractors with their own incentives. The data needs for understanding · Education about complementary feeding of children has what works in a large-scale program--effectiveness studies been shown to increase height for age in populations 6 | What Can We Learn from Nutrition Impact Evaluations? with sufficient food; the same result requires food supple- concluded that there was evidence that CCTs raised the use ments (with or without nutrition education) in popula- of health and nutrition services and reduced disparities in tions with inadequate food. the use of services by income group. However, the evidence of impact on final nutrition outcomes, such as child growth, · The case-fatality rate can be reduced by more than half by was variable. Three of the four evaluations of programs in managing severe acute malnutrition following the World Mexico showed positive impacts on height or change in Health Organization guidelines. height, though not necessarily of great magnitude, and a · Iron folate supplements can increase hemoglobin in preg- fourth evaluation showed no long-run impact on height. nant women, and micronutrients reduce the risk of LBW. Two evaluations showed a significant positive impact of the Despite the large number of studies reviewed, these conclu- CCT on height for age, but in three cases there was no ef- sions were based on a much smaller group of evaluations of fect; in Brazil, the impact on weight for age was negative. the same intervention that measured outcomes in the same way (Bhutta and others 2008, p. 421).8 There was no attempt Large-scale programs with many activities to compare the effectiveness of different interventions to are evaluated less frequently. achieve the same outcome. Meta-analyses are heavily influenced by the results of Unfortunately, these meta-analyses provide limited guid- randomized evaluations that shed little light on the im- ance on what is likely to work in large-scale programs as plementation or programmatic factors that led to success implemented in the conditions of developing countries. or lack of it. The medical literature in particular tends to Most of the studies reviewed by Bhutta and others (2008) focus on the difference in mean health outcomes between consisted of smaller-scale, often randomized, pilot efficacy treatment and control groups. Very little is typically learned studies of single interventions; fewer than 3 percent of the about the performance of the intervention itself--what parts of the causal chain worked and what parts did not; Most of the research literature on nutrition this type of information, however, is important in under- impacts is based on randomized controlled standing how to improve effectiveness. Fiszbein and Schady trials. (2009) comment, for example, that it is not clear whether the variation and in many cases lack of results for CCTs-- interventions were assessed as part of effectiveness studies which generally are large-scale programs--reflect "dif- of large-scale programs. Allen and Gillespie (2001) admit ferences in the data and estimation choices or underlying that there were "few published examples of well designed differences in population characteristics and program de- evaluations of community-based nutrition interventions" sign or implementation" (p. 151). They speculate that the (as opposed to those based in health facilities) and that "it is rare to find a rigorous evaluation which has demonstrated Randomized evaluations rarely provide plausibly the net effects that are clearly attributable to a information on what part of an intervention community-based nutrition intervention" (p. 69). Bhutta worked. and others (2008) caution that the results of efficacy studies can overstate potential benefits of scaled-up interventions, reason for lack of impact could have to do with "important as they "fail to include the reality of lower coverage and constraints at the household level that are not addressed by technical and logistical difficulties that hamper implemen- CCTs as currently designed, perhaps including poor par- tation in health systems" (p. 434).9 enting practices, inadequate information, or other inputs The evidence of nutrition impact from large-scale pro- into the production of . . . health" (p. 163). grams with multiple interventions is more ambiguous. A The usefulness of meta-analyses for those interested in recent review assessed the impact of conditional cash trans- understanding the impact of large-scale government fers (CCTs) on utilization of health care and on final nutri- nutrition programs of the type typically supported by tional outcomes, among other variables, using information the World Bank is further limited by their lack of fo- from eight evaluations of seven programs in five countries, cus on the range of results, on the distribution of im- almost all of them in Latin America (Fiszbein and Schady pacts, and on cost-effectiveness. The emphasis in the meta- 2009).10 Most of the programs were implemented on a large evaluation by Bhutta and others (2008) was on characterizing scale, providing to the poorest households cash transfers the average effect across studies, rather than on explaining that represented from 7 percent to 27 percent of per capita the variation in results. The range of impact estimates is income, conditioned on use of health or nutrition services. typically large, but the specific contexts and differences in Both the additional income and the conditionality could the interventions underlying this variability are rarely dis- have an impact on anthropometric outcomes. The authors cussed. The reviews are often organized to examine the Introduction | 7 impact of individual interventions; they rarely compare the Bank. Underweight--low weight for age--is also the indi- impact of alternative interventions to achieve the same out- cator for one of the MDGs. Finally, in contrast to the meta- come. Meta-evaluations typically do not report on findings evaluations of the literature, the report organizes the evi- on the distribution of impacts across study subjects--that is, dence so that the impacts of diverse programs can be who benefits and who does not.11 Further, very few studies compared with respect to a common outcome.13 present evidence on the cost-effectiveness of interventions, Chapter 2 reviews the methodology and findings of 46 eval- alone or comparatively.12 uations published since 2000 that measured the impact of various interventions on child anthropometry and LBW. In Objectives of This Study addition to reviewing the average effects found by these As the World Bank moves to expand its efforts to address evaluations, it asks the following questions: How do results malnutrition--both by financing programs and by incor- vary across studies, and what explains the variation? How porating more rigorous impact evaluation--it is important are the impacts distributed across individuals? What do the to understand in greater detail what the impact evaluation results tell us about the effectiveness of specific program el- research has found and how future nutrition impact evalu- ements? How much did the interventions cost in relation to ations can be made more relevant and useful for policy their impact? The review does not attempt to be exhaustive; makers. its purpose is to shed light on these other questions that often are not addressed in the meta-evaluation literature, This report addresses neglected issues in using a limited number of recent evaluations that assessed recently completed evaluations of impacts the impact of interventions on some of the most commonly researched nutrition outcomes. on child height and weight. This report addresses four questions not addressed in the The report also reviews the results of and recent meta-evaluations of nutrition impact evaluations. lessons from impact evaluations of World First, what can be said about the impact of different inter- Bank nutrition support. ventions on children's anthropometric outcomes? Second, how do these findings vary across settings and within target Chapter 3 reviews in depth the experience of a subset of the groups, and what accounts for this variability? Third, what 46 impact evaluations--those linked to World Bank sup- is the evidence of the cost-effectiveness of these interven- port for nutrition outcomes. The review of 12 nutrition tions? Finally, what have been the lessons from implement- impact evaluations of Bank support in eight countries ad- ing impact evaluations of Bank-supported programs with dresses such issues as the relation between the project de- anthropometric impacts? sign and the impact evaluation, the use of the data, the use of routine administrative data, the role of local researchers, The report focuses on impact evaluations completed since the impact of the evaluation results on the implementation 2000 that assess the impact of interventions on child of the program, and the impact of the evaluation on local anthropometric measures in developing countries. Impact capacity and public policy. The findings are based on a re- evaluations are defined as those that measure an effect of an view of World Bank project documents, impact evaluation intervention by constructing a counterfactual--what would reports, and interviews with those involved (World Bank have happened to similar individuals in the absence of the task managers, researchers, and country policy makers). intervention--and comparing outcomes under the coun- terfactual with the outcomes in the treatment group. They Chapter 4 summarizes the findings. It suggests that, going include evaluations using a variety of experimental and forward, we should not be asking simply what works in re- quasi-experimental methods. The report focuses on evalu- ducing malnutrition, but rather under what conditions it ations of the impact of programs on child anthropometric works, for whom, what part of the intervention works, and outcomes, including weight, birthweight, and height, be- for how much. These are important questions that manag- cause these are the most common nutrition outcome indi- ers should be asking in reviewing the literature; addressing cators in the literature and those most commonly moni- them will improve the relevance and utility of future nutri- tored in national nutrition programs supported by the tion impact evaluations. 8 | What Can We Learn from Nutrition Impact Evaluations? Chapter 2 EVALUATION HIGHLIGHTS · A wide range of interventions has been evaluated with respect to impact on child anthropometric outcomes. · Many programs have shown positive impacts, yet the findings show great variability, even controlling for the intervention and the age of the child. · Results are sensitive to local context, age group, duration of exposure, and evaluation methods. · Few of the evaluations measure the distri- bution of impacts by gender, education, or poverty. · Most of the nutrition impact evaluations Photo by Curt Carnemark, courtesy of the World Bank Photo Library. lack evidence on outputs and intermediate outcomes; very few measure costs or cost-effectiveness. Findings from Recent Nutrition Impact Evaluations This chapter systematically reviews impact evaluations of interventions to improve child anthropometric outcomes in developing countries since 2000. It compares the average program impacts across evaluations as well as--where possible--the heterogeneity of impacts in the beneficiary population and the cost-effectiveness of interventions. Most interventions have positive impacts on anthropomet- and publications of the World Bank, the International Food ric outcomes in some settings and age groups, yet there is Policy Research Institute, and the Integrated Management considerable variation in the results. The review finds evi- of Childhood Illness (IMCI) program. Evaluations that did dence that this variation is partly explained by local con- not measure weight, height, or birthweight were excluded.1 text, the choice of the age group, the duration of exposure Evaluations of water supply and sanitation were explicitly to the intervention, and the evaluation method. The evi- excluded to keep the sample to a reasonable size and in light dence shows no clear pattern across interventions--in of other recent reviews of that literature (IEG 2008). Also every intervention group there are examples of programs excluded were evaluations that did not use experimental or that did and did not have an impact on a given indicator. The review concludes that results are context specific and All the evaluations tried to measure impact that it is not possible to point to certain interventions by comparing program outcomes with a that are systematically more effective than others in re- counterfactual--what would have happened ducing malnutrition across diverse settings. without the intervention. Methodology quasi-experimental methods--such as randomization, pro- pensity score matching, double-differencing, instrumental This review is based on 46 impact evaluations published variables, or regression discontinuity methods--to con- since 2000 of interventions to improve child anthropom- struct the counterfactual. The final set of 46 evaluations in- etry and birthweight in developing countries. An impact cludes 35 articles from peer-reviewed journals (76 percent), evaluation is defined as one that attempts to construct a 7 World Bank working papers (15 percent), and 4 working counterfactual as the basis for measuring changes in nutri- papers from other institutions (9 percent). tional outcomes attributable to the program or intervention. Because there has already been a large recent meta-analysis Description of the sample of evaluations of nutrition interventions (Bhutta and others 2008), this re- A list of the 46 evaluations reviewed, by country, type of view focuses on a subset of the literature that measured the intervention, evaluation method, and anthropometric out- impact of interventions and programs on child anthropo- come indicators analyzed, is presented in table 2.1. metric outcomes--indicators based on child weight, height, Geographic distribution and income level. The evalua- and birthweight. These are among the most common out- tions represent evidence from 25 developing countries. come indicators in World Bank­supported nutrition proj- About half (52 percent) are of interventions in countries ects. The review assesses the impact on undernutrition; from Latin America and the Caribbean, 28 percent in Afri- studies of obesity have not been included. The review is not can countries, and 20 percent in East and South Asian intended to be comprehensive, but rather to identify a sub- countries. There are no evaluations from the Middle East set of the recent nutrition impact evaluations for closer ex- and North Africa or from Eastern Europe and Central Asia. amination of issues often not sufficiently covered in larger About half of the evaluations (54 percent) took place in meta-analyses. low-income developing countries; the remainder were con- ducted in middle-income countries. Selection criteria The Independent Evaluation Group (IEG) conducted an on- The programs evaluated were in Latin line search of Pub Med, J-Stor, and Google Advance Scholar America and the Caribbean, Sub-Saharan using relevant key words for the year 2000 through mid- 2009. Other databases searched were the working papers Africa, and East and South Asia. 10 | What Can We Learn from Nutrition Impact Evaluations? TABLE 2.1 Interventions, Components, Countries, Evaluation Method, and Outcomes Analyzed Intervention/ Evaluation Anthropometric program Country Componentsb Source methodc outcomes analyzedd Conditional cash transfers (9 evaluations) Bolsa Alimentação Brazil CT, F, G, M, NE, P, T Morris and others 2004 IVe HAZ, WAZ Familias en Acción Colombia CT, F, G, M, NE, T Attanasio and others 2005 PSM, DID HAZ*, BW* Oportunidades Mexico CT, F, G, M, NE, P, T Leroy and others 2008 PSM, DID Height*, weight*, HAZ*, WHZ* Oportunidades Mexico CT, F, G, M, NE, P, T Behrman and Hoddinott 2005 R, FE Height* Oportunidades Mexico CT, F ,G, M, NE, P Barber and Gertler 2008 R, IV BW*, LBW* Oportunidades Mexico CT, F, G, M, NE, P, T Rivera and others 2004 R Height* Oportunidades Mexico CT, F, G, M, NE, P, T Gertler 2004 R Height*, stunting Atención a Crisis Nicaragua CT, F, G, M, NE, P, T Macours, Schady, and Vakis 2008 R HAZ, WAZ, BW, LBW Red de Protección Nicaragua CT, G, M, N E, P, T Maluccio and Flores 2005 R, DID HAZ, stunting*, Social underweight*, wasting Unconditional cash transfers (3 evaluations) Bono Solidario Ecuador CT Leon and Younger 2007 IV HAZ*, WAZ* Bono de Desarrollo Ecuador CT Paxson and Schady, forthcoming R Height, HAZ Humano Child Support Grant South Africa CT Agüero, Carter, and Woolard 2007 PSM HAZ* Community-based nutrition (8 evaluations) Bangladesh Bangladesh F, G, M, NE, P Hossain and others 2005 Matching Stunting, underweight, Integrated Nutrition wasting Project Bangladesh Bangladesh F, G, M, NE, P White and Masset 2007/IEG 2005 PSM, other HAZ*, WAZ*, WHZ* Integrated Nutrition Project World Vision Haiti F, G, M, NE Ruel and others 2008 R HAZ*, WAZ*, WHZ*, programs stunting*, underweight*, wasting* SEECALINEa Madagascar F, G, M, NE, P Galasso and Umapathi 2009 PSM, DID HAZ*, WAZ*, stunting*, underweight* SEECALINEa Madagascar F, G, M, NE, P, S Galasso and Yau 2006 PSM Underweight* Programme de Senegal D, G, M, NE, P Linnemayr and Alderman 2008 PSM, DID WAZ* Renforcement de la Nutrition Programme de Senegal D, G, M, NE, P Alderman and others 2009 DID Underweight* Renforcement de la Nutrition Community Vietnam D, G, F, NE Schroeder and others 2002 R HAZ, WAZ, WHZ, Empowerment and stunting, underweight, Nutrition Project wasting Early child development (4 evaluations) Proyecto Integral de Bolivia DC, F, G, M Behrman, Cheng, and Todd 2004 PSM Height, weight Desarrollo Infantil Hogares Colombia DC, F, G, M Attanasio and Vera-Hernandez IV HAZ*, WAZ Comunitarios 2004 Early Child Philippines F, G, M, NE, P, T Armecin and others 2006 DID, PSM HAZ, WHZ*, stunting, Development wasting* Early Child Uganda D, G, NE Alderman 2007 DID WAZ* Development Feeding/Food transfers (5 evaluations) School meals and Burkina Faso F, THR Kazianga, de Walque, and R, DID HAZ, WAZ*, WHZ* take-home rations Alderman 2009 Food aid Ethiopia FFW, FD Yamano, Alderman, and IV Height* Christiaensen 2005 Food aid Ethiopia FFW, FD Quisumbing 2003 Other HAZ, WHZ* NGO feeding post Tanzania F Alderman, Hoogeveen, and IV HAZ*, WAZ* (Partage) Rossi 2006 Vaso de Leche Peru FT Stifel and Alderman 2006 IV HAZ (continued on next page) Findings from Recent Nutrition Impact Evaluations | 11 TABLE 2.1 (continued) Components, Countries, Evaluation Method, and Outcomes Analyzed Interventions, Intervention/ Evaluation Anthropometric program Country Componentsb Source methodc outcomes analyzedd Integrated health services (3 evaluations) Integrated Brazil NE Santos and others 2001 R Height, weight*, HAZ, Management of WAZ*, WHZ* Childhood Illness Integrated Child India Variousf Das Gupta and others 2005 PSM HAZ, WAZ Development Services Integrated Tanzania Not clearg Masanja and others 2005 Matching Stunting*, Management of underweight*, wasting Childhood Illness De-worming (3 evaluations) Primary school Kenya D, hygiene education Miguel and Kremer 2004 R HAZ*, WAZ de-worming Pratham Delhi India D, M Bobonis, Miguel, and Sharma R, DID HAZ, WAZ*, WHZ* Preschool Health 2006 Program ECD/De-worming Uganda D, DC, G, M, NE, P Alderman and others 2006 R Weight* Micronutrient only (7 evaluations) Micronutrient China M (iron, folic acid, Zeng and others 2008 R BW*, LBW multiple) Micronutrient India M (multiple Gupta and others 2007 R BW, LBW* containing 29 vitamins and minerals) Micronutrient Mexico M (iron, multiple) Ramakrishnan and others 2003 R BW, LBW h Micronutrient Nepal M (multiple ) Osrin and others 2005 R BW*, LBW* Micronutrient Nepal M (folic acid, iron, Christian and others 2003 R BW*, LBW* zinc, multiple) Micronutrient Peru M (zinc) Iannotti and others 2008 R Height, weight*, BW Micronutrient Zimbabwe M Friis and others 2004 R BW*, LBW (multimicronutrientI) Others (4 evaluations) Nutrition education Peru NE Waters and others 2006 Other HAZ*, WAZ, stunting*, underweight Nutrition education Peru NE Penny and others 2005 R HAZ*, WAZ*, WHZ, height*, weight* Malaria Mozambique Sulphadoxine- Menéndez and others 2008 R LBW* pyrimethamine with insecticide-treated nets Gardening Thailand Mixed gardening Schipani and others 2002 Matching HAZ, WAZ, WHZ, stunting, underweight, wasting Source: IEG analysis. Note: * = statistically significant positive impact. a. SEECALINE = Projet de Surveillance et Éducation des Écoles et des Communautés en Matière d'Alimentation et de Nutrition Élargi. b. CT = cash transfer; D = de-worming; DC = day care; F = feeding; FD = free food distribution; FFW = food for work; FT = food transfer; G = growth monitoring; M = micronutrients; NE = nutrition education; P = prenatal services; T = treatment of illness; THR = take-home rations. c. DID = difference-in-difference; FE = fixed effects; IV = instrumental variable; Matching= simple comparison of program and nonprogram areas; Other = Heckman two-step maximum likelihood estimation; PSM = propensity score matching; R = randomized. d. BW = birthweight; HAZ = height-for-age z-score; LBW = low birthweight; WAZ = weight-for-age z-score; WHZ = weight-for-height z-score. e. The control is the group that was excluded because of "random administrative error." f. Growth monitoring, supplementary feeding, preschool education, basic health services for young children, pregnant or lactating women. g. Elements are not described in the evaluation; however, the IMCI strategy involves a number of complementary services at health facilities and communities (http://www.who.int/imci-mce/). h. Vitamins A, E, D, B2, B12, and C; zinc; copper; selenium. i. Vitamins A, -carotene, thiamine, riboflavin, B6, B12, niacin, C, D, and E; zinc, copper, selenium. 12 | What Can We Learn from Nutrition Impact Evaluations? Interventions evaluated. The interventions and programs HAZ, weight-for-height z-score [WHZ], or birthweight). assessed can be classified into several broad groups: The number of studies presenting results on each of the large-scale CCTs; unconditional cash transfers (UCT); outcome indicators is shown in figure 2.1. community-based nutrition; early child development; inte- grated health services; school feeding and food transfers; Program impacts were measured for 10 de-worming; micronutrients; and others.2 The interven- anthropometric indicators of weight, tions consist of numerous component activities, as noted in height, and birthweight. table 2.1. Programs of the same type may include a different mix of activities, or cash or food transfers of different Although many of these indicators are related, they do not amounts; they may also be targeted to specific population measure the same thing: a change in height or weight is a groups.3 It is important to note that all the evaluations of measure of absolute growth; HAZ, WHZ, and HAZ are community-based nutrition programs and of de-worming relative to the median of another population; and stunting, were in low-income countries and all the evaluations of underweight, and wasting measure the most malnourished cash transfer programs (conditional and unconditional) segment of the distribution. It is possible to affect average were in middle-income countries, all but one of which were height or HAZ, for example, without affecting the share of in Latin America and the Caribbean. All the cash transfer children stunted. To facilitate comparisons and avoid dis- programs were targeted to women or mothers. crepancies based solely on the choice of indicator, the anal- ysis compares results for all evaluations and interventions The interventions can be classified by for each outcome indicator. broad type, but even those of the same type Evaluation method. Half of the 46 evaluations used an ex- involved different activities. perimental design in which recipients (individuals or com- munities) were randomly assigned to a treatment or control Anthropometric outcome indicators. The evaluations re- group (R); the impact was measured as the difference be- ported results across some 10 indicators related to height tween the outcome in the treatment and control groups.4 The and weight (table 2.2). Some of the evaluations presented remaining evaluations used quasi-experimental methods, results for only 1 of these 10 indicators; others presented including propensity score matching (PSM), instrumental multiple indicators in the same dimension (for example, variables (IV), difference-in-difference (DID), or other height, height-for-age z-score [HAZ], and stunting) or dif- matching to establish the counterfactual.5 These methods ferent dimensions (such as weight-for-age z-score [WAZ], are explained in appendix F. TABLE 2.2 Definition and Interpretation of Anthropometric Indicators Used by the Nutrition Impact Evaluations Indicator Definition and interpretation Height or These are all absolute measures of height, weight, or birthweight. Recumbent length is measured instead of height recumbent length for the youngest children. Studies using these measures report the centimeters of growth in a given population, or the grams or kilograms of weight gain or birthweight. These measures are reported as mean levels in the population, Weight with no comparison to a well-nourished reference population and no indication of the distribution of outcomes. Birthweight Height-for-age These three indicators compare a child's weight or height with the median values of a well-nourished reference z-score population of the same age or height, and sex. The z-score measures the number of SDs above (+) or below (­) the reference population median. A child with a HAZ of ­1.5 is 1.5 SDs below the median of the reference population of Weight-for-age the same age and gender. Low HAZ is considered a measure of chronic malnutrition, while low WHZ is a measure of z-score acute malnutrition and can change quite quickly. Low WAZ is affected by both. Weight-for-height z-score Stunting These are the percentages of children with z-score values below ­2 in HAZ, WAZ, and WHZ, respectively. In other words, they are children whose measurements are more than 2 SDs below the reference population median. In the Underweight reference population, only 2.3 percent of children would normally fall below a z-score of ­2. The choice of a z-score of ­2 as the cutoff point is somewhat arbitrary, but these indicators are flagging the size of the group of children who Wasting are most malnourished in each dimension. Low birthweight Defined as the percentage of children less than 2,500 grams at birth. This is a measure of the most severely affected children. Source: Authors, based on WHO 1995. Note: HAZ = height-for-age z-score; SD = standard deviation; WAZ = weight-for-age z-score; WHZ = weight-for-height z-score. Findings from Recent Nutrition Impact Evaluations | 13 FIGURE 2.1 Number of Evaluations Reporting 1.2 indicator. However, the evaluations still vary in terms of the Each of 10 Anthropometric Outcome age group of the children they analyze, and this can affect Indicators the findings, in light of the specific biological windows of opportunity for affecting anthropometric outcomes. 30 Height, height for age, and stunting 25 25 Thirty-three evaluations were reviewed with respect to Number of evaluations 20 their impact on children's height, height for age, or stunt- 20 ing; 18 evaluations (54 percent) show positive and sig- nificant results for at least one group of children and one 15 of these indicators; that is, either the program has signifi- 11 11 10 10 10 10 cantly improved height or HAZ or reduced the proportion 8 of stunting in program areas compared to nonprogram ar- 7 6 5 eas (table 2.3). However, 15 of the evaluations (46 percent) found no impact of the program on the selected height- 0 related indicators for any of the age groups studied. De- un Z He g ht rw AZ W ht t W HZ g t W tailed findings of all evaluations of height, HAZ, and stunt- gh gh St HA tin tin LB ig g de W W ei ei ei as w ing are presented in appendix B. rth Bi Un Anthropometric measure A little more than half of the evaluations Source: IEG analysis. Note: HAZ = height-for-age z-score; LBW = low birthweight; that used height indicators found a WAZ = weight-for-age z-score; WHZ = weight-for-height z-score. program impact. Height/linear growth. Most of the evidence on program Height for age and weight for height were impacts on height or linear growth comes from evalua- the most commonly used indicators. tions of two cash transfer programs in Latin America-- one that affected height and one that did not. Four evalu- Finally, it is important to note that these impact evalua- ations of Mexico's CCT program, Oportunidades, found tions, which primarily aim to affect anthropometric out- positive impacts on child height. In rural areas children comes of young children, are measuring impacts over a aged 12­36 months exposed to the program were about one relatively short time frame--a few years at most. The evalu- centimeter taller than those not exposed (Gertler 2004; Ri- ations do not capture long-run impacts of undernutrition. vera and others 2004; and Behrman and Hoddinott 2005, respectively). In urban areas, children who were younger Half of the evaluations used an than six months at enrollment grew 1.5 centimeters more experimental design; all evaluations than children in the control group after two years (Leroy measured short-term nutritional impacts, and others 2008). However, Ecuador's Bono de Desarrollo Humano (BDH), a UCT, had no impact on the height of not long-term consequences. children aged three to seven years. The following sections summarize and compare the im- A CCT program in Mexico increased pacts found in these evaluations; the extent to which they are explained by evidence of a causal chain of program in- height; a UCT program in Ecuador did not. puts, outputs, and intermediate outcomes; evidence of the costs and cost-effectiveness of the interventions; and the Of the five remaining programs, each a different type, factors underlying the variability in results. only two had an impact on child height. In Ethiopia, chil- dren aged 6­24 months in the communities that received food aid grew 2 centimeters faster over 6 months, compared Programmatic Impacts on Anthropometric with the counterfactual of no aid (Yamano, Alderman, and Outcomes Christiaensen 2005). In Peru, children aged 0­18 months The 46 impact evaluations present diverse results, in part whose mothers were exposed to nutrition education were because they assessed the impacts on groups of children of 0.71 centimeter longer than children in the control area different ages and used different nutritional outcome mea- (Penny and others 2005). However, three programs had no sures. The findings below are contrasted for all interven- impact on height--a nutrition education intervention as tions that present results for a common anthropometric part of the IMCI program in Brazil (Santos and others 14 | What Can We Learn from Nutrition Impact Evaluations? TABLE 2.3 Share of Evaluations with Positive Impacts on Height, HAZ, or Stunting, by Indicator and Program Total: Height, HAZ, Program Height HAZ Stunting or stunting Conditional cash transfers 4/4 2/5 1/2 6/8 Unconditional cash transfers 0/1 2/3 -- 2/3 Community-based nutrition -- 3/4 2/4 3/5 Early child development 0/1 1/2 0/1 1/3 Feeding/food transfer 1/1 1/4 -- 2/5 Integrated health services 0/1 0/2 1/1 1/3 De-worming -- 1/2 -- 1/2 Micronutrient supplementation 0/1 -- -- 0/1 Others 1/1 2/3 1/2 2/3 Total 6/10 12/25 4/10 18/33 Source: IEG analysis. Note: -- = There were no evaluations of the intervention with respect to this outcome variable. HAZ = height-for-age z-score. Interpretation: 4/4 = The number of evaluations that found impact (the numerator) out of the total that analyzed the outcome (the denominator). 2001), an early child development program in Bolivia, Three of the four community nutrition programs im- Proyecto de Desarrollo Infantil (PIDI) (Behrman, Cheng, proved HAZ. In Madagascar, the HAZ of both treatment and Todd 2004), and a micronutrient intervention in Peru and control groups declined, but the Projet de Surveillance (Iannotti and others 2008). et Éducation des Écoles et des Communautés en Matière HAZ. HAZ is the most frequently used indicator, ana- d'Alimentation et de Nutrition Élargi (SEECALINE) pro- lyzed in 25 studies, of which 12 reported program im- gram slowed the deterioration in the treatment group pacts. As noted earlier, height for age is an indicator of (Galasso and Umapathi 2009). The Bangladesh Integrated chronic malnutrition. More programs can be compared in Nutrition Project (BINP) had a modest impact on HAZ of terms of their impact on HAZ than for any other indicator. children between 6 and 23 months (IEG 2005; White and Masset 2007). In Haiti, age-based targeted interventions Only two of the five CCTs had an impact on HAZ, and in had a greater impact on HAZ of children in the preventive different age groups. Colombia's Familias en Acción im- program model than on children in the traditional recu- proved HAZ of children 0­24 months old, but not of chil- perative program (Ruel and others 2008).9 However, the dren 24­72 months (Attanasio and others 2005). In urban Community Empowerment Nutrition Program (CENP) in areas, Mexico's Oportunidades improved HAZ of children Vietnam had no impact on the HAZ of children age 5­30 0­6 months, but not of those 6­12 or 12­24 months (Leroy months (Schroeder and others 2002). and others 2008). However, neither Atención a Crisis nor Red de Protección Social (RPS), both in Nicaragua, had an Two of the community nutrition programs impact on the HAZ of children in any age group.6 Brazil's Bolsa Alimentação likewise found no such impact.7 improved height for age, one showed modest results, and one had no impact. Conditional and unconditional cash transfer programs did not consistently Only one of the four feeding and food transfer (FFT) affect height for age. programs had an impact on HAZ. The Partage feeding program in Tanzania was found to have improved the HAZ Two of the three UCT programs had impacts on HAZ. The of children under five (Alderman, Hoogeveen, and Rossi South African Child Support Grants had positive impacts on 2006). However, three other primarily food transfer pro- HAZ on children 0­36 months,8 as did Ecuador's Bono Soli- grams did not improve HAZ: food distribution and food for dario UCT program on children under five years of age, al- work (FFW) in Ethiopia on the HAZ of children aged 0­9 though the impact in the latter case was modest (Leon and years (Quisumbing 2003);10 school meals and take-home Younger 2007). However, Ecuador's subsequent uncondi- rations (THRs) in Burkina Faso on the HAZ of children tional transfer program, BDH, which was better targeted to 6­60 months (Kazianga, deWalque, and Alderman 2009); the poor, had no effect on the HAZ of children between three and the Vaso de Leche program in Peru on the HAZ of chil- and seven years of age (Paxson and Schady, forthcoming). dren 0­59 months (Stifel and Alderman 2006). Findings from Recent Nutrition Impact Evaluations | 15 Among the early child development programs that mea- Community nutrition programs in sured HAZ, one had a sizable impact and the other had none. Colombia's Hogares Comunitarios early child devel- Madagascar and Haiti reduced stunting; opment program had an impact on HAZ of children six those in Bangladesh and Vietnam did not. years old and younger (Attanasio and Vera-Hernandez 2004).11 Participation in the program (captured by current Among the remaining programs, the nutrition education attendance), the months in the program, and program ex- program in Peru prevented 11.1 cases of stunting per 100 posure (months in program adjusted for age) all had posi- children age 0­18 months, according to one evaluation (Wa- tive impacts on HAZ. However, enhancements to the early ters and others 2006), whereas in Tanzania stunting declined child development program in the Philippines had very more in the IMCI integrated health districts than in non- little impact on HAZ; it worsened in both program and IMCI districts among children under five years of age be- nonprogram areas among children two to seven years of tween 1999 and 2002 (Masanja and others 2005). However, age (Armecin and others 2006). the enhanced Philippines early child development program had mixed impacts on children age two to seven years com- An early child development program in pared to children in nonprogram areas with the standard Colombia had a large impact on height program (Armecin and others 2006)14 and the gardening in- tervention in Thailand had no impact on stunting (Schipani for age, but one in the Philippines did not. and others 2002). De-worming interventions had a modest impact on HAZ Weight, weight for age, and underweight in one case and no impact in the other. Mass de-worming Twenty-eight evaluations were reviewed with respect to of school children 6­18 years old in Kenya, accompanied program impact on children's weight, weight for age, or un- with hygiene education, produced a small and marginally derweight. Seventeen (61 percent) reported an impact on at significant difference in the HAZ of children in the treat- least one of these indicators in children of at least one age ment group compared with the controls (­1.13 versus group (table 2.4). One evaluation in Brazil found negative ­1.22, respectively) (Miguel and Kremer 2004).12 In India, a program impact (Morris and others 2004); the remaining de-worming intervention of a similar design--but includ- 10 (36 percent) report no significant program effects on the ing iron supplementation for the treatment group and vita- selected weight-related indicator. Detailed findings of the min A for the treatment and control--had no impact on evaluations reporting results on weight, WAZ, and under- HAZ of children between the ages of two and six years weight are in appendix C. (Bobonis, Miguel, and Sharma 2006). Weight. Five of six evaluations found positive program In Peru, a nutrition education intervention improved the HAZ of children 0­18 months by about 0.3 (Penny and oth- impacts on the weight of children in different age groups ers 2005; Waters and others 2006). However, two other in diverse programs. The Oportunidades CCT program in programs--the Integrated Child Development Services urban Mexico improved the weight of children aged zero to (ICDS) program in India and mixed gardening in Thai- six months at the time of enrollment by 0.77 kilogram; the land--had no impact on HAZ.13 weight of children from the lowest-income group also in- creased (Leroy and others 2008). The IMCI nutrition edu- Stunting. Stunting is analyzed in 10 evaluations, 5 of cation component in Brazil raised the weight of children which report program impacts. Half of the four commu- 12­18 months but not that of children 0­6 and 6­12 months nity nutrition programs had an impact on stunting. Mada- of age (Santos and others 2006). Periodic de-worming of gascar's SEECALINE program reduced stunting by about 3 Ugandan preschool children aged one to seven years in- percent (Galasso and Umapathi 2009). The World Vision creased their weight by 10 percent per year when given community nutrition program in Haiti reduced stunting twice a year, and by 5 percent when given annually (Alder- among children in the preventive model compared with the man and others 2006). In Peru, nutrition education raised traditional recuperative model (Ruel and others 2008). the weight of children in the intervention area by 0.199 ki- However, neither the BINP in Bangladesh (Hossain and logram compared with children in the control area (Penny others 2005) nor the CENP in Vietnam (Schroeder and and others 2005), and a micronutrient-supplementation others 2002) had an impact on stunting. program raised the weight of children under 12 months by Similarly, among CCT programs, the RPS program in Nica- 0.58 kilogram (Iannnotti and others 2008). However, the ragua reduced stunting by 5.2 percentage points among PIDI early child development program in Bolivia had no children younger than five years of age (Maluccio and impact on children's weight in any age group (6­24, 25­36, Flores 2005), but Mexico's Oportunidades had no impact on 37­41, 42­58, and >59 months), even though the interven- stunting of children 12­36 months old (Gertler 2004). tion included feeding (Behrman, Cheng, and Todd 2004). 16 | What Can We Learn from Nutrition Impact Evaluations? TABLE 2.4 Share of Evaluations with Positive Impacts on WAZ, Underweight, or Weight, by Indicator and Program Total: Weight, WAZ, or Program Weight WAZ Underweight underweight a Conditional cash transfers 1/1 1/3 1/1 2/4a Unconditional cash transfers -- 0/1 -- 0/1 Community-based nutrition -- 4/5 4/6 6/8 Early child development 0/1 1/2 -- 1/3 Feeding/food transfer -- 2/2 -- 2/2 Integrated health services 1/1 1/ 2 1/1 2/3 De-worming 1/1 1/ 2 -- 2/3 Micronutrient supplementation 1/1 -- -- 1/1 Others 1/1 1/3 0/2 1/3 Total 5/6 11/20 6/10 17/28 Source: IEG analysis. Note: -- = There were no evaluations of the intervention with respect to this outcome variable. Interpretation: 1/1 = The number of evaluations that found impact (the numerator) out of the total that analyzed the outcome (the denominator). a. In addition to these positive results, an additional evaluation (in the denominator) found a negative impact of Brazil's Bolsa Alimentação on WAZ of children seven years of age or younger (Morris and others 2004). WAZ. Evidence of program impact on WAZ comes from In contrast, two of four cash transfer programs had no almost all intervention types, but the largest group rep- impact on WAZ, and in one CCT, WAZ actually wors- resented is community-based nutrition programs. Four ened. Nicaragua's Red de Protección Social CCT improved of the five programs improved WAZ: the BINP in Bangla- the WAZ of children under six years of age (Maluccio and desh, SEECALINE in Madagascar, the World Vision nutri- Flores 2005). However, Atención a Crisis, another Nicara- tion program in Haiti, and the Programme de Renforcement guan CCT, had no impact on the WAZ of children of any de la Nutrition (PRN) in Senegal. However, the CENP age group (Macours, Schady, and Vakis 2008), nor did Ec- community-based nutrition program in Vietnam had no uador's UCT, Bono Solidario (Leon and Younger 2007). impact on WAZ. BINP had a modest impact (0.07 to 0.09) However, each additional month of exposure to Brazil's on WAZ of children aged 6­23 months (IEG 2005; White Bolsa Alimentação CCT was associated with a 0.13 lower and Masset 2007). SEECALINE increased the WAZ of chil- WAZ than that observed in children of the same age in the dren under five years by 0.15 to 0.22 (Galasso and Umap- control group (Morris and others 2004).15 athi 2009). The Haiti program raised the WAZ of children Similarly, the impact of two early child development pro- 12­41 months in preventive communities by 0.24, com- grams on WAZ varied. The program in Uganda raised the pared with children in the recuperative communities (Ruel WAZ of children less than one year of age; no program ef- and others 2008). Senegal's PRN increased WAZ for chil- fect was found in WAZ of children 12­24 months, 24­36 dren 0­6 months, but not for children aged 0­36 months months, 36­48 months, or >48 months, however (Alder- (Linnemayr and Alderman 2008). man 2007). The author noted that one would expect the younger children to experience the greatest impact because Four of five community nutrition programs their mothers were exposed to the intervention during improved weight-for-age scores. pregnancy. However, the Hogares Comunitarios early child development program in Colombia had no impact on WAZ Both of the food transfer programs that measured WAZ of children 0­72 months, even though food was distributed had an impact. In Burkina Faso, take-home rations (THRs) as a component (Attanasio and Vera-Hernández 2004). at primary school improved the WAZ of preschool children De-worming of preschool children in India improved in school-age children's homes, but neither THR nor school WAZ, but de-worming of school-age children in Kenya feeding improved WAZ for school-age children (box 2.1). did not. In India, the de-worming program brought about In Tanzania, presence of a Partage feeding post in the com- a 0.31 improvement in WAZ for children between two and munity was associated with higher WAZ (Alderman, Hoo- six years of age, which is equivalent to an average weight geveen, and Rossi 2006). gain of 0.5 kilogram (Bobonis, Miguel, and Sharma 2006). Findings from Recent Nutrition Impact Evaluations | 17 BOX 2.1 The Impact of School-Based Feeding Interventions in Burkina Faso on School-Age and Preschool Children A school feeding program implemented in Burkina Faso offered two interventions: school meals and take-home rations (THR). The school meals component was a lunch provided daily to attending students. The THR component was a 10-kilogram bag of cereal flour to girls, given every month conditional on a 90 percent attendance rate. The program targeted school-age children and therefore the recipients of the school meal and THR were children aged 6­15 years. Kazianga, de Walque, and Alderman (2009) evaluated several schooling and health outcomes of these school-age chil- dren as well as the impact of this program on the nutritional status of preschool children in the same households. The underlying assumption is that the dry THRs issued to school-age children would increase food availability and hence improve the nutritional status of preschool children in the same household. The assumption in the school meals case is that the preschool children at home would receive more food than would have been the case had their older siblings not participated in the school meals program. The evaluation found that in the THR villages, WAZ increased by 0.36 for preschool children, but there was no impact on school-age children. In the school meals villages, there was an impact on WAZ of school-age children, but not on preschool children. There was no impact on HAZ of either group of children in either program, although WHZ increased for preschool children in the THR villages. Source: Kazianga, de Walque, and Alderman 2009. However, there was no impact of de-worming on WAZ of additional year of exposure reduced underweight rates for school children aged 6­18 years in Kenya (Miguel and Kre- children aged 0­6 months by about 8 percent and for chil- mer 2004).16 dren 7­12 months by 4 percent; two additional years of ex- posure reduces underweight by 8 percent in all age groups The impact of CCTs, early child (Galasso and Yau 2006). However, the impacts varied ac- development programs, and de-worming cording to the child's age when the intervention started: reductions in underweight for children aged 12­36 months on weight for age was variable. are observed only after two extra years of exposure. Sene- Of the remaining four programs, only two had an impact gal's PRN community nutrition program (Alderman and in raising WAZ, and one of those is in question. The nu- others 2009) and the World Vision nutrition program in trition education component of the Brazil IMCI program Haiti (Ruel and others 2008) both reduced underweight improved WAZ among children 12­18 months, but not among younger children in program villages, compared to among children 0­6 or 6­12 months (Santos and others children in nonprogram villages. However, neither the 2001); the ICDS health intervention in India found no im- Vietnam CENP (Schroeder and others 2002) nor the Ban- pact on the WAZ of preschool children in the mid-1990s gladesh BINP community nutrition program had an im- (Das Gupta and others 2005). In Peru, a nutrition educa- pact on underweight (Hossain and others 2005).17 tion program roughly halved the (negative) WAZ of chil- Among the remaining programs evaluated on underweight, dren age 18 months in the intervention area compared with two had an impact and two did not. In Nicaragua, the children in the control area (mean values of ­0.34 and ­0.62, RPS CCT program reduced underweight of children 0­60 respectively) (Penny and others 2005). However, using the months to 9.8 percent in the program areas, and under- same data set, a second evaluation found that this impact weight increased to 16.6 percent in nonprogram areas (Ma- disappears when other maternal and household character- luccio and Flores 2005).18 The Tanzania IMCI program also istics are controlled for in a multivariate regression analysis reduced underweight in program areas (Masanja and oth- (Waters and others 2006). ers 2005). However, neither mixed gardening in Thailand (Schipani and others 2002) nor nutrition education in Peru Underweight. Six of the ten studies that investigated un- (Waters and others 2006) was found to have had an impact derweight are community-based nutrition programs and on underweight. three of the six programs had an impact. At the individual child level, Madagascar's community-based SEECALINE Weight for height and wasting nutrition program reduced underweight among children Weight for height and wasting are not as commonly mea- younger than five years of age by 5.2­7.6 percentage points sured as other anthropometric indicators. Only 14 of (Galasso and Umapathi 2009). At the community level, an the 46 evaluations (30 percent) selected for this review 18 | What Can We Learn from Nutrition Impact Evaluations? presented impacts on WHZ or wasting (table 2.5). The de- 72 months raised the WHZ of children by 0.52 five months tailed findings of these studies are in appendix D. Wasting after the intervention began (Bobonis, Miguel, and Sharma is less prevalent than stunting and underweight.19 In addi- 2006). The nutrition education component of the IMCI tion, a child's WHZ can change in a very short time because program in Brazil improved the WHZ of children 12­18 of acute illness, for example, which can easily overwhelm months, but not those of children 0­6 and 6­12 months program effects. The community-based nutrition evalua- (Santos and others 2001). However, two other programs-- tions were most likely to measure WHZ or wasting (half of nutrition education in Peru (Penny and others 2005) and them did so), but only two of the nine evaluations of CCTs mixed gardening in Thailand (Schipani and others 2002)-- reported results on one of the two outcomes. Surprisingly, had no impact on WHZ. The first of these was aimed at chil- only two of the food-based programs measured WHZ, and dren 0­18 months of age and the second at children between none measured wasting, even though this type of interven- the ages of 1 and 7 years. tion conceivably could have important short-run impacts Wasting. Only seven studies analyzed wasting, and only on weight. two reported program impacts. Three of the seven were community-based nutrition programs. As was the case for Weight for height and wasting are not WHZ, only the World Vision community-based program in often measured in the impact evaluation Haiti, where 9 percent of children are wasted, had an impact literature. on wasting (Ruel and others 2008).21 Neither the CENP community-based nutrition program in Vietnam (Schroeder WHZ. Only one of the three community-based nutrition and others 2002) nor the Bangladesh BINP (Hossain and programs that measured WHZ had an impact on it. The others 2005)22 had an impact on wasting. World Vision community nutrition programs in Haiti-- The other program that had an impact on wasting--the with relatively high levels of wasting--raised the WHZ of Philippines comprehensive early child development pro- children in the preventive communities by 0.24 compared gram--had predominantly positive program impacts on the with the children in the recuperative communities (Ruel wasting of children aged 2, 3, 4, 5, and 6 years for different and others 2008). However, the community-based programs durations of exposure to the program (4­12, 13­16, and >17 in Bangladesh (BINP) (IEG 2005; White and Masset 2007) months) (Armecin and others 2006).23 However, the Nicara- and Vietnam (CENP) (Schroeder and others 2002) had little gua CCT, RPS (Maluccio and Flores 2005), the Tanzania or no impact. IMCI health program (Masanja and others 2006), and the Both of the food aid programs that measured WHZ had mixed-gardening program in Thailand (Schipani and others an impact on it. In Ethiopia, food distribution raised the 2002) had no impact on wasting. The finding in Nicaragua is WHZ of children zero to five and five to nine years of age in high-asset households, and FFW had a similar impact on TABLE 2.5 Share of Evaluations with Positive young children in low-asset households (Quisumbing 2003). Impacts on WHZ or Wasting, The THR program in Burkina Faso raised WHZ of children by Indicator and Program 12­60 months by 0.33 after about a year (Kazianga, de Total: WHZ Program WHZ Wasting or wasting Walque, and Alderman 2009). However, the result is signifi- cant only at the 10 percent level, and it disappears when the Conditional cash 1/1 0/1 1/2 transfers sample includes all children from 6­60 months. The school Community-based 1/3 1/3 1/4 meals component of the program had no impact on WHZ. nutrition Only one evaluation each measured WHZ for a CCT pro- Early child 1/1 1/1 1/1 gram, an early child development program, integrated development health services, or de-worming interventions, but all four Feeding/ food transfer 2/2 -- 2/2 of these programs had an impact on WHZ. In urban areas, Integrated health 1/1 0/1 1/2 Mexico's CCT, Oportunidades, raised WHZ by 0.47 among services children 0­6 months old in program areas, but not for those De-worming 1/1 -- 1/1 aged 6­12 or 12­24 months (Leroy and others 2008). The Others 0/2 0/1 0/2 enhanced early child development program in the Philip- Total 7/11 2/7 7/14 pines had predominantly positive impacts on the WHZ of Source: IEG analysis. children of different ages (2, 3, 4, 5, and 6 years) for different Note: -- = There were no evaluations of the intervention with this outcome variable. Interpretation: 1/1 = The number of evaluations durations of exposure in the program (4­12, 13­16, and that found impact (the numerator) out of the total that analyzed >17 months) (Armecin and others 2006).20 A de-worming the outcome (the denominator). intervention in India on children between the ages of 24 and Findings from Recent Nutrition Impact Evaluations | 19 not altogether unexpected, as only 1 percent of children weight but had no impact on LBW. However, neither a were wasted (less than the 2.3 percent in the reference popu- Peruvian program that offered only zinc (Iannotti and oth- lation). The predominance of impact evaluations from Latin ers 2008) nor a Mexican intervention that provided iron America, where wasting is low, may explain in part why so and a multiple micronutrient (Ramakrishnan and others few of the 46 evaluations measured this indicator. 2003) had an impact on birthweight. It is interesting to note Birthweight and LBW that the two programs with no impact on birthweight were Micronutrient interventions dominate the programs for in middle-income Latin American countries, whereas those which birthweight impacts were measured (table 2.6). that did were in low-income countries. This review identified 11 recent impact evaluations of birth- Three CCT programs measured impacts on birthweight, weight or LBW from nine countries--China, Colombia, as did one malaria program. In the case of Mexico's Oportu- India, Mexico, Mozambique, Nepal, Nicaragua, Peru, and nidades, "beneficiary status predicts 127.3 g[rams] higher Zimbabwe. Birthweight and the incidence of LBW respond birth weight . . . and a 4.6 percentage point reduction in low to activities targeted to pregnant women, including micro- birth weight" (Barber and Gertler 2008, p. 1409). The im- nutrient and energy supplements and other prenatal ser- pacts were greater among women who spent more time in vices aimed at improving dietary practices and living conditions (Allen and Gillespie 2001; Bhutta and others the CCT program and those who received more cash.25 2008). Seven of the 11 evaluations of birthweight and LBW Colombia's Familias en Acción CCT also had an impact on measured the impact of micronutrient interventions; the raising birthweight. However, Nicaragua's Atención a Crisis only other interventions represented are CCTs and a single had no impact on birthweight (Macours, Schady, and Vakis program targeting malaria. Notably, 10 of the 11 studies of 2008).26 birthweight or LBW had experimental (randomized) de- Finally, a program in Mozambique that provided two doses signs.24 The detailed findings of evaluations that measured of sulphadoxine-pyrimethamine and insecticide-treated the impact on birthweight and LBW are in appendix E. bednets reduced LBW among women who had had four or more pregnancies (Menendez and others 2008).27 Most of the programs affecting birthweight involved micronutrient interventions, and Heterogeneity in Impacts most worked. Aside from analyzing the average impacts of interven- tions across age groups, fewer than half of the studies Five of the seven micronutrient programs had impacts examined the distribution of effects on the nutritional on birthweight or LBW. Although the specific micronutri- outcomes of different beneficiary groups--the impact on ents provided varied across the programs, most offered multiple micronutrient supplementations during preg- the poor and the nonpoor, the children of educated and nancy to the treatment groups, compared with the standard uneducated mothers, or boys and girls. Only 40 percent folic acid and/or iron supplementations in the controls. (19 of the 46 evaluations) examined the variation (hetero- Interventions offering multiple micronutrients in India geneity) of the impact of the interventions by characteris- (Gupta and others 2007) and Nepal (Christian and others tics other than age group. These included income and pov- 2003; Osrin and others 2005) both raised birthweight and erty or any other measure of socioeconomic status (9 reduced LBW. Programs in China (Zeng and others 2008) evaluations), maternal education (6), gender (6), place of and Zimbabwe (Friis and others 2004) raised average birth- residence or region (3), and other characteristics (8). TABLE 2.6 Share of Evaluations that Found Impacts on Measures of Birthweight Total--birthweight Program Birthweight Low birthweight or low birthweight Conditional cash transfers 2/3 1/1 2/3 Micronutrient supplementation 5/7 3/6 5/7 Others--malaria -- 1/1 1/1 Total 7/10 5/8 8/11 Source: IEG analysis. Note: -- = There were no evaluations of this intervention for this outcome measure. Interpretation: 2/3 = Of the three evaluations that measured BW, two reported statistically significant impacts. There were no evaluations of the impact of UCT, community-based nutrition, early child development, food transfers, integrated health services, or de-worming on birthweight. 20 | What Can We Learn from Nutrition Impact Evaluations? had a greater impact on the HAZ of children whose moth- Fewer than half of the evaluations looked at ers had no education (Attanasio and Vera-Hernandez 2004). the distribution of impacts. A de-worming program in India had a larger impact on the WHZ of children whose mothers had less than three years Among the nine evaluations that examined impacts by of schooling (Bobonis, Miguel, and Sharma 2006). Neither socioeconomic status, most found that children from the the Ugandan early child development program nor the poorest households benefit more than those from less Bangladesh community nutrition program (BINP) had dif- poor households. Although programs often target the ferential program impacts on WAZ by mother's education poorest group of the society, the relative differences in in- (Alderman 2007; IEG 2005). come or socioeconomic status within the targeted group affect the magnitude and significance of impacts. Children whose mothers had more Mexico's Oportunidades CCT program had a positive impact education were more likely to benefit in on height among rural children from the poorest house- Mexico and Madagascar, but less likely to holds, but not on children from relatively better-off house- benefit in Colombia or India. holds (Rivera and others 2004). In urban areas Oportuni- dades also had a stronger impact on child growth (measured The six evaluations that examined the differing impacts by both height and weight) for children from the poorest of programs by gender produced quite variable results, households (Leroy and others 2008). Among Ethiopian chil- depending on the country and the intervention. The BDH dren younger than 5, food for work improved WHZ in low- unconditional cash transfer program in Ecuador benefited but not high-asset households (Quisumbing 2003). girls more than boys for several health and educational out- In contrast, free distribution of food raised WHZ of chil- comes, although there were no impacts on the height of dren younger than 5 in high-asset Ethiopian households, girls or boys (Paxson and Schady, forthcoming). Food for but not in low-asset households (Quisumbing 2003). Mad- work in Ethiopia--where boys under nine have lower nu- agascar's SEECALINE, though targeted to the poorest ar- tritional status than girls--appears to improve boys' WHZ eas, tended to benefit the nutritional status of children in more than girls', among children under five, and it improves better-off communities (Galasso and Umapathi 2009).28 boys' HAZ more than girls' in children between the ages of five and eight in low-asset households (Quisumbing 2003).29 Four programs had no differential impact on children's nu- However, the gender effects depend on the modality of food tritional status across income groups or household wealth: aid (FFW versus free distribution of food), the age groups, Nicaragua's Bono de Desarrollo Humano, a UCT (Paxson household assets, and the specification; in most cases there and Schady, forthcoming); Uganda's early child develop- are no gender effects of food aid. The ICDS program in In- ment program (Alderman 2007); and the community nu- dia tended to improve the HAZ of boys more than girls in trition programs in Bangladesh (IEG 2005) and Senegal 1992, but there were no differences in impact by gender in (Linnemayr and Alderman 2008). 1998, nor were there any differences in impact by gender of In Mexico and Colombia, the poorest WAZ in either year (Das Gupta and others 2005). The In- dian de-worming program improved the WHZ of both children benefited the most. boys and girls, but the magnitude of the impacts was larger and stronger for girls (Bobonis, Miguel, and Sharma 2006). Evaluations in Mexico and Madagascar suggest that chil- dren with more educated mothers benefit more than In contrast, there were no differential impacts on HAZ, those with less educated mothers. The impact of Mexico's WAZ, or WHZ by gender of the Red de Protección Social in Oportunidades CCT on height was larger for children Nicaragua (Maluccio and Flores 2005). A micronutrient whose mothers had better education (Behrman and Hod- program in Peru reported different impacts by gender but dinott 2009). Madagascar's SEECALINE community-based did not explain them (Iannotti and others 2008). nutrition program improved the HAZ, WAZ, and under- Evaluations have also looked at impacts by other benefi- weight of children whose mothers had secondary or higher ciary and program characteristics, such as place of resi- education; the program also raised WAZ for children whose dence, community infrastructure, number of prior preg- mothers had primary schooling but had no impact on chil- nancies, anemia, or human immunodeficiency virus (HIV) dren whose mothers had no education (Galasso and Uma- status. The ICDS program tended to improve the WAZ of pathi 2009). children from the northern (poor) region of India in 1998, In contrast, in Colombia and India the children of the but there were no differences in impact by region in 1992, least educated mothers benefitted the most. In Colombia, nor were there any regional differences in impacts on HAZ the Hogares Comunitarios early child development program in either year (Das Gupta and others 2005). Findings from Recent Nutrition Impact Evaluations | 21 WHZ improved both in children who were anemic at base- pact or small impacts can be the result of shortcomings in line and in those who were not; however, the impact of the implementation, which cannot be assessed without infor- Pratham Delhi Preschool Program was greater for children mation from the causal chain. Many nutrition interventions who were anemic at baseline (Bobonis, Miguel, and Sharma involve multiple activities, and managers want to under- 2006). The SEECALINE community-based nutrition pro- stand which of these activities contributed to outcomes. gram in Madagascar had greater impacts in villages with For community-based nutrition programs, for example, better proximity to a road, a hospital, electricity, and access managers want to understand the contribution of feeding to safe water source (Galasso and Umapathi 2009). How- (the most expensive component) to better outcomes. In the ever, Mexico's Oportunidades, a CCT, had no differential case of CCTs, policy makers want to understand whether it program impact on height by access to community infra- was the cash transfer or the conditionality that was respon- structure (Behrman and Hoddinott 2005). sible for outcomes. There was an enormous increase in the Colombia's Familias en Acción, a CCT, had impact on birth- uptake of iron supplement (ferrous sulfate) as a result of the weight in urban but not in rural areas (Attanasio and others RPS conditional cash transfer in Nicaragua in the treatment 2005). A malaria intervention in Mozambique reduced in- areas relative to the control areas between 2000 and 2002 cidence of LBW for women with four or more prior preg- (Maluccio and Flores 2005). Both stunting and underweight nancies (Menendez and others 2008). However, no differ- declined in the treatment areas relative to the controls. De- ential impact was found by HIV status of women. Similarly, spite this, there were no significant reductions in anemia in Tanzania, there was no difference in the impact of multi- between the treatment and control children over time. Rich micronutrient supplementation on birthweight by HIV sta- data on the causal chain could offer an explanation for un- tus of the woman (Friis and others 2004). expected results, such as the worsening of WAZ in Bra- zil's Bolsa Alimentação program (Morris and others 2004). Greater attention to tracking intermediate outcomes and Understanding the Causal Chain a process evaluation to assess implementation difficulties Impact evaluations have as an objective to be able to at- would have shed light on the causes of these counterintui- tribute an outcome to an intervention. If the control and tive results. treatment groups are identical in their composition and there is no attrition or crossover between groups, then any Only about half of the evaluations difference between outcomes in the two groups can be at- documented at least one intermediate tributed to the program. outcome. However, there are a number of reasons why it is not only prudent but highly advisable to document the causal chain Despite these benefits, only about half of the 46 impact of the program or intervention--from the inputs to outputs evaluations (24) documented at least one intermediate and intermediate outcomes. First, in the real world it is of- outcome. The most commonly measured intermediate out- ten difficult to prevent attrition, crossover, or other exoge- comes were micronutrient intake or status (13); illness (12); nous events (such as an economic or a political crisis) that use of health care (9); dietary intake (7); and breastfeeding can compromise an experimental design and confound the knowledge and practice (7).30 findings. Documenting implementation of the intervention A few evaluations were able to infer the effectiveness of and intermediate outputs and outcomes lends plausibility the different parts of the intervention by pointing to in- to the findings. It establishes whether the intervention was termediate outcome indicators in the causal chain. In Sen- fully implemented, providing insight as to whether the im- egal, the positive impact of PRN, a community-based nutri- pact might have been even larger had it been implemented tion program, on the WAZ of the youngest group of children correctly. was validated and explained by a concomitant increase in breastfeeding and weaning practices in program areas for Documenting the causal chain helps the youngest children (Linnemayr and Alderman 2009). re no explain why outcomes were or were not Bangladesh's BINP community-based nutrition program achieved. had a small impact on nutritional outcomes, at best. Data on intermediate outcomes showed that women in the BINP s explain Second, documenting the causal chain helps expla why areas had greater knowledge than women in control areas achiev the anticipated outcomes were or were not achieved, the as a result of the program; however, for some reason they n actually extent to which each part of the intervention was a had not been able to translate that information into changes ost implemented, which part contributed the most or least to in practice that would improve nutrition outcomes (Hos- d. outcomes, and how impact might be increased. Lack of im- sain and others 2005; White and Masset 2007). 22 | ations? What Can We Learn from Nutrition Impact Evaluations? In Peru, the improvements in children's nutritional status 18 months of follow-up of 338 children from birth, the nu- could be explained in part by an increase in health care use trition education program was found to have averted 11.1 in areas covered by the nutrition education program (Wa- cases of stunting per 100 children in the 0- to 18-month age ters and others 2006). Colombia's Familias en Acción CCT range. The estimated marginal cost, including external costs, program had an impact on intermediate outcomes, such as training, health education materials, and extra travel and improved probability of compliance with preventive health equipment, was $6.12 per child, or $55.16 per case of stunt- care, lower morbidity, and improved food intakes. HAZ im- ing averted (Waters and others 2006). proved among children younger than 2 years old, but not Three evaluations assessed the costs and benefits of the for older children (24­48 months and >48 months), even interventions by examining payoffs in the long run. The though the food intake of the older children was improved anthropometric improvements attributable to Mexico's by the program (Attanasio and others 2005). Similarly, Oportunidades CCT in rural areas were estimated to be Atención a Crisis in Nicaragua had an impact on dietary intakes and health care utilization, although this apparently Photo by Julio Pantoja, courtesy of the World Bank Photo Library. did not lead to an impact on any of the child anthropomet- ric indicators (Macours, Schady, and Vakis 2008). In Bangladesh, women participating in the BINP community nutrition program acquired knowledge, but this did not change their behavior. The Kenya primary school de-worming program included both de-worming and preventive health education, either or both of which could have accounted for the improve- ment in HAZ. However, because the evaluators were able to document no difference between the control and treatment groups in hygiene behavior, they argue that the nutritional outcome was likely a result of the de-worming drugs (Miguel and Kremer 2004). equivalent to a 2.9 percent increase in lifetime earnings Program Costs and Cost-Effectiveness (Behrman and Hoddinott 2005). The present value of the Impact evaluations provide an opportunity to measure investment in human capital resulting from the South Af- the impact as well as the costs of programs, providing rica Child Support Grants exceeded by more than 60 per- insights into both efficiency and sustainability. Cost- cent the cost of the program (Agüero, Carter, and Woolard effectiveness analysis of specific elements of complex inter- 2007). The benefit-cost ratio of the PIDI preschool program ventions is often constrained, however, by the fact that in Bolivia was calculated by estimating the benefits and evaluations do not isolate the component that matters for costs to the child, assuming that he or she attained interme- the measured impact. diate and secondary education (Behrman, Cheng, and Todd Among the 46 evaluations reviewed, only a handful doc- 2004). In a hypothetical setting,32 the benefit-cost ratio is umented the costs or cost-effectiveness of the interven- estimated to be 1.37­2.48 at a 5 percent discount rate; how- tions evaluated. In Uganda, a de-worming intervention ever, improved anthropometric outcomes were not among was implemented with preschool children as part of "child the benefits. health days" in the early child development program, which also offered polio inoculations and vitamin A supplementa- Accounting for the Variability in Results tion (Alderman and others 2006). The cost of the health day When comparing results of evaluations with similar inter- event was estimated at $1.33 per child and the de-worming ventions on identical outcomes, the analysis of these 46 intervention at $0.25 per child per event. evaluations leads to the conclusion that there is enormous In Kenya, a de-worming program helped avert 649 disability- variability. This review finds evidence that some of the vari- adjusted life years (DALYs), equivalent to a cost of $5 per ation can be explained by differences in context, the age DALY averted,31 but this value underestimated the health group studied, the duration of the intervention, and the spillover benefits (Miguel and Kremer 2004). In Peru, after evaluation method. Findings from Recent Nutrition Impact Evaluations | 23 then baseline levels of maternal education will affect the Photo by Ami Vitale, courtesy of the World Bank Photo Library. average impact. The impact of de-worming in India was greatest among children with the most severe anemia. The availability of complementary infrastructure--not often measured in these evaluations--can also affect pro- gram impact. This review found systematic differences in the distribution of interventions by region. It is perhaps no accident that all the CCTs, in which transfers to the poorest people are conditioned on the use of health or education services, were in middle-income countries, where access to basic health services is not generally constraining. Even the UCT program in Ecuador, Bono de Desarrollo Humano, raised utilization of health care. However, in low-income countries health care is less accessible. Community infra- Context mediates the impact of nutrition structure not only augments the impact of Madagascar's interventions. SEECALINE community-based nutrition program but also Impact evaluations of similar programs offer different complements mother's education (Galasso and Umapathi results because of differences in context. The variability of 2009).33 the impacts of similar programs implemented in different countries or the same country in different periods or set- Implementation capacity is another dimension of con- tings is evident for all types of interventions and anthropo- text, though the evaluations reviewed here had very little metric indicators. The programs have important differences information to document the extent of implementation. that arise from baseline beneficiary characteristics, country, Poorly implemented interventions can be indistinguishable and program area, all of which can affect outcomes. from no intervention at all. The causal chain was rarely doc- umented in these evaluations, but it is reasonable to expect The variation in nutrition impacts of that in some cases the lack of impact could be caused by the same programs can be explained by poor implementation. The PIDI early child development program in Bolivia, for example, showed no impact on different contexts, exposure, age groups, height or weight, even though the intervention provided and evaluation methodologies. food to the children; however, no information was available on the extent to which the food was delivered, the quality of Baseline characteristics or initial conditions can affect home care and stimulation provided the children, the num- the magnitude of the impact. The evaluation of de-worm- ber of children per caretaker, or other indicators to under- ing in Uganda, for example, took place in the region with stand to what extent the intervention was implemented as the highest burden (Alderman and others 2006); both the planned (Behrman, Cheng, and Todd 2004). results and cost-effectiveness would likely be different in other parts of Uganda where the burden is less severe. A Lack of impact of large-scale nutrition community-based nutrition program had an impact on programs can be due to shortfalls in WHZ and wasting in Haiti, with high baseline levels of both (Ruel and others 2008). program implementation. In contrast, there was no impact of nutrition education on Finally, women's status can strongly condition the out- WHZ in Peru, which could be attributable to the interven- comes of nutrition programs. Most of the impact eval- tion or to the fact that it is at such a low level (less than in uations were of interventions targeted to women, on the as- the reference population) (Penny and others 2005). The im- sumption that they are the main decision makers concerning pact of a de-worming program in India on WHZ was higher children's welfare. However, this may not always be the case. among children with the most severe anemia at baseline Evaluations of Bangladesh's BINP community nutrition pro- (Bobonis, Miguel, and Sharma 2006). If certain interven- gram found that although women in program communities tions predominantly have an impact among children with had higher levels of knowledge than women in nonprogram educated mothers (as was found in several evaluations), areas, the impact of the program on nutritional outcomes 24 | What Can We Learn from Nutrition Impact Evaluations? was small (IEG 2005; White and Masset 2007).34 There are age, corresponding to the critical window of opportunity to factors constraining women from acting that are not gen- prevent malnutrition (Agüero, Carter, and Woolard 2007; der related (for example, resources, time), but the authors Allen and Gillespie 2001; World Bank 2006a). of one study point to evidence from a Demographic and Health Survey (DHS) that women are often not the main Short durations of exposure to the decision makers with respect to nutrition decisions in Ban- programs may explain low impacts in gladesh (IEG 2005).35 In many cases, men do the shopping some cases. and mothers-in-law make meal decisions. Differences in the age of the children studied Increased exposure raises impact. are partly responsible for the variability in Impacts are affected by duration of exposure to the pro- results. gram. Interventions that are implemented for a few months If there truly are certain ages at which children are more may not have a discernible effect on linear growth. Some of susceptible to nutritional shocks and more likely to re- the reviewed evaluations mention short duration of expo- cover from them, then programs would be expected to sure as a justification for lack of impact on stunting (for have different impacts, depending on the age of the target example, Bobonis, Miguel, and Sharma 2006; Kazianga, de group. The evaluations reviewed here did not consistently Walque, and Alderman 2009; and Santos and others 2001). report results for similar age groups. The three evaluations Differences in duration of exposure can result in differences of de-worming, for example, examined the impact on chil- dren 1­7 years old in Uganda, 2­6 years old in India, and in magnitude and significance of impacts of the same pro- 6­18 years old in Kenya (respectively: Alderman and others gram (Agüero, Carter, and Woolard 2007; Armecin and 2006a; Bobonis, Miguel, and Sharma 2006; Miguel and others 2006; Galasso and Yau 2006). Kramer 2004). These results are not easily compared with Evaluation methodologies can affect the findings on community-based nutrition programs, which results. measured impacts on children under 3 years (3 evalua- Studies that evaluated the same program using different tions), under 5 years (1 evaluation), 6 months­2 years methods arrive at different results. On the basis of experi- (2 evaluations), and 5­30 months (1 evaluation). mental results of a nutrition education intervention in Peru, Penny and others (2005) report a significant difference in Some of the variation in results is due the WAZ of children aged 18 months in control and inter- to evaluation of impacts in different age vention areas. However, in a multivariate analysis of the groups. same program, Waters and others (2006) show that the dif- ference disappears when controls are included for selected Many of the studies measured impacts only on a rela- socioeconomic characteristics. tively large age spread, such as 0­60 months, without re- porting disaggregated results for children under 2 or 3 The impact of nutrition education in Peru years old. This points to the possibility that some of the depended on which estimation method was statistically insignificant findings for broad age groups used. might have yielded different findings had the age groups been disaggregated. For example, there was no program The evaluations of the BINP community-based nutrition impact of the Uganda early child development program on program in Bangladesh on nutritional status of children WAZ of children aged 0­48 months, but when the author under two years old tell a similar story (figure 2.2). Early studied only children under 12 months of age, WAZ im- project monitoring data showed substantial reductions in proved (Alderman 2007). malnutrition, especially in severe malnutrition, in project Although the age group of analysis is contributing to the areas and convinced the World Bank and the government variability in results in the aggregate, there is still vari- to scale up the intervention in the National Nutrition Proj- ability in results among children of the same age. Com- ect (Karim and others 2003). A subsequent evaluation of paring all studies that examined age groups under 36 the program that compared program and nonprogram ar- months and controlling for the anthropometric outcome eas found no difference in stunting, underweight, or wast- measure, evaluations even of the same intervention show ing between the program and nonprogram areas (Hossain inconsistent results, with some showing impacts and others and others 2005). However, it was unclear how well matched none. The results and the age groups studied are sufficiently the program and nonprogram areas were in terms of their variable that this review could not confirm a pattern of baseline characteristics before the program was launched. higher program impact for children under three years of Using propensity score matching, IEG's reanalysis of the Findings from Recent Nutrition Impact Evaluations | 25 FIGURE 2.2 FIGURE 2.2 Child Anthropometry Findings of Three Evaluations of the BINP 0.2 0.17*** 0.15 BINP Endline (children Single difference scores 6­23 months) Karim 0.1 0.09** and others 2003 0.08** 0.07 Save the Children UK 0.05 0.05 0.05 (children 0­23 months) 0.03 0.03 Hossaim and others 2005 0 IEG Endline PSM (children HAZ WAZ WHZ 6­23 months), IEG 2005 ­0.05 ­0.1 ­0.08** Measure Source: IEG 2005, Tables G.17 and G.19. Note: Significance levels: ** p .05; *** p .01. IEG endline PSM estimates are average treatment effects on the treated, one-to-one matching. BINP = Bangladesh Integrated Nutrition Project; HAZ = height-for-age z-score; WAZ = weight-for-age z-score; WHZ = weight-for-height z-score. same data suggested that the project had a modest impact The results are enormously variable, which is partly ex- at best (IEG 2005; White and Masset 2007). plained by context, the child's age, duration of exposure to the program, and the analytical methods used. Although Conclusions there may be biological factors that justify early action, the evaluations of the programs reviewed here do not consis- This chapter synthesizes evidence from 46 recent evaluations tently show short-term impacts over the window of oppor- that analyzed the impact on child anthropometric outcomes tunity among the youngest children, during which time of interventions implemented in 25 developing countries. impacts are anticipated to be greatest. More than half of the studies show impacts on at least one anthropometric indicator for some children. However, the Finally, most of the evaluations focused on average impacts; lack of disaggregated results for common age groups makes among the minority that measured the distribution of im- it difficult to compare results across evaluations, and inade- pacts there were differential impacts by socioeconomic sta- quate evidence on the causal chain and cost-effectiveness of tus and mother's education. Only 1 in 8 of the evaluations the programs makes it difficult to synthesize the lessons. addressed impacts by gender. 26 | What Can We Learn from Nutrition Impact Evaluations? Chapter 3 EVALUATION HIGHLIGHTS · Twelve nutrition impact evaluations evaluated interventions or programs in eight countries receiving World Bank support. · Cash transfers, community nutrition, and early child development programs were evaluated. · A large majority of evaluations used quasi-experimental methods. · Evaluating large programs presented many challenges. · The degree of implementation of the interventions was not well documented. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. · Only half of the evaluations examined the heterogeneity of impacts; fewer documented costs. · The impact evaluations in two of the eight countries plausibly had an impact on policy. Evaluations of World Bank Nutrition Support In an effort both to increase knowledge and to improve the effectiveness of programs, the World Bank has embarked on major initiatives to support rigorous impact evalu- ations, often embedded in World Bank projects.1 IEG's recent evaluation of the Bank's support for health, nutrition, and population (HNP) found that though nearly a third of HNP projects called for impact evaluations or evaluation of pilot projects in their design, only about 1 in 20 actually conducted one (IEG 2009). Thus, a review of the characteris- tics, implementation experience, and ultimate impact of nutrition impact evaluations on policy is likely to lead to valuable insights on how to improve their effectiveness. This chapter reviews the experience of the 12 evaluations or children. The services were delivered by community that assessed World Bank­supported interventions to re- workers, supervised by NGOs. duce malnutrition from among the 46 reviewed in chap- · Evaluations in Bolivia, the Philippines, and Uganda ter 2. Specifically, it reviews the characteristics of the pro- measured the impact of early child development in- grams evaluated, the challenges of designing and imple- terventions on nutritional outcomes. The program in menting impact evaluations of large government programs Bolivia consisted of informal, home-based day care that to reduce malnutrition, the evaluations' findings, the im- included nutrition supplements, stimulation, and access pact of the evaluations on programs and policy, and the to health care. The early child development programs in lessons that can be drawn. The evidence is culled from a the Philippines and Uganda had community-level work- review of project documents, the evaluations, and inter- views with project managers, evaluators, and country pol- ers providing nutrition services, in addition to early child icy makers.2 education interventions. An ancillary impact evaluation embedded in the Uganda early child development evalu- Twelve evaluations measured the impact ation assessed the impact of de-worming on the weight of preschool children. of Bank support on nutrition outcomes in eight countries. Bank-supported cash transfers, community nutrition, and early child development The Programs Evaluated programs were evaluated. Twelve of the 46 recent nutrition impact evaluations re- With only one exception, the World Bank­supported pro- viewed for this study could be linked to interventions supported by eight projects financed by the World Bank grams that were evaluated were large-scale government (table 3.1). programs with multiple interventions and a very long causal chain that involved the compliance of implementers · Evaluations in Colombia and Ecuador examined the im- as well as beneficiaries to ensure effective implementation. pact of CCTs and UCTs, respectively, on child nutritional Only the de-worming program for preschool children in and development outcomes. Uganda had a relatively short results chain and comprised · Evaluations in Bangladesh, Madagascar, and Senegal a single intervention implemented in a discrete region. All measured the impact of community nutrition inter- programs evaluated were implemented by developing coun- ventions. These programs involved growth monitoring try governments (national or local) or by NGOs on contract promotion for young children, nutrition education for to government. This is in contrast with the larger body of the mothers (including breastfeeding messages), micro- nutrition impact evaluations reviewed by Bhutta and others nutrient supplements, and, in Bangladesh and Madagas- (2008), most of which involved randomized controlled trails car, food supplements for severely malnourished women (RCTs) of discrete interventions with a short causal chain. 28 | What Can We Learn from Nutrition Impact Evaluations? TABLE 3.1 Impact Evaluations of Programs and Interventions Supported by World Bank Projects World Bank support Country/ (total proj- Impact intervention Project ect cost)a Program/nutrition intervention evaluated evaluations Cash transfers Colombia Human Capital $152.5 Familias en Acción CCT: Subsidy of $15.38/month/mother of children 0­6, conditioned on growth control and development check- Attanasio and Protection ($369.3) ups every 2 months for children <1 year, three times a year for children <2 years, and twice a year for those 2­6. Subsidy of $4.61 others 2005 (2001­05) for a child in primary school and $9.23 for secondary school, ages 8­17, conditioned on 80% attendance.b Targeted to the poorest quintile of the population in program municipalities. Ecuador First Programmatic $50.0 Bono de Desarrollo Humano UCT: $15/month/mother, $11.5/month for senior and disabled for the lowest two poverty index quin- Paxson and Human Develop- ($50.0) tilesc (amounts to a 10% increase in monthly income for average eligible family). Schady, ment (2003) forthcoming Community nutrition Bangladesh Integrated Nutrition $58.6 Monthly growth monitoring and promotion for children <2 and pregnant or lactating women (PLW); supplementary feeding of mal- White and (1995­2002) ($65.7) nourished PLW and severely malnourished and growth-faltering children <2; nutrition education for pregnant women, mothers of Masset 2007/IEG children <2, and adolescent girls at community nutrition centers.d Implemented by community nutrition promoters linked to NGOs 2005; Hossain in some thanas, by government in others. and others 2005 Madagascar Community $46.7f Community nutrition worker supervised by an NGO provides growth monitoring of children under 3, vitamin A to children and PLW, (1) Galasso and Nutrition II ($67.9) education and cooking demonstration, and food supplementation for severely malnourished children. Evaluation of program from Umapathi 2009; (1998­2009)e (1) 1999­2002 and (2) 1997­2007. (2) Galasso and Yau 2006 Senegal Nutrition Enhance- $14.4 Growth monitoring and promotion, behavior change communication through female CNAs for pregnant women and children <3, Alderman and ment (2002­06) ($23.1) basic health services by CNAs (managed by NGOs) and district health staff (vitamin A, iron, de-worming, insecticide-treated bednets, others 2009; sick child consultations), for children <5 and PLW. Evaluation of program 2004­06. Linnemayr and Alderman 2008 Early child development Philippines Early Childhood $17.3 Enhanced early child development services to pregnant women and children <7. Integrated and multisectoral approach to deliver- Armecin and Development ($49.6) ing center-based and home-based interventions, linked by a child development worker who offers food and nutrition supplements, others 2006 (1998­2006) monitoring of health status, and parent education. Upgraded facilities. Uganda Nutrition and Early $35.1 (1) Community-based growth monitoring and promotion carried out by childcare workers, and "child days" offering vitamin A, rou- (1) Alderman Childhood Develop- ($40.6) tine and catch-up immunization, and promotion of family care practices, children 0­6. 2007 Evaluations of World Bank Nutrition Support ment (1997­2005) (2) De-worming of children 1­7 years old, in addition to the standard community nutrition program. Evaluated from 2000 to 2003. (2) Alderman and others 2006 Bolivia Integrated Child $25.6 Nonformal, home-based day care centers (PIDIs) for children between the ages of 6 months and 6 years in poor families in 34 low- Behrman, Development ($60.6) income urban areas. Services included food supplements for malnourished children (70%­100% of daily requirement), access to Cheng, and Todd (1993­2004) health care, and early child education. Women trained to offer childcare for up to 15 children in their homes, with a grant/loan of 2004 up to $500 to bring the home to minimum standards of safety and hygiene; two to three caregivers per home from the community trained by the project. Evaluated for 1996­98.g Sources: Project Appraisal Documents and Implementation Completion and Results Reports. Note: CCT = conditional cash transfer; CNA = community nutritiion aide; NGO = nongovernmental organization; PIDI = Integrated Child Development Project; PLW = pregnant or lactating women; UCT = unconditional cash transfer. a. Expressed in $ millions. Final disbursements, except for Madagascar Community Nutrition II, which was still active as of October 2009. b. 2002 data. c. BDH was supposed to be conditioned on children between 0 and 5 years of age receiving bimonthly visits to health posts for growth and development checkups and immunizations and 90% attendance for children aged 6­15 years. However, these conditions were not enforced. d. There were several other activities at the national level (information, education, and communication, vitamin A supplementation, and salt iodization, gardens, and poultry) that either never materialized, had weak capacity building compared with what was envisaged, or were implemented on a limited scale (Pelletier and others 2005). e. Project still active. | f. Originally $26.7 million, plus two additional financings of $10 million each. 29 g. Beginning in 1997, the intervention underwent substantial changes to reduce the cost and increase the coverage. yond normal project monitoring and evaluation. However, Almost all the programs were large- one of the triggers for moving from Phase I to Phase II of scale government programs with many the Adaptable Program Loan was an independent evalua- interventions and long results chains. tion of Phase I. In Madagascar, the impact evaluation was not foreseen until well after the Community Nutrition II Three-quarters of the programs evaluated were com- Project was approved. The BINP conducted an evaluation pletely new government programs. Familias en Acción in at the end of the project that was said to measure impact Colombia was a CCT program that had only been piloted (Karim and others 2003), but the two external evaluations in a few towns and was to be launched on a large scale. Ec- reviewed here (Hossain and others 2005; White and Masset uador's BDH was to be a better-targeted CCT, replacing an 2007) were conducted after the project closed and were not unconditional, poorly targeted program (Bono Solidario).3 foreseen.8 The community nutrition programs in Senegal (PRN) and Most of the impact evaluations involved Madagascar (SEECALINE) had been previously piloted World Bank researchers. and were evaluated in the first major scale-up phase. Three-quarters of the evaluations, representing six of the However, in Uganda there had been no pilot for the early eight projects, were led by or done in coordination with child development program. It was evaluated in one region researchers in the Bank's Development Research Group. of the country, while the program was national in scope, The three exceptions were the evaluations of BINP by Hos- targeted to the most malnourished areas. The PIDI child sain and others (2005), sponsored by Save the Children care program in Bolivia was also totally new, based on only Federation/UK (SCF), and by White and Masset (2007), a year's experience with pilot activities and modeled after a sponsored by IEG; and the evaluation of Familias en Ac- successful program in Colombia.4 ción in Colombia, for which the government contracted In contrast, two of the programs were ongoing when with a consortium of research groups (Attanasio and others evaluated. The early child development program in the 2005).9 Philippines aimed to improve ongoing services through World Bank researchers were involved in better inputs and a multisectoral delivery mechanism that used a new type of community worker. The IEG-financed conducting the evaluations of six of the impact evaluation of the community nutrition activities of eight projects. the BINP (White and Masset 2007) arose out of a need to reconcile conflicting findings of impact evaluations gener- World Bank research evaluators often participated in ated by the project and by the Save the Children Federation project preparation or supervision, but not directly in (Hossain and others 2005), neither of which had robust data collection. In Ecuador and Uganda, the researchers control groups. participated in project appraisal missions, and in all six countries they participated in supervision missions (fi- Most of the impact evaluations were foreseen nanced through Bank operational budgets), either to su- at project appraisal. pervise the implementation of the impact evaluation or the Most of the impact evaluations were foreshadowed in the other project monitoring and evaluation activities.10 Project Appraisal Document (PAD) as part of the proj- ect's monitoring and evaluation plan.5 The PAD for the Other than influencing the timing of the rollout of the inter- Bolivian early child development project, PIDI, defined ventions in Ecuador and Senegal, the evaluation designs were the intervention group and two control groups; the impact not reported to have affected the design of the project or the evaluation of PIDI was part of the project's monitoring and intervention. In Colombia, Ecuador, and Senegal, household evaluation component.6 The Colombian Human Capital surveys were contracted out to private firms, some of which Protection Project PAD called for an evaluation with "a had experience in implementing the DHS. In Bolivia and comparison group that will provide a counterfactual for Madagascar, the data were collected by national statistical what would have occurred had the Project not been imple- offices. Only in the Philippines and Uganda were university mented"; the evaluation was to be external. Preparation research institutes directly responsible for data collection.11 and implementation milestones of the impact evaluation Some of the evaluations were linked to of Ecuador's BDH were triggers for the approval of each of program monitoring data. the three planned operations in the Programmatic Human The evaluations drew to varying degrees on program Development Reform series.7 monitoring data. The cash transfer evaluations in Colom- In contrast, the Senegal Nutrition Enhancement Project did bia and Ecuador used data from banking and administra- not explicitly mention an impact evaluation above and be- tive systems to verify the timing and amount of the transfers 30 | What Can We Learn from Nutrition Impact Evaluations? to beneficiaries. The evaluation of early child development collection across multiple studies. The time costs of the in the Philippines used administrative data to document World Bank researchers and academic evaluators are not exactly when the improved services became available. The easily documented. However, it is possible to document the evaluation of the community nutrition intervention in sources of funding for these impact evaluations (table 3.2). Madagascar used aggregated program data by site on the percentage of children who were malnourished (as col- They were financed by projects, lending lected by the community nutrition worker). operations, World Bank budget, and trust funds. The evaluations used program monitoring data to different degrees. Governments financed at least part or most of the im- pact evaluations--usually data collection--through the The two evaluations of BINP drew on program data for the lending operation, whereas the data analysis was often analysis of supplemental feeding of severely malnourished subsidized from other sources. Seven of the eight projects or growth-faltering children, and the White and Masset financed data collection and, in some cases, analysis of the evaluation (2007) used the project's midterm and end- data used for the 12 impact evaluations. The Senegal PRN line household survey data. The evaluation of the impact project financed $700,000 for the first- and second-round of adding de-worming for preschool children to Uganda's surveys for the impact evaluation (World Bank 2007b). early child development program relied on the program's Only the evaluation of Colombia's CCT, Familias en Acción, child-weight monitoring data. However, the evaluations of was completely funded by the project, including data col- Photo by Yuri Kozyrev, courtesy of the World Bank Photo Library. early child development programs in Bolivia and Uganda lection and analysis. In contrast, very little of the evalua- and of community nutrition in Senegal reportedly did not tion of Ecuador's BDH, an unconditional cash transfer pro- link to any program monitoring data. gram, was financed by the government.12 The data used by Galasso and Yau (2006) in Madagascar were entirely from The evaluations were financed from diverse routine administrative sources and entailed no additional sources. data collection expenditure. It is difficult to obtain exact information on the costs of most of the evaluations, because all but one (Familias en Acción, The World Bank research budget supported Colombia) received funding from multiple sources. Further, evaluations in six of the eight countries. in some cases more than one evaluation was conducted us- ing the same data set (for example, the BINP evaluations Evaluations in six of the eight countries also received sup- by Hossain and others [2005] and by White and Masset port from grants by the World Bank Research Committee [2007]), or one of the evaluations piggybacked on the other for two research proposals for a total of $600,000.13 Other (the de-worming and early child development evaluations sources of finance for either data collection or analysis in Uganda). One would have to allocate the costs of data included World Bank project supervision budget support Evaluations of World Bank Nutrition Support | 31 TABLE 3.2 Sources of Funding for Evaluations of the Impact of World Bank­Supported Programs on Nutrition Outcomes Sources of funding World Bank supervision World Bank World Bank Country Projecta Project budget researcher time research support Trust fund Other Cash transfers Colombia Human Capital Protection/FA Ecuador First Programmatic b Human Develop- ment Reform/BDH Community nutrition Bangladeshc Integrated d Nutrition/BINP IEG, SCF Madagascar Community f UNICEF Nutrition II/ SEECALINEe Senegal Nutrition Enhancement/PRNe Early child development Bolivia PIDI g Philippines Early Childhood g Uganda Nutrition and Early g Childhoode Total 7 5 5 6 3 5 Sources: Interviews with task team leaders and evaluators, research committee funding proposals, and PADs. a. BDH = Bono de Desarrollo Humano; BINP = Bangladesh Integrated Nutrition Project; FA = Familias en Acción; PIDI = Proyecto Integral de Desarrollo Infantil; PRN = Programme de Renforcement de la Nutrition; SEECALINE = Projet de Surveillance et Éducation des Écoles et des Communautés en Matière d'Alimentation et de Nutrition Élargi. b. Japanese Policy and Human Resources Development Fund Grant, Spanish Impact Evaluation Fund. c. Includes sources of funding for all three BINP evaluations--by the project team, by SCF, and by IEG. d. Department for International Development partnership, Danish Trust Fund. e. Includes funding sources for more than evaluation of the program. f. Bank Netherlands Partnership Program Trust Fund. g. Co-investigators brought funding from additional sources. (Ecuador, Madagascar, the Philippines, Senegal, Uganda), cash transfer was randomized such that the communities trust funds (Bangladesh, Ecuador, Madagascar),14 IEG receiving the intervention in future years could serve as the budget (Bangladesh),15 and research funds from academic control group for the communities that received the inter- co-investigators. vention at the start (Paxson and Schady, forthcoming). The random assignment to the rollout was maintained. The Design and Implementation of the However, in Uganda and Senegal, the randomized as- Evaluations signments did not go according to plan. A de-worming intervention for young children in Uganda was randomly Most evaluations used quasi-experimental assigned among areas already receiving an early child de- designs. velopment intervention, but some households in the con- Few of the nutrition impact evaluations attempted to trol group nevertheless increased purchase of de-worming randomize the assignment of the program; those that did medicine on their own (Alderman and others 2006). so randomized assignments at the community, not the individual, level. Only three of the evaluations attempted The attempt to use a randomized program rollout to provide to randomly assign the program, and of these only one was for treatment and control groups in Senegal for the PRN able to maintain a relatively clean design during project was foiled when the NGOs responsible for implementing implementation. The rollout of the BDH unconditional the program did not adhere to the plan--postponing its 32 | What Can We Learn from Nutrition Impact Evaluations? launch in some of the treatment areas and implementing it ing of households and children nationwide with character- earlier than planned in some control areas (Alderman and istics similar to those of the treatment group and the other others 2009; Linnemayr and Alderman 2008).16 Neverthe- of households in the same neighborhood that did not enroll less, the evaluators in both of these cases were able to use their children in the program (Behrman, Cheng, and Todd the randomized assignment as an instrumental variable 2004). These groups were not found to be sufficiently similar; to predict treatment, purging the impact estimates of self- consequently, the authors used matching methods to con- selection bias. trol for selectivity into the program. In fact, 7 of the 12 evalu- ations employed PSM, either because they had no control or Three of the impact evaluations had comparison group or because the selected control groups were randomized designs, but the designs for found to be inadequate (Armecin and others 2006; Attanasio and others 2005; Behrman, Cheng, and Todd 2004; Galasso two were not fully realized. and Umapathi 2009; Galasso and Yau 2006; Linnemayr and For either political or practical reasons, most of the eval- Alderman 2008; White and Masset 2007).19 uations used quasi-experimental methods for estimating Evaluations in Bolivia and Madagascar program impact. Policy makers in Colombia, for example, were unwilling to embrace randomized rollout of interven- compared cohorts exposed to the programs tions at a time of political crisis. In the evaluation of Fa- for different amounts of time. milias en Acción, Attanasio and others (2005) compared randomly selected treatment municipalities with matched Evaluations in Bolivia and Madagascar estimated mar- control municipalities on the basis of geographic region, ginal impacts of program exposure by comparing co- education and health infrastructure, population, and other horts of participants who had been in the program for characteristics.17 They estimated the impact based on the different amounts of time with those who had only re- difference-in-difference between treatment and control ar- cently joined. In Bolivia, children enrolled in PIDI for two eas over time. However, there were still fundamental dif- months or more (up to more than 25 months) were com- ferences between the baseline treatment and control areas pared with children enrolled for a month or less. In Mad- that led to the use of propensity score matching to generate agascar, communities that had participated for two years a control group.18 In Uganda, Alderman (2007) compared were compared to matched communities that had partici- project areas with controls that were nonproject subcoun- pated for one year, and both were compared with commu- ties adjacent to each subcounty in the study; the areas were nities that had just enrolled. The evaluation used regularly found to be sufficiently similar in characteristics to sim- collected administrative data of the community nutrition ply compare the mean effects between the treatment and program, supplemented in later phases by household sur- controls. veys (box 3.1). One of the advantages of this approach is that examining the effects of additional exposure does not Most of the evaluations of the nutrition require a control group.20 impact of Bank-supported programs had a People in control groups spontaneously quasi-experimental design. adopted the same activities as those The evaluations that drew on existing data sets or programs assigned to the treatment groups in the already under way did not have the option of a prospective Philippines and Uganda. experimental design. For example, the BINP in Bangladesh was ongoing when evaluated by two sets of researchers, Crossover effects were experienced in evaluations with which led them to choose matching methods. The proj- comparison groups as well as in those with control ect and nonproject comparison areas used by Hossain and groups to which the intervention was assigned on a ran- others (2005) were not good matches; White and Masset dom basis. Parents of about a third of the Ugandan children (2007) used the BINP project survey data for the treatment in the control group got their children de-wormed (Alder- areas but used PSM to generate a control group using a man and others 2006). In the Philippines, nonproject areas third, nonproject data set. spontaneously adopted some of the activities of the early child development program being evaluated (Armecin and The evaluation design for the PIDI program in Bolivia others 2006). In both cases, these crossover effects resulted called for comparing a random sample of program partici- in muting the difference between the treatment and control pants with two matched comparison groups--one consist- or comparison areas. Evaluations of World Bank Nutrition Support | 33 BOX 3.1 Measuring the Impact of Additional Exposure to a Community Nutrition Program Using Program Data in Madagascar The second Community Nutrition Project in Madagascar supported a community-based nutrition program implemented by community nutrition workers (CNWs) supported by NGOs. According to the PAD, the CNW is elected by the com- munity, trained, and receives an annual salary of about $350­$400. The community identifies a nutrition center, can get a grant of up to $200 to furnish it, and receives basic weighing and measuring equipment. The centers are to cover a population of 2,000 with the capability of covering 226 children within 5 kilometers. Social workers are also recruited by local NGOs. The CNW conducts a census of all children under three years of age at the outset and annually thereafter. The CNW weighs all children under three monthly and gives the mothers nutrition education and a cooking demonstration. Malnourished children get food supplements and are monitored every two weeks. Children who weigh in at < ­3 SD WHZ are sent to the health system for rehabilitation. Vitamin A supplements are given once a year to children under 24 months, twice a year to children 24­36 months, and to lactating women within six months of delivery. The impact evaluation used aggregated routine monitoring data from 1999 to 2002 from four main provinces, from about 3,600 sites and about a quarter of all communities in the country. The authors used the time delay involved in the rollout of the program to compare participating communities with one or two years of intervention with communities just starting. Because the phase-in began with the most severely affected communities that also had NGOs, the authors used PSM to adjust for selection bias. The evaluation found that two years' exposure to the program reduced the per- centage of children under three years or age who were underweight by 7­9 percentage points, from an initial level of 46 percent. Sources: Galasso and Yau 2006; World Bank 1998. The evaluations measured short-run program is known, yet the program was supposed to be impacts. phased in over a five-year period. This was apparently not Most of the evaluations assessed impact over a relatively controlled for in either evaluation. short period following the launch of the intervention. Evaluation of large-scale government Two-thirds of the evaluations measured impact after no programs presented challenges. more than three years of implementation and, in 7 of the 12 The evaluations faced major challenges because of delays in cases, two years or less. project launch, disruptions in delivery, political pressures The quality of the service may improve over time following not to follow the plan, and disruptions caused by political a learning curve, and longer exposure may independently pressure, natural disasters, and other breakdowns in pro- affect the impact if there is a dose-response relationship. For gram implementation. these two reasons, somewhat less impact may be expected Delay in launching the intervention. In Uganda the early for certain interventions (for example, for an intervention child development program baseline survey was done in to affect chronic malnutrition) over a relatively short imple- mentation period. Failure to control for the actual launch January­March 2000, but the growth-promotion interven- date can result in an underestimate of the impact or to a tion did not begin until late 2001 and the community nu- finding of no impact at all. trition grants started in 2002. As a result, the intervention had been operational for only a little more than a year by The evaluations measured short-run the time of the endline survey in January­March 2003. impacts on malnutrition, generally within Delays in project effectiveness delayed two years of program start-up. baseline surveys in Bolivia and the The initial findings of the Philippines early child develop- Philippines. ment impact evaluation found little or no impact; not until the researchers went through administrative records to pin- In Bolivia the baseline data collection was postponed two point when services became available for each community years because of a delay in project effectiveness. Partly be- did significant results appear (Armecin and others 2006). cause of the extensive delays in launching the early child The two evaluations of BINP may suffer from this problem; development project in the Philippines, the results of the only the rough starting date--about 1996--of the overall first round of evaluation found very little impact. This led 34 | What Can We Learn from Nutrition Impact Evaluations? the researchers to seek from administrative records exactly financings, in response to cyclone damage in 2000 and 2004 when the intervention had been launched in each site, to and to political turmoil in 2002. The first of the restructur- be assured that the project areas were, in fact, exposed. The ings added rural areas of 16 more districts to the 52 districts delays, however, were not always detrimental: in Senegal, already targeted and urban areas of 6 districts, representing the delay in project implementation allowed time to design 550 more sites. The 2006 amendment expanded the pro- the impact evaluation. gram to include children under five in selected communi- ties in all 110 districts of the country. Beyond this, there Political pressures not to follow the plan. The evaluation were regular disruptions in the availability of food for the of BDH in Ecuador was supposed to have three arms--a take-home rations that were to be issued to children who CCT, a UCT, and a control group. However, the government did not gain weight for two months. never enforced the conditionality, so for all intents and pur- poses, it was an unconditional transfer and there were twice In Bolivia, within two months of approval of the project in as many treatment households as there were controls. 1993, a new administration took office that had concerns about the scale and financing of PIDI. In 1994, the Decen- In Colombia, there was an election and political change tralization and Popular Participation Laws were enacted, shortly after the researchers were awarded the contract to which made municipalities and departments responsible collect the baseline data. This created pressure to scale up for social service investment decisions, and at the end of the program before the baseline data could be collected. At 1995, the implementing agency was dissolved and the proj- the time that the baseline data were being collected, in 2002, ect was assigned to the Social Investment Fund. In the Phil- some towns were already participating in the program. In ippines early child development project, there were several Uganda, pressure from parliament led to the expansion of changes in the Project Management Unit. Following each the project to more districts than planned without increas- change, the researchers had to rebuild support for the im- ing the budget. Although this did not expand the scope of pact evaluation. the impact evaluation, it reduced the resources for imple- menting the project, and the project ran out of money be- Findings fore many activities could be completed. Three-quarters of the 12 impact evaluations found a In Colombia the program was expanded positive impact on anthropometric outcomes of chil- before the baseline could be implemented. dren in at least one age group, although the magnitude was in some cases not large or the impact applied to a In Senegal, during the delay in project implementation, narrow age group.21 The evaluations are notable not only NGOs conducted social mobilization to prepare and orga- for the variability in their findings (discussed below) but nize the communities destined to participate in the proj- also for the extent to which the complex results chain was ect. This made it difficult, once the evaluation design was documented, so as to put forward a plausible story of cau- finalized, for the researchers to explain to some commu- sation and to understand the extent to which the interven- nities that in fact the services would be delayed a year or tions were actually implemented. When implementation two because of the need to randomize the rollout. In fact, is spotty, it can be as if there is no intervention at all. The the NGOs in charge of implementing the intervention did anthropometric impacts and the extent to which the evalu- not respect the randomization of communities, electing to ations documented program outputs and intermediate be- launch the intervention in some phase 2 areas and delay it havioral outcomes are summarized in table 3.3. in the phase 1 areas. As a result, 30 percent of the villages that had been randomly selected to get the intervention in Three-quarters of the evaluations found the first round did not get it, and eight of the control vil- program impacts, but little is known about lages in the first round did (Alderman and others 2009). what part of the intervention worked. NGOs in Senegal mobilized communities Average impacts for similar interventions were before the impact evaluation design was variable; links to the underlying causal chain finalized, making it difficult to respect the were weakly documented. randomization plan. Cash transfers. Colombia's Familias en Acción, a CCT pro- gram, would be expected to affect nutrition status through Disruption in service delivery caused by changes in the the additional income of the cash transfer and the condi- political context, natural disasters, or breakdowns in tionality on use of health and education services. However, program implementation. In Madagascar the SEECA- in Ecuador's BDH, an unconditional cash transfer program, LINE project was amended five times, with two additional only the income effect would be operating. The evaluations Evaluations of World Bank Nutrition Support | 35 TABLE 3.3 Nutrition Impact Evaluations and the Results Chain for World Bank Projects Intermediate Was there an Heterogeneity of Evaluation Program output outcomes anthropometric impacts Program type Country Evaluation period data analyzed? analyzed? impact? analyzed? Conditional cash Colombia Attanasio 2002­06 Yes. Administra- Yes. Diphtheria, Yes. HAZ (espe- No transfer and others tive data on pay- pertussis, and cially for children 2005 ments, health, tetanus vac- <24 months) and and education cination rate; newborn weight. service data. reported food intake; participation in growth monitoring. Unconditional Ecuador Paxson and October 2003/ Yes. Bank records Yes. Participa- No. HAZ Yes. Household cash transfer Schady, September 2004­ of transfers and tion rate; use of (ages 3­7). poverty; gender.a forthcoming September 2005/ when started. health clinics January 2006 for growth monitor- ing; sought treatment for helminth infections. Community Bangladesh Hossain and 1996­2002b Yes. Children Yes. Mother's No. WAZ, HAZ, No nutrition, others 2005 receiving food; nutrition WHZ (ages 6­23 including food effectiveness knowledge months). supplements among those and reported enrolled; food practice. leakage; food substitution; village health worker quality. White and November/ Yes. Receipt Yes. Participa- Yes. WAZ and Yes. House- Masset 2007/ December of counseling; tion rate for HAZ (age 6­23 hold assets IEG 2005 1998­January/ receipt of food; weighing; months), but and mother's March 2003 targeting of nutrition small in education. food; duration of knowledge; magnitude. food. practice. Madagascar Galasso and 1999­2002 Yes. Characteris- Yes. Registra- Yes. Underweight Yes. Community Yau 2006 tics of the NGOs. tion rate. (age <3 years), poverty; cyclone- relatively large in prone areas; magnitude. length of lean season; access to safe water. Galasso and 1997­2007 Yes. Receipt Yes. Breastfeed- Yes. WAZ, Yes. Mother's Umapathi of vitamin A ing; feeding underweight, education; low- 2009c and message; practices; hy- HAZ, stunting, poverty areas; tetanus injection giene practices; relatively large proximity to road, during preg- diarrhea. (age < 5). hospital; access nancy; assisted to safe water, delivery; posses- electricity. sion of health card; receipt of nutritional counseling. 36 | What Can We Learn from Nutrition Impact Evaluations? TABLE 3.3 (continued) Nutrition Impact Evaluations and the Results Chain for World Bank Projects Intermediate Was there an Heterogeneity of Evaluation Program output outcomes anthropometric impacts Program type Country Evaluation period data analyzed? analyzed? impact? analyzed? Community Senegal Alderman 2004­06 No Yes. Receipt Yes. Underweight No nutrition, and others of iron supple- (age <3). without food 2009 ments; malaria supplements pills by mothers; receipt of vitamin A; de-worming; ownership of bednets. Linnemayr 2004­06 No Yes. Health Yes. WAZ (age <3). Yes. Villages with and inputs; nutri- seasonal roads; Alderman tion knowledge villages with lower 2008 of mother; average wealth at breastfeeding baseline practices. Early child Bolivia Behrman, 1996­98 No No No. Weight No development, Cheng, and percentile and with food Todd 2004 height percentile (ages 6 months to 6 years). Philippines Armecin and 1996­98 Yes. Early child No Yes. WHZ and No others 2006 development wasting. Mixed worker training results­HAZ and and func- stunting (age <7). tions; feeding programs; parent educa- tion seminars; home-based day care; exact onset of program. Early child Uganda Alderman 2000­03 No Yes. Breastfeed- Yes. WAZ among Education; development, 2007 ing and wean- those <12 imputed without food ing practices; months. expenditure reported foods fed to children. De-worming Uganda Alderman 2000­03 Yes. Number Yes. Uptake. Yes. Weight (age No and others of child health 1­7 years). 2006 days; treatment intervals. Source: IEG analysis. Note: HAZ = height-for-age z-score; WAZ = weight-for-age z-score; WHZ = weight-for-height z-score. a. The interactions for heterogeneity are not for HAZ individually, but rather a synthetic variable for physical development. b. The authors assumed that the intervention began in 1996, the year the project was approved. However, implementation was supposed to be phased, and it is not clear when the intervention actually became available to the survey villages. Thus, the exposure may be significantly less than six years. c. The outputs and intermediate outcomes are presented in Galasso and Umapathi 2009, a working paper that was revised for publication, from which this information was dropped. Evaluations of World Bank Nutrition Support | 37 of both programs document the disbursement of the trans- actually exposed, a factor that can result in underestimating fers and changes in intermediate outcomes that would be impact.22 consistent with improved nutrition outcomes--an increase Food supplements for women or children typically account in vaccination rates and reported food intakes in Colombia, for a very large share of the costs of nutrition programs treatment for helminth infections in Ecuador, and partici- and are logistically difficult, yet their effectiveness was as- pation in growth monitoring in both countries. sessed only in the two BINP evaluations, even though food In Colombia there was an impact on HAZ for children was an element in the programs for half of the eight coun- younger than two but not of children two to four or older. tries.23 White and Masset (2007), using data collected from There was no impact of Ecuador's Bono de Desarrollo health centers by the SCF authors (Hossain and others Humano on HAZ, but the children studied in that evalu- 2005), found important targeting problems. Only 16 per- ation were between three and seven years of age. Thus, cent of children receiving the food should not have re- the finding of no impact for children over two is consis- ceived it, whereas more than two-thirds of the children tent across the two programs; it cannot be compared for who were eligible (that is, those with severe malnutrition younger children. or growth faltering) were not fed. Among those receiving Another factor contributing to different findings could be food, only a quarter received the supplements for the rec- that the transfers had been in place in BDH for two years or ommended three months. More than 40 percent of the less, half the exposure of the Familias en Acción at the time children who were receiving supplements were not mal- of the evaluation. Different access to health care in the two nourished, but were receiving them because their growth countries could also have played a role, though that infor- was faltering. The authors note, however, that growth fal- mation was not presented. tering is normal. Community-based nutrition programs. The six evalua- An evaluation of Bangladesh's BINP was tions of community-based nutrition programs in Bangla- the only one that assessed the impact desh, Madagascar, and Senegal generally found positive effects on weight and, when measured, height, though the of supplemental food for malnourished size of the impact varied and many of the evaluations children. suffered from a lack of information on the extent to which the interventions were implemented. Community nutrition programs in Madagascar and Sene- gal had positive impacts on WAZ or underweight, primar- The impact of BINP on WAZ and HAZ was small, even ily for children under three. In the case of Senegal (a nu- though the mothers' knowledge improved. It is not clear trition program that does not dispense food), Alderman why. Many possible implementation factors could have and others (2009) and Linnemayr and Alderman (2008) been responsible; for example, the performance of the com- track important intermediate outcomes to explain those munity nutrition promoters (CNPs) and the large number improvements--receipt of iron supplements and malaria of people that each CNP was supposed to serve (more than pills by the mothers, receipt of vitamin A, de-worming, 1,000). However, the evaluation did not explore this issue. It ownership of bednets, and breastfeeding. Yet neither of is also not known how long each of the communities was the evaluations for Senegal documents the extent to which the interventions were actually implemented. One of the evaluations of Madagascar's SEECALINE program docu- Photo by Curt Carnemark, courtesy of the World Bank ments changes in intermediate outcomes that are consis- tent with improved nutrition found in the evaluation-- breastfeeding, hygiene, and feeding practices (Galasso and Umapathi 2009). Early child development. The results of the evaluations of the early child development programs were likewise vari- able. Bolivia's PIDI, a nonformal, home-based day care pro- gram, had no effect on any anthropometric indicators for children six months to six years old, despite the fact that the program provided meals to the children amounting to 70 Photo Library. percent to 100 percent of their daily needs. There were sig- nificant program impacts on weight and wasting in the Philippines and Uganda, although only for children less than one year of age in Uganda,24 and mixed effects on HAZ 38 | What Can We Learn from Nutrition Impact Evaluations? and stunting among children under seven, depending on gion with the highest worm load, and both the treatment the age group, in the Philippines. and control areas had access to the early child develop- ment intervention. Despite the large number of activities embedded in these programs--including growth monitoring and food supple- Only half of the evaluations documented ments found in the community nutrition programs--the heterogeneity in impacts. results chain of program outputs and intermediate out- Only half of the impact evaluations explored the distri- comes for these three evaluations is weak. bution of impacts across individuals or communities.25 The greatest challenge was for the evaluation in the Philip- The coverage of heterogeneity and the variables considered pines of the improvement and reorientation of an existing by each study are presented in the last column of table 3.3. early child development program. The research teams had Poverty. Six of the 12 evaluations assessed whether poorer to go from center to center to assemble the necessary ad- households or communities benefited more than the non- ministrative data documenting exactly when the interven- poor. In Ecuador, the impacts were larger among the lowest tion began. The evaluation shows convincingly that in the quartile of eligible families (Paxson and Schady, forthcom- program areas the number of trained workers, feeding pro- ing).26 In Madagascar, the SEECALINE program, which grams, day care centers, and other activities increased rela- was targeted to the poorest and most malnourished areas, tive to the control areas. Even then, there is little evidence had the largest impact on all four anthropometric out- provided to demonstrate how well the services were deliv- comes in the better-off communities; in the communities ered, and no information was presented on intermediate with the highest poverty rates, only children of the most outcomes that might logically be linked to the nutritional educated mothers had better anthropometric outcomes outcomes observed. (Galasso and Umapathi 2009). In contrast, the sites with the highest poverty rates had higher returns to program The impact of Uganda's early child exposure over two years (Galasso and Yao 2006). However, development program is supported by in Bangladesh, Senegal, and Uganda, there was no differ- changes in breastfeeding and weaning ence in impact in less wealthy households (IEG 2005), in practices. poorer communities (Linnemayr and Alderman 2008), or in households with lower imputed expenditures (Alder- In contrast, the evaluation of early child development in man 2007), respectively. Uganda presents no evidence on program outputs but does document changes in breastfeeding and weaning practices In Ecuador the benefits were greatest and in the foods reportedly fed to children (Alderman for the lowest income families, whereas 2007). The evaluation of Bolivia's PIDI program, which in Madagascar children in better-off found no anthropometric impacts, provides no informa- communities in the targeted poor areas tion on either program outputs or intermediate behavioral outcomes that might explain this result (Behrman, Cheng, benefited the most. and Todd 2004). Mother's education and child's gender. Three of the eval- The PIDI early child program in Bolivia uations assessed whether the impacts were greater for chil- had no impact on height or weight, even dren of more educated mothers than for children of less educated mothers. In Madagascar, results suggested that though the children were fed. children of educated mothers benefited more from the in- De-worming. The single study that tested the impact of terventions (Galasso and Umapathi 2009); the impact of adding de-worming to the ongoing early child develop- neither the Bangladesh community nutrition program nor ment intervention in Uganda found weight gains among the Uganda early child development program varied with preschool-age children (1­7 years) (Alderman and others mother's education (IEG 2005; Alderman 2007). Only one 2006). The results chain for this particular intervention of the evaluations examined the impact according to the was short. The evaluation used administrative records to child's gender, finding that impacts were greater for girls document each participating child's weight gain and the than for boys (Paxson and Schady, forthcoming). receipt of the de-worming drugs. There are few inter- mediate behaviors to document. However, it should be In Madagascar, program impact was greater noted that the results are likely underestimates, as a sizable in communities with roads . . . share of the parents in the control area spontaneously in- creased their purchase of de-worming medicine for their Availability of public services. Surprisingly, only three of children. Further, the evaluation was launched in the re- the evaluations examined the relation between the program's Evaluations of World Bank Nutrition Support | 39 impact and the availability of public services.27 Even women ous assumptions and discount rates, the benefit-cost ratio with better knowledge of good child nutrition practices was estimated to be between 1.7 and 4.5. may be limited in their ability to act on this knowledge if they lack access to complementary services such as health Costs and cost-effectiveness of the care or to markets. programs were rarely assessed, and cost- Galasso and Umapathi (2009) found that the impact of the benefit analyses were rarely performed. Madagascar SEECALINE community nutrition program on all of the anthropometric outcomes was greater with Cost-effectiveness, in contrast to cost-benefit, can be more proximity to a road or hospital, and that the WAZ impact easily calculated in the context of an impact evaluation was greater with access to a safe water source. However, the based on local data, actual implementation costs, and ef- other evaluation in Madagascar, which used aggregated fects. Because the impact of the Bangladesh BINP as imple- data across sites, found no difference in the returns to pro- mented is found to be so small, the cost to achieve a given gram exposure for communities with better access to safe outcome is high. The cost of preventing a child from being water (Galasso and Yau 2006). underweight was calculated to be $187­$333 per year, and for stunting $241­$490 annually, with an estimated cost per In contrast, the Senegal PRN community nutrition program life saved ranging from $2,328 to $4,095 (IEG 2005).30 had greater impact in more isolated villages not served by all-weather roads. That implies that the services of the nu- The marginal cost of adding de-worming medicine to the trition worker may have been substituting for services out- (then ongoing) early child development program in Uganda side the villages (Linnemayr and Alderman 2008). was calculated. Because the program was already distrib- uting vitamin A to the children, only the marginal cost of . . . but in Senegal community workers $0.42 was included for twice-yearly de-worming treatment that would result in a 10 percent increase in weight gain (or substituted for the availability of services. half that amount for once-a-year de-worming) (Alderman Program costs and cost-effectiveness were and others 2006). rarely assessed. The impact evaluations rarely remarked on the program The Impact of the Evaluations costs per beneficiary or conducted cost-benefit or cost- Is there any evidence that the findings of these 12 impact effectiveness analyses. In only three cases were costs pre- evaluations were used? This section pulls together evi- sented in the published evaluations (or their antecedents), dence of the use of the data and other impacts from these and in a fourth case (Madagascar), the analysis was done evaluations based on a review of the projects' Implementa- informally for the government based on the impact evalua- tion Completion and Results Reports (ICRs), the PADs of tion, but was not published. follow-on projects, any impacts of the findings mentioned The cost of the Bolivia early child development program in the evaluation reports, and interviews with key infor- was estimated by various sources to be as high as $43/ mants for each project--the World Bank project leaders, month and as low as $22/month per child enrolled (Beh- the evaluators, and at least one policy maker from six of the rman, Cheng, and Todd 2004). Either cost clearly would be eight countries.31 unsustainable for large numbers of children in Bolivia, with Because it was not possible to conduct country visits, these a gross domestic product/capita at that time of $800.28 Nev- findings should be considered partial and suggestive. None- ertheless, the cost-benefit analysis done by the authors sug- theless, the findings across the documents and individuals gests a benefit-cost ratio (under varying assumptions and consulted for each project were generally consistent. Table discount rates) between 1.37 and 3.66. This is based on the 3.4 summarizes evidence on the impact of the evaluations. extrapolation of future benefits for the nonanthropometric impacts, however, as the study found no impact on HAZ or The impact evaluations plausibly had an impact on pol- WAZ.29 From the perspective of the actual nutrition out- icy in two of the eight countries. In both countries, the comes, the benefit-cost ratio would be zero. intervention had a positive effect on child anthropometric outcomes. In unpublished calculations for government, the lead au- thor for the two Madagascar evaluations calculated the unit In Colombia, a new political administration came into cost of the SEECALINE program to be on the order of $7/ power in 2002, only a year after the project was approved. child/year and the cost of preventing one child from being There was reportedly great concern at that time about the stunted as $219/child/year (Emanuela Galasso, personal severe fiscal situation that affected all government pro- communication). After discounting the benefits with vari- grams and the high cost of the impact evaluation. However, 40 | What Can We Learn from Nutrition Impact Evaluations? TABLE 3.4 Summary of the Impact of the Nutrition Impact Evaluations Impact Were evaluation Was the Results found there a results Country-- reported nutrition follow-on in the interventiona in ICR? impact? project? PAD? Reported policy or program impact of the evaluation Cash transfers Colombia-- Yes Yes Yes Yesb Yes. Generated political support to continue funding when new president Familias en came into power and to scale up; the evaluation also "contributed to defin- Acción ing the larger social protection and evaluation agenda in the country" (World Bank 2006, p. 11). Findings supported dropping the restriction that children born since program launch be excluded and including children also enrolled in Hogares Communitarios. Ecuador BDH No No Yes Noc No. Government did not add conditionality and did not drop the next-to- poorest quintile, even though there were no benefits of targeting them. However, the evaluation greatly raised capacity in the ministry for conducting impact evaluations. Ecuador is pursuing impact evaluations of other programs. Community nutrition Bangladesh No Small Yesd No No. Respondents report that the program has not changed. However, one BINP respondent remarked that the Bank is paying more attention to the quality of service delivery as a result of the two evaluations. e e e Madagascar Yes Unclear. The project was expanded; the prime minister wrote a letter to The SEECALINE Lancet, along with the prime minister of Senegal. However, it appears that the program was politically popular even without the evaluation, so it is unclear whether it was really the impact evaluation that changed things. Senegal PRN No Yes Yes No Unclear. The program was scaled up in the second operation, which was the second phase of an Adaptable Program Loan; however, the results were not available at the time that decision was made. The evaluation may have been reaffirming. Early child development Bolivia PIDI Yes No No n/a No. The model evaluated was excessively expensive and subsequently adapted to a model quite different from the one evaluated. "All activities were ended as of December 2003 and none . . . were included or absorbed by other ongo- ing programs." "The family/home-based day care centers . . . have practically disappeared and most have been converted into community centers. Yet . . . they still have a high cost compared to other similar programs" (World Bank 2004, p. 30). Philippines Nof Yes No n/a Yes. Was reportedly used to justify expanding program innovations. Strong ECD ownership of the impact evaluation; the ECD head presented results at the 2004 World Bank Conference on Scaling Up Poverty Reduction in Shanghai, China. However, the ECD program had strong support even before the evalua- tion showed some impacts. Possibly reaffirmed existing support. Uganda ECD Yes Yes No n/a No. Community nutrition has been dropped from the program, although child days have continued. (The idea of child days was mentioned as attributed to UNICEF.) None of the ministries, especially the Ministry of Health, ever owned the project. De-worming Uganda Yesg Yes No n/a Unclear. The evaluation implies that the government expanded the ECD de-worming policy following release of results, but others report that the deci- sion to expand de-worming to preschool children had already been made. The evaluation may have influenced other African countries. Source: IEG analysis. Note: ICR = Implementation Completion and Results Report; PAD = Project Appraisal Document. a. BDH = Bono de Desarrollo Humano; BINP = Bangladesh Integrated Nutrition Project; ECD = early child development program; PIDI = Proyecto Integral de Desarrollo Infantil); PRN = Programme de Renforcement de la Nutrition; SEECALINE = Projet de Surveillance et Éducation des Écoles et des Communautés en Matière d'Alimentation et de Nutrition Élargi. b. There were two follow-on projects, both of which mentioned the results--the Social Safety Net Project (2005) and the Second Phase of the Program of Conditional Transfers--Familias en Acción (2008). c. The PAD for the follow-on project discusses at great length the results for the impact evaluation on education, but not the results (or lack of results) on health and nutrition. The follow-on project was canceled following a change in government. d. The follow-on project that scaled up BINP, the National Nutrition Project, was launched following the positive results reported for the BINP midterm review and before either of the evaluations (Hossain and others 2005; White and Masset 2007) was published. e. This project was scheduled to close in December 2009. There was not yet an ICR at the time of this review. Whether there will be a follow-on project is not known. f. The evaluation is mentioned and the trends in the treatment and control areas are charted, but the final evaluation results, as put forth in Armecin and others (2006), are not mentioned in the ICR. g. The results of the de-worming are inaccurately conveyed in the ICR, which says that the largest impact (a 10% increase in weight) was among the youngest children (that is, those under 12 months). The magnitude is correct, but it was for children aged one to seven years; infants were not given de-worming medicine. Evaluations of World Bank Nutrition Support | 41 the results of the first wave of the evaluation of Familias en its more than decade-long lifetime, and the project sup- Acción, which became available shortly thereafter, showed porting it still had not closed as of October 2009. impacts on schooling, health, labor supply, and consump- There have been changes in government following a period tion.32 The government not only expanded the program to of unrest; however, elements of the community nutrition new areas and broadened the eligibility to additional chil- activities have been incorporated into the new National dren within the original areas but also embraced a program Nutrition Program. The program was politically popular, of rigorous impact evaluation more generally in developing even before the evaluation, so it is unclear whether the evi- its social safety net program. dence from the evaluation contributed to its expansion. The impact evaluations of Bank support In Madagascar and Senegal, positive plausibly had an impact on policy in two of impacts may have helped maintain support the eight countries. for the programs. World Bank support was enlisted for two follow-on safety The Bank's support for Senegal's PRN was packaged as part net projects, including additional financing for Familias. of a multiple-phased Adaptable Program Loan. The evalu- The PAD for one of two follow-on projects (Social Safety ation found evidence of impact and has reaffirmed the ex- Net, approved in 2005) notes that "the program credi- isting government support for the program. However, the bility has . . . been fostered by the very positive results of findings were not available at the time of the decision to the conditional cash transfer evaluation that has been con- move to the second phase, making it unclear whether the tinuously disseminated since the early stages of program impact evaluation per se merely validated an ongoing com- implementation" (World Bank 2005, p. 15). The full results mitment or played a role in decision making. are cited in the rationale for the Second Phase of the Pro- gram of Conditional Cash Transfers/Familias en Acción The positive findings of the two impact evaluations the (approved in 2008, two years after the last round of data early child development program in Uganda and of the de- collection). worming for preschool children within that program were available at the time of the project's completion and cited in Early results from the Colombia's the ICR. However, the project, which was initially moved Familias en Acción helped convince a new from a multisectoral entity to the Ministry of Health soon after it was approved, never had strong support from the administration not to cancel it. latter. Further, it ran out of money and closed before being fully implemented. Child days have continued nationwide The complete findings of the impact evaluation of the Phil- even after the end of the project, although it was unclear ippines early child development project were not available whether this was the result of the evaluation of the early at the close of the project; as the ICR was being written, child development program or of efforts by UNICEF. The only the trends in the project and nonproject areas were government also introduced de-worming of preschool-age cited. There was already strong political commitment for children, although it was unclear whether this decision was the ongoing early child development program even as the taken before the impact evaluation results were known. program upgrades were introduced. Reportedly, since the project closed, many of the innovations have been incor- In three countries where the evaluations found no or porated more widely into the program. It is difficult to tell very small impact there was compelling evidence that the in this instance whether the evaluation merely reaffirmed impact evaluations had no effect. An evaluation finding of the wisdom of something that government was already set small impact or no impact should not necessarily lead to to do or whether it had a role in the decision to expand the the cancellation of a program--it could point to the need to innovations. introduce course corrections. However, this apparently did not occur in these three cases. In Madagascar, Senegal, and Uganda, evaluations found positive impacts on nutrition outcomes, but it was un- The BINP evaluation found a small positive impact of the clear whether subsequent program decisions were due community nutrition component on anthropometric out- to the evaluations. In Madagascar, following dissemina- comes and pointed to a number of weak links in the causal tion of the results of the evaluation of SEECALINE, the chain that could be addressed for greater impact or cost- prime minister wrote a letter to The Lancet (cosigned by the effectiveness (White and Masset 2007). The prior evalua- prime minister of Senegal) extolling the positive impacts of tion sponsored by SCF pointed to some of these weak links community nutrition programs (Sall and Sylla 2005). The as well, but concluded that BINP had no impact on nutri- SEECALINE program was expanded multiple times over tion outcomes (Hossain and others 2005). The decision to 42 | What Can We Learn from Nutrition Impact Evaluations? scale up the community nutrition activities in the form of of impact evaluations of social sector programs is being the National Nutrition Program was taken at the midterm pursued in Colombia with World Bank support; since 2002 of BINP, based only on trends in project areas and before the number of evaluations launched by the government has either of the impact evaluations had been issued. Nutrition risen from 3 to 30 to 46. has subsequently been absorbed into Bangladesh's sector- In Ecuador, respondents underscored that the experience wide program. with the impact evaluation greatly increased the capacity of Respondents indicated that the activities included in the the social sector ministry secretariat through their involve- community nutrition part of the program are basically un- ment in the design, piloting, and sample-selection phases. It changed and that the evaluations had had no real impact. reportedly led to a large change in the capacity to think about One respondent noted, however, that at least on the part Photo by Simone D. McCourtie, courtesy of the of the Bank there was much greater attention to the quality of implementation of the program, a point that was high- lighted in the evaluations. The evaluations in Bangladesh, Ecuador, World Bank Photo Library. and Bolivia found low impact, and the evaluations had little influence. In Ecuador, the evaluation concluded that BDH was better targeted than its predecessor, Bono Solidario; the evalua- tion found impacts on a number of dimensions, though not specifically for HAZ (only when aggregated with two other measures). The program was targeted to all households in the two lowest quintiles of the population--40 percent of and offer impact evaluations and, although the Bank's support the population overall; however, the benefits were demon- for this program and others was discontinued, the secretariat strated only in the lowest quintile. The recommendation has reportedly launched impact evaluations on its own. In to drop the second-lowest quintile from the program was the Philippines, the evaluation--which had strong local own- not taken, nor was the suggestion that impact might be in- ership--was reported by one respondent to have had broad creased by introducing conditionality based on enrollment impacts on the design of future government programs. and use of public health and education services. However, more recently, conditional transfers are being introduced in On the basis of the experience with the three provinces with the highest stunting rates.33 evaluating Familias en Acción, Colombia adopted a large program of impact The results of the impact evaluation of Bolivia's PIDI pro- gram were available in time for the ICR. The evaluation evaluations for other social programs. found impacts in a number of areas, though not on nu- tritional outcomes. Although there were political changes The scaling up of programs was often cited as evidence of during the course of the project, almost from the outset it the impact of the evaluations, but the features of programs was clear that the model was extremely expensive (about that were scaled up were often substantially different from $30/child/month) and not sustainable on a large scale in a those that were evaluated. For example, the findings of the country of the income level of Bolivia. As a result, the inter- evaluation in Colombia demonstrated impact in rural areas, vention initially evaluated was altered in major ways, such but the scaling up was done in urban areas. The need for an that what was ultimately adopted was much cheaper ($2/ urban pilot was recognized, but in the face of an election, the child/month) and sustainable, and not evaluated. The ICR intervention was expanded and the evaluation of the urban noted that "all activities were ended as of December 2003 pilot was canceled. and none . . . were included or absorbed by other ongoing The National Nutrition Program in Bangladesh scaled up programs" (World Bank 2004, p. 30). the BINP community nutrition interventions, but some Several of the impact evaluations were reported to NGOs in the new areas were less experienced. In Mada- have increased evaluation capacity or commitment to gascar program coverage has been extended to the whole evidence-based decision making, irrespective of the find- country, but the government has dropped key elements to ings. These included evaluations in Colombia, Ecuador, cut costs. These substantially different interventions have and the Philippines. The commitment to a broader agenda not been evaluated and their effectiveness is unknown. Evaluations of World Bank Nutrition Support | 43 Lessons · Evaluators must thoroughly understand the interven- tions being evaluated and when delivery of the in- The findings in this chapter underscore important lessons tervention effectively took place. Failure to take into for both program managers and evaluators that can guide account the timing of implementation can mute the mea- future evaluations of the impact of large-scale government sured impact of the intervention. programs on nutritional outcomes. · Impact evaluations need to collect rich data to docu- For managers: ment the delivery of program outputs, their quality, · Impact evaluations of interventions that are clearly be- and their intermediate outcomes to establish the plau- yond the means of the government to sustain are of lim- sibility of evaluation results and to point to parts of ited relevance. The complexity, absolute costs, and poten- the program that work and do not work. The nutri- tial sustainability of finance of the intervention should play tion impact evaluations reviewed here have generally into the decision as to whether it should be evaluated. failed to collect sufficiently rich data, including process · Impact evaluations are often launched for the pur- evaluations in parallel, to help identify what parts of the pose of evaluating completely new programs, but they program are working and to explain why some program may be equally or even more useful in improving the elements are ineffective. Too often, the lack of impact is effectiveness of ongoing programs. The prospects for not sufficiently followed up with an understanding of updating an existing program with broad political and how effectiveness can be improved. Any significant im- institutional support may be greater than those for a to- pact, even a small one for a subgroup, is often hailed as tally new program that has less ownership and may be evidence that the program worked, without understand- more politically contentious. ing how impacts can be enhanced. · There are ways of obtaining reliable results, even when · Evaluations need to provide evidence for timely de- randomized assignment of the intervention is not feasi- cision making, but with sufficient elapsed time for a ble for political, ethical, or practical reasons. Correctly plausible impact to have occurred. There is clearly ten- executed experimental designs are valuable for establish- sion between the need to report results quickly and to ing internal validity of the evaluation, but randomization ensure that the intervention has had time to work. There is not always possible, and even when attempted, it can are benefits to disseminating early baseline and midterm be derailed in implementation of large-scale programs. results prospectively, along with process data and inter- Quasi-experimental methods can also be used, alone or mediate outcome data that can point to changes along as backup to experimental evaluations, to address the is- the results chain, even when longer-term rounds of data sue of the counterfactual--for example, through match- collection are planned. ing techniques and analyzing the marginal impact of · Nutrition impact evaluations need to invest more in longer exposure to a program. documenting the targeting and cost-effectiveness of For evaluators: supplemental feeding for malnourished or growth- · Evaluators would be well advised to do an ex ante risk faltering children; the food element of the community analysis in designing impact evaluations of large gov- nutrition and early child development programs often ernment programs to anticipate how the risks to im- accounted for half or more of the total cost of the pro- plementing the evaluation can be reduced and to chart gram. Food distribution is often politically popular, but out a contingency plan in the event that risk mitiga- it creates many logistical problems and is demanding tion is not successful. Large public nutrition programs of implementers, who must prevent leakage. Different are sensitive to political changes and budget crises; these delivery mechanisms for feeding need to be evaluated factors should be considered in the planning of impact as well (for example, observed by a health worker versus evaluations to maximize the success of the evaluation take-home rations). (beyond any project-related risk analysis). · Evaluations of interventions to improve nutrition need · Nutrition impact evaluations, in their design and anal- to assess systematically the distribution of the benefits ysis of the data, need to take into account the sensitivity and the complementarities with public health and of different age groups to the interventions. Interven- other services. Too few evaluations assessed the extent tions found to be ineffective for a large age range may to which the poor disproportionately benefit in relation nonetheless be important for children at certain points in to the nonpoor, or the impact of the availability or quality their development, particularly during gestation and in of health services on the ability of the poor to act on the the first two years of life. information they receive on better nutrition. 44 | What Can We Learn from Nutrition Impact Evaluations? Chapter 4 Photo by Curt Carnemark, courtesy of the World Bank Photo Library. Conclusions High rates of childhood malnutrition in developing countries are raising mortality and present long-term consequences for survivors. Progress in reducing child malnutrition has been slow, and the global food and financial crises have no doubt created setbacks. In this context, the World Bank is expanding its support for nutrition and, in parallel, has launched several new impact evaluation initiatives. This review has attempted to inform these new efforts to Many things can go wrong, both in the quality of imple- improve the impact of nutrition support through a two- mentation of the intervention on the supply side and in the pronged approach. response of households on the demand side. · First, IEG reviewed the recent impact evaluation research This has several implications: on the effectiveness of interventions and programs in im- · It should not be assumed that an intervention found proving nutrition outcomes, focusing on child anthropo- effective in an RCT in the medical literature will have metrics and birthweight. Forty-six recent nutrition im- the same effects when implemented under field condi- pact evaluations were reviewed, representing evidence tions as part of a large program with a mix of interven- from 25 developing countries and a variety of interven- tions and in a population for which the underlying tions, including large-scale social programs of conditional factors affecting malnutrition may be fundamentally and unconditional cash transfers, community-based nu- different. trition, integrated health services, early child develop- ment, food transfers, de-worming, and micronutrient · It is important for the design of both the program and supplementation, among others. the evaluation to understand the prevailing underly- · Second, IEG examined in detail the experience from im- ing causes of malnutrition in any given setting. When pact evaluations embedded in World Bank projects that there are multiple channels and several are equally im- sought to affect anthropometric outcomes. Twelve im- portant, addressing only one of them may have limited pact evaluations reviewed in the first part could be linked impact. to evaluation of Bank support to eight countries. The re- · Impact evaluations need to collect rich data on pro- view examined the design, implementation difficulties, gram service delivery and demand-side behavioral findings, and impact of the impact evaluations, based on outcomes to explain nutrition impacts. Irrespective of a review of project documents, the evaluation results, and the evaluation design, it is critically important to under- interviews with Bank staff, the evaluators, and individu- stand not only whether the outcome is different between als from the borrowing countries. a treatment and comparison or control group but also The overarching conclusion of the review is that context why. When an evaluation finds no significant impact of matters. A wide range of interventions was found to have an an intervention that theoretically should have an effect, it impact on indicators related to height, weight, wasting, and is important to find out where in the causal chain the birthweight. In many settings, however, similar interventions program broke down. This involves conducting process had no effect. The magnitude of program impacts was not evaluations and collecting data to document the causal only difficult to compare across studies but also variable. chain in parallel. In particular, many interventions in- volve costly food supplementation, but the functioning, The findings overall do not lend themselves easily to gener- targeting, and impact of food supplementation are not alizations about what works and does not work in reducing tracked with respect to how it contributes to outcomes. malnutrition--particularly as applied in field conditions of developing countries. Some results are based on RCTs with Evaluations need to look more closely at the distribution short results chains. But when it comes to evaluation of of impacts. Very few of the evaluations reviewed examined more complex programs implemented outside of a research who is benefiting and who is not. Just because malnutrition setting the evaluation must document a long causal chain. is more common among the poor does not mean that they 46 | What Can We Learn from Nutrition Impact Evaluations? will disproportionately benefit from a nutrition program, and planning mitigation measures can help keep an evalua- particularly if acting on new knowledge or different incen- tion on course. Nutrition impact evaluations, in their design tives relies on access to education or quality services. Very and analysis, need to take into account the sensitivity of dif- few of the evaluations assessed whether the impact differed ferent age groups to the interventions. Evaluators also need according to the availability of complementary health ser- to understand exactly when delivery of the intervention ef- vices. Several found, in fact, that the children of more edu- fectively took place. Evaluation results need to be delivered cated mothers are benefiting the most. in time to provide evidence for decision making, but with sufficient elapsed time for a plausible impact to have oc- A number of lessons for development practitioners and curred. Impact evaluations provide a rare opportunity to evaluators arose from the review of impact evaluations document both costs and effects, yet cost-effectiveness is of World Bank nutrition support. Impact evaluations rarely analyzed. With these factors in mind, impact evalua- should be prioritized for relevant interventions that are tions of World Bank­supported programs to affect nutrition within the capacity and budget of the country to implement can have a far greater impact on program effectiveness. and sustain. Though most evaluations are of completely new programs, there is considerable scope for improving In sum, in approaching the impact evaluation literature and the conduct of nutrition impact evaluations, we shouldn't be program effectiveness through impact evaluations of en- asking simply, "What works?" but rather, "Under what con- hancement of ongoing programs.1 There are ways of obtain- ditions does it work, for whom, what part of the interven- ing reliable results, even when randomized assignment of tion works, and for how much?" These are important ques- the interventions is not feasible. tions that development practitioners should be asking in There are many challenges to implementing evaluations of reviewing the literature and that evaluators should be ad- large-scale programs with a long results chain; assessing the dressing to improve the relevance and impact of nutrition risks to the evaluation design and implementation ex ante impact evaluations. Photo by Curt Carnemark, courtesy of the World Bank Photo Library. Conclusions | 47 48 | APPENDIX A What Can We Learn from Nutrition Impact Evaluations? The Impact Evaluations Reviewed Significant impact on Height, Birth- Inter- Country-- Study Evaluation HAZ, Weight, WAZ, WHZ, weight, mediate Hetero- Studya Program Nameb Interventionc duration Sample size methodd stunting underweight wasting LBW outcomee geneityf Agüero, Carter, and South Africa--Child CT 18 monthsg 1,606 children aged 0­36 PSM with dose Yes Woolard (2007) Support Grant months response Alderman (2007) Uganda--Early Child D, G, NE 1 year 9,073 children aged 0­60 DID Yes B, D E, I Development months Alderman and others Senegal--PRN D ,G, M, NE, P 3 yearsh 10,378 children less than 3 DID Yes B, I, M, U (2009) years of age Alderman and others Uganda--Early Child D, G, M, NE 4 yearsi 27,995 children aged 1­7 R Yes (2006) Development and years de-worming Alderman, Tanzania--Partage F 1,140 children <5 years of IV Yes Yes Hoogeveen, and age surveyed between Rossi (2006) one and four times Armecin and others Philippines--Early Child F, G, M, NE, P, T 36 months 6,693 children aged 24­84 DID, PSM Yesj Yes I, M (2006) Development months Attanasio and Vera- Colombia--Hogares DC, F, G, M n.a. 4,147 children aged 0­6 IV, PSM, Yes No E Hernández (2004) Comunitarios years quantile regression Attanasio and others Colombia--Familias en CT, F, G, M, NE, T 4 years 8,919 children aged 0­6 PSM, DID Yes Yes D, I, U R (2005) Acción years (0­83 months) Barber and Gertler Mexico--Oportunidades CT, F, G, M, NE, P 20 monthsk 840 women in rural areas; R, IV Yes U (2008) mean age 29 years Behrman and Mexico--Oportunidades CT, F, G, M, NE, P, 24 months 601 children aged 4­48 PSM, DID Yes - E, O Hoddinott (2005) T months Behrman, Cheng, and Bolivia--PIDI DC, F, G, M 2­25+ Participating children PSM with dose No No Todd (2004) months aged 6­72 months: 1,198 response in the first round, 2,420 in the second round, and 364 in both rounds Bobonis, Miguel, and India--Pratham Delhi D, M 5 months 2,383 children aged 24­72 R, DID No Yes Yes I, M E, G, O Sharma (2006) Preschool Health months Program Christian and others Nepal M 29 monthsh 4,926 pregnant women R Yes (2003) and 4,130 live-born infants Das Gupta and India--ICDS Variousl n.a. 90,000 households; ages PSM No No G, R others (2005) of children ranged from 0­4 years in the 1992 survey and 0­3 years in the 1998 survey Friis and others (2004) Zimbabwe M Starting 1,669 women; birth data R Yesm O from week available for 1,106 women 22­35 week of gestation Galasso and Madagascar-- F, G, M, NE, P 8 years 14,000 households and PSM , DID Yes Yes B, D, H, I, E, I, R, O Umapathi (2009) SEECALINE 12,367 (at follow-up); M, P, U children aged 0­60 months Galasso and Yau Madagascar-- F, G, M, NE, P, S 24 months Children aged 0­36 PSM Yes I, O (2006) SEECALINE months in 2,697 communities Gertler (2004) Mexico--Oportunidades CT, F, G, M, NE, P, 24 months 1,552 children aged 12­36 R Yes I T months Gupta and others India M 37­77 days 200 pregnant women R Yes I (2007) randomized into treatment and control groups Hossain and others Bangladesh--BINP F, G, M, NE, P 6 years 2,338 children aged 6­23 Matching No No No B, P (2005) months program and nonprogram Appendix A: The Impact Evaluations Reviewed areas Iannotti and others Peru M 15.6 weeks 1,295 women enrolled; R No Yes No B, D, I G (2008) 1,079 neonates measured for anthropometry at birth; 546 children included for growth analysis from birth to 12 months (continued on next page) | 49 50 | APPENDIX A (continued) What Can We Learn from Nutrition Impact Evaluations? Significant impact on Height, Birth- Inter- Country-- Study Evaluation HAZ, Weight, WAZ, WHZ, weight, mediate Hetero- Studya Program Nameb Interventionc duration Sample size methodd stunting underweight wasting LBW outcomee geneityf Kazianga, de Walque, Burkina Faso--School F, THR Not clearn Covers 46 new schools R, DID No Yes Yes M and Alderman (2009) meals and THRs that were opened in 2005­06 school year; 2,208 households (having a total of 4,140 school-age [6­15 years] children) around these schools were surveyed León and Younger Ecuador--Bono CT n.a. 5,824 households and IV Yes No O (2007) Solidario approximately 3,000 children under 5 years of age Leroy and others Mexico--Oportunidades CT, F, G, M, NE, P, 24 months 432 children younger than PSM, DID Yes Yes Yes I (2008) T 24 months Linnemayr and Senegal--PRN D, G, M, NE, P 1 year 4,296 baseline and 6,144 PSM, DID Yes B, I, M, U I Alderman (2008) endline children 0­35 months of age Macours, Schady, Nicaragua--Atención a CT, F, G, M, NE, P, 1 yearh 3,506 children aged 0­83 R No No No D, U and Vakis (2008) Crisis T months Maluccio and Flores Nicaragua--RPS CT, G, M, NE, P, T 24 months 987 children (at follow-up) R, DID Yes Yes No M, U G, I (2005) 5 years and younger Masanja and others Tanzania--IMCI n.a. n.a. 2,006 (1999 survey) and Matching IMCI Yes Yes No (2005) 1,924 (2002 survey) and non-IMCI children 24­59 months of areas age Menéndez and others Mozambique Sulphadoxine- 1,027 pregnant women R Noo I, M O (2008) pyrimethamine randomized, 990 births with insecticide- analyzed treated nets Miguel and Kremer Kenya D and hygiene 2 yearsp 30,000 school children R Yes No I, M (2004) education 6­18 years of age Morris and others Brazil--Bolsa CT, F, G, M, NE, P, 6+ months 1,889 children 0­84 IVq No Yes j (2004) Alimentação T months of age Osrin and others Nepal M 5 months 1,200 pregnant women R Yes (2005) (final sample 1,052) Paxson and Schady Ecuador--BDH CT 17 months 1,479 (baseline full R No U, M I, G (forthcoming) sample) children 3­7 years of age Penny and others Peru NE 2 yearsh 377 children followed R Yes Yes No B, D (2005) from birth to 18 months Quisumbing (2003) Ethiopia--Food Aid FFW, FD n.a. 1,500 households and MLE Yes j Yes I,G 2,968 children 0­9 years of age Ramakrishnan and Mexico M 873 pregnant women R No others (2003) Rivera and others Mexico--Oportunidades CT, F, G, M, NE, P, 24 months 650 children 0­12 months R Yes I (2004) T of age Ruel and others (2008) Haiti--World Vision F, G, M, NE 9­18 1,500 children per survey, R Yes Yes Yes Programs months 12­41 months of age Santos and others Brazil--IMCI NE 6 months 33 doctors with 12­13 R No Yes Yes B, D, M (2001) patients; about 424 children <18 months old at baseline Schipani and others Thailand Mixed gardening 12 monthsr 60 households; 85 Matchings No No No M (2002) children aged 1­7 years Schroeder and others Vietnam--CENP D, G, F, NE 12 months 238 children, 5­30 R No No No (2002) months of age Stifel and Alderman Peru--Vaso de Leche FT n.a. 19,053 children aged 0­59 IV No (2006) months Waters and others Peru NE 18 months 338 children 0­18 months R Yes No U (2006) of age Appendix A: The Impact Evaluations Reviewed White and Masset Bangladesh--BINP F, G, M, NE, P Data from three sources;t PSM Yes Yes Yes P E, O (2007)/IEG (2005) children 6­23 months of age Yamano, Alderman, Ethiopia--Food Aid FFW, FD n.a. 2,089 children aged 6­60 IV Yes and Christiansen months (2005) Zeng and others China M 6 monthsu 5,828 pregnant women R Yes I, M (2008) and 4,697 live births (continued on next page) | 51 52 Significant impact on | APPENDIX A (continued) Height, Birth- Inter- What Can We Learn from Nutrition Impact Evaluations? Country-- Study Evaluation HAZ, Weight, WAZ, WHZ, weight, mediate Hetero- Studya Program Nameb Interventionc duration Sample size methodd stunting underweight wasting LBW outcomee geneityf Source: IEG analysis. Notes: Blank space indicates that no data were collected. HAZ = height-for-age z-score; LBW = low birthweight; WAZ = weight-for-age z-score; WHZ = weight-for-height z-score; n.a. = not available or not mentioned in the reviewed material. a. Studies in bold were carried out by, or in collaboration with, World Bank researchers. b. BDH = Bono de Desarrollo Humano; BINP = Bangladesh Integrated Nutrition Project; CENP = Community Empowerment and Nutrition Project; ICDS = Integrated Child Development Services; IMCI = Integrated Management of Childhood Illness; PIDI = Proyecto Integral de Desarrollo Infantil; PRN = Programme de Renforcement de la Nutrition; RPS = Red de Protección Social; SEECALINE = Projet de Surveillance et Éducation des Écoles et des Communautés en Matière d'Alimentation et de Nutrition Élargi. c. CT = cash transfer; DC = day care; D = de-worming; F = feeding; FT = food transfer; FD = free food distribution; FFW = food for work; G = growth monitoring/promotion; M = micronutrient supplement; NE = nutrition education; P = prenatal services; T = treatment of illness; THR = take-home rations. d. DID = difference-in-difference/double difference; FE = fixed effects; IV = instrumental variables/two-stage least squares; PSM = propensity score matching, R = randomized. e. B = breastfeeding, colostrums knowledge, practice; D = dietary intake; H = hygiene; I = illness; M = micronutrient intake or status; P = pregnancy knowledge/practice; U = health care use. f. E = mother's education; G = child's gender; I = household income or socioeconomic status; R = region/location or place of residence; O = other. g. The treated children in their three-year window. Values are computed based on the percentage of exposure in the three-year window reported in the study. h. Study period. i. Five rounds of health days in four years. j. Negative impact on some specifications. k. April 1998­November 1999 for the treatment group. The control group started receiving benefits in December 1999. l. Child growth monitoring, supplementary feeding, preschool education to young children, and some basic health services to young children, pregnant women, and lactating mothers. m. Significant impact on birthweight but not on incidence of LBW. n. One school year. o. No significant impact on birth outcomes. However, when the results are disaggregated by gravidity, low birthweight is lower for women with four or more pregnancies. p. 1998 and 1999 school years. The program is phased in three groups; Group 3 (the control) received treatment in 2001. q. The control is the group that was excluded because of "random administrative error." r. The study considered children from mixed-gardening households as the treatment group and children from non-mixed-gardening households as the controls group. The data were collected in three different seasons of the year to account for seasonal variation. s. Mixed-gardening and nongardening households were randomly selected and matched for comparison. t. BINP data from Karim and others (2003) on pregnant mothers and infants (6­24 months), the Save the Children data on 1,450 children aged 6­59 months, and the Helen Keller International Nutritional Surveillance Project data of about 10,000 rural households with at least one child under 59 months prior to 2000. u. Approximation based on the mean enrollment of gestation week, which is about 14 weeks. APPENDIX B Impact Evaluations of Height, Height for Age, and Stunting Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb methodc Baseline level Impactsd Poverty level education Other Remarks Agüero, South Africa-- CT PSM with n.a. Children's age: 0­36 months Carter, and Child Support dose response Significant impact on HAZ, and Woolard Grant the gains are maximum when (2007) treatment covers around two- thirds of the nutritional window or the first three years of the child's life. Treatment effect is analyzed by age group: <12 months, 12­24 months, and 24­36 months. The effect is the largest when treatment is started at the youngest age. Appendix B: Impact Evaluations of Height, Height for Age, and Stunting Alderman, Tanzania F IV n.a. Children's age: 0­60 months Hoogeveen, Availability of feeding post in the and Rossi village is significant in the HAZ (2006) equation. Armecin Philippines-- D, DC, F, G, M, PSM, DID HAZ Children's age: 24­84 months The study estimates and others ECD NE, P, T T = ­1.57 HAZ 15 models for each (2006) C = ­1.67 T = ­1.79 outcome, that is, five Stunting (%) C = ­1.88 age groups (2, 3, 4, T = 34.9 No impact in 14 of the 15 5, and 6+ years) by C = 38.7 specifications. One negative three duration-of- impact. exposure categories Stunting (%) (4­12 months, 13­16 T = 43.4 months, 17+ months). C = 48.6 Mixed impact on stunting. Of the total 15 specifications, 3 show improvement, 5 show worsening, and 7 show no impact. (continued on next page) | 53 54 | APPENDIX B (continued) What Can We Learn from Nutrition Impact Evaluations? Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb methodc Baseline level Impactsd Poverty level education Other Remarks Attanasio Colombia-- CT, F, G, M, NE, PSM, DID n.a. Children's age: 0­83 months and others Familias en P, T Program impact on HAZ by age (2005) Acción group: <24 months = 0.161* 24­48 months = 0.011 (ns) >48 months = 0.012 (ns) Attanasio Colombia-- DC, F, G, M IV, PSM, HAZ for Children's age: 0­72 months Children whose Identification based and Vera- Hogares quintile children <6 Measured impacts on HAZ for mothers had on IV; attendance is Hernández Comunitarios regression years three measures of program no education number of months in (2004) = ­1.24 take-up: benefited most. the program; exposure Attendance = 0.486** is number of months No. of months = 0.013** adjusted by age of the Exposure = 0.78** child Behrman, Bolivia--PIDI DC, F, G, M PSM with n.a. Children's age: 6­72 months Cheng, and dose response Insignificant impact on height Todd (2004) for all age groups. Cumulative impact estimated for combined age cohorts (6­24, 25­36, 37­41, 42­58, 59+ months) and duration of exposure (1­6, 7­12, 13­18, 19­24, 25+ months). Behrman Mexico-- CT, F, G, M, NE, R, FE HAZe Children's age: 4­48 months Significant Other significant and Oportunidades P, T 4­12 months Height in cm program impact impacts of Hoddinott = ­0.949 4­12 months = 0.503 (n.s.) for children household head (2005) 12­36 months 12­36 months = 1.016** 12­36 months characteristics, = ­1.928 36­48 months = ­0.349 (n.s.) whose mothers not by 36­48 months have >5 years of community = ­1.947 schooling. infrastructure. Bobonis, India--Pratham D, M R, DID HAZ Children's age: 24­72 months Treatment and control Miguel, and Preschools Delhi T = ­0.79 HAZ declined by 0.19 (n.s.) groups were not Sharma C = ­0.45. balanced at baseline. (2006) The treatment Randomization did group had not eliminate the HAZ statistically differences between significantly them. Estimated DID lower z-scores to control for initial at baseline. differences. Das Gupta India--ICDS F, G, M, NE, P, T PSM n.a. Children age 0­4 years in the 1992 Impact for boys' All results reported and others survey, and 0­3 years in the 1998 sample in 1992, here are based on (2005) survey not in 1998. For matched treated and HAZ girls' sample no control groups. 1992: T = ­1.877, C = ­1.933 impact in either The study is not 1998 : T = ­1.807, C = ­1.832 year. longitudinal; based No impact. No differential on two cross-sectional impacts by studies from 1992 and region. 1998. Gertler Mexico-- CT , F, G, M, NE, R n.a. Children's age: 12­36 months (2004) Oportunidades P, T Children in the treatment group grew about 0.96 cm more than those in the control group.*** Treatment children were 8.6% less likely to be stunted (n.s.) Galasso and Madagascar-- F, G, M, NE, P PSM HAZ (1997­ Children's age: 0­60 months HAZ by poverty 2004 HAZ: HAZ Deterioration Umapathi SEECALINE 98) HAZ (2004) incidence: Unschooled = Proximity to a concentrated (2009) T = ­1.942 T = ­2.017 Lowest poverty 0.044 (n.s.) rural road: among children with C = ­1.831 C = ­1.943 incidence = Primary Yes = 0.118* unschooled mothers. HAZ (2007) 0.287*** = 0.097 (n.s.) No = 0.061 (n.s.) T = ­2.012 Middle = ­0.001 Secondary & Hospital Appendix B: Impact Evaluations of Height, Height for Age, and Stunting C = ­1.903 (n.s.) higher = 0.141* Yes = 0.160* Highest poverty 2007 HAZ: No = 0.084 (n.s.) Deteriorated in both program incidence = Unschooled = No differential and nonprogram areas; 0.020 (n.s.) ­0.151 (n.s.) impact in deterioration significantly less in Stunting Primary communities program areas. by poverty = 0.120 (n.s.) with and without incidence: Secondary & electricity or Lowest poverty higher = 0.323** access to a safe incidence = 2004 Stunting: water source. ­0.084*** Unschooled Middle = 0.022 = ­0.005 (n.s.) (n.s.) Primary Highest poverty = ­0.019 (n.s.) incidence = Secondary & ­0.032 (n.s.) higher = ­0.051 (n.s.) 2007 Stunting: Unschooled = 0.104 (n.s.) Primary = 0.005 (n.s.) Secondary & higher = ­0.103 (n.s.) (continued on next page) | 55 56 | APPENDIX B (continued) What Can We Learn from Nutrition Impact Evaluations? Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb methodc Baseline level Impactsd Poverty level education Other Remarks Hossain Bangladesh-- F, G, M, NE, P Matching n.a. Children's age: 6­23 months and others BINP program and Severe stunting (<­3 SD) (%) (2005) nonprogram T = 11.6 areas C = 12.4 Moderate Stunting: (between ­2 SD and ­3 SD) (%) T= 27.5 C = 27.6 No significant program impact found. Iannotti Peru M R Children's age: 0­12 months Intervention during and others No impact on linear growth. pregnancy. Treatment (2008) = zinc + iron + folic acid. Control = iron + folic acid. Neonates followed from birth to 12 months for growth analysis. Kazianga, de Burkina Faso F R, DID HAZ all Children's age: 6­60 months Walque, and children Impact on HAZ Alderman School meals School meals (2009) = ­2.351 6­60 months = ­0.19 (n.s.) THRs = ­2.086 12­60 months = ­0.135 (n.s.) Control = ­2.317 Take-home rations Stunting (% of 6­60 months = 0.212 (n.s.) all children) 12­60 months = ­0.189 (n.s.) School meals = 50.9 THRs = 60.0 Control = 61.7 Leon and Ecuador--Bono CT IV n.a. Children's age: 0­60 months The impact of Bono Younger Solidario The Bono increases expenditure income works through (2007) by about 11%. Observed mean the expenditure per value for HAZ is ­1.081. When capita and the income Bono income is set to zero, mean share Using two-stage value for HAZ is ­1.121. In other least squares (2SLS) words, the grant improves HAZ estimates, a doubling by 0.04. Statistically significant of household but small. expenditures per capita would increase HAZ by 0.85. Targeting the Bono cash grant to mothers was no more efficacious at reducing malnutrition than was other household income. Leroy and Mexico-- CT, F, G, M, NE, PSM, DID Height (cm) Children's age: 0­24 months Height gain (cm) Income/poverty levels others Oportunidades P, T T = 70.9 Height gain (cm) T1 = 0.86(n.s.) given in tercile (T1 to (2008) C = 70.2 0.47(n.s.) T2 = 0.22 (n.s.) T3); T1 is the poorest. Appendix B: Impact Evaluations of Height, Height for Age, and Stunting HAZ gain T3 = 0.74 (n.s.) HAZ 0.11(n.s.) HAZ gain T =­1.9 Height gain (cm) T1 = 0.27** C = ­1.4 0­6 months = 1.53 ** T2 = 0.00 (n.s.) 6­12 months = 0.73 (n.s.) T3 = 0.13 (n.s.) 12­24 months = ­0.07 (n.s.) HAZ gain 0­6 months = 0.41** 6­12 months = 0.23 (n.s.) 12­24 months = 0.02 (n.s.) Macours, Nicaragua-- CT , F, G, M, NE, R HAZ Children's age: 12­35 and 60­83 Impacts on other age Schady, and Atención a Crisis P, T Children 0­5 months groups calculated by Vakis (2008) months HAZ decreased by 0.052 (n.s.) Fiszbein and Schady T = ­1.28 Treatment effect (2009), using Macours C = ­1.10 12­35 months = ­0.116 (n.s.) data: 60­83 months = ­0.013 (n.s.) 0­23 months = ­0.14 (n.s.) 24­47 months = ­2 (n.s.) 48­71 months = ­0.03 (n.s.) (continued on next page) | 57 58 | APPENDIX B (continued) What Can We Learn from Nutrition Impact Evaluations? Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb methodc Baseline level Impactsd Poverty level education Other Remarks Maluccio Nicaragua--RPS CT , G, M, NE, R, DID Stunting (%) Children's age: 0­60 months Significant effect No diferential and Flores P, T T = 39.8 Stunting* of the program effects by gender. (2005) C = 39.5 T = 36.5 on HAZ among HAZ C = 41.7 the poor: 0.22. T = ­1.73 HAZ­no impact C = ­1.77 T = ­1.63 C = ­1.8 Masanja Tanzania--IMCI Various Compared Stunting (%) Children's age: 24­59 months Health equity and others trends and T = 59 Stunting (%)** improved. (2005) inequalities of C = 51 T = 43 health using C = 40 two surveys Miguel and Kenya D, Hygiene R n.a. Children's age: 72­216 months Authors argue that Kremer HAZ the impact is the (2004) T = ­1.13 result of the use of C = 1.22 antihelminths, not of Mean difference = 0.09* the health education component of the program because there is no significant difference in behavior. Morris and Brazil--Bolsa CT, F, G, NE, P, R n.a. Children's age: 0­84 months others Alimentação M, T Program impact for all children (2004) younger than 84 months was ­0.11 (a worsening outcome) but not significant. Impact on HAZ of children aged <24 months = ­0.11 (n.s.) 24­48 months = ­0.19 (n.s.) 48-84 months = ­0.04 (n.s.) Paxson and Ecuador--BDH CT, F, G, M, NE, R, DID HAZ Children's age: 36­84 months On average, Program effects Baseline levels and Schady P, T T = ­1.22 No impact on height or HAZ. children in on physical impacts on HAZ by (2009) C = ­1.20 the lowest measures are age are taken from expenditure consistently Fiszbein and Schady quartile have larger among (2009). 24.3% higher girls than boys physical both for poorest Physical outcomes outcomes than and relatively include three those in the better-off measures of physical control group. children. In some development: the cases the gender child's hemoglobin differences are level, height for age, significant. and fine motor control Penny and Peru NE R n.a Children's age: 0­18 months Birth cohort followed others Height at age 18 months (cm) to 18 months (2005) T = 79.36 C = 78.29 Adjusted difference = 0.714 (p = 0.014) HAZ at 18 months T = ­0.81 Appendix B: Impact Evaluations of Height, Height for Age, and Stunting C = ­1.19 Adjusted difference = 0.272 (p = 0.002) Quisumbing Ethiopia--Food FD, FFW Heckman n.a. Children's age: under 9 years Impact reported 1. Gender of the Main message is that (2003) Aid maximum Children <5 in low-asset by low and child. HAZ is not responsive likelihood households: high asset Children <5, In to food aid in the estimate FFW has direct negative impact households. low-asset house- short run. on HAZ. FD and all (FD + FFW) do (See the column holds, FFW and not have impact on HAZ to the left) FD have no dif- ferential impact. Children 5­8 in low-asset The total value households: (FD + FFW) has Food aid does not have an negative lagged impact on HAZ; The direct and impact on girls. lagged effects of all FFW, FW, and FFW + FD are insignificant. In high-asset households, no differential im- pact of food aid for all children. (continued on next page) | 59 60 | APPENDIX B (continued) What Can We Learn from Nutrition Impact Evaluations? Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb methodc Baseline level Impactsd Poverty level education Other Remarks Quisumbing Children <5 in high-asset 2. Gender of the (Continued) households: recipient Food aid has no impact on HAZ; In low-asset house- The direct and lagged effects of holds, gender of all FFW, FD, and FFW + FD are the child inter- insignificant. acted with the gender of the re- Children 5­8 in high-asset cipient (GirlsXFe- households: male recipient) FFW has no impact. FD as well not significant for as total value (FD + FFW) have 5­8 year olds and lagged but negative impact on inconsistent (posi- HAZ. tive and nega- tive impact) for younger children. In high-asset households, the gender of the recipient has no effect in either group of children. Rivera and Mexico-- CT, F, G, M, NE, R HAZ Children's age: 0­12 months Impact larger for others Oportunidades P, T 0­6 months: Significant impact on height for children from (2004) T = ­0.45 some children. Children (0­6 the poorest C = ­0.25 months) in the treatment group households 6­12 months: grew about 1.1 cm more than T = ­1.04 those in the control group (26.4 C = ­1.03 cm in the intervention group vs. 25.3 cm in the control group). Ruel and Haiti (two World F, G R HAZ Children's ages: 12­41 months others Vision programs Preventive HAZ (2008) on maternal and = ­1.69 Preventive = ­1.53 child health and Recuperative Recuperative = ­1.67 nutrition) = ­1.65 By age group, significant (higher for preventive group) for age Stunting (%) group 24­35 months, but not for Recuperative 12­23 months or 36­41 months. = 37.4 Stunting Preventive = Significantly lower (by 4 36.7 percentage points) in the preventive communities. Santos Brazil NE R HAZ Children's age: 0­18 months and others All: Impact on length (cm) (2001) T = ­0.23 All: ­0.12 (n.s.) C = ­0.04 By age group 0­6 months = ­0.27 (n.s.) 6­12 months = ­0.46 (n.s.) 12­18 months = 0.38 (n.s.) Impact on HAZ All: ­0.04 (n.s.) By age group 0­6 months = ­0.01 (n.s.) 6­12 months = ­0.13 (n.s.) 12­18 months = ­0.17 (n.s.) Schipani Thailand Mixed Matching n.a. Children's age: 12­84 months Results reported No baseline and others gardening program and No impact by season. Not information is used in (2002) nonprogram Rainy season (%): significant in any the analysis. areas T = 20 season. C = 26.6 Cool season (%): T = 16.7 C = 23 Hot season (%): Appendix B: Impact Evaluations of Height, Height for Age, and Stunting T = 22 C = 34 Schroeder Vietnam D, F, G, NE R, multivariate HAZ Children's age: 5­30 months Children both in and others T = ­1.65 HAZ the treatment and (2002) C = ­1.67 T = ­1.66 control groups were Stunting (%) C = ­1.66 (Difference n.s.) de-wormed. T = 35.5 Stunting (%) C = 42.9 T = 36.0 C = 33.1 (Difference n.s. ) Stifel and Peru--Vaso de FT IV n.a. Children's age: 0­60 months Alderman Leche Negative but insignificant impact (2006) in all models. The parameter estimates are also small in magnitude. Waters Peru NE Multivariate HAZ Children's age: 0­8 months Significant differences and others T = ­0.50 HAZ between treatment (2006) C = ­0.50 The coefficient of intervention and control in 2 Stunting compared with control is 0.31 of the 11 baseline T = 0.05 and significant. characteristics. The C = 0.04 Stunting treatment group was The coefficient of intervention better in terms of compared with control is 0.33 proportion of mothers and significant. The intervention completing secondary prevented 11.1 cases of stunting education and | per 100 children. economic quintile. (continued on next page) 61 62 | APPENDIX B (continued) What Can We Learn from Nutrition Impact Evaluations? Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb methodc Baseline level Impactsd Poverty level education Other Remarks White and Bangladesh-- F, G, NE, P, M PSM n.a. Children's age: 6­23 months Average treatment Masset BINP HAZ effect on the treated, (2007)/IEG Midterm: 0.10*** one-to-one matching (2005) Endline: 0.08** (IEG 2005, p. 170) Endline HAZ n.s. in any other models. Yamano, Ethiopia--Food FD, FFW IV n.a. Children's age: 0­60 months Alderman, Aid Food aid has a positive impact on and height of children 6­24 months. Christiansen Impact more than doubles when (2005) program placement effects are controlled using IV (coefficient: 0.42 vs. 0.09). Equivalent to 2 cm faster growth in 6-month period among children in food aid communities. No impact on older groups (25­60 months). Source: IEG analysis. Notes: C = control group; DID = difference-in-difference; HAZ height-for-age z-score; T = treatment group. Blank space indicates that no data were collected; n.a. = not available or not mentioned in the reviewed material; n.s. = not statistically significant. Statistical significance: * = p 0.10, ** = p 0.05, *** = p 0.01. a. BDH = Bono de Desarrollo Humano; BINP = Bangladesh Integrated Nutrition Project; ECD = early childhood development program; ICDS = Integrated Child Development Services; IMCI = Integrated Management of Childhood Illness; PIDI = Proyecto Integral de Desarrollo Infantil; RPS = Red de Protección Social; SEECALINE = Projet de Surveillance et Éducation des Écoles et des Communautés en Matière d'Alimentation et de Nutrition Élargi. b. CT = cash transfer; DC= day care; D = de-worming; F = feeding; FD = free food distribution; FFW = food for work; FT = food transfer; G = growth monitoring; M = micronutrient supplement; NE = nutrition educa- tion; P = prenatal services; T = treatment of illness; THR = take-home rations. c. DID = difference-in-difference/double difference; FE = fixed effects; IV = instrumental variable/2SLS; PSM = propensity score matching, R = randomized. d. cm = centimeter; SD = standard deviation. e. Computed based on HAZ information reported in the study. APPENDIX C Impact Evaluations of Weight, Weight for Age, and Underweight Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb Methodc Baseline Impactsd poverty level education Other Remarks Alderman Uganda--ECD D, G, M, NE, P DID WAZ Children's age: 0­60 months No differential No differential (2007) T = ­1.052 WAZ impacts by impacts. C = ­1.060 T = ­1.105 household C = ­1.108 expenditure. On average, there was no program impact. However, WAZ significantly improved for children <12 months in the T compared with the C (p = 0.08). Alderman, Tanzania-- F IV n.a. Children's age: 0­60 months Hoogeveen, Partage Availability of feeding post in the Appendix C: Impact Evaluations of Weight, Weight for Age, and Underweight and Rossi village is significant in the WAZ (2006) equation. Alderman Uganda ECD/ D, G, M, NE, P R Children's age: 12­84 months and others De-worming An increase in weight gain of (2006) about 10% (about 166 g per child per year, 95%, CI 16 to 316 g) above expected weight gain when treatments were given twice a year, and an increase of 5% when the treatment was given annually. Alderman Senegal--PRN D, G, M, NE, P DID Children's age: 0­36 months and others Children in program villages are (2009) less likely to be underweight (odds ratio 0.83, 95% CI: 0.686, 1.000). Attanasio Colombia-- DC, G, F, M IV, quintile WAZ = ­0.80 Child age: 0­72 months and Vera- Hogares regression Results pooled Impact computed for three Hernández Comunitarios from baseline measures of program take-up: (2004) (2002) and attendance (participation), follow-up number of months in the (2003). program, and exposure (number of months adjusted by age of the child). Attendance = 0.247 (n.s.) No. of months = 0.001 (n.s.) Exposure = 0.132 (n.s.) | (continued on next page) 63 64 | APPENDIX C (continued) What Can We Learn from Nutrition Impact Evaluations? Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb Methodc Baseline Impactsd poverty level education Other Remarks Behrman, Bolivia--PIDI DC, G, F, M PSM with dose n.a. Children's age: 6­72 months Cheng, and response Insignificant impact on weight Todd (2004) for all age groups (6­24, 25­36, 37­41, 42­58, 59+ months) and durations of exposure (2­6, 7­12, 13­18, 19­24, 25+ months). Bobonis, India--Pratham D, M R, DID WAZ Children's age: 24­72 months T and C groups Miguel, and Delhi) T = ­1.41 WAZ were not balanced Sharma C = ­1.15 improved by 0.31. Child weight at baseline. (2006) The difference gains--roughly 0.5 kg (1.1 Randomization did was pounds) on average--in the not eliminate the WAZ statistically treatment schools relative to differences between T significant. comparison schools. and C. Estimated DID to control for initial differences. Das Gupta India--ICDS Various PSM n.a. Children's ages: 0­4 years in the No differential Baseline not available. and others 1992 survey, and 0­3 in the 1998 impact by Results reported here (2005) survey gender. are based on matched WAZ Some impacts treatment and control 1992 by region, but groups. T = ­1.917 negative. Impacts seen at two C = ­1.873 different points in 1998 time, in 1992 and T = ­1.789 1998. C = ­1.788 No impact. Galasso and Madagascar-- F, G, M, NE, P PSM WAZ Children's age: 0­60 months WAZ: 2004 WAZ: WAZ and Overall, differential Umapathi SEECALINE T = ­1.789 WAZ Impact by Unschooled = underweight: impacts by (2009) C = ­1.667 T = ­1.584 incidence of 0.136 (n.s.) Impacts are maternal education C = ­1.574 poverty: Primary = 0.121** also greater and community The program improved WAZ by Lowest poverty Secondary with respect to socioeconomic 0.15­0.22, and the incidence of incidence = &higher = proximity to rural characteristics. underweight declined by 5.2­7.5 0.330*** 0.214*** road, hospital, Positive trends for percentage points. Middle = 0.001 2004 electricity, and WAZ are observed (ns); Highest Underweight: type of water only for educated poverty Unschooled = source in the mothers. Impacts incidence = ­0.04 (n.s.) community. widened when 0.144** Primary = ­0.03 socioeconomic Underweight: (n.s.) disparities are Lowest poverty Secondary & reinforced. incidence = higher = Nutritional gains are ­0.130*** ­0.111*** larger in better-off Middle = 0.001 2007 WAZ: communities. (n.s.) Unschooled = Highest poverty ­0.05 (n.s.) incidence = Primary = 0.152* ­0.041 (n.s.) Secondary & higher = 0.279** Appendix C: Impact Evaluations of Weight, Weight for Age, and Underweight 2007 Underweight Unschooled = 0.008 (n.s.) Primary = ­0.044 (n.s.) Secondary & higher = ­0.07 (n.s.) Galasso and Madagascar-- F, G, M, NE, P PSM Underweight Children's age: 0­36 months "Sites with a Program effect On average, there are Yau (2006) SEECALINE 0­6 months = Results are reported by duration higher poverty responsive to positive returns to 0.31 of exposure and age. Statistically rate have duration of exposure. 7­12 months = significant reductions in both higher returns exposure. 0.46 cases. to exposure 13­36 Reduction of malnutrition by over two years months = 0.51 child age: relative to sites For one-year exposure (%) with lower 0­6 months = 8 poverty rates." 7­12 months = ­4 (p. 22) 13­36 months = 0 For two-year exposure (%) 0­6 months = 9 7­12 months = 8 13­36 months = 8 (continued on next page) | 65 66 | APPENDIX C (continued) What Can We Learn from Nutrition Impact Evaluations? Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb Methodc Baseline Impactsd poverty level education Other Remarks Hossain Bangladesh F, G, M, NE, P Matching n.a. Severely low WAZ (<­3 z-scores) T = project area and others --BINP program and (%) C = nonproject area (2005) nonprogram T = 11.4 areas C = 12.1 Moderately low WAZ (­3 and <­2 z-scores) (%) T = 35.2 C = 36.3 Iannotti Peru M R Children's age: 0­12 months Gender affects Compared zinc + iron and others Positive impact on weight. impact, but result + folic acid versus iron (2008) Treatment group was heavier by not reported. + folic acid in pregnant 0.58 kg.*** women. Neonates followed from birth to 12 months for growth analysis. Kazianga, de Burkina Faso F R WAZ all Children's ages: 6­60 months and Walque, and children 6­10 years Alderman School meals = Impact on WAZ (2009) ­2.202 School meals Take-home 6­60 months = 0.219** rations = 12­60 months = 0.172 ­2.521 6­10 years = 0225* Control = Take-home rations ­2.394 6­60 months = 0.355* Underweight 12­60 months = 0.376* (% of children) 6­10 years = 0.153 (n.s.) All children School meals = 52.6 Take-home rations = 56.2 Control = 55.3 Leon and Ecuador--Bono CT IV n.a. Children's age: 0­60 months The impact of Bono Younger Solidario There is no impact of the Bono income works through (2007) share of household expenditure the expenditure-per- on WAZ. Household expenditure capita variable and the per capita does have a significant income-share variable. impact, however, and Bono income increases expenditure by 11%. Leroy and Mexico-- CT, F, G, M, NE, PSM, DID Weight (kg) Children's age: 0­24 months Weight gain (in Income/poverty levels others Oportunidades P, T T = 8.62 Weight gain terciles [T]) given in tercile (T1­ (2008) C = 8.46 0.202 kg (ns) T1 = 0.523** T3); T1 is the poorest. Weight gain by age (kg) T2 = 0.079 (n.s.) 0­6 months = 0.763 (n.s.) T3 = 0.138 (n.s.) 6­12 months = 0.026 (n.s.) 12­24 months = 0.09 (n.s.) Linnemayr Senegal--PRN D, G, M, NE, P DID WAZ score Children's age: 0­36 months No impact by and T = ­1.352 No significant program impact on wealth. Alderman C = ­1.276 WAZ overall. A dummy variable (2008) for full exposure is created for younger children (0­6 months). Impact is significant for this group. Macours, Nicaragua-- CT, G, M, NE, R Average WAZ Children's age: 12­35 months and Schady, and Atención a Crisis P, T for children 60­83 months Vakis (2008) 0­5 months WAZ decreased by 0.052 (n.s.) old at baseline were ­0.91 and Impact on WAZ by age Appendix C: Impact Evaluations of Weight, Weight for Age, and Underweight ­1.06 for C 12­35 months = ­0.114 (n.s.) and T groups, 60­83 months = 0.046 (n.s.) respectively. The difference is not statistically significant (p = 0.168). Maluccio Nicaragua--RPS CT, F, G, M, NE, R, DID Baseline Children's age: 0­60 months No differential and Flores P, T in 2000. The program also improved effects by gender. (2005) Proportion of underweight in the intervention children under areas. Percentage underweight 5 who were declined from 13.7 to 9.8 in the underweight treatment group and increased was 13.7% in from 14.3 to 16.6 in the control the treatment group.** group and 14.3% in the control group. Masanja Tanzania--IMCI Various Compared Underweight Children's age: 24­59 months The program started and others trends and (%) Underweight (%)** earlier and the authors (2005) inequalities In 1999 in 2002 compared trends of health T = 30 T = 23 and inequalities of for children C = 27 C = 19 health using surveys 24­59 months Difference in differences (IMCI conducted in 1999 using two districts [T] minus non-IMCI (n = 2,006) and 2002 surveys districts [C]) = 0.044. (n = 1,924) for children | under 5 years of age. (continued on next page) 67 68 | APPENDIX C (continued) What Can We Learn from Nutrition Impact Evaluations? Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb Methodc Baseline Impactsd poverty level education Other Remarks Miguel and Kenya D, hygiene R WAZ Children's age: 72­216 months Estimated Kremer T = ­1.39 WAZ score externalities by (2004) C = ­1.40 T = ­1.25 treatment status. C = ­1.25 Difference was not (no difference) statistically significant. Morris and Brazil--Bolsa CT, F, G, M, NE, IV Children's age: 0­84 months The authors indicated others Alimentação P, T WAZ for all children in beneficiary that negative impact (2004) households was 0.13 lower. might be "due to No significant impact on WAZ perceptions that is found when the sample is benefits would be disaggregated by age groups. For discontinued if the three age groups, <24 months, child started to grow 24­48 months, and 48­84 well." months, project impacts on WAZ were 0.25, ­0.11, and ­0.08, respectively. Penny and Peru NE R n.a. Children's age: 0­18 months Birth cohort followed others Mean weight at 18 months (kg) from birth to age 18 (2005) T = 10.77 months. C = 10.48 Adjusted difference = 0.199 (p = 0.093) WAZ at 18 months T = ­0.33 C = ­0.62 Adjusted difference = 0.194 (p = 0.041) Ruel and Haiti F, G, M, NE, P R WAZ Children's age: 12­41 months others Preventive = WAZ (2008) ­0.97 Preventive = ­0.96 Recuperative Recuperative = ­1.2 = ­1.02 At the end of three years, children from preventive communities had significantly higher mean WAZ (+0 · 24) than the recuperative group. By age group, the difference was significant (higher for preventive group) for age groups 12­23 months and 24­35 months, but not for 36­41 months. Underweight Significantly lower (6 percentage points) in the preventive model. Santos Brazil NE R WAZ Children's age: 0­18 months and others All: Impact on Weight (kg) Appendix C: Impact Evaluations of Weight, Weight for Age, and Underweight (2001) T = 0.06 All: ­0.01 (n.s.) C = 0.31 By age group 0­6 months = ­0.16 (n.s.) 6­12 months = 0.07 (n.s.) 12­18 months = 0.34** Impact on WAZ All: 0.07 (n.s.) By age group 0­6 months = ­0.09 (n.s.) 6­12 months = 0.13 (n.s.) 12­18 months = 0.31** Schipani Thailand Mixed Matching n.a. Children's age: 1­84 months No baseline and others gardening program and No significant impact on information is used (2002) nonprogram underweight. No impact on in the analysis. The areas wasting. study is based on Results for % of WAZ 2 in three single difference from seasons: a cross-sectional data Rainy season collected in three T = 10.8 seasons. C = 16.6 Cool season T = 6.8 C = 16.6 Hot season T = 14.8 C = 20.6 (continued on next page) | 69 70 | APPENDIX C (continued) What Can We Learn from Nutrition Impact Evaluations? Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb Methodc Baseline Impactsd poverty level education Other Remarks Schroeder Vietnam--CENP D, F, G, NE, R, multivariate WAZ Children's age: 5­30 months and others T = ­1.51 WAZ (2002) C = ­1.68 T = ­1.92 Underweight C = ­2.06. The difference is not (WAZ < ­2 SD) significant (p = 0.19). T = 30.3 Underweight C = 35.3 T = 46.5 C = 55.9. The difference is not significant (p = 0.15). Children who were younger (<15 months) and more malnourished (less than ­2 WAZ) at baseline had significantly better growth than similarly young, malnourished children in the control group. Waters Peru NE R, multivariate WAZe Children's age: 0­18 months and others T = ­0.26 The coefficients of the (2006) C = ­0.34 intervention on WAZ and Underweight underweight regressions are T=3 insignificant. C=4 White and Bangladesh-- F, G, M, NE, P PSM n.a. Children's age: 6­23 months Interactions with Interactions Average trreatment Masset BINP WAZ wealth quantiles with mother's effect on the treated, (2007)/IEG Midterm: 0.12*** at endline n.s. education at one-to-one matching (2005) Endline: 0.09** endline n.s. (IEG 2005, p. 170). Interactions reported in IEG 2005, p. 177. Source: IEG analysis. Note: C = control group; T = treatment group; WAZ = weight-for-age z-score; n.a. = not applicable; n.s. = not significant. Statistical significance: * = p 0.10; ** = P 0.05; *** = p 0.01. a. BINP = Bangladesh Integrated Nutrition Project; CENP = Community Empowerment and Nutrition Project; ECD = early childhood development program; ICDS = Integrated Child Development Services; IMCI = Integrated Management of Childhood Illness; PIDI = Proyecto Integral de Desarrollo Infantil; PRN = Programme de Renforcement de la Nutrition; RPS = Red de Protección Social; SEECALINE = Projet de Surveillance et Éducation des Écoles et des Communautés en Matière d'Alimentation et de Nutrition Élargi. b. CT = cash transfer; DC = day care; D =de-worming; F = feeding; FD = free food distribution; FFW = food for work, FT = food transfer; G = growth monitoring; M = micronutrient supplement; NE = nutrition education; P = prenatal services; T = treatment of illness. c. DID = difference in difference/double difference; FE = fixed effects; IV = instrumental variable/2SLS; PSM = propensity score matching, R = randomized. d. C = nonproject area; CI = confidence interval; g = gram; kg = kilogram; T = project area; WAS = weight-for-age z-score. e. Significant differences between treatment and control in 2 of the 11 baseline characteristics included in the study. The treatment group is better off in terms of proportion of mothers completing secondary education and economic quintile. APPENDIX D Impact Evaluations of Weight for Height and Wasting Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb Methodc Baseline Findingsd poverty level education Other Remarks Armecin Philippines-- D, DC, F, G, M, PSM, DID WHZ Children's age: 24­84 months The study estimates and others Comprehensive NE, P, T T = ­0.638 WHZ 15 models for each (2006) ECD C = ­0.696 T = ­0.355 outcome, that is, five Wasting (%) C = ­0.526. age groups (2, 3, 4, T = 7.4 Wasting (%) 5, and 6+ years) by C = 6.1 T = 2.4 three duration-of- C = 2.5. exposure categories Results reported by age group (4­12 months, 13­16 and duration of exposure. months, 17 months). Overall, mixed but predominantly The authors positive results. summarize the By age group: number of positive 2- and 3-year-olds, mixed and significant (negative and positive), or no specifications for impact. WHZ and wasting, Appendix D: Impact Evaluations of Weight for Height and Wasting 4-, 5-, and 6+ year olds, positive fine and gross motor impacts. skills, cognitive skills, The mean impact of all outcomes and self-help, as well is the biggest for the youngest as expressive and age group (2 years). This group receptive language. has also the largest number of significant cases. Bobonis, India--Pratham D, M R, DID WHZ Children's age: 24­72 months School Gender Miguel, and Delhi T = ­1.12 WHZ <3 years Boys = 0.29* Sharma C = ­1.02 Improved by 0.52*** = 0.63** Girls = 0.67*** (2006) WHZ by age group >3 years Probability of being 2­3 years = 0.57*** = 0.37* anemic at baseline 4­6 years = 0.46** High = 0.62** Low = 0.32* Hossain Bangladesh F, G, M, NE, P Matching n.a. Children's age: 6­23 months and others --BINP program and Severe wasting (WHZ < ­3) (%) (2005) nonprogram T = 1.0 areas C = 1.1 Moderate wasting ( WHZ ­3 and < ­2) (%) T = 13.4 C = 14.3 (continued on next page) | 71 72 | APPENDIX D (continued) What Can We Learn from Nutrition Impact Evaluations? Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb Methodc Baseline Findingsd poverty level education Other Remarks Kazianga, de Burkina Faso F, THR R, DID WHZ all Children's age: 6­60 months Walque, and children WHZ Alderman School meals School meals (2009) = ­0.786 6­60 months = 0.005 (n.s.) Take-home 12­60 months = 0.062 (n.s.) rations = Take-home rations ­1.125 6­60 months = 0.291 (n.s.) Control = 12­60 months = 0.333* ­0.903 Wasting (% of children) All children School meals = 29.5 Take-home rations = 32.3 Control = 42.7 Leroy and Mexico-- CT, F, G, M, NE, PSM, DID WHZ Children's age: 0­24 months WHZ gain Income/poverty levels others Oportunidades P, T T = 0.30 WHZ gain for all age groups (in terciles [T]) given in tercile (T1­ (2008) C = 0.33 0.085 (ns) T1 = 0.118 (ns) T3); T1 is the poorest. WHZ gain by age T2 = 0.074 (ns) 0­6 months = 0.465** T3 = ­0.164 (ns) 6­12 months = ­0.172 (n.s.) 12­24 months = 0.103 (n.s.) Maluccio Nicaragua--RPS CT, F, G, M, NE, R, DID Wasting (%) Children's age: 0­60 months. No differential effects As shown by baseline and Flores P, T T = 0.8 Wasting (%) by gender. and follow-up (2005) C = 0.4 T = 0.2 values, wasting is C = 0.2 not a problem in the Wasting declined in both T and C intervention area. groups. Project impact is n.s. Masanja Tanzania--IMCI n.a. Compared Wasting (%) in Children's age: 12­23 months The authors compared and others trends and 1999 Wasting (%) in 2002 trends and inequalities (2005) inequalities T = 13 T=7 of health using of health C = 11 C=5 surveys from 1999 for children (n = 2006) and 2002 24­59 months (n = 1,924) for children of age using under 5 years of age. two surveys Penny and Peru NE R Children's age: 0­18 months Birth cohort followed others WHZ at 18 months from birth to age 18 (2005) T = 0.15 months. C = 0.05 Adjusted difference = 0.048 (p = 0.609) Quisumbing Ethiopia--Food FD, FFW Heckman n.a. Children's age: under 9 years Impact 1. Gender of child. (2003) Aid maximum WHZ reported Children <5 years in likelihood Children <5 years in low-asset by low- and low-asset households. estimate households high-asset FFW has positive FFW has impact; FD and all (FD + households impact on boys but FFW) do not. (see column to negative impact Children 5­8 years in low-asset the left). on girls; FD has no households differential impact; Separately, FFW and FD have no the total value (FD + impact on WHZ. However, the FFW) has no differential total value (FFW + FD) has lagged impact either. impact. In high-asset Children <5 years in high-asset households, FFW has households lagged negative impact FFW has no impact. FD as well on girls. FD has lagged as total value (FD + FFW) have impact. No differential lagged impact. impact is observed Children 5­8 years in high-asset when the total value of households food aid (FFW + FD) is Appendix D: Impact Evaluations of Weight for Height and Wasting FFW has no impact. FD as well considered. as total value (FD + FFW) have Children 5­8 years: lagged impact. No differential impact by gender on any specifications in low- asset households. Lagged impact in FD and total value (FD + FFW) on girls. 2. Gender of recipient Children <5 years: gender of the child interacted with the gender of the recipient (Girls interacted with female recipient) not significant in low- and high-asset households. Children 5­8 years: interaction term is insignificant and inconsistent (positive and negative impact) in both low- and high- asset households. (continued on next page) | 73 74 | APPENDIX D (continued) What Can We Learn from Nutrition Impact Evaluations? Heterogeneity of impacts Country-- Evaluation Income/ Maternal Study Programa Interventionb Methodc Baseline Findingsd poverty level education Other Remarks Ruel and Haiti F, G, M, NE, P R WHZ Children's age: 12­41 months Coefficients by age are others Preventive = WHZ not reported. (2008) ­0.18 Preventive = ­0.22 Recuperative Recuperative = ­0.46 = ­0.18 At the end of the three-year intervention, children from preventive communities had significantly higher mean z scores WHZ (0.24) than the recuperative group By age group, significant (higher for preventive group) for age group 12­23 months and 24­35 months, but not for 36­41 months. Wasting Significantly lower (by 4 percentage points) in the preventive model. Santos Brazil NE R WHZ Children's age: 0­18 months and others All: WHZ (2001) T = 0.37 All: 0.09 (n.s.) C = 0.55 By age group 0­6 months = ­0.02 (n.s.) 6­12 months = 0.24 (n.s.) 12­18 months = 0.28** Schipani Thailand Mixed Matching n.a. Children's age: 12­84 months No baseline and others gardening program and No significant impact on wasting information is used (2002) nonprogram (WHZ ­2): in the analysis. The areas Rainy season study is based on T = 10.0 single difference from C = 6.6 a cross-sectional data Cool season collected in three T = 0.0 different seasons. C = 0.0 Hot season T = 3.7 C = 0.0 Schroeder Vietnam--CENP D, F, G, NE R, multivariate WHZ Children's age: 5­30 months Unbalanced baseline. and others T = ­0.66 WHZ The difference is (2002) C = ­0.90 T = 1.25 significant. T better Wasting (%) C = ­1.35 The difference is not off. Small number (WHZ <­2 SD) significant (p = 0.12). of observations in T = 1.7 Wasting (%) the T at baseline for C = 10.1 T = 14.9 wasting. C = 12.7 The difference is not significant (p = 0.63). White and Bangladesh-- F, G, M, NE, P PSM Children's age: 6­23 months Average treatment Masset BINP WHZ. effect, on the treated, (2007)/IEG Midterm: 0.10*** one-to-one matching (2005) Endline: 0.03 (n.s.) (IEG 2005, p. 170) Source: IEG analysis. Notes: C = control group; T = treatment group; WHZ = weight-for-height z-score; n.a. = not available or not mentioned in the reviewed material; n.s. = not statistically significant. Statistical significance: * = p < =.10, ** = p <=.05, *** = p <=.01. a. BINP = Bangladesh Integrated Nutrition Project; CENP = Community Empowerment and Nutrition Project; ECD = early childhood development program; IMCI = Integrated Management of Childhood Illness; RPS = Red de Protección Social. b. CT = cash transfer; DC= day care; D = de-worming; F = feeding; FD = free food distribution; FFW = food for work; G = growth monitoring; M = micronutrient supplement , NE = nutrition education; P = prenatal services; T = treatment of illness; THR = take-home rations. Appendix D: Impact Evaluations of Weight for Height and Wasting c. DID = difference in difference/double difference; PSM = propensity score matching; R = randomized. d. kg = kilogram; IMCI = Integrated Management of Childhood Illness | 75 76 | APPENDIX E What Can We Learn from Nutrition Impact Evaluations? Impact Evaluations of Birthweight and Low Birthweight Country-- Evaluation Heterogeneity Study Program Interventiona Methodb Impactsc of impacts Remarks Attanasio and Colombia-- CT, NE PSM, DID Significant impact on weight gain in urban areas. Weight gain by The program does not require pregnant others (2005) Familias en newborns in urban areas women to visit clinics to obtain prenatal Acción is 0.578 kg (significant) care. Attendance at meetings at which and in rural areas 0.176 health, hygiene, and nutrition issues are kg (n.s.). discussed are not compulsory, although the program encourages women to attend such meetings. Barber and Gertler Mexico-- CT, F, M, NE, P IV Beneficiaries had 127.3 g (95% CI: 21.3, 233.1) higher (2008) Oportunidades birthweight among participating women and a 4.6 percentage point reduction in LBW. Program impact using the average beneficiary time on program amounts to 68.3 g (p = 0.05) and program impact from cash received amounts to 78.2 g (p = .07). Christian and Nepal M R Folic acid-iron increased mean birthweight by 37 g Folic acid­iron and multiple others (2003) (95% CI: 16, 90) and reduced the percentage of micronutrients increased head and chest LBW babies (< 2,500 g) from 43% to 34% (16%; relative circumference of babies, but not length. risk = 0.84, 0.72­0.99). Multiple micronutrient supplementation increased birthweight by 64 g (12­115 g) and reduced the percentage of LBW babies by 14% (0.86, 0.74 to 0.99). Friis and others Zimbabwe M R Micronutrient supplementation was associated with No difference by human (2004) increased birthweight, 49 g (95% CI: ­6 to 104 g; immunodeficiency virus p <0.08), but was not associated with LBW. status of women. Gupta and others India M R Incidence of LBW reduced in the treatment group. No Facility-based randomized trial on (2007) difference in birthweight. malnourished pregnant women. The T (micronutrient) received 29 vitamins and minerals once a day for 37­77 days; the C received placebo for 15­66 days. All participants received diet advice; usual prenatal services, including immunization; and iron and folic acid supplements. Iannotti and others Peru M R LBW = 2.2% in both the T and C groups. No significant The T received zinc, iron and folic acid; (2008) difference on weight at birth. the C received iron and folic acid. Macours, Schady, Nicaragua-- CT, F, M, NE, P R No significant program effect on LBW. Treatment effect and Vakis (2008) Atención a Crisis (weight gain) for 0­5-month-old infants was 0.161 kg (insignificant). Mean birthweight in the control group was 2.987 kg. Menéndez and Mozambique Sulphadoxine- R No significant impact on LBW for the whole sample, Impacts on LBW by HIV All participants in the T and C groups others (2008) pyrimethamine but significant in subsamples (see column at right). status and gravidity. No received long lasting insecticide-treated with insecticide- change by HIV status, but nets. Those in the T received two doses treated nets. by gravidity, incidence of sulphadoxine-pyrimethamine. Pregnant LBW was lower among women with HIV infection were not women with four or more given sulphadoxine-pyrimethamine. prior pregnancies. Osrin and others Nepal M R Significant impact on birthweight. The mean (±SD) of T The T received multiple micronutrient (2005) and the C groups are 2.810 (± 0.453) and 2.733 supplementations (recommended daily (± 0.422), respectively. Mean difference = 77g (95% CI allowance of 15 vitamin and minerals) 24­130; p = 0.004). Fall (25%) in the incidence of LBW. with iron and folic acid; the C received iron and folic acid only. Ramakrishnan and Mexico M R No significant difference between the T and C groups The T group received iron and multiple others (2003) in birthweight or incidence of LBW. The mean (±SD) of micronutrient supplementation; the C the T and C groups are 2.981 (± 0.391) and 2.977 received iron-only supplementation (± 0.393), respectively. Appendix E: Impact Evaluations of Birthweight and Low Birthweight Zeng and others China M R Micronutrients had significant but modest impact Participants were randomized into (2008) on birthweight. Birthweight was 42 g (95%CI 7­78 g) three groups. The C received folic acid higher in the multiple micronutrient group than in the alone. In the two T groups, one received folic acid group. No significant difference between the daily folic acid + iron, and the other T and C groups for LBW. received multiple micronutrient with recommended daily allowance of 15 vitamin and minerals. Source: IEG analysis. Note: C = control group; g =gram; HIV = human immunodeficiency virus; kg = kilogram; LBW = low birthweight; T = treatment group; n.s = not significant. a. CT = cash transfer; F = feeding; M = micronutrient supplement; NE = nutrition education; P = prenatal services. b. DID = difference-in-difference/double difference; IV = instrumental variable/2SLS; PSM = propensity score matching; R = randomized. c. CI = confidence interval; SD = standard deviation | 77 APPENDIX F Impact Evaluation Basics Program impact in this review is defined as the difference in child anthropometric out- comes of two statistically comparable groups--one with the program (the treatment group) and the other without it (the control group). The magnitude of impact can be either an intent-to-treat or a treatment-on-the treated estimate. The average intent-to- treat effect is an estimate of the average impact of the availability of the program on eligible beneficiaries in treatment areas, whether or not they were actually treated. Including the untreated in the treatment group may bias the results downward. In contrast, the average treatment-on-the treated parame- It is important to note that in practice, particularly in devel- ter is the effect of the program on those who actually re- opment applications, randomization can be difficult to im- ceived the treatment. The intent-to-treat estimate can be a plement (Baker 2000; Ravallion 2009a). First, it may not be parameter of interest in nutrition impact evaluations. For ethical to deny treatment to otherwise eligible individuals example, a cost-effectiveness analysis of a school-based de- or to provide treatment to those who do not need it. Sec- worming or supplementation program needs to consider ond, it is not always politically possible to provide treat- the fact that all children may not be at school on the day of ment to one group and to deny or delay treatment to the treatment and that tracking children at home may not another. Third, not all interventions are amenable to ran- be practical. Therefore, in this case, the parameter of inter- domized evaluation. For example, some interventions are est is intent-to-treat (Duflo and others 2007). There are conducted at the national level, and the scope may mean many cases where other data-related and methodological that there is no possibility for randomization. Fourth, re- concerns (mainly self-selection into the program) make sults could be invalidated or contaminated as a result of using intent-to-treat estimations better than the treatment- spillovers and changes in the behavior of individuals in the on-the treated effect. treatment group or the control group. Fifth, the generaliz- Experimental or randomized design is regarded as the most ability (external validity) of the results may be a source of robust of impact evaluation methodologies. Because the concern. Sixth, randomized designs can be expensive and beneficiaries of a program cannot be both receiving and time consuming. not receiving it, the control group must be constructed Proponents of randomization challenge some of these limi- from a group that is very similar. One critical difference tations (Duflo and others 2007). For example, on ethics, it is between a reliable and an unreliable impact evaluation, argued that it would be wrong "to assume that one would be therefore, is how well this counterfactual approximates the denying the poor a beneficial intervention until an idea has treatment group in the absence of the intervention. Ran- been properly evaluated" (World Bank 2007c). Moreover, dom assignment to the program ensures initial equivalence other ethical and political issues can be addressed by ex- of the beneficiary (treatment) and nonbeneficiary (control tending the program in the control areas at a later stage and or comparison) groups. It implies that both observable and by selecting the treatment and control groups in a politi- unobservable characteristics in the two groups are statisti- cally transparent manner (Baker 2000). cally identical. In that case, the impact of the program is measured by the difference in mean outcomes between the Concerning contamination, Duflo and others (2007) argue treatment and the control groups. In addition to this sim- that spillover effects can be captured if randomization oc- plicity in interpreting and conveying the results, a ran- curs at a higher level. For example, Miguel and Kremer domized evaluation design eliminates the possibility that (2004) randomized at the school level and found larger ef- specification error is influencing the results (Duflo and fects of de-worming drugs than other evaluations did based Kremer 2003; Duflo and others 2007). In this review, the on individual-level randomization. Regarding costs, Duflo primary identification strategy of 21 evaluations (46 per- and Kremer (2003) argue that evaluation costs can be re- cent of those reviewed) is based on randomization. duced by conducting a series of evaluations in the same 78 | What Can We Learn from Nutrition Impact Evaluations? area. Finally, problems of external validity also apply to served characteristics. For example, PSM would lead to nonexperimental methods. regression toward the mean if the worst cases of the un- treated were compared with the best cases of the treated Quasi-experimental designs comprise a class of causal eval- group. Other drawbacks of PSM and other matching uation designs that define a control group through methods include the need for large samples, the strong some nonrandom process. The identification strategy assumption that individuals in the matched control in 25 of the 46 reviewed evaluations (54 percent) is group did not choose to be untreated, and hidden bias based on these nonrandom processes. Econometric tech- that might remain because of differences between the niques are used to generate comparison groups that resem- treated and the untreated groups in unobservable char- ble the treatment group, at least in observed characteristics. acteristics. For example, in the Hogares Comunitarios Among the advantages of these approaches are that they can program, Attanasio and Vera-Hernandez (2004) show use existing data and are cheaper and quicker to implement. that PSM would show counterintuitive results on the im- However, one critical problem with quasi-experimental ap- pact of the program. They argue that a comparison of at- proaches is selection bias. Randomization balances the se- tending and nonattending children based on observables lection bias between the treated and the untreated samples alone would be misleading "as it ignores the endogeneity (Heckman and Smith 1995), but nonrandomized approaches of the participation decisions." In this review, 12 of the 46 use complex methods to correct it. Quasi-methods include evaluations (26 percent) used PSM. matching techniques, difference-in-difference (DID) or double-difference methods, instrumental variables meth- · Double difference or DID--This method compares the ods, regression discontinuity, and reflexive comparisons. treatment and control groups (first difference) before and after the intervention (second difference). The validity of The following methods were used by one or more of the this analysis depends on the assumption on the parallel reviewed studies. evolution of the outcome in the absence of the treatment. · Matching methods or constructed controls--The main task Eleven evaluations (24 percent) reviewed for this study is to pick an ideal comparison group that matches the used DID in combination with other methods. treatment group. The most widely used type of matching · Instrumental variables--The instrumental variables method is propensity score matching (PSM), in which the com- recognizes that program placement is not random, but parison group is matched to the treatment group on the purposive. Therefore, this method identifies the exoge- basis of a set of observed characteristics. In this method, nous component of the variance in program placement treated and untreated cases are matched on the basis of by using instrumental variables that matter to participa- propensity scores (the predicted probability of partici- tion to the program but not to outcomes, given participa- pating in the intervention, given observed characteris- tion. The validity of this method depends on the quality tics). The closer the score, the better the match. of the instrument. The instrumental variables method However, PSM can introduce error if the treated and the was used in six evaluations (13 percent) reviewed for this untreated groups do not have substantial overlap in ob- study. Appendix F: Impact Evaluation Basics | 79 Endnotes Chapter 1 for which the evidence showed little or no impact. These 1. Estimates are for 2005. recommendations are summarized in appendix A. 2. De Onis and Blössner (2000), based on an analysis of 160 10. It is important to note that a primary objective of CCTs national surveys from 94 countries. Overweight is defined is to affect poverty, as well as human development outcomes as a weight that is more than two standard deviations above such as nutrition. that of the reference population for a given height. Among 11. This may be due in part or mostly to a failure by the the regions with the highest rates of overweight are North- studies themselves to examine the heterogeneity of impacts ern Africa (8.1 percent), Southern Africa (6.5 percent), and (Heckman and Smith 1995; Ravallion 2009). However, nu- Latin America and the Caribbean (4.4 percent). trition impact evaluations often do present results across 3. This is the share of the lending portfolio managed by different age groups--the main exception. the Health, Nutrition, and Population Sector with nutri- 12. Bhutta and others (2008) highlight this evidence on tion objectives; the share of projects managed by other sec- "effectiveness and cost-effectiveness of nutritional interven- tors that have nutrition objectives or components was not tions in national health systems, single and packaged, for quantified. impact on stunting and weight gain." 4. The renewed commitment is evidenced in part by the 13. In this regard, it is important to note that child anthro- recent recruitment of six nutrition specialists to address pometric outcomes were often not the only outcomes antic- malnutrition, particularly in Africa and South Asia. ipated from these interventions. A comparative assessment 5. More recently, the Poverty Reduction and Economic of interventions across their other major objectives (both Management Network issued a handbook entitled Meth- in terms of other nutritional outcomes, as well as cognitive odologies to Evaluate the Impact of Large-Scale Nutrition and poverty reduction outcomes) is beyond the scope of Projects. this paper. 6. Because of this complexity, Bhutta and others (2008) Chapter 2 note that "the choice [of intervention] will depend on the actual nature and distribution of the malnutrition prob- 1. Studies of the impact of interventions on other anthro- lem, its causes, and the type of resources that are available" pometric outcomes, such as upper-arm circumference and (p. ix). skinfold thickness, were excluded. 7. As an exception, in China and Madagascar, where the 2. Most of these evaluations measured program effects of edible-salt industry is concentrated in a few producers, salt the interventions on several other schooling and health iodization can be nearly universalized and little choice is outcomes. Further, for some of the interventions (such as exercised by households (Goh 2001). CCTs and micronutrient interventions), improving anthro- 8. The conclusions on breastfeeding promotion, comple- pometric outcomes was not the primary objective. Inter- mentary feeding, and food supplementation in populations ventions with little impact on anthropometric outcomes with and without sufficient food, for example, were based might have significant impacts on these other primary out- on 10 studies--3 in food-secure populations (defined as comes; however, these are not reviewed here. having average income of more than $1/day) and 7 in non- 3. CCTs and UCTs, for example, are generally offered to food-secure populations (Bhutta and others 2008). low-income households. 9. Despite the lack of data on the effectiveness of large-scale 4. These evaluations nevertheless often control for dem- interventions, the authors nevertheless classify a relatively ographic and socioeconomic characteristics to reduce long list of specific nutrition interventions into four catego- idiosyncratic variation and to improve the power of the ries as the basis for their recommendations on scaling up: estimates (for example, Bobonis, Miguel, and Sharma 2006; (a) interventions for which "evidence was sufficiently ro- Gertler 2004; Morris and others 2004; Paxson and Schady, bust to recommend their use in most countries with high forthcoming). burdens of undernutrition"; (b) those that might be recom- 5. Quasi-experimental methods may be adopted when mended for countries in specific situational contexts; (c) randomization fails to equate the treatment and control or those with insufficient or variable evidence; and (d) those when no baseline information is available. 80 | What Can We Learn from Nutrition Impact Evaluations? 6. Macours and others (2008) studied impacts on children 18. At baseline in 2000, 13.7 percent and 14.3 percent of 0­23, 24­47, and 48­71 months old; Maluccio and Flores the children in the program and nonprogram areas, respec- (2005) studied children age 0­60 months. tively, were underweight, respectively, with just a ­0.6 in- 7. Morris and others (2004) found no impacts on children significant difference between them. The net underweight 0­23, 24­47, and 48­84 months old. averted by the program was 5.5 percentage points. 8. Agüero and others (2007) consider the first three years 19. For example, the average regional prevalence of stunt- of life as a "nutritional window" vital for larger program ing, underweight, and wasting for 2000­07 based on the impact. They argue that a treatment that covers much of National Center for Health Statistics reference population the child's early age boosts the HAZ, and there are no gains is as follows: Sub-Saharan Africa (38 percent, 28 percent, for treatments covering less than 20 percent of the child's and 9 percent); Latin America and the Caribbean (16 per- nutritional window. cent, 6 percent, and 2 percent); and South Asia (46 percent, 9. The program included a behavior change and communi- 45 percent, and 18 percent). http://www.childinfo.org/ cation component. The preventive model targeted all chil- index.html. dren age 6­23 months, and the recuperative model targeted 20. Of the 15 results, 7 had significant and positive impacts. underweight children age 6­60 months 21. The prevalence of wasting in Haiti is for children 10. In fact, FFW had a negative impact on HAZ for chil- younger than 0­59 months in 2000. http://www.childinfo. dren under five and for girls five to nine years of age in low- org/undernutrition_wasting.php. asset households (p < 0.10 and p < 0.05, respectively). Lagged 22. However, their control areas were less than ideal. Un- food distribution had a negative impact on HAZ for children fortunately, White and Masset (2007) did not report find- five to nine years of age in high-asset households (p < .001), ings on wasting using more robust PSM techniques. although the magnitude of the impact is quite small. 23. All in all, they report 15 results, with 9 showing impact. 11. Participation in the program, captured by the current Of the 9, 6 were with their expected negative signs. attendance measure, was associated with an increase in 24. Only the evaluation of Colombia's CCT, Familias en Ac- the HAZ by 0.486, which is equivalent to 2.36 centimeters ción, by Attanasio and others 2005 used a quasi-experimen- in height for a boy or 2.39 centimeters for a girl at age 72 tal design (PSM and difference-in-difference techniques). months. The exposure model suggests that impact increas- 25. The average beneficiary time in the CCT program es when participation is adjusted by age. The age-adjusted contributes 68 grams, and the amount of cash received is increase is 0.78 in HAZ, which is equivalent to a 3.78- associated with a 78.2-gram weight gain. Program time centimeter increase in height for a boy or 3.83-centimeter measures the number of months between the date of receipt increase for a girl 72 months old. of the first cash transfer and the date of birth. 12. The finding was statistically significant at the p = 0.10 26. The sample size (including treatment and control) for level. this part of the analysis is 174. The authors suggest that lack 13. Das Gupta and others (2005) (ICDS, for children age 0­3 or 0­4 years), Schipani and others (2002) (gardening, of significant impact might be due to the small size of the for children age 1­7 years). sample. 14. Of 15 coefficients representing children of different 27. These impacts were not found for all women (just for ages and exposures to the program, 3 indicated a signifi- this subgroup), although the evaluation did find impacts on cant reduction in stunting and 5 indicated an increase. The malaria and anemia. remaining 7 coefficients were insignificant. 28. The "better-off " communities were the third of com- 15. The authors speculate that this counterintuitive re- munities with the lowest incidence of poverty. sult might be caused by a perception by beneficiaries that 29. In fact, table 6 of Quisumbing (2003) shows that FFW "benefits would be discontinued if the child started to grow improves the WHZ of boys under five in low-asset house- well." holds and worsens the WHZ of girls. 16. The evaluations in Kenya and India had similar designs 30. The other intermediate outcomes measured were preg- and found impacts on other educational and health out- nancy knowledge and practice (three evaluations) and hy- comes. However, in India the program raised WAZ but not giene behavior (one evaluation). HAZ for children age 2­6, while the opposite was the case 31. The cost of a de-worming program per pupil per year is in Kenya for children aged 6­18. $0.49, and the authors show that 99 percent of the reduction 17. However, the comparability of the program and non- in DALYs was attributable to the averted schistosomiasis. program areas was not well established. The subsequent 32. A scenario that is taken into consideration is a pre- evaluation by White and Masset (2007) with a more rigor- school program that results in a 2 percent increase in height ous methodology that used propensity score matching did at childhood, a 5 percent increase in cognitive skills and a not report results on underweight. one-year increase in grades completed, and a corresponding Endnotes | 81 one-year increase in the age of school completion. However, 6. In fact, many different evaluative activities were pro- the program did not improve child nutritional status. grammed into the Bolivia project. 33. They estimate program impact on WAZ of children and 7. The triggers included selection of a firm for the base- show that gains are larger for more educated mothers for line survey of BDH; a methodology and implementation villages with better infrastructure. schedule (first loan); adequate progress in implementation 34. Hossain and others (2005) also found an increase in of the evaluation, according to the plan (second loan); and knowledge in project areas, compared with nonproject ar- changes in the design, budget, and implementation of BDH eas, but concluded that there was no impact on child nutri- based on the results of the impact evaluation (third loan) tion outcomes. However, the project and nonproject areas (World Bank 2003a). may not have been comparable. 8. Karim and others (2003) measured the impact of the 35. See in particular the DHS evidence presented in ap- project as the difference in outcomes between the baseline pendix E. However, the surveys used for the impact evalu- and endline surveys in project areas; there was no attempt ation did not include these measures, so it was not possible to compare results with nonproject areas. to examine BINP impacts for women who did and did not 9. The consortium included Econometria Consultores; the face these constraints. Institute for Fiscal Studies at University College, London; Chapter 3 and Sistemas Especializados de Informacion. 10. The non-Bank researchers involved in the evaluation 1. The Development Impact Evaluation Initiative (DIME) of Familias en Acción and BINP were not involved in the is a Bank-wide collaboration involving thematic networks, design of the projects they evaluated. Regional units, and the research group under the guidance 11. The Office of Population Studies, San Carlos Univer- of the World Bank's Chief Economist. There are 27 com- sity, Cebu, Philippines, and the Institute of Public Health pleted or ongoing evaluations reported on the DIME Web at Makerere University, Kampala, Uganda. The White and site that measure impacts on anthropometric outcomes, 6 Masset (2007) evaluation of BINP used existing data sets; of which are reviewed in this study. Of the 21 remaining, the evaluation by Hossain and others (2005) financed their two-thirds measure the impact of health or nutrition inter- own data collection, but it is unclear which organization ventions, and a third measure the impact of social protec- collected the data. tion interventions (CCTs, social funds). About half involve 12. The evaluation was nevertheless part of the policy a randomized design, a quarter used a quasi-experimental matrix for the First Programmatic Human Development design, three use both methods, and for three the method- Reform Project. ology was not reported. A third are in Sub-Saharan Africa, 13. The two research proposals and funding were for com- and a quarter each are in Latin America and the Caribbean and South Asia; none measures nutrition outcomes in East- munity nutrition program impact evaluations in Mada- ern Europe and Central Asia or the Middle East and North gascar and Senegal (Alderman and Rokx 2003, request Africa. More than half of these nutrition impact evaluations for $207,200) and for evaluation of the three early child are linked to World Bank projects. Six have been completed. development programs in Bolivia, the Philippines, and The Spanish Impact Evaluation Fund directly funds impact Uganda (Alderman and van der Gaag circa 1997, request evaluations, preferring experimental designs, but to date it for $395,500). has funded none of the proposals on nutrition. 14. The impact evaluation of BDH in Ecuador received 2. IEG was able to interview project team leaders and $400,000 from the Spanish Impact Evaluation Fund and a evaluators for all eight programs; policy makers were inter- $1 million grant from the Japanese Trust Fund; additional viewed for six of the eight countries (Bolivia and the Philip- data collection by Galasso and Umapathi (2009) of commu- pines were not reached). nity nutrition in Madagascar was funded with grants from 3. The conditionality was announced but never enforced. the Bank­Netherlands Partnership Program and UNICEF; So for all intents and purposes, the program was an uncon- the evaluation of BINP and other maternal and child health ditional transfer. programs by IEG was supported by $230,000 from a De- 4. Hogares de Bienestar Infantil, in Colombia, had been partment for International Development partnership and evaluated in 1992 and was found to be successful (World $23,400 from a Danish trust fund. Bank 1993, p. 14). 15. The IEG budget supported the BINP evaluation (which 5. Projects in Colombia, Ecuador, Bangladesh, and the was combined with the evaluation of several other maternal Philippines incorporated impact evaluations explicitly in and child health programs) to the sum of $165,625. the PAD; the other projects all called for baseline, midterm, 16. The NGOs had launched sensitization and mobilization and endline surveys or evaluations. activities in the communities before the impact evaluation 82 | What Can We Learn from Nutrition Impact Evaluations? design was finalized, putting them in an awkward position Yet low access to health care conceivably could be a rea- vis-à-vis communities previously mobilized for which im- son for nonparticipation or nonadherence in the Colombia plementation would have to be deferred. CCT program, and, in the case of Ecuador, the availability 17. The main difference between the treatment and control and quality of health services is likely to affect the extent municipalities was that the controls lacked a bank, which to which additional cash income is translated into health was essential for processing the transfer. outcomes. 18. Orazio Attanasio, personal communication. 28. The estimate of $43 is attributed by Behrman, Cheng, 19. Alderman (2007), Armecin and others (2006), Lin- and Todd (2004) to Ruiz (1996). The Implementation Com- nemayr and Alderman (2008), Behrman, Cheng, and Todd pletion and Results Report for the project put the cost at (2004), Galasso and Umapathi (2009), Galasso and others $30/month/child initially, which was brought down to $22/ (2009), White and Masset (2007). Matching methods also month/child. Subsequent changes to the program (after the have limitations, however. It is possible to match only on impact evaluation) brought the cost down to $2/month/ the basis of characteristics that are observed in both the child, based on eight months of implementation. treatment and control populations. 29. One of the difficulties in conducting cost-benefit analy- 20. However, it is important to control for the characteris- sis is that there is often no country-specific data on how nu- tics of the communities or individuals enlisted at different tritional and other impacts from the program affect long-run times. For example, the program may have initially targeted earnings, on the basis of which to calculate the benefits. Thus, the neediest individuals or communities. they are often extrapolated from studies in other settings. 21. The cash transfer and early child development interven- 30. The authors calculate, according to simulations (not tions often aimed to affect other outcomes, including edu- based on the impact evaluation parameters), that the cost cational attainment and cognitive outcomes, and, in some of preventing one case of underweight by simply financing cases, other health outcomes. However, this section focuses a rice ration would be on the order of $110 per year and the narrowly on the findings on child anthropometric status. cost per life saved $2,223. 22. The authors point to cultural factors--the lack of 31. IEG was unable to interview policy makers from Bo- control of women in decisions regarding food purchase livia and the Philippines. and preparation--as possibly explaining the fact that bet- 32. Cited in the Implementation Completion and Results ter knowledge does not seem to have led to much better Report. In retrospect, it is fortunate that some of the mu- outcomes. nicipalities in the impact evaluation baseline survey had 23. In the Bolivia early child development project, food ac- already been enlisted into the program. Had that not been counted for about half of the total project cost of $36/child/ the case, there would have been no quick evidence that the month. program was effective to provide to the new government. 24. This perhaps is not surprising, given the short implemen- It was also reported by informants that evidence from the tation period (18 months) and the well-documented finding Progresa evaluation in Mexico was influential in the deci- in the literature that the weight and height of children under sion to continue the program. two are particularly sensitive to nutritional inputs. 33. According to informants, the cash transfers for rural 25. Almost all of the studies examined the impacts across families in the most vulnerable municipalities are condi- different age groups of children (the exception being the tioned on the number of annual visits for children under de-worming evaluation in Uganda). Here we review het- two on their "healthy child card" and on the weight register erogeneity in impacts across socioeconomic characteristics at the health facility. Children under one year of age must and access to services. The evaluation of BINP by White show at least six visits, and children between one and two and Masset presented results on the heterogeneity of inter- years must show at least three visits. mediate behavioral outcomes but not nutrition impacts. 26. Note, however, that this result does not apply to HAZ Chapter 4 individually but rather to a synthetic index of three "physi- 1. This point is also made in a 2008 letter to the editor of cal" outcome measures that included HAZ. The Lancet, in which Shekar and 17 signatories highlight 27. The evaluations of the cash transfer programs in Co- the need to expand the research agenda to include the "de- lombia and Ecuador are among those that did not examine livery science" to "understand implementation and cost ef- impacts as a function of the availability of public services. fectiveness at scale" of nutrition interventions. Endnotes | 83 Bibliography *Agüero, Jorge M., Michael Carter, and Ingrid Woolard. Allen, Lindsay H., and Stuart R. Gillespie. 2001. "What 2007. "The Impact of Unconditional Cash Transfers on Works? A Review of the Efficacy and Effectiveness of Nutrition: The South African Child Support Grant." In- Nutrition Interventions." ACC/SCN Nutrition Policy ternational Poverty Centre Working Paper No. 39. Inter- Paper 19; Asian Development Bank (ADB) Nutrition national Poverty Centre, United Nations Development and Development Series 5. United Nations Administra- Programme, Brasilia, Brazil. tive Committee on Coordination/Sub-Committee on *Alderman, Harold. 2007. "Improving Nutrition through Nutrition and the Asian Development Bank, Geneva Community Growth Promotion: Longitudinal Study of and Manila. the Nutrition and Early Child Development Program in *Armecin, Graeme, Jere R. Behrman, Paulita Duazo, Sha- Uganda." World Development 35 (8): 1376­89. ron Ghuman, Socorro Gultiano, Elizabeth M. King, and Alderman, Harold, and Jere R. Behrman. 2006. "Reducing Nanette Lee. 2006. "Early Childhood Development Pro- the Incidence of Low Birthweight in Low-Income Coun- grams through an Integrated Program: Evidence from tries Has Substantial Economic Benefits." World Bank the Philippines." Policy Research Working Paper 3922, Research Observer 21 (1): 25­48. World Bank, Washington DC. *Alderman, Harold, Hans Hoogeveen, and Mariacristina *Attanasio, Orazio, Luis Carlos Gómez, Patricia Heredi, Rossi. 2006. "Reducing Child Malnutrition in Tanzania: and Marcos Vera-Hernández. 2005. "The Short-term Combined Effects of Income Growth and Program In- Impact of a Conditional Cash Subsidy on Child Health terventions." Economics and Human Biology 4: 1­23. and Nutrition in Colombia." Institute of Fiscal Studies, *Alderman, Harold, Joseph Konde-Lule, Isaac Sebuliba, London. Donald Bundy, and Andrew Hall. 2006. "Effect on *Attanasio, Orazio, and M. Vera-Hernández. 2004. "Me- Weight Gain of Routinely Giving Albendazole to Pre- dium and Long-Run Effects of Nutrition and Child Care: school Children during Child Health Days in Uganda: Evaluation of a Community Nursery Program in Rural Cluster Randomized Controlled Trial." British Medical Colombia." IFS Working Paper EWP04/06, Institute for Journal 333: 122­26. Fiscal Studies, London. *Alderman, Harold, Brian Ndiaye, Sebastian Linnemayr, Baker, J. 2000. Evaluating the Impact of Development Proj- Abdoulaye Ka, Claudia Rokx, Khadidiatou Dieng, ects on Poverty: A Handbook for Practitioners. Washing- and Menno Mulder-Sibanda. 2009. "Effectiveness of a ton, DC: World Bank. Community-Based Intervention to Improve Nutrition in *Barber, Sarah L., and Paul J. Gertler. 2008. "The Impact of Young Children in Senegal: A Difference-in-Difference Mexico's Conditional Cash Transfer Programme, Opor- Analysis." Public Health Nutrition 12 (5): 667­73. tunidades, on Birthweight." Tropical Medicine & Inter- Alderman, Harold, and Claudia Rokx. 2003. "Community national Health 13 (11): 1405­14. Nutrition: Evaluation of Impacts in Africa." Request *Behrman, Jere R., Yingemei Cheng, and Petra E. Todd. for Research Support Budget Grant: Administrative 2004. "Evaluating Preschool Programs When Length of Summary. June 30. Exposure to the Program Varies: A Nonparametric Ap- Alderman, Harold, and Elizabeth King. Undated. "Evalua- proach." Review of Economics and Statistics 86 (1): tion of the Impact of Investments in Early Child De- 108­32. velopment on Nutrition and Cognitive Development." *Behrman, Jere R., and John Hoddinott. 2005 "Programme World Bank Research Committee Project Completion Evaluation with Unobserved Heterogeneity and Selec- Report No. P052067, Washington, DC. tive Implementation: The Mexican PROGRESA Impact Alderman, Harold, and Jacques van der Gaag. Undated, on Child Nutrition." Oxford Bulletin of Economics and circa 1997. "Evaluation of the Impact of Investments in Statistics 67 (4): 547­69. Early Child Development on Nutrition and Cognitive Bhutta, Zulfiqar A., Tahmeed Ahmed, Robert E. Black, Si- Development: A Research Proposal." Research Proposal mon Cousens, Kathryn Dewey, Elsa Giugliani, Batool A. Submitted to the World Bank Research Committee. Haider, Betty Kirkwood, Saul S. Morris, H. P. S. Sachdev, *These references are the 46 impact evaluations that the Independent Evaluation Group reviewed for this study. 84 | What Can We Learn from Nutrition Impact Evaluations? and Meera Shekar, for the Maternal and Child Undernu- grammes for Child Health, Growth, and Development: trition Study Group. 2008. "What Works? Interventions An Analysis of Mexico's Oportunidades."The Lancet 371: for Maternal and Child Undernutrition and Survival." 828­37. The Lancet 371 (9610): 417­40. Fiszbein, Ariel, and Norbert Schady with Francisco H. G. Black, Robert E., Lindsay H. Allen, Zulfiqar A. Bhutta, Ferreira, Margaret Grosh, Nial Kelleher, Pedro Olinto, Laura E. Caulfield, Mercedes de Onis, Majid Ezzati , Co- and Emmanuel Skoufias. 2009. Conditional Cash Trans- lin Mathers, and Juan Rivera. 2008. "Maternal and Child fers: Reducing Present and Future Poverty. Washington, Undernutrition: Global and Regional Exposures and DC: World Bank. Health Consequences." The Lancet 371: 243­60. *Friis, Henrik, Exnevia Gomo, Norman Nyazema, Patricia *Bobonis, Gustavo, Edward Miguel, and Charu Puri Ndhlovu, Henrik Krarup, Pernille Koestel, and Kim F. Sharma. 2006. "Anemia and School Participation." Jour- Michaelsen. 2004. "Effect of Multimicronutrient Sup- nal of Human Resources 41 (4): 693­721. plementation on Gestational Length and Birth Size: A Caulfield, L., S. Richard, J. Rivera, P. Musgrove, and R. Randomized, Placebo-Controlled, Double-Blind Ef- Black. 2006. "Stunting, Wasting, and Micronutrient fectiveness Trial in Zimbabwe." American Journal of Deficiency Disorders." In Disease Control Priorities in Clinical Nutrition 80: 178­84. Developing Countries, ed. Dean T. Jamison, Joel G. Bre- *Galasso, Emanuela, and Nithin Umapathi. 2009. "Improv- man, Anthony R. Measham, George Alleyne, Mariam ing Nutritional Status through Behavioral Changes: Claeson, David B. Evans, Prabhat Jha, Anne Mills, Lessons from Madagascar." Journal of Development Ef- Philip Musgrove, chapter 28. Washington, DC: World fectiveness 1 (1): 60­85. Bank. *Galasso, Emanuela, and Jeffrey Yau. 2006. "Nutritional *Christian, Parul, Subarna K. Khatry, Joanne Katz, Eliza- Gains from Extended Exposure to a Large-Scale Nutri- beth K. Pradhan, Steven C. LeClerq, Sharada Ram tion Program." Policy Research Working Paper 4508, Shrestha, Ramesh K. Adhikari, Alfred Sommer, and World Bank, Washington, DC. Keith P. West Jr. 2003. "Effects of Alternative Maternal *Gertler, Paul. 2004. "Do Conditional Cash Transfers Im- Micronutrient Supplements on Low Birthweight in Ru- prove Child Health? Evidence from PROGRESA's Con- ral Nepal: Double-blind Randomized Community Trial." trol Randomized Experiment." The American Economic British Medical Journal 326: 571­74. Review 94 (2): 336­41. *Das Gupta, Monica, Michael Lokshin, Michele Gragnolati, Goh, Chorching. 2001. "An Analysis of Combating Iodine and Oleksiy Ivaschenko. 2005. "Improving Child Nutri- Deficiency: Case Studies of China, Indonesia, and Mad- tion Outcomes in India: Can the Integrated Child De- agascar." IEG Working Paper Series, No. 18, World Bank velopment Services Program Be More Effective?" Policy Independent Evaluation Group, Washington, DC. Research Working Paper 3647, World Bank, Washing- *Gupta, Piyush, Mily Ray, Tarun Dua, Gita Radhakrishnan, ton, DC. Rajeev Kumar and H. P. S. Sachdev. 2007. "Multimicro- De Onis, Mercedes, and Monika Blossner. 2000. "Preva- nutrient Supplementation for Undernourished Pregnant lence and Trends of Overweight among Preschool Chil- Women and Birth Size of Their Offspring." Archives of dren in Developing Countries." American Journal of Pediatric Adolescent Medicine 161: 58­64. Clinical Nutrition 72: 1032­39. Habicht, Jean-Pierre, Reynaldo Martorell, and Juan A. Deaton, Angus. 2008. "Instruments of Development: Ran- Rivera. 1995. "Nutritional Impact of Supplementation in domization in the Tropics, and the Search for the Elusive the INCAP Longitudinal Study: Analytic Strategies and Keys to Economic Development." The Keynes Lecture in Inferences." Journal of Nutrition 125(4): 1042S­50S. Economics, British Academy, October 9. Heckman, James J., and Jeffrey A. Smith. 1995. "Assessing Duflo, E., and M. Kremer. 2003. "Use of Randomization in the Case for Social Experiments." Journal of Economic the Evaluation of Development Effectiveness." Paper Perspectives 9 (2): 85­110. prepared for the World Bank Independent Evaluation Hoddinott, John, John A. Maluccio, Jere R. Behrman, Ra- Group, Conference on Evaluation and Development Ef- fael Flores, and D. Reynaldo Martorell. 2008. "Effect of a fectiveness, Washington, DC, July 15­16. Nutrition Intervention during Early Childhood on Eco- Duflo, E., M. Kremer, and R. Glennerster. 2007. "Using nomic Productivity in Guatemalan Adults." The Lancet Randomization in Development Economics Research: 371: 411­16. A Toolkit." Discussion Paper 6059, Center for Economic *Hossain S., M. Moazzem, Arabella Duffield, and Anna Policy Research, London. Taylor. 2005. "An Evaluation of the Impact of a US$60 Fernald, Lia C. H., Paul J. Gertler, and Lynnette M. Neufeld. Million Nutrition Programme in Bangladesh." Health 2008. "Role of Cash in Conditional Cash Transfer Pro- Policy and Planning 20 (1): 35­40. *These references are the 46 impact evaluations that the Independent Evaluation Group reviewed for this study. Bibliography | 85 *Iannotti, Lara L., Nelly Zavaleta, Zulema Leon, Anuraj H. *Maluccio, John A., and Rafael Flores. 2005. "Impact Evalu- Shankar, and Laura E. Caulfield. 2008. "Maternal Zinc ation of a Conditional Cash Transfer Program: The Ni- Supplementation and Growth in Peruvian Infants." caraguan Red de Protección Social." Research Report 14, American Journal of Clinical Nutrition 88: 154­60. International Food Policy Research Institute, Washing- Independent Evaluation Group (IEG). 2009. Improving Ef- ton, DC. fectiveness and Outcomes for the Poor in Health, Nutri- *Masanja, H., J. Armstrong Schellenberg, D. De Savigny, H. tion, and Population. Washington, DC: World Bank. Mshinda, and C. Victora. 2005. "Impact of Integrated ------. 2008. What Works in Water Supply and Sanitation? Management of Childhood Illness on Inequalities in Lessons from Impact Evaluation. Washington, DC: World Child in Child Health in Rural Tanzania." Health Policy Bank. and Planning 20 (Supplement 1): 77­84. *------. 2005. Maintaining Momentum to 2015? An Impact McGregor, Sally, Yin Bun Cheung, Santiago Cueto, Paul Evaluation of Interventions to Improve Maternal and Glewwe, Linda Richter, Barbara Strupp, and the Inter- Child Health and Nutrition in Bangladesh. Washington, national Child Development Steering Group. 2007. "De- DC: World Bank. velopment Potential in the First 5 Years for Children in Karim, Rezaul, Sascha A. Lamstein, M. Akhtaruzzaman, Developing Countries." The Lancet 369: 60­70. Khan Maududur Rahman, and Nurul Alam. 2003. The *Menéndez, C., A. Bardaji, B. Sigauque, C. Romagosa, S. Bangladesh Integrated Nutrition Project Community- Sanz, E. Serra-Casas, E. Macete, A. Berenguera, C. Da- Based Nutrition Component: Endline Evaluation. Final vid, D. Dobaño, D. Naniche, A. Mayor, J. Ordi, I. Man- Report. Institute of Nutrition and Food Science, Univer- domando J. J. Aponte, S. Mabunda, and P. L. Alonso. 2008. "A Randomized Placebo-Controlled Trial of Inter- sity of Dhaka, Dhaka, Bangladesh; and Friedman School mittent Preventive Treatment in Pregnant Women in of Nutrition Science and Policy, International Food and the Context of Insecticide-Treated Nets Delivered Nutrition Center, Tufts University, Medford, MA. through the Antenatal Clinic." PLoS ONE, 3 (4): e1934. *Kazianga, Harounan, Damien de Walque, and Harold Al- *Miguel, Edward, and Michael Kremer. 2004. "Worms: derman. 2009. "Educational and Health Impacts of Two Identifying Impacts on Education and Health in the School Feeding Schemes: Evidence from a Randomized Presence of Treatment Externalities." Econometrica 72 Trial in Rural Burkina Faso." Policy Research Working (1): 159­217. Paper 4976, World Bank, Washington, DC. *Morris, S., P. Olinto, R. Flores, E. Nilson, and A. Figueiró. King, Elizabeth M., and Jere R. Behrman. 2009. "Timing 2004. "Conditional Cash Transfers Are Associated with and Duration of Exposure in Evaluations of Social Pro- a Small Reduction in the Rate of Weight Gain of Pre- grams." World Bank Research Observer 24 (1): 55­82. school Children in Northeast Brazil." The Journal of Kramer, Michael S. 1987. "Determinants of Low Birth Nutrition 134: 2336­41 Weight: Methodological Assessment and Meta-Analysis." *Osrin, David, Anjana Vaidya, Yagya Shresta, Ram Baha- Bulletin of the World Health Organization 65 (5): 663­737. dur Baniya, Dharma S. Manandhar, Ramesh K. Adhikari, *Leon, Mauricio, and Stephen D. Younger. 2007. "Transfer Suzanne Filteau, Andrew Tomkins, and Anthony M. de Payments, Mothers' Income and Child Health in Ecua- L. Costello. 2005. "Effects of Antenatal Multiple Micro- dor." Journal of Development Studies 43 (6): 1126­43. nutrient Supplementation on Birthweight and Gesta- *Leroy, Jeff L., Armando Garcia-Guerra, Raquel Garcia, tional Duration in Nepal: Double-Blind, Randomized Clara Dominguez, Juan Rivera, and Lynnette Neufeld. Controlled Trial." The Lancet 365: 955­62. 2008. "The Oportunidades Program Increases Linear *Paxson, Christina, and Norbert R. Schady. Forthcoming. Growth of Children Enrolled at Young Ages in Urban "Does Money Matter? The Effects of Cash Transfers on Mexico." The Journal of Nutrition 138: 793­98. Child Health and Development in Rural Ecuador." Eco- Levinson, F. J., and J. E. Rohde. 2005. Letter to the Editor. nomic Development and Cultural Change. Health Policy and Planning 20 (6): 405­06. *Penny Mary E., Hilary M. Creed Kanashiro, Rebecca C. *Linnemayr, Sebastian, and Harold Alderman. 2008. "Almost Robert, M. Rocio Narro, Laura E. Claufield, and Robert Random: Evaluating a Large-Scale Randomized Nutrition E. Black. 2005. "Effectiveness of an Educational Interven- Program in the Presence of Crossover." Policy Research tion Delivered through the Health Services to Improve Working Paper 4784, World Bank, Washington, DC. Nutrition in Young Children: A Cluster-Randomized *Macours, Karen, Norbert R. Schady, and Renos Vakis. Controlled Trial." The Lancet 365: 1863­71. 2008. "Cash Transfers, Behavioral Changes and the Cog- Pelletier, David, Meera Shekar, L. Du, and Kees Koster- nitive Development of Young Children: Evidence from a mans. 2005. "Bangladesh Integrated Nutrition Project: Randomized Experiment." Policy Research Working Effectiveness and Lessons." Bangladesh Development Paper 4759, World Bank, Washington, DC. Series, Paper No. 7. World Bank, Dhaka, Bangladesh. *These references are the 46 impact evaluations that the Independent Evaluation Group reviewed for this study. 86 | What Can We Learn from Nutrition Impact Evaluations? *Quisumbing, Agnes R. 2003. "Food Aid and Child Nutrition Nutritional Status of Young Children in Families Prac- in Rural Ethiopia." World Development 31 (7): 1309­24. ticing Mixed Home Gardening in Northeast Thailand." *Ramakrishnan, Usha, Teresa González-Cossío, Lynnette Food and Nutrition Bulletin 23(2): 175­180. M. Neufeld, Juan Rivera, and Reynaldo Martorell. 2003. *Schroeder, Dirk G., Helena Pachón, Kirk A. Dearden, Tran "Multiple Micronutrient Supplementation during Preg- Thu Ha, Tran Thi Lang, and David R. Marsh. 2002. "An nancy Does Not Lead to Greater Infant Birth Size Than Integrated Child Nutrition Intervention Improved Does Iron-Only Supplementation: A Randomized Con- Growth of Younger, More Malnourished Children in trolled Trial in a Semi-Rural Community in Mexico." Northern Viet Nam." Food and Nutrition Bulletin 23 (4) American Journal of Clinical Nutrition 77: 720­25. (supplement): 50­58. Ramakrishnan, Usha. 2004. "Nutrition and Low Birth Shekar, Meera, on behalf of 17 signatories. 2008. "Delivery Weight: From Research to Practice." American Journal of sciences in nutrition." Letter to the editor. The Lancet Clinical Nutrition 79: 17­21. 371: 1751. Ravallion, Martin. 2009a. "Should the Randomistas Rule?" Shrimpton, Roger, Cesar G. Victora, Mercedes de Onis, Ro- Economists' Voice 6 (2): 1­5. sangela Cost Lima, Monika Blossner, and Graeme ------. 2009b. "Evaluation in the Practice of Development." Clugston. 2001. "Worldwide Timing of Growth Falter- World Bank Research Observer 24 (1): 29­54. ing: Implications for Nutritional Interventions." Pediat- ------. 2005. "Evaluating Anti-Poverty Programs." Policy rics 107 (5): e75. Research Working Paper 3625, World Bank, Washing- Smith, Lisa C., and Lawrence Haddad. 2002. "How Potent ton, DC. Is Economic Growth in Reducing Undernutrition? What Are the Pathways of Impact? New Cross-Country Evi- *Rivera, Juan A., Daniela Stores-Alvarez, Jean-Pierre dence." Economic Development and Cultural Change 51 Habicht, Teresa Shamah, and Salvador Villalpando. (1): 55­76. 2004. "Impact of the Mexican Program for Education, *Stifel, David, and Harold Alderman. 2006. "The `Glass of Health, and Nutrition (PROGRESA) on Rates of Growth Milk' Subsidy Program and Malnutrition in Peru." World and Anemia in Infants and Young Children: A Random- Bank Economic Review 20 (3): 421­48. ized Effectiveness Study." Journal of the American Medi- UNICEF (United Nations Children's Fund). 2007. "Prog- cal Association 291 (21): 2563­70. ress for Children: A World Fit for Children Statisti- Ruel, Marie, and John Hoddinott. 2008. "Investing in Early cal Review." http://www.unicef.org/publications/index_ Childhood Nutrition." IFPRI Policy Brief 8. Interna- 42117.html. tional Food Policy Research Institute, Washington, DC. ------. 1990. "Strategy for Improved Nutrition of Children *Ruel, Marie, Purnima Menon, Jean-Pierre Habicht, Cor- and Women in Developing Countries." UNICEF Policy nelia Loechl, Gilles Bergeron, Gretel Pelto, Mary Ari- Review, New York. mond, John Maluccio, Lesly Michaud, and Bekele Han- UNICEF (United Nations Children's Fund) and World kebo. 2008. "Age-Based Preventive Targeting of Food Health Organization. 2004. Low Birthweight: Country, Assistance and Behavior Change and Communication Regional and Global Estimates. UNICEF: New York. for Reduction of Childhood Under Nutrition in Haiti: A Van de Poel, Ellen, Ahmad Reza Hosseinpoor, Niko Spey- Cluster Randomized Trial." The Lancet 371 (9612): broeck, Tom Van Ourti, and Jeannette Vega. 2008. "So- 588­95. cioeconomic Inequality in Malnutrition in Developing Ruiz, Fernando. 1996. Estudio de Costos del Proyecto Inte- Countries." Bulletin of the World Health Organization 86 gral Desarrollo Infantil (PIDI). La Paz, Bolivia: Ruiz (4): 282­91. Guissani Consultores. Victora, Cesar, Jean-Pierre Habicht, and Jennifer Bryce. Sack, D. A., S. K. Roy, T. Ahmed, and G. Fuchs. 2005. Letter 2004. "Evidence-Based Public Health: Moving Beyond to the Editor. Health Policy and Planning 20 (6): 406­07. Randomized Trials." American Journal of Public Health Sall, Macky, and Jacques Sylla. 2005. "African Prime Minis- 94 (3): 400­05. ters Take Lead in Child Survival." The Lancet 366: Walker, Susan P., Theodore D. Wachs, Julie Meeks Gardner, 1988­89. Betsy Lozoff, Gail A. Wasserman, Ernesto Pollitt, Julie *Santos, Iná, Cesar G. Victora, José Martines, Helen Gon- A. Carter, and the International Child Development çalves, Denise P. Giante, Neiva J. Valle, and Gretel Pelto. Steering Group. 2007. "Child Development: Risk Fac- 2001. "Nutrition Counseling Increases Weight Gain tors for Adverse Outcomes in Developing Countries." among Brazilian Children." Journal of Nutrition 131: The Lancet 369: 145­57. 2866­73. *Waters, Hugh R., Mary E. Penny, Hilary M. Creed- *Schipani, Steven, Frits van der Haar, Sangsom Sinawat, Kanashiro, Rebecca C. Robert, Pocio Naro, Jeff Willis, and Kandavasee Maleevong. 2002. "Dietary Intake and Laura E. Caulfield, and Robert E. Black. 2006. "The Cost- *These references are the 46 impact evaluations that the Independent Evaluation Group reviewed for this study. Bibliography | 87 Effectiveness of a Child Nutrition Education Programme ------. 2006a. Repositioning Nutrition as Central to Devel- in Peru." Health Policy and Planning 21: 257­64. opment: A Strategy for Large-Scale Action. Washington, West, Stephen G., Naihua Duan, Willo Pequegnat, Paul DC: World Bank. Gaist, Don C. Des Jarlais, David Holdgrave, Jose Szapoc- ------. 2006b. "Implementation Completion Report znik, Martin Fishbein, Bruce Rapkin, Michael Clatts, (FSLT-70500) on a Loan in the Amount of US$150.0 and Patricia Dolan Mullen. 2008. "Alternatives to the Million to the Republic of Colombia for a Human Capi- Randomized Controlled Trial." American Journal of tal Protection Project." Report No. 35614, World Bank, Public Health 98 (8): 1359­66. Washington, DC. *White, Howard, and Edoardo Masset. 2007. "Assessing In- ------. 2005a. "The Bangladesh Integrated Nutrition Proj- terventions to Improve Child Nutrition: A Theory-Based ect: Effectiveness and Lessons." Bangladesh Develop- Impact Evaluation of the Bangladesh Integrated Nutri- ment Series, Paper No. 8, Dhaka, Bangladesh. tion Project." Journal of International Development 19 ------. 2005b. "Project Appraisal Document on a Pro- (5): 627­52. posed Loan in the Amount of US$86.4 Million to the White, Howard. 2005. "Comment on contributions regard- Republic of Colombia for a Social Safety Net Project." ing the impact of the Bangladesh Integrated Nutrition Report No. 31880-CO, World Bank, Washington, DC. Project." Health Policy and Planning 20: 408­410. ------. 2004. "Implementation Completion Report (IDA- WHO (World Health Organization). 1995. "Physical Status: 25310) on a Credit in the Amount of SDR 35.8 Million The Use and Interpretation of Anthropometry." Report (US$50.7 million equivalent) to the Republic of Bolivia of a WHO Expert Committee, Technical Report Series for an Integrated Child Development Project." Report 854, Geneva. No. 30209, World Bank, Washington, DC. World Bank. 2009a. Global Monitoring Report 2009: A De- ------. 2003. "Program Document for a Proposed Pro- velopment Emergency. Washington, DC: World Bank. grammatic Human Development Reform Loan I in the ------. 2009b. "Implementation of the World Bank's Strat- Amount of US$50.00 Million to the Republic of Ecua- egy for Health, Nutrition, and Population (HNP) Results: dor." Report No. 25791-EC, World Bank, Washington, Achievements, Challenges, and the Way Forward-- DC. Progress Report." Health, Nutrition, and Population, Hu- ------. 2001. "Project Appraisal Document on a Proposed man Development, World Bank, Washington, DC. Loan in the Amount of US$150.0 Million to the Repub- ------. 2009c. "Methodologies to Evaluate the Impact of lic of Colombia for a Human Capital Protection Project." Large-Scale Nutrition Projects." Doing Impact Evalua- Report No. 21608-CO, World Bank, Washington, DC. tion, No. 13. World Bank, Washington, DC. ------. 1998. "Project Appraisal Document on a Proposed ------. 2008. "Project Appraisal Document on a Proposed Credit in the Amount of SDR 20.4 Million Equivalent to Loan in the amount of US$636.5 million to the Republic the Republic of Madagascar for a Community Nutrition of Colombia for a Support for the Second Phase of the II Project." Report No. 17507, World Bank, Washington, Expansion of the Program of Conditional Transfers- DC. Familias en Accion Project." Report No. 45377-CO, ------. 1993. "Staff Appraisal Report: Bolivia Integrated World Bank, Washington, DC. Child Development Project." Report No. 11905-BO, ------. 2007a. Healthy Development: The World Bank World Bank, Washington, DC. Strategy for Health, Nutrition, and Population Results. *Yamano, Takashi, Harold Alderman, and Luc Christi- Washington, DC: World Bank. aensen. 2005. "Child Growth, Shocks, and Food Aid in ------. 2007b. "Implementation Completion and Results Rural Ethiopia." American Journal of Agricultural Eco- Report on a Credit in the Amount of SDR 11.8 Million nomics 87 (2): 273­88. (US$14.7 million equivalent) to the Republic of Senegal *Zeng, L., Y. Cheng, S. Dang, H. Yan, M. J. Dibley, S. Chang, in Support of the First Phase Nutrition Enhancement and L. Kong. 2008. "Impact of Micronutrient Supple- Program." Report No. ICR000107, World Bank, Wash- mentation during Pregnancy on Birthweight, Duration ington, DC. of Gestation, and Perinatal Mortality in Rural Western ------. 2007c. "Impact Evaluation for Microfinance." Do- China: Double-Blind Cluster Randomized Controlled ing Impact Evaluation, No. 7. Washington, DC: World Trial." British Medical Journal 337: a2001. Bank. *These references are the 46 impact evaluations that the Independent Evaluation Group reviewed for this study. 88 | What Can We Learn from Nutrition Impact Evaluations? IEG Publications Annual Review of Development Effectiveness 2009: Achieving Sustainable Development Addressing the Challenges of Globalization: An Independent Evaluation of the World Bank's Approach to Global Programs Assessing World Bank Support for Trade, 1987­2004: An IEG Evaluation Books, Building, and Learning Outcomes: An Impact Evaluation of World Bank Support to Basic Education in Ghana Bridging Troubled Waters: Assessing the World Bank Water Resources Strategy Climate Change and the World Bank Group--Phase I: An Evaluation of World Bank Win-Win energy Policy Reforms Debt Relief for the Poorest: An Evaluation Update of the HIPC Initiative A Decade of Action in Transport: An Evaluation of World Bank Assistance to the Transport Sector, 1995­2005 The Development Potential of Regional Programs: An Evaluation of World Bank Support of Multicountry Operations Development Results in Middle-Income Countries: An Evaluation of World Bank Support Doing Business: An Independent Evaluation--Taking the Measure of the World Bank­IFC Doing Business Indicators Egypt: Positive Results from Knowledge Sharing and Modest Lending--An IEG Country Assistance Evaluation 1999­2007 Engaging with Fragile States: An IEG Review of World Bank Support to Low-Income Countries Under Stress Environmental Sustainability: An Evaluation of World Bank Group Support Evaluation of World Bank Assistance to Pacific Member Countries, 1992­2002 Extractive Industries and Sustainable Development: An Evaluation of World Bank Group Experience Financial Sector Assessment Program: IEG Review of the Joint World Bank and IMF Initiative From Schooling Access to Learning Outcomes: An Unfinished Agenda--An Evaluation of World Bank Support to Primary Education Hazards of Nature, Risks to Development: An IEG Evaluation of World Bank Assistance for Natural Disasters How to Build M&E Systems to Support Better Government IEG Review of World Bank Assistance for Financial Sector Reform An Impact Evaluation of India's Second and Third Andhra Pradesh Irrigation Projects: A Case of Poverty Reduction with Low Economic Returns Improving Effectiveness and Outcomes for the Poor in Health, Nutrition, and Population Improving the Lives of the Poor through Investment in Cities Improving Municipal Management for Cities to Succeed: An IEG Special Study Improving the World Bank's Development Assistance: What Does Evaluation Show: Maintaining Momentum to 2015: An Impact Evaluation of Interventions to Improve Maternal and Child Health and Nutrition Outcomes in Bangladesh New Renewable Energy: A Review of the World Bank's Assistance Pakistan: An Evaluation of the World Bank's Assistance Pension Reform and the Development of Pension Systems: An Evaluation of World Bank Assistance The Poverty Reduction Strategy Initiative: An Independent Evaluation of the World Bank's Support Through 2003 The Poverty Reduction Strategy Initiative: Findings from 10 Country Case Studies of World Bank and IMF Support Power for Development: A Review of the World Bank Group's Experience with Private Participation in the Electricity Sector Public Sector Reform: What Works and Why? An IEG Evaluation of World Bank Support Small States: Making the Most of Development Assistance--A Synthesis of World Bank Findings Social Funds: Assessing Effectiveness Sourcebook for Evaluating Global and Regional Partnership Programs Using Knowledge to Improve Development Effectiveness: An Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000­2006 Using Training to Build Capacity for Development: An Evaluation of the World Bank's Project-Based and WBI Training The Welfare Impact of Rural Electrification: A Reassessment of the Costs and Benefits--An IEG Impact Evaluation World Bank Assistance to Agriculture in Sub-Saharan Africa: An IEG Review World Bank Assistance to the Financial Sector: A Synthesis of IEG Evaluations World Bank Group Guarantee Instruments 1990­2007: An Independent Evaluation The World Bank in Turkey: 1993­2004--An IEG Country Assistance Evaluation World Bank Engagement at the State Level: The Cases of Brazil, India, Nigeria, and Russia All IEG evaluations are available, in whole or in part, in languages other than English. For our multilingual section, please visit http://www.worldbank.org/ieg. ISBN 978-0-8213-8406-0 90000 9 780821 384060 SKU 18406