Report No: AUS0000448 . India Mobilizing Communities for a Healthier Future: Impact Evaluation of Social Accountability Interventions in Uttar Pradesh, India . 09/20/2018 . HNP . © 2017 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved This work is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Attribution — Please cite the work as follows: “World Bank. 2018 . Mobilizing Communities for a Healthier Future: Impact Evaluation of Social Accountability Interventions in Uttar Pradesh, India. © World Bank.� All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. 1 ACKNOWLEDGEMENTS This report was co-authored by the principal investigators, Prof. Manoj Mohanan (Duke University), Prof. Harsha Thirumurthy (University of Pennsylvania), and Vikram Rajan (World Bank). Significant data cleaning and analysis were carried out by Kendal Swanson (Duke University). Additional inputs were given by Amith Nagaraj (World Bank) and Matt Bunyi (Duke University). The team is thankful for research collaboration on social networks from Prof. Arun Chandrasekar (Stanford University) and Prof. James Moody (Duke University). The team is grateful to the invaluable partners across the Government of Uttar Pradesh (GoUP), including Ms.V Hekali Zhimomi, I.A.S. and Dr. Harsh Sharma at Uttar Pradesh Health Systems Strengthening Project (UPHSSP) as well as Mr. NS Ravi and Dr. Om Prakash Pandey at the Deendayal Upadhyaya State Institute or Rural Development. The team would also like to thank the earlier leadership at these institutions that has supported the current evaluation. TARU Leading Edge Pvt ltd conducted the data collection for the endline surveys analyzed in this report. Morsel India Pvt Ltd and Sambodhi Research and Communications Pvt Ltd conducted fieldwork for information interventions and network data studies. This evaluation research was made possible through the generous support of the World Bank Group (Strategic Impact Evaluation Fund). The team would also like to thank Junaid Ahmad, Country Director, India, Rekha Menon, Practice Manager, Health, Nutrition and Population Global practice, and Alaka Holla, Program Manager, SIEF, for their continuous support and guidance during the study. Additional funding through the National Institutes of Health (NIH), the Duke University Population Research Institute (DUPRI), Bass Connections, and the Sanford School of Public Policy at Duke University made complementary research possible, including study components related to social networks. 2 EXECUTIVE SUMMARY Social Accountability has received much attention as a potentially promising method to engage community members in monitoring the delivery of public services. Recent efforts in Uganda and Nepal found that community monitoring mechanisms led to large improvements in child health outcomes. The government of Uttar Pradesh (UP), as part of the World Bank supported Uttar Pradesh Health Systems Strengthening Project (UPHSSP), implemented Social Accountability (SA) interventions on a large scale. In contrast to previous impact evaluation efforts led by researchers on a smaller scale, this novel effort represents the first large-scale impact evaluation of SA and community based monitoring that is designed and implemented by the government. SA interventions typically include providing information to constituents (including information about entitlements, rights, and responsibilities) as well as facilitation of community meetings to enable grievance redressal. Out of the 12 districts where UPHSSP is currently implementing the SA interventions, two of the districts – Sultanpur and Fatehpur – were selected to implement the program to enable research on the underlying mechanisms through which SA might impact outcomes. Specifically, this impact evaluation sheds light on whether the benefits of SA accrue due to delivery of information about health outcomes and services to households or due to facilitation of community-wide meetings regarding public services. The SA interventions in UP focus on the critical policy issue of public sector health service delivery, which remains a challenge in UP as well as India more generally. Despite decades of investments, the availability and utilization of key services including maternity care, immunization, primary care for children and nutritional supplementation remains low in rural areas of UP. The SA interventions aim to mobilize community participation in the functioning of the Village Health, Sanitation, and Nutrition Committees (VHSNCs), and delivery of critical services during the Village Health and Nutrition Days (VHNDs) that are supposed to be held every month. This evaluation used a randomized evaluation of UP’s SA interventions implemented across 120 villages in the two districts. A third of the villages were randomly assigned to an information only arm, a third to an information plus facilitation arm and a third to a control arm. In the Information only arm, community members received key information about local health indicators as well as facts about the VHSNC. In the information plus facilitation arm, Gram Panchayat Coordinators (GPCs) helped activate and organize VHSNC meetings in addition to the standard information provision to community members. The SA interventions were implemented starting in May 2016. GPCs were recruited by GoUP for a period of one year. In the initial months of the implementation, intensive information dissemination was done and data was collected on community member participation in VHSNC meetings. Two years after introduction of the SA interventions, the impact evaluation (IE) team collected follow up data in the 120 villages to evaluate the impact of SA on a range of child health and nutrition outcomes. Nutrition outcomes included measures of stunting and underweight prevalence. Health outcomes included measures of child mortality, diarrhea incidence, and duration of illness. The evaluation also looks at a series of intermediate outcomes related to health and nutrition service utilization including, immunization rates, institutional delivery, ante- 3 natal checkups, receipt of nutrition supplements, satisfaction with frontline health workers, and frequency of VHND. Overall, both intervention arms had positive effects that were significant with the information plus facilitation intervention arm demonstrating consistently larger effects than the information only intervention arm. The effects were seen for stunting and underweight prevalence, treatment patterns for childhood diarrhea, immunization rates, institutional delivery but not for childhood mortality or antenatal care utilization. • Both intervention arms had impacts on important nutrition outcomes. The information plus facilitation arm reduced stunting rates among children <2 years of age by almost 11 percent (5.6 percentage point reduction, relative to 51.3 percent in the control arm). In the 2-5 year age group, the information only interventions reduced both stunting and underweight rates, by almost 11 and 13 percentage points, respectively. The information plus facilitation intervention arm had similar sized effects in reducing underweight among 2-5 year old children. • The significant improvements in nutritional status were accompanied by striking improvements in treatment patterns for childhood diarrhea (although there was no effect on prevalence). Among children aged <2 years, the information plus facilitation intervention increased likelihood that child was treated within the first day by almost 14 percentage points (an increase of more than half relative to 20.8 percent in control). The evaluation did not detect any effects on mortality rates in the study sample. • The SA interventions also led to dramatic improvements in vaccination rates. Among children aged 12-24 months, the share with full immunization in control is comparable to NFHS averages. Full immunization rates increased by 7.2 percentage points in the Info only arm, and by 11.8 percentage points in the information plus facilitation arm. Relative to 44.5 percent in the control arm, these are approximately 16 and 27 percent increases. • The SA interventions also improved indicators of maternal health care. Institutional delivery rates increased by over 5.6 percentage points in both arms. Improvements in rates of women receiving three or more ANC were relatively small, and not statistically significant. • Both interventions led to significant increases in awareness of entitlements in intervention villages. Households in information only arm were 9.3 percentage points more likely to be aware of their entitlements (6.7 percentage points in information plus facilitation arm). There were no effects on the likelihood of actual receipt of nutritional supplementation though, suggesting little to no improvement in supply of nutritional services. • Analysis of satisfaction with services from providers reveals that the intervention areas saw significant increases in satisfaction with ASHA workers’ performance, but not for other providers (ANM or AWW). The 6.4 percentage point increase in information plus facilitation areas represents a 9 percent increase, relative to 70.1 percent in control villages. 4 • Finally, the evaluation finds that the information plus facilitation interventions led to a 1.2 percentage point increase in share of households that reported that VHNDs had been organized every month for the past six months. Taken together, the SA interventions in UP led to systematic improvements in health delivery and utilization for maternal and child health, improving key health and nutrition outcomes (See Summary Table). The large improvements in immunization rates suggest that further improvements might be expected over time. There have also been significant increases in frontline health worker satisfaction (for ASHAs), community health service delivery organization (VHNDs) as well as increased awareness of the community on their entitlements. This evaluation’s findings strongly suggest that information and facilitation jointly have larger effects than provision if information alone, leading the IE team to strongly recommend that UPHSSP and SIRD continue its policy of investing in community facilitation mechanisms as part of larger SA efforts. The government-run program currently being implemented in the other 10 districts in UPHSSP could be improved by incorporating design features from this 2-district evaluation, especially in terms of information delivery and facilitation of VHSNC meetings. A key feature of the evaluated intervention was its government-led design and implementation, enabling replication at scale by government. Hence, the evaluation’s findings are also relevant for scale up across India, and other countries with a similar context within the South Asia Region, and beyond. The ongoing implementation of SA interventions by UPHSSP in 10 districts, which also follows a block-wise randomized design, is an incredible opportunity to study and document the impacts of SA, when implemented at large-scale. The IE team strongly recommends that the state government continue to improve and implement the SA interventions robustly, and invest in efforts to monitor and evaluate the impacts of this large-scale implementation of SA. 5 Effect of Information plus Facilitation on Key Outcomes Summary Table 5.6pp* (Con.: 51.3%) Nutrition 0.6pp (Con.: 45.7%) Reduction in 2.4pp (Con.: 55.7%) 6.5pp** (Con.: 51.9%) -0.4pp (Con.: 15.0%) Reduction in Diarrhea 0.03 days (Con.: 4.9 Days) 13.8pp* (Con.: 20.8%) Increase in 4.1pp ( Cont: 29.2%) Mortality Infant Reduction in -0.2pp (Cont:: 0.02 ) 11.8pp*** (Con.: 48.3%) Vaccinations 10.4pp*** (Con.: 77.4%) Increase in 11.0pp*** (Con.: 53.9%) -4.0pp (Con.: 96.9%) 5.6pp*** (Con.: 81.9%) Maternal Health Increase in 3.2pp (Con.: 29.6%) 0.07 visits Supplements Nutritional 6.7pp** (Con.: 59.7%) Increase in -3.4pp (Con.: 33.6%) Satisfaction 6.4pp*** (Con.: 62.4%) w/ AAA Increase in 2.6pp (Con.: 68.0%) -0.7pp (Con.: 62.3%) VHND Increase in 1.2pp (Con.: 15.5%) -30 -20 -10 0 10 20 30 PERCENTAGE POINTS Notes: Statistically significant difference relative to the control group is noted at the 1% (***), 5%(**), or 10% (*) level. Confidence intervals of impact estimates are designated at the 95 percent level. Numbers in parentheses () denote mean value of control arm. of ANC (visits). Death to births Effect is shown as percentage point changes (pp), except duration of illness (days) and No. 6 ratio control mean is the ratio of household deaths to household births. Table of Contents ABBREVIATIONS ................................................................................................................................................. 10 1. INTRODUCTION ............................................................................................................................................. 11 1.1 Country and Sectoral Context .......................................................................................................... 11 1.2 Importance of Social Accountability in India’s Health System ................................................ 12 1.3 Social Accountability in Uttar Pradesh ............................................................................................ 13 2. Evaluation Design and Intervention Overview ................................................................................... 15 2.1 Evaluation Design .................................................................................................................................. 15 2.2 Study Sites and Randomization ......................................................................................................... 16 2.3 Evaluation Motivation and Policy Significance ............................................................................ 16 2.4 Intervention Description ...................................................................................................................... 19 2.5 Theory of Change ................................................................................................................................. 20 2.6 Outcomes of Interest ............................................................................................................................ 22 3. Data Collection Overview ........................................................................................................................ 23 3.1 Target Population & Sampling ........................................................................................................... 23 3.2 Ethical Review and Research Approvals ....................................................................................... 24 3.3 Baseline Data and Balance across Intervention Arms ............................................................... 24 4. Evaluation Findings...................................................................................................................................... 26 4.1 Nutritional Status of Children under Five Years of Age ............................................................... 27 4.2 Childhood Diarrhea and Intervention ............................................................................................. 29 4.3 Mortality ................................................................................................................................................... 30 4.4 Immunization .......................................................................................................................................... 32 4.5 Maternal Care ........................................................................................................................................ 34 4.6 Nutritional Supplementation – Awareness and Utilization ......................................................... 35 4.7 Frontline Health Worker Satisfaction ................................................................................................ 36 4.8 Village Health and Nutrition Days..................................................................................................... 39 4.9 Embedded studies on information provision and role of social networks ............................ 39 4.9.1 Information Seeding Experiment ............................................................................................... 40 4.9.2 The Network Study ......................................................................................................................... 42 5. Conclusion and Policy Recommendations ......................................................................................... 44 5.1 Recommendations ............................................................................................................................... 46 Appendices .......................................................................................................................................................... i 7 Appendix 1 – VHSNC Description ........................................................................................................... i Appendix 2 – Intervention Description ................................................................................................ iii Appendix 3 - Modified Theory of Change for SA ............................................................................vii Appendix 4 – Implementation Overview .......................................................................................... viii Appendix 5 – Full Regression Tables on Outcomes ......................................................................... xv Appendix 6 - Power Calculation Estimates Prior to Intervention ............................................... xxv References ...................................................................................................................................................... xxvii Figures Figure 1: Map of social accountability intervention districts in Uttar Pradesh ............................ 18 Figure 2: Theory of change for studied social accountability interventions ............................... 21 Figure 3: Nutrition outcomes, by district and intervention arm ..................................................... 28 Figure 4: Treatment of diarrhea in a day, by gender ..................................................................... 30 Figure 5: Full immunization rates, by district ..................................................................................... 33 Figure 6: Full immunization rates, by gender and intervention arm ............................................. 33 Figure 7: ANC and institutional delivery rates, by intervention arm ............................................. 34 Figure 8: ANC and Institutional Delivery, by gender of the child ................................................. 35 Figure 9: Nutritional supplementation and utilization, by gender ................................................ 36 Figure 10: Satisfaction AAA, by intervention arm ........................................................................... 37 Figure 11: Satisfaction with AAA, by district ..................................................................................... 38 Figure 12: Satisfaction with AAA, by gender ................................................................................... 38 Figure 13: VHND frequency in last six months, by intervention arm ............................................. 39 Figure 14: Sample blobbogram depicting social network data from a village ......................... 43 Figure 15: Participation rates of VHSNC members in activated VHSNCs (Jul ’16 – Feb ’17) ...... iii Figure 16: Modified theory of change for social accountability interventions............................vii Figure 17: Social Accountability Intervention Timeline (2015-2018) .............................................. ix Figure 18: Percentage of GP’s conducting VHSNC meetings (Jun ‘2016 - Mar ’17) ................. x Figure 19: Average number of VHSNC attendees in activated VHSNCs (Sep ’16 – Dec ’17) . xii Figure 20: Participation rates of VHSNC members in activated VHSNCs (Jul ’16 – Feb ’17) .... xiii 8 Tables Table 1: Primary Outcomes ................................................................................................................ 22 Table 2: Intermediate Outcomes ...................................................................................................... 23 Table 3: Baseline summary statistics.................................................................................................. 24 Table 4: Intermediate Outcomes ...................................................................................................... 26 Table 5: Rates of stunting and underweight, by arm and age .................................................... 27 Table 6: Effect of SA interventions on nutrition, by age ................................................................. 28 Table 7: Rates of stunting and underweight, by gender ............................................................... 29 Table 8: Childhood diarrhea and treatment rates, by treatment arms ...................................... 29 Table 9: Effect of interventions on diarrhea treatment within 1 day ........................................... 30 Table 10: Deaths as a proportion of births in the last 5 years, by intervention arms .................. 31 Table 11: Full immunization rates, by arm ........................................................................................ 32 Table 12: Effect of interventions on immunization rates ................................................................ 32 Table 13: Effect of interventions on institutional delivery ............................................................... 34 Table 14: Effect of interventions on ANC visits................................................................................. 35 Table 15: Effect of interventions on nutritional supplementation awareness and utilization ... 36 Table 16: Effect of interventions on AAA satisfaction..................................................................... 37 Table 17: Results from month-wise VHSNC attendance ................................................................ 41 Table 18: Effect of network interventions on VHSNC attendance ............................................... 44 Table A1: Effect of Interventions on Rates of Stunting and Underweight .................................... xv Table A2: Effect of Interventions on Diarrhea Prevalence ............................................................ xvii Table A3: Effect of Interventions on Diarrhea Duration ............................................................... xviii Table A4: Effect of Interventions on Diarrhea Treatment (Treatment sought within 1 day) ...... xix Table A5: Effect of Interventions on Immunization Rates for Children Aged 12-23 months .......xx Table A6: Effect of Interventions on Maternal Care ........................................................................ xxi Table A7: Effect of Interventions on Awareness, Receipt, and Difficulty Accessing Supplementation ................................................................................................................................. xxii Table A8: Effect of Interventions on Satisfaction with AAA (ASHA, ANM, and AWW) .............. xxiii Table A9: Estimates for minimum detectable effect size ............................................................xxv 9 ABBREVIATIONS AAA (Auxiliary Nurse Midwife, Accredited Social Health Activist, & Anganwadi Worker) ANM (Auxiliary Nurse Midwife) ASHA (Accredited Social Health Activist) ATR (Action Taken Reports) AWW (Anganwadi Worker) BCG (Bacillus Calmette–Guérin vaccine) DDSIRD (Deendayal Upadhyaya State Institute of Rural Development) DPT (Diphtheria, Pertussis, and Tetanus vaccine) GoUP (Government of Uttar Pradesh) GP (Gram Panchayat) GPC (Gram Panchayat Coordinator) ICC (Intra-cluster Correlation) ICDS (Integrated Child Development Services) IE (Impact Evaluation) IMR (Infant Mortality Rate) MDG (Millennium Development Goal) NFHS (National Family Health Survey) NHM (National Health Mission) PMU (Project Management Unit) PP (percentage point) PRI (Panchayati Raj Institutions) SA (Social Accountability) SIEF (Strategic Impact Evaluation Fund) U5 (Under-5) U5MR (Under-5 Mortality Rate) UNICEF (United Nation’s Children Fund) UP (Uttar Pradesh) UPHSDP (Uttar Pradesh Health Systems Development Project) UPHSSP (Uttar Pradesh Health Systems Strengthening Project) VHND (Village Health and Nutrition Day) VHSNC (Village Health, Sanitation, and Nutrition Day) 10 1. INTRODUCTION 1.1 Country and Sectoral Context While India has made rapid economic progress in the past three decades, health and nutrition outcomes have not improved at the same pace. According to India’s National Family Health Survey, India’s infant mortality rate (IMR) has fallen from 78.5 deaths per 1,000 live births in 1991 to 41 deaths per 1,000 live births in 2016, a 48 percent decrease. Likewise, the under-five mortality rate (U5MR) has fallen from 109.3 deaths per 1,000 live births in 1991 to 50 deaths per live births in 2016, a 54 percent decrease[1,2]. While both indicators have improved, India’s Millennium Development Goal (MDG) target of reducing U5MR by two thirds between 1990 and 2015 was not achieved. Given that over half of under-5 child deaths occur due to easily preventable diseases like diarrhea and infectious disease, addressing these issue will be crucial to tackling childhood mortality [3]. India is also lagging in its performance on nutrition indicators, with the county being home to 40 percent of the world’s stunted children and nearly half of the world’s wasted children under the age of 5 [4]. According to NFHS-4, the percentage of children under 5 years qualifying as wasted has increased from 19.8 percent in 2006 to 21.5 percent in 2016 [1]. A closer look at nutrition and health indicators across the country reveal large amounts heterogeneity in India’s outcomes: states such as UP have indicators comparable to those in the world’s poorest countries, while other states like Kerala have indicators comparable with developed economies. This variation points to the role of geographic, social, cultural, and institutional factors in determining these outcomes. The states with better health outcomes are also the ones with well-functioning, responsive health systems and better health care provider performance (supply-side factors), as well higher levels of income and education (demand- side factors). Improving population health outcomes in the state of Uttar Pradesh, which has a population of over 200 million, is essential for achieving India’s global development goals. Despite declines in maternal and child mortality rates in India, Uttar Pradesh has continued to perform poorly. For instance, maternal mortality rate (MMR) in India declined from 437 deaths per 100,000 live births in 1990 to 190 by 2015. While India’s MMR fell short of its MDG target of 109, UP’s MMR in 2013 was a much higher 285 [5,6,7]. The infant mortality rate (IMR) in UP is also high – at 64 per 1,000 live births in 2016 – and well above India’s national average of 41 [1]. Uttar Pradesh lags behind in these health maternal and child outcomes despite the fact that most of these deaths are preventable. The National Health Mission (NHM) of the Government of India recommends a list of interventions to improve maternal and newborn health outcomes, including delivery of essential obstetric care, improving quality of antenatal care, and providing skilled birth attendants and postnatal care for mothers and newborns [8]. While these are proven interventions to improve maternal and child health, ensuring effectiveness at scale has been challenging due to weak health systems that fail to deliver services in states like UP [9]. Additionally, limited success of departments that focus on nutrition, like the Integrated Child Development Services (ICDS) compound the effects of weak health systems. ICDS’ effectiveness has been more limited in poor states with high under-nutrition such as UP. 11 Weaknesses in the implementation of these services, including corruption in the distribution of supplements, have been attributed as reasons for this failure [10]. 1.2 Importance of Social Accountability in India’s Health System Low provider effort and quality of care limit the success of interventions in the health sector. Recent studies have documented widespread problems in the health system including absenteeism in the public sector, low levels of provider knowledge, and even lower quality of care delivered to patients due to gaps between what providers know and what they do in practice [11,12,13,14,15]. Factors such as failure to align incentives of providers and patients, lack of institutional monitoring, weak governance, and lack of accountability have contributed to poor implementation of programs and limited effectiveness of many interventions in the health sector in India (and in developing countries more generally) [16,17,18,19]. Interventions that can enhance the accountability of the health systems and of local elected officials have the potential to make health systems more responsive and deliver health care that improves outcomes at the population level. There is growing recognition among governments and donors that citizens in developing countries can be instrumental in enhancing the accountability of public officials, reducing leakage of funds, and ultimately improving health service delivery [20,21]. As a result, social accountability (SA) has become an attractive approach for improving service delivery and health outcomes. SA interventions typically seek to enable community members to hold service providers accountable at the local level. Key components of SA interventions include: (a) information provision; and, (b) facilitating engagement through community meetings where grievances with the health care system might be redressed publicly. Evidence in SA from previous trials in Uganda, Nepal and India have shown dramatic improvements in service delivery, with over 30 percent reduction in under-five and neonatal mortality and large impacts on stunting rates [22,23,24]. In both India and Uganda, intervention implementation was overseen by research teams that directly facilitated community meetings and ensured their frequent occurrence. However, SA at scale, and implemented by the Government itself, is yet to be evaluated. The SA component of the World Bank supported Uttar Pradesh Health Systems Strengthening Project (UPHSSP) aims to implement SA interventions at scale, by the Government itself, and also facilitate rigorous evaluation of the program to fill a critical evidence gap. The SA interventions in UP build on previous efforts by the Indian government to facilitate community monitoring of providers and local elected officials in rural areas. In 2005, NHM guidelines defined the formation of the Village Health, Sanitation, and Nutrition Committee (VHSNC) as a local governance platform [25]. The VHSNCs aimed to help local community members to monitor delivery of local health services and provide feedback to health workers. The SA interventions in UPHSSP focused on activating the VHSNCs, creating community awareness about the role of VHSNC, and providing trained facilitators to conduct monthly meetings to improve governance and accountability of local level health care delivery. The overall SA intervention in UP is being implemented and evaluated at two levels: the state level implementation that will be evaluated in 2019, and the two district SIEF-funded evaluation of SA interventions in two districts. The key objectives of the SIEF-funded evaluation in two districts are: 12 1. To evaluate the impact of the state-run SA interventions in a randomized field trial across 120 villages 2. To test the underlying mechanisms through which SA interventions might lead to changes in population health outcomes. Specifically, the SIEF-funded evaluation aims to estimate (A) the effect of only providing information on accountability and health; and (B) the effect of providing information in addition to facilitation support to conduct the VHSNC meetings, relative to (C) a control arm where neither interventions were introduced. 3. To study the role of information and social networks on participation for collective action The following sections of this report present draft findings from the SIEF-funded evaluation of SA interventions in two districts in UP. Section 2 describes the overall evaluation design and provides an overview of the intervention along with details on the VHSNC. Section 3 describes data collection methodology and specific outcomes of interest. Section 4 describes the findings of the evaluation. Section 5 concludes with overall observations and recommendations regarding the SA intervention in UP. 1.3 Social Accountability in Uttar Pradesh Mirroring NHM’s effort to increase local level community engagement in health, several previous efforts in UP have contained some elements of SA. However, there is very little rigorous evidence regarding the impact of these programs. An Integrated District Planning project led by the United Nation’s Children Fund (UNICEF) in Lalitpur sought to improve health and development indicators through behavior change communication and the mobilization of panchayat level committees. Panchayat level committees included the village Pradhan, all village level functionaries, and local village volunteers. Village volunteers acted as important members of the committee that helped schedule meetings, advertise meetings to the public, and voice village needs/priorities. Typical themes brought up in meetings in Lalitpur focused on the performance and coordination of ASHA workers and Anganwadi workers [26]. The project found association between the integrated district planning and improvements in prevalence of hand washing, breastfeeding, and ANM visits [27,28]. However, these findings were not rigorously tested. The previous Uttar Pradesh Health Systems Development Project (UPHSDP) also attempted to implement community-based accountability interventions to address maternal deaths. The proposed system recommended conducting “social reviews� of maternal deaths to improve outcomes and health system performance. The program included using Citizens’ Charters at health facilities and grievance redress mechanisms to improve the quality of services [29]. However, this recommended system has proven difficult to implement and was not yet functioning effectively [30]. The Uttar Pradesh Health Systems Strengthening Project (UPHSSP) was conceived as a strategic initiative to enhance people’s access to quality health care services in the state of Uttar Pradesh. To this end, this World Bank funded project has aimed to improve the Department of Health’s capacity by: (i) strengthening the Department’s management and accountability systems; and (ii) improving the Department’s ability to perform quality assurance and engage the private sector. UPHSSP planned to achieve the first goal by: a) supporting strategic 13 planning functions; b) improving the use of data for program management; c) strengthening accountability in management systems – including financial, procurement, and supply chain management –; and, d) introducing and strengthening SA and provider incentives [26]. In line with introducing and strengthening SA mechanisms, UPHSSP has sought to stimulate community action improving the use of information gathered by community assessments of health systems and health care. In the context of local health systems, community assessments focus on areas critical to improving health outcomes, including the performance of frontline healthcare workers and management of community resources [23]. Among Frontline healthcare workers, the relevant local actors are the following functionaries: • Accredited Social Health Activist (ASHA): a female resident of the village who is recruited and trained to deliver first-contact health care, health information, and help community members access health services. ASHAs are tasked with creating awareness about health and disseminating information about publicly provided health services. • Auxiliary Nurse Midwife (ANM): the main provider of primary care in the community, including maternal and child health services, family planning, immunizations, and treatments of minor conditions such as childhood diarrhea and respiratory illnesses. The ANM also trains the ASHA and guides all activities performed by the ASHA and oversees the arrangement of prescribed monthly Village Health and Nutrition Days (VHND) during which immunizations and other services are offered. • Anganwadi worker (AWW): a community-level health worker who manages the village- level ICDS centers. Anganwadi workers are tasked with dispensing supplementary nutrition packages, weigh each child to monitor growth, track malnourished children, and provide health and nutrition education. Given that frontline healthcare workers report to different line ministries (ASHAs and ANMs to the Department of Health and Family Welfare, and AWWs to the Department of Women and Child Development), SA interventions provide a common platform to monitor all local public healthcare providers, regardless of department.1 Using information available as part of SA interventions, community members may be able to demand better services and take necessary actions to sanction non-performing and underperforming staff. Particular areas of community monitoring could include health worker attendance, completion of assigned activities (e.g. distribution of nutrition supplementation, conducting immunization camps, enrolling women and children in government programs, etc.), and/or ensuring beneficiaries do not pay out of pocket expenses. In addition to leveraging community participation to hold healthcare workers accountable, the SA intervention in UPHSSP could enable local management of community resources. Local management could enable communities to better target pertinent local issues in health, sanitation, and nutrition. In particular, community participation could improve the usage of the Village Untied Fund. This Village Untied Fund is an amount of INR 10,000 allotted to every village to spend on health related inputs, like vector control, information campaigns, or other similar activities. 1This report refers to the Auxiliary Nurse Midwife, Accredited Social Health Activist, and Anganwadi Worker collectively as the AAA. 14 With these key areas of focus for improving healthcare performance, UPHSSP designed pilot SA interventions to improve community management of health inputs. Box 1. VHSNC: A Social Accountability Platform for India A key innovation in the UPHSSP SA intervention was to leverage an already existing public health body, the Village Health, Sanitation, and Nutrition Committee (VHSNC). As per its design by the National Health Mission, the VHSNC offers an open platform where citizens can publicly raise concerns regarding local problems relating to health, sanitation, and nutrition. VHSNC meetings are intended to be open to the public, providing an inclusive forum for villagers from all backgrounds to voice complaints and concerns. Meetings should typically happen in regular monthly intervals in centrally located venues [31]. As the village’s principal body for resolving village-specific public health problems, the VHSNC aims to bring together key stakeholders to implement relevant solutions. This includes members from local elected bodies, healthcare providers, and patients/local villagers [31]. By including key elected officials together with healthcare providers, decision makers can directly and transparently address concerns raised by citizens. This is especially relevant in cases related to healthcare provider performance, absenteeism, and provision of free services. Some examples of potential actions that VHSNCs can take to improve local health and nutrition include, but are not limited to, conducting awareness campaigns, supervising the Village Health and Nutrition Day (VHND), utilizing the village untied fund, providing feedback to healthcare providers. Most importantly, VHSNCs can generate Action Taken Reports (ATRs), which can request support of officials at the block level to take more decisive action regard a GP-level issue. For a more in-depth description of the VHSNC (Appendix 1). 2. Evaluation Design and Intervention Overview 2.1 Evaluation Design The impact evaluation of SA interventions in Fatehpur and Sultanpur was designed as a cluster randomized field experiment. A cluster was defined as a GP comprising approximately 1,000 households. A concern in randomized evaluation of policy interventions implemented by government is the possibility of design contamination if the implementing agency deviates from planned implementation due to political reasons after randomization. One solution to deal with this specific issue is to adopt a matched cluster-randomized design [32,33].2 To study the effectiveness of SA interventions with two intervention arms (information only versus information plus facilitation) and a control arm, the IE team implemented a matched trio randomization of clusters, which is an extension of pairwise matched randomization. Although there were no instances of political intervention to modify randomized assignment in the evaluation, the IE team’s analysis accounts for the matched clusters. 2 One advantage of such cluster-matched designs is that they provide additional empirical approaches to estimate the key treatment effect of interest in the event that politically motivated interference in implementation of the randomized program should occur. 15 With this randomized design, the IE team is able to estimate the causal impact of being in an intervention arm (information only or information plus facilitation by GPCs) relative to being in a control arm where no interventions were introduced. The randomized design provides the key identification assumption that whether or not the selected interventions were implemented in a village due to randomized assignment was independent of community characteristics, which might be related to nutrition and health outcomes and health service delivery. The IE team has undertaken an intent-to-treat analysis in order to ensure that the evaluation’s results are not biased as a result of possible endogenous variation.3Although assignment into intervention arm 1 or 2 was random, there was likely to be variation in how intensively the intervention was implemented in specific villages. This variation, however, is likely to be endogenous and be related to underlying community characteristics that influence how extensively an intervention is implemented in a community. 2.2 Study Sites and Randomization The impact evaluation focused on 120 villages across two districts, Fatehpur and Sultanpur. These two districts were part of 12 districts that were identified by Government of Uttar Pradesh for SA interventions as part of the UPHSSP intervention (Figure 1). Out of 1,642 GPs in these two districts, the IE team randomly selected 120 GPs based on stratified sampling and probability proportional to population. Within each GP (which is comprised of 2-3 villages on average in UP), the IE team then randomly selected one village to focus on for purposes of this evaluation. The research team then generated matched trios of the selected GPs based on selected baseline characteristics. Specifically, trios of villages were matched based on key village-level indicators such as child nutrition status, awareness of roles and responsibilities of local health providers and health services, and household characteristics. These indicators were collected at baseline. Within each trio, villages were then assigned to one of the three intervention arms. 2.3 Evaluation Motivation and Policy Significance The design and context of this evaluation provide the IE team with a significant opportunity to advance important discussions in both policy and research. The evaluation was designed to evaluate the impact of two key components of SA interventions - information provision and community facilitation. In terms of policy significance, this evaluation provides insight into the ability of the Government of Uttar Pradesh to effectively implement SA interventions in public health. As previous studies have focused on SA interventions implemented by the non-profit sector, this study provides an idea of SA’s scalability when by the public sector. Unlike project implemented by NGOs or in small research settings, evaluation projects that are implemented in partnership with the government encounter real life implementation challenges including logistics, human resources, project coordination, and as well as policy priorities. Indeed, the IE team went to great lengths to understand and document these challenges to provide more 3 Rather than analyzing impact on only the villages where take up of the information plus facilitation intervention worked well (i.e., GPCs successfully activated VHSNCs on a monthly basis), the IE team’s Intent to Treat (ITT) analysis looks at all villages regardless of apparent take up. An analysis based on Treatment of Treated (TOT) would potentially introduce endogenous factors that will bias estimates of intervention impact. As such, the evaluation’s findings provide an estimate of average causal impact across villages that had varying levels of VHSNC activation. 16 context for the generalizability of this evaluation (Appendix 4). In addition to understanding the potential feasibility of SA within the public sector, this evaluation specifically looks at the ability of poorer states, like UP, to implement SA. In India, promising results from UP, could provide a general idea of whether other Empowered Action Group4 states could replicate similar SA interventions. In addition to the policy relevance, this evaluation tests whether information provision has an independent effect on outcomes compared to the combined effect of information provision and community engagement. This approach contrasts with previous effort in Uganda where researchers tested the effect of facilitation alone relative to information plus facilitation [34]. A key design feature in this evaluation of SA in the two districts is that it builds on the government’s main SA program through additional information interventions prior to introduction of the government appointed facilitators to help meetings for the villages’ health and sanitation committee. As described in the previous section, the IE team conducted a matched cluster randomization where 1/3 of the 120 villages in the evaluation were assigned to the information only intervention arm, 1/3 were assigned to the information plus facilitation intervention arm, and the remaining were assigned to the control arm, which had no interventions. A description of the two intervention arms follows. 4 Empowered Action Group (EAG) states refer to a series of eight states that perform poorly on a series of socioeconomic indicators. In particular, these states have the highest rates of infant mortality in India. The eight EAG states are Bihar, Chhattisgarh, Jharkhand, Madhya Pradesh, Orissa, Rajasthan, Uttaranchal and Uttar Pradesh. 17 Figure 1: Map of social accountability intervention districts in Uttar Pradesh The map to the left provides an overview of intervention districts with Social Accountability interventions implemented by UPHSSP, in partnership with SIRD. SIEF-Evaluation Districts (red): 1. Fatehpur (120 villages) 2. Sultanpur (120 villages) UPHSSP Evaluation Districts (yellow): 3. Ambedkar Nagar 4. Banda 5. Basti 6. Bhadodi 7. Chandauli 8. Hapur 9. JP Nagar 10. Kushinagar 11. Mau 12. Moradabad 2.4 Intervention Description The following section provides a brief overview of activities that took place in the SA intervention’s intervention arms in Sultanpur and Fatehpur. For more a more in-depth description of the intervention, see Appendix 2. Intervention Arm 1: Information Only • Information provision on village-level health indicators for U5 children relative to the district average. • Information provision on roles and responsibilities of the VHSNC given through visits to all households in each village. • Information dissemination regarding VHSNCs through phone calls and Interactive Voice Response (IVR) calls. Intervention Arm 2: Information plus Facilitation The villages in the information plus facilitation arm received ALL information interventions that were given to Intervention arm 1. In addition, these villages also had government- appointed facilitators to help activation and facilitation of monthly village health and sanitation committee meetings. The key components of this intervention arm include: • Information provision as implemented in Intervention Arm 1. • Active engagement from a Gram Panchayat Coordinator (GPC), hired and trained by UPHSSP and the Deen Dayal Upadhyaya State Institute of Rural Development (SIRD) activated VHSNCs and to assist in facilitating monthly meetings and other functions of the committee. o The GPCs in the 40 intervention villages also received additional handholding support and training to help improve fidelity of implementation in the 40 information plus facilitation villages. • Monthly VHSNC meetings, where village members were encouraged to attend and discuss issues in the village related to health, sanitation, and nutrition. • Household visits from the Morsel field team to remind households about upcoming VHSNC meetings as well as small incentives for some households to attend the meetings. This was conducted as part of the supplementary research on social networks as explained later in the study. The evaluation was designed to test the effect of information provision alone (Arm 1) relative to information provision combined with facilitation (Arm 2). The IE team chose to test this design rather than attempting to isolate the effect of facilitation alone (without information provision) because it was not possible to conduct facilitation of community meetings without providing any information. Especially in the context of community based social accountability interventions implemented through government institutions such as VHSNCs, it is critical to provide households as well as VHSNC members information in order to convene meetings. It is, however, feasible to provide information about rights, responsibilities and health outcomes in these settings making it possible to estimate the effect of an information only arm (Arm 1) relative to the control arm. Further, although the difference between Arm 1 and Arm 2 can 19 be interpreted as the marginal effect of facilitation, it is likely that the effect observed in this study is the result of information combined with facilitation, rather than that of facilitation alone. As a result, in this evaluation we only describe separately the estimated effects of Arm 1 and Arm 2. One challenge for implementing SA through VHSNCs was that most of these committees were inactive. Data from the baseline survey conducted in April 2015, among 4,856 households in the 12 implementing districts, suggest that VHSNCs were largely inactive or had not yet been implemented in these areas. Only 8 percent of households were aware of a VHSNC existing in their village (AMS Final Report). Hence, a critical first step in the Intervention Arm 2 was to activate the VHSNC in each of these villages. In order to carry out these interventions, UPHSSP collaborated with the Deen Dayal Upadhyaya State Institute of Rural Development (SIRD) to manage implementation of SA interventions in the selected villages.5 SIRD recruited a project management unit (PMU) based at their state headquarters to direct intervention implementation. UPHSSP and SIRD jointly developed protocols and procedures for GPC activities on the ground, such as VHSNC activation forms, Action Taken Report documents, and other monitoring formats for GPCs to fill each month. 2.5 Theory of Change The theory of change for how SA interventions could lead to improvements in health outcomes focuses on two aspects: (a) the effects on service delivery by village-level healthcare workers (AAA) who provide critical primary health care services at the village level, and (b) active participation and community members’ response to the health interventions. Figure 2 presents the underlying theory of change that was part of IE team’s original proposal to SIEF. It informs the evaluation’s choice of outcomes for measuring the impact and performance of the proposed interventions. Under the proposed model, the evaluation anticipates that providing village specific information about the performance of key health and nutrition outcomes, and subsequently informing community members about key health entitlements could improve knowledge of healthcare programs among community members. With improved understanding of what healthcare services should look like in villages, community members may be more willing to engage in monitoring of staff and resources related to healthcare services (e.g. the performance of the AAA, usage of the untied fund, etc.). In addition to information potentially causing increased monitoring from community members, community members could also have greater awareness about health services to which they are entitled and seek out care more often (demand-side response to increase service uptake). This would result in increased rates of antenatal checkups (ANC), institutional delivery, and immunization among children. Additionally, nutrition behavior may be positively influenced, including exclusive breastfeeding and utilization of nutritional supplementation. These outcomes would then have positive effects on child health (infant and under-5 mortality rates) and nutrition (lower rates of stunting and wasting). 5 For details about Deen Dayal Upadhyaya State Institute of Rural Development please visit: http://www.sirdup.in/ 20 Information provision can increase demand for healthcare services, but it is important to have corresponding supply-side improvements. Community facilitation focuses on helping to close a feedback loop and improve the responsiveness of the local health system. In the case of villages that have an active VHSNCs, this platform would make healthcare providers directly accountable to community members and empower the community to take direct action in improving the local healthcare system. As a case in point, the three key health workers at the village level (ASHA, ANM, and AWW) report to the village level elected representatives and block level authorities, who receive feedback from the community at the VHSNC. A failure to improve both could result in dismissal or transfer of staff if the VHSNC recommends such an action. This would lead to more responsiveness from providers to the needs of the community, and ultimately contributing to improved health outcomes both for child and mother. Score Cards (informa on Interven on 1 Interven on 2 about HS performance in village) Ci zen Charter Community (informa on Par cipa on / about rights and Engagement responsibili es) Demand for be er services (to Household GP reps or to Behavior HC provs) change Improved Service Delivery Outcomes (health) Increased u liza on of services Figure 2: Theory of change for studied social accountability interventions This theory of change assumes that the information interventions are able to appropriately target and reach individuals who have a lack of awareness regarding health programs and 21 duties among frontline healthcare workers. In this specific evaluation, the IE team is able to ensure this level of targeting and reach. Furthermore, it also assumes that the facilitated meetings are conducted in a regular manner and with active participation from community members and VHSNC members. In this evaluation, the handholding support provided to the GPCs appointed by the government aimed to ensure that these meetings were held. Finally, IE team also studies how participation in community meetings can be increased by information interventions and the role of social network connections. This evaluation describes two sets of embedded studies that the IE conducted to understand how information interventions in the context of social networks help improve participation for collective action (details in Section 4.9). Further details on the various channels through which SA interventions could lead to improvements in key health outcomes are described in the expanded theory of change in Appendix 3. With this described theory of change in mind, the IE team anticipates that improvements in delivery of services and increased utilization of public services will lead to improvements in key health and nutrition outcomes. The IE team also developed a modified theory of change that incorporates changes in the program design implemented by the government (Appendix 3). This modified theory of change enables the IE team to identify inputs, outputs, outcomes, and goals of the SA interventions. Based on this TOC, the evaluation focuses on measurable outcomes along the causal chain to estimate the impact of the information only intervention arm, relative to the impact of information plus facilitation intervention arm, on a series of health and nutrition outcomes described in the following section. 2.6 Outcomes of Interest This evaluation investigates the effect of the two described interventions on a set of i) primary outcomes and ii) intermediate level outcomes. In both of these cases, these outcomes are highly related to the previously described theory of change. Table 1: Primary Outcomes Outcome Indicators Frequency Measurement instruments (Impact outcomes) Z-scores for height and Household survey, Baseline and Endline weight anthropometrics Child mortality in last 5 Household Survey, and GP Baseline and Endline years (study sample) administrative records Incidence of Diarrhea Baseline and Endline Household Survey Duration of diarrhea illness Baseline and Endline Household Survey This evaluation investigated the program’s effect on key outcome indicators related to children’s health. Children’s heights and weights were measured as part of household surveys in order to calculate Z-scores to estimate stunting and underweight. All deaths of U5 children were captured from household surveys to create a mortality rate for households in our study sample. In addition to mortality, the IE team collected data on all diarrhea cases occurring 22 among children under 5 years of age, and the duration of the illness in days, within the two weeks prior to surveying. Table 2: Intermediate Outcomes Outcome Indicators Frequency Measurement instruments (Intermediate outcomes) Info / Awareness of VHND Monthly Cell Phone Cell Phone and VHND Participation in VHND Monthly monitoring Service Quality, reliability, Baseline and Endline Household Surveys perception, satisfaction Change in receipt and Baseline and Endline Household Surveys utilization of services As described in the theory of change, the IE team expected that changes in health seeking behavior and/or quality of services would subsequently lead to improvements in child health and nutrition outcomes. In order to capture changes on quality of services, the IE team assessed availability of services and perceptions of quality for the health practitioners in the village (AWW, ANM, and ASHA). The IE team measured patient perceptions of quality using Likert scale questions on satisfaction with services. The team also examined utilization of services by determining if parents sought treatment for fever and diarrhea (such as children receiving ORS and Zinc for diarrhea), and if parents utilized basic maternal and child health services, including immunization, antenatal care, postnatal care, and supplemental nutrition. With these focus areas in mind, the IE team conducted an endline household survey with the female head of household to capture both quality and utilization of services. 3. Data Collection Overview The randomized evaluation of SA interventions in the 120 villages relies on analysis of data collected from these villages at end line. The SA interventions were initiated in the third quarter of 2016 and GPC-led interventions were implemented by SIRD for a year. Since it was not feasible to implement household surveys at the end of 2017 due to elections, the IE team implemented endline data collection in spring 2018. 3.1 Target Population & Sampling The interventions covered and included all population subgroups, castes, and religious minorities in the intervention villages. Data was also collected from all of these subgroups. The target population for this evaluation was U5 children and their mothers or primary caregivers. All households in the 120 villages selected for the impact evaluation were listed again at the time of follow up. Household details, including whether and how many U5 children were residing in the household as well as other socio-economic characteristics, were captured during this listing. This data was then de-identified and sent to the IE team along with unique household identification codes. Restricting the listing data to eligible households (those containing at least one child under 5 who resides in the household), the IE team used the statistical software STATA to randomly select 40 households (if 40 eligible households existed) in each of the 120 villages. In total, 4,800 households were selected for surveys that assessed key 23 study outcomes. The sample size was selected based on the IE team’s power calculations estimated prior to the evaluation. A sample of this size provides over 90% power at 5% level of statistical significance to detect a change of a quarter of a standard deviation in Z-score measures of nutritional status, and 80% power to detect a 5 percentage point reduction in diarrhoea incidence in the past two weeks. Enumerators from TARU Leading Edge were deployed to these households to collect data on health-seeking behavior, self-reported health outcomes, satisfaction with AAA, and other household characteristics. Surveys were designed to capture household characteristics that the IE team hypothesized could influence or correlate with the evaluation’s outcomes of interest, including caste, religion, education, as well as agricultural and household assets. The survey instrument is included in Appendix B for reference. 3.2 Ethical Review and Research Approvals The evaluation questionnaires, protocols, and methods described in this report were reviewed and approved by the Institutional Review Board (IRB) based at Duke University [D0168]. The IE team also provided the Duke University IRB with regular updates and amendments to the original approved protocols. These updates and amendments were subsequently recorded and approved by the IRB. The IE team also registered their protocols at the NIH clinical trial registry and at the American Economic Association’s RCT registry.6 In addition, all investigators and key research personnel have completed ethical research training. This includes CITI courses in Human Subject Protection and Good Clinical Practices. 3.3 Baseline Data and Balance across Intervention Arms Prior to implementation of the SA interventions and the beginning of this evaluation, UPHSSP commissioned data collection at baseline. In addition to documenting baseline level of awareness of health service delivery at the primary care level in a random sample of households in the 12 districts as well as knowledge of rights and responsibilities, the baseline survey aimed to: a. Collect data on characteristics of villages randomized to the three arms to test balance. b. Generate summary statistics on key outcomes at baseline, in order to inform design of SA interventions. Importantly, in the SIEF districts, the evaluation design aimed to provide information on health outcomes (collected at baseline) to households in intervention villages as part of the information interventions. Table 3 shows summary characteristics of villages assigned to the three arms at baseline. Table 3: Baseline summary statistics Village Characteristic Info Only Info + Control p-value of Facilitation difference Household is SC/ST 0.451 0.449 0.433 0.657 6The study’s registration with the American Economic Association’s RCT registry can be found at the following address: https://www.socialscienceregistry.org/trials/1393 24 Household is Hindu 0.873 0.871 0.883 0.670 5 quantiles of wealth score 2.699 2.545 2.573 0.028 Child's gender 0.519 0.531 0.517 0.778 Child's age in years 2.132 2.131 2.151 0.941 Mother's age 27.494 27.695 27.797 0.378 % children who are stunted (low HfA) 0.469 0.483 0.486 0.743 % children who are underweight (low WfA) 0.324 0.347 0.333 0.578 p-value from joint test of orthogonality of intervention arms The test of joint orthogonality across the intervention arms demonstrates balance across the three randomized intervention arms. Furthermore, attrition is not a major concern in this study since there were no instances of villages deciding they did not want to be part of the intervention (the intervention was implemented by the state). 25 4. Evaluation Findings Table 4 provides summary statistics of household level characteristics in Fatehpur Table 4: Intermediate Outcomes and Sultanpur districts. The table reports household characteristics that are Mean relevant to the outcomes of interest for this Caste Category evaluation as well as potential factors in SC 0.361 mechanisms related to SA. These variables ST 0.016 are included as controls in all regression OBC 0.471 models described in the following General 0.151 analyses. Others 0.001 Religion Household Characteristics Hindu 0.895 The majority of the sample belong to Muslim 0.105 Other Backward Caste (OBC) category Other 0 and almost 90 percent of the sample is BPL Card Status Hindu. More than 60 percent of the APL 0.217 sample has BPL cards. Mother’s education BPL 0.602 – a key factor for child health outcomes – No card 0.182 has a wide distribution, with almost a third Mother's level of education of the sample having no formal education Illiterate and/or no formal education 0.329 and almost a quarter that has above Completed primary (5th standard) 0.139 secondary education. The IE team Completed secondary (10th includes all of these key variables as standard) 0.286 controls in regressions even if they are Beyond secondary 0.245 balanced across intervention arms (as Number of U5 kids 1.261 demonstrated in previous section). Observations 4216 Organization of Findings For each of the subsequent focus areas in this evaluation, the IE team presents mean values of the outcomes of interests across the three evaluation arms. The IE team then presents the estimated causal impact from regression analysis. While this section includes abbreviated regression tables, Appendix 4 provides regression tables with control values. The team also comments on any interesting trends observed based on gender. In two cases, nutrition outcomes and immunization rates, the IE team also presents findings broken down by age to focus on relevant age groups beyond solely U5 children. 26 4.1 Nutritional Status of Children under Five Years of Age One of the key outcomes that the SA intervention aimed to address was child malnutrition, namely rates of stunting and underweight7 These findings are presented for aged <2 years and those aged 2-5 years. This evaluation does so for two reasons: (a) the SA interventions were introduced two years ago in May 2016, so children born after the interventions began received maximum exposure to the interventions; (b) Effects of early childhood exposure to inadequate nutrition, illnesses, and other growth disturbances are harder to recover from. Hence, the evaluation expects smaller effects among older children. Overall, the IE team found lower prevalence of stunting and underweight due to information only and information plus facilitation (Table 5). In both cases, stunting rates were approximately 4 percentage points lower relative to 54.4 percent in control arm villages. Underweight rates were also approximately 5 percentage points lower in intervention areas. Table 5: Rates of stunting and underweight, by arm and age Info + Info Only Control Total Facilitation Stunting Rates All 50.1% 50.3% 54.4% 51.4% Age <2yr 50.1% 45.8% 50.9% 48.9% Age 2-5 50.1% 53.2% 57.1% 53.1% Underweight Rates All 44.9% 44.6% 49.7% 46.1% Age <2yr 45.2% 44.4% 46.0% 45.2% Age 2-5 44.6% 44.8% 52.4% 46.8% The IE team investigated the observed differences in means for malnutrition in regressions analyses. The analysis used probit regression models that controlled for a vector of child and household control variables (age, gender, mother’s education, caste, religion). Analysis also accounted for the matched trio study design by including trio level fixed effects and district fixed effects. Regression clustered standard errors at the level of the village, corresponding to the unit of randomization in the evaluation. The adjoining tables only show the main results for intervention arms (marginal effects and standard errors); for full set of results, see Appendix 5. Consistent with unadjusted mean differences in outcomes between the intervention and control groups, adjusted regression analyses revealed approximately 4 percentage point reductions in both outcomes over all age groups (Table 6). Furthermore, information plus facilitation resulted in significant reduction in stunting prevalence (5.6 percentage point reduction, p<0.05) – an 11 percent reduction relative to the 51 percent stunting prevalence in control arm. In the older age groups, the information only interventions reduced both stunting 7Stunting and underweight are based on WHO definitions as height-for-age (HfA) and weight-for-age (WfA) z-scores that are 2 standard deviations below the WHO growth standards median. 27 and underweight by almost 11 and 13 percent, respectively. The information plus facilitation intervention had similar sized effects in reducing underweight among 2-5 year old children. Table 6: Effect of SA interventions on nutrition, by age Full Sample Under 2 years 2-5 years Stunting Rates b/se b/se b/se Info Only -0.039 -0.009 -0.064** 0.021 0.028 0.023 Info + Facilitation -0.038 -0.056* -0.024 0.020 0.028 0.023 Control Arm Mean 0.544 0.509 0.571 Underweight Rates Info Only -0.034 0.013 -0.067** 0.019 0.025 0.025 Info + Facilitation -0.039* -0.006 -0.065** 0.018 0.025 0.023 Control Arm Mean 0.497 0.460 0.524 Observations 4874 2000 2874 Note: Marginal effects reported from probit; SEs clustered at village level; matched trio and district FEs The IE team found some differences in nutrition outcomes based on geography. Malnutrition in study villages across the two districts were slightly higher than state averages reported in NFHS- 4 (48.5 percent stunted and 41 percent underweight in rural UP) [35], control villages in two districts were similar in terms of stunting, with slightly higher underweight rates in Fatehpur (Figure 3). Figure 3: Nutrition outcomes, by district and treatment arm Stunting Rates, by District and Arm Underweight Rates, by District and Arm 60.0% 60.0% 50.0% 50.0% 40.0% 40.0% 30.0% 30.0% 20.0% 20.0% 10.0% 10.0% 0.0% 0.0% Infor mation Only Info + Facilit ation Control Infor mation Only Info + Facilit ation Control Sultanpur Fatehpur Sultanpur Fatehpur 28 The IE team also found that stunting rates varied by gender. These findings are in line with other studies that look at gender differences in child health and nutrition. The overall lower levels of stunting and underweight among girls has been documented in other research that suggests that the difference could be attributed to lower survival rates among girls, so the marginal surviving infant is healthier [36]. Additionally, when compared against the control mean, the effect of the interventions appear larger among boys than among girls. Table 7: Rates of stunting and underweight, by gender Info + Info Only Control Total Facilitation Stunting Rates Girls 49.4% 50.7% 52.3% 50.7% Boys 50.8% 49.9% 56.2% 52.0% Underweight Rates Girls 44.4% 42.7% 45.5% 44.1% Boys 45.3% 46.3% 53.2% 47.9% 4.2 Childhood Diarrhea and Treatment Childhood diarrhea is one of the most common causes of disease burden and child mortality globally, and in rural Uttar Pradesh. 15 percent of children under 5 years of age had suffered from diarrhea in the past two weeks, both in this evaluation’s sample and in NFHS-4. Since the SA intervention was focused on primary care, the main anticipated effects are better treatment outcomes. However, the information provided to households also included summaries and relative comparisons on childhood diarrhea rates. There are large differences in rapid treatment seeking across arms: the share of children with diarrhea in the past two weeks who received treatment within one day after symptoms began. In the information plus facilitation arm, the increase was over 11.1 percentage points (an increase of almost half, albeit from a low base of 24.4 percent in control). Table 8: Childhood diarrhea and treatment rates, by treatment arms Info Only Info + Facilitation Control Prevalence of Diarrhea in last two 14.7% 15.4% 15.0% weeks % treated in 1 day 22.4% 35.5% 24.4% % treated in 2 days 48.4% 58.9% 50.7% Duration of Diarrhea 5.2 4.8 4.9 The results from regression analyses confirm these differences in means. The direction of effects is consistent across all age groups, including a significant and large improvement of 13.8% in rapid treatment of children in the information plus facilitation arm increased by 13.8% (p<0.10). However, the evaluation’s smaller sample among the 15 percent children who had diarrhea results in lack of precision for other estimates. 29 Table 9: Effect of interventions on diarrhea treatment within 1 day Full Sample Under 2 years 2-5 years b/se b/se b/se Info Only -0.023 0.043 -0.065 0.040 0.054 0.065 Info + Facilitation 0.075 0.138* 0.041 0.042 0.063 0.050 Control Arm Mean 0.244 0.208 0.292 Observations 661 337 285 Note: margins reported from probit; SEs clustered at village level; matched trio and district FEs It is possible that the improvements in diarrhea treatment rates could contribute to improvements in stunting and underweight, however this evaluation does not allow the IE team to test this causal mechanism. This is due to the possibility of several other driving factors of nutrition outcomes – including provision of other health and nutrition services - potentially having been affected by the SA interventions. Finally, there are substantial gender differences in time to treatment for diarrhea (Figure 4). Treatment of diarrhea was faster for boys (22.6 percent), compared to girls (21.6 percent) in the control arm. A similar pattern was present in the information only arm, which had no significant changes. The information plus facilitation arm had large increases: 33.6 percent of boys and 37.6 percent of girls with diarrhea received treatment in the first day. Figure 4: Treatment of diarrhea in a day, by gender Sought treatment or treated at home within one day 40% 35% 30% 25% 20% 15% 10% 5% 0% Boys Girls Boys Girls Boys Girls Info Only Info + Facilitation Control 4.3 Mortality Previous efforts at introducing community monitoring and SA have led to large reductions in under five mortality rates. The IE team investigates the impact of SA interventions in UP on mortality. However, it is important to note that due to sampling differences, the evaluation’s 30 estimates of program impact are not directly comparable to effects seen in Uganda or Nepal. In order to focus on child health outcomes (nutrition in particular), the household survey focused on households that had children under 5 years of age and collected data on mortality of all children previously born in those households. As a result, the evaluation’s sample does not include data from households where there were no surviving children – this leads to an underestimate of mortality rates in study villages. The omission of households with no surviving children means that intervention impacts on child mortality rates are likely to be biased downward.8 Summary statistics for child deaths in last five years in the evaluation’s sample of households that have ≥1 surviving child under 5 years of age show no difference in child mortality rates between arms (Table 10). The mortality per thousand live births in the three arms are 42, 41 and 41 (information only, information plus facilitation, and Control). However, at the time of writing this report, the evaluation’s mortality data is still incomplete – the survey agency was unable to reach approximately 10 percent of households in the sample when they conducted revisits to collect information on mortality. In the event that this last 10 percent of data will change estimates, the IE team will update its analysis. Table 10: Deaths as a proportion of births in the last 5 years, by intervention arms Deaths to births Info Only 2.07% Info Plus 1.88% Facilitation Control 1.90% Total 1.95% 8 The downward bias stems from the fact that any reductions in mortality due to the intervention will be less likely to be detected since control group households that experienced child mortality and have no children will not appear in the evaluation sample. 31 4.4 Immunization Despite major investments, immunization rates in UP continue to remain low. The NFHS-4 reported that 50.4 percent of children aged 12-23 months had received full immunization. This evaluation compares full immunization rates among children in the 12-23 month age group. Overall, the full immunization rates in control villages in the evaluation sample are comparable to the rural UP average. The interventions (both arms) lead to significantly higher rates of immunization rates (Table 10). These differences are especially evident in the case of polio immunizations and Diphtheria, Pertussis, and Tetanus (DPT) immunizations. The rate of Bacillus Calmette–Guérin (BCG) immunization was found to be relatively high across all arms. Table 11: Full immunization rates, by arm Info + Info Only Control Total Facilitation Full Immunization 55.8% 59.9% 48.3% 54.8% 3 DPT doses 88.5% 88.4% 77.4% 84.9% 3 Polio 59.3% 64.2% 53.9% 59.2% BCG 97.9% 95.3% 96.9% 96.7% Note: Full immunization is defined (following NFHS) as 3 doses each of DPT and Polio, and BCG. In analyses with probit regressions that control for a full set of child and household level characteristics and account for sampling and study design, the IE team found that improvements in immunization rates are large and statistically significant (Table 11). Full immunization rates increased by 7.2 percentage points in the information only arm, and by 11.8 percentage points in the information plus facilitation arm (p<0.05). Relative to a mean of 48.3 percent in the control arm, these are approximately 20 and 33 percent increases. Given the high coverage of BCG in control (over 95 percent), there was little to no improvement in BCG rates. The large improvements were driven by both polio and DPT immunization rates. This is of key policy significance as higher full immunization rates could result from lower drop-outs after BCG vaccination at birth. Table 12: Effect of interventions on immunization rates Fully Vaccinated DPT Polio BCG b/se b/se b/se b/se Info Only 0.072 0.109*** 0.057 0.021 0.045 0.028 0.045 0.017 Info + Facilitation 0.118** 0.110*** 0.104* -0.04 0.044 0.03 0.043 0.022 Control Arm Mean 0.483 0.774 0.539 0.969 Observations 1005 1005 1005 531 Note: Marginal effects reported for probit; SEs clustered at household-level; matched trio and district FEs 32 Immunization Rates, by District Immunization Rates, by District Sultanpur - BCG Sultanpur - BCG ultanpur - BCG Fatehpur - BCG Fatehpur - BCG In addition to the noted improvements in immunization rates, there are some differences across Fatehpur - BCG Immunization districts.Sultanpur - 3Polio rates are higher in Sultanpur, especially for those receiving three doses Sultanpur - 3Polio of polio (Figure 5). Overall, rates for BCG immunizations were consistently the highest across tanpur - 3Polio Fatehpur - 3Polio both districts while polio immunization rates were the lowest across the two districts. The Fatehpur - 3Polio regression model controls for these differences through district fixed effects. tehpur - 3Polio Sultanpur - 3DPT Sultanpur - 3DPT Figure 5: Full immunization rates, by district ultanpur - 3DPT Fatehpur - 3DPT Fatehpur - 3DPT 100% atehpur - 3DPT Sultanpur - Full Imm. Sultanpur - Full Imm. 80% pur - Full Imm. Fatehpur - Full Imm. Immunization Rates, by District Info Only Fatehpur - Full Imm. 0% 20% 40% 60% 80% 100% 60% pur - Full Imm. 0% 20% 40% 60% 80% 100% Control Info + Facilit ation Info Only Info + Facilitation 0% 20% 40% 60% 80% ation 100% Info Only 40% Control Info + Facilit Control Info + Facilit ation Info Only 20% 0% Control Fatehpur - Full Imm. Fatehpur - BCG Fatehpur - 3DPT Sultanpur - BCG Sultanpur - Full Imm. Fatehpur - 3Polio Sultanpur - 3DPT Sultanpur - 3Polio Despite the overall positive effect observed in both intervention arms, there are persistent gender differences in immunization rates, with girls doing worse than boys do. While the interventions lead to overall improvements in immunization, it only partially closes the gap in the information plus facilitation arm (Figure 6). The difference in control arm is 6.9 percentage points (40.4 percent for boys versus 33.5 percent for girls) and the difference in information plus facilitation is 4.6 percentage points (51.2 percent for boys versus 46.6 percent for girls). Figure 6: Full immunization rates, by gender and treatment arm Full Immunization Rates, by gender and arm 0.6 0.5 0.4 0.3 0.2 0.1 0 Info Only Info + Facilit ation Control Girls - Full Immu. Boys - Full Immi. 33 4.5 Maternal Care This section studies the potential impact of the SA interventions on key indicators of maternal care, focusing on institutional delivery rates and ANC care. In the evaluation’s study areas, similar to what was seen in large representative surveys, institutional delivery are relatively ratesDelivery Insitutional and 3 ANC visits but ANC care remains much lower (Figure 6). In high 7the case of institutional Insitutional 0.9 Delivery delivery, and 3 ANC both visits arms have higher rates of institutional delivery with Insitutional approximately 0.9 Delivery and 3 ANC visits 89 percent of women and 88 percent having institutional births in the 0.8 0.9 information 0.8 only arm and the information plus facilitation arm, respectively. 0.7 0.8 0.7 0.6 Figure 7: ANC and institutional delivery rates, by treatment arm 0.7 0.6 0.5 0.6 0.5 Insitutional Delivery and 3 ANC visits 0.4 0.5 0.9 0.4 0.3 0.4 0.8 0.3 0.2 0.3 0.7 0.2 0.1 0.2 0.6 0.1 0 0.1 0.5 0 3 ANC V isits Institutional Deliv ery 0 3 ANC V isits Institutional Deliv ery Info Only 0.4 ation Info + Facilit Control 3 ANC V isits Info Only Institutional Deliv ery Info + Facilit ation Control 0.3 Info Only Info + Facilit ation Control 0.2 0.1 0 3 ANC V isits Institutional Deliv ery Testing these differences in regression analyses reveals that both intervention arms lead to Control Info Only Info + Facilit ation statistically significant improvements in institutional delivery rates (Table 12). Institutional deliver rates improved by 5.7 percentage points in the information only arm, and 5.6 percentage points in the information plus facilitation arm (both statistically significant at p<0.05). Table 133: Effect of interventions on institutional delivery Institutional Delivery b/se Info Only 0.057** 0.021 Info + Facilitation 0.056** 0.020 Control Arm Mean 0.819 Observations 2079 Note: Marginal effects reported for probit; SEs clustered at household-level; matched trio and district FEs The two intervention arms’ effects on ANC visits, in contrast, were small and not statistically significant (Table 13). It is unclear exactly why the intervention arms did not affect the average number of ANC visits but did affect institutional delivery rates, but several explanations include 34 the possibility that the frontline healthcare workers provided antenatal advices to pregnant women in the communities rather than at health facilities, or that the ANC utilization variable was not measured correctly. Table 14: Effect of interventions on ANC visits At least 3 ANC visits Number of ANC visits b/se b/se Info Only -0.031 0.019 0.032 0.062 Info + Facilitation 0.032 0.070 0.033 0.063 Control Arm Mean 0.296 2.208 Observations 2114 2027 Note: Marginal effects reported for probit; SEs clustered at household-level; matched trio and district FEs The IE team did not expect to see large gender differences for maternal care, since the gender of the child is not supposed to be known to households even if there are gender preferences. Looking only at control arm villages, there are some differences across gender: institutional delivery rates are slightly higher for mothers who gave birth to girls (85 percent for girls versus 80 percent for boys) and ANC rates are higher for mothers who gave birth to boys (33.1 percent for boys versus 25.6 percent for girls) (Figure 8). However, these differences are statistically insignificant and the IE team believes they are not a result of the intervention (Appendix 4). Figure 8: ANC and Institutional Delivery, by gender of the child ANC & Institutional Delivery, by Gender 1 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 ANC3-Girls ANC3-Boys Inst.Del - Gir ls Inst.Del - Boys Info Only Info + Facilit ation Control 4.6 Nutritional Supplementation – Awareness and Utilization A potential mechanism through which SA interventions could improve nutritional outcomes is by improving awareness of entitlements under programs such as ICDS, increasing availability of nutritional supply at Aanganwadi centers and receipt of the supplies. Among households in control villages, 60 percent were aware of entitlements for nutritional supplementation for both 35 pregnant / lactating mothers as well as young children (Table 14). The awareness levels were comparable across both districts. Findings from regression analyses reveal that awareness of entitlements increased significantly in intervention villages (Table 14). Households in the information only arm were 9.3 percentage points more likely to be aware of their entitlements compared to 6.7 percentage points in the information plus facilitation arm (both statistically significant at p<0.05 and p<0.1 levels, respectively). Table 15: Effect of interventions on nutritional supplementation awareness and utilization Full awareness Receipt of nutritional Difficulty of nutritional supplementation in obtaining NS in entitlements past 1 month past 1 year b/se b/se b/se Info Only 0.093** -0.062 0.028 0.030 0.037 0.031 Info + Facilitation 0.067* -0.034 -0.014 0.030 0.039 0.027 Control Arm Mean 0.597 0.336 0.323 Observations 4213 2733 4213 Note: margins reported from probit; SEs clustered at village level; matched trio and district FEs There were no statistically significant effects on households reporting receipt of nutritional supplementation in the last month, or whether they had difficulty obtaining the supplements in the last year (Table 14). The negative coefficients, although small and imprecisely estimated, could be due to increased awareness about eligibility for nutritional supplementation programs as a result of the interventions. All measures of nutritional supplementation and utilization were comparable regardless of the gender of the respondent’s child (Figure 9). Figure 9: Nutritional supplementation and utilization, by gender Nutritional Supplementation and Utilization, by gender 1 0.8 0.6 0.4 0.2 0 Full Awareness of Receipt of Supplements Difficulty in Obtaing Entitlements Supplements Boys Girls 36 4.7 Frontline Health Worker Satisfaction To better understand potential service delivery improvements, the IE team measured households’ satisfaction with the three key primary care providers in the village – the ANM, the ASHA, and the AWW. The evaluation’s surveys collected data on the experience of the household when accessing services. The evaluation found that female head of households were most satisfied on average with ASHA worker performance. In contrast, satisfaction with AWW performance was the lowest among respondents (Figure 10). For both the ASHA and the ANM, the information plus facilitation arm had the highest mean satisfaction among the three evaluation arms. Figure 10: Satisfaction AAA, by treatment arm Analysis of satisfaction with services from providers reveals that the both intervention arms saw significant increases in satisfaction with ASHA workers’ performance, but not for other providers (Table 15). The 6.4 percentage point increase in information plus facilitation areas represents a 9 percent increase, relative to 70.1 percent in control villages (p<0.01). While the information only arm showed negative effects for the ANM and the AWW, these findings are not statistically significant. Table 16: Effect of interventions on AAA satisfaction ASHA ANM AWW b/se b/se b/se Info Only 0.045* -0.017 -0.026 0.018 0.022 0.022 Info + Facilitation 0.064*** 0.026 -0.007 0.019 0.024 0.024 Control Arm Mean 0.701 0.690 0.627 Observations 4048 4098 4213 Note: Marginal effects for Satisfaction with AAAs (1- score of 4+; 0-score of 1-3) reported from probits; SEs clustered at village level; matched trio and district FEs 37 Unlike indicators of health and nutrition outcomes and health care utilization, where Fatehpur was marginally worse than Sultanpur, these two districts differed greatly in terms of satisfaction with services provided by these workers (Figure 11). The share of households who reported being satisfied or highly satisfied with the services they received in Fatehpur was almost half that reported by households in Sultanpur. This is a strikingly large difference that appears to be invariant to the intervention. While our study cannot identify specific factors to explain this difference, it supports the inclusion of district fixed effects in our analyses. Figure 11: Satisfaction with AAA, by district Satisfaction with AAA, by district ASHA (Fatehpur) ASHA (Sultanpur) ANM (Fatehpur) ANM (Sultanpur) AWW (Fatehpur) AWW (Sultanpur) 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Control Info + Facilit ation Info Only Satisfaction did not differ between the respondent’s based on the gender of the child (Figure 12). Mirroring the overall perceptions in the overall evaluation sample, satisfaction with ASHA workers was highest among respondents with either boy or girl children. Likewise, respondents with either boy or girl children had the lowest satisfaction with AWW. Figure 12: Satisfaction with AAA, by gender Satisfaction with AAA, by Gender 0.8 0.6 0.4 0.2 0 ASHA ANM AWW Boys Girls 38 4.8 Village Health and Nutrition Days A key service delivery-related focus of the SA interventions was organizing village health sanitation days (VHNDs) every month. The evaluation’s surveys asked households whether they recalled VHNDs being organized in their village. Specifically, surveys asked how many VHNDs had been organized during the past six months. Just over 15 percent the households in control areas reported that there were VHNDs every month in their village (figure 13). The share of households reporting VHNDs being held every month was larger in the information plus facilitation villages where 18 percent of households reported monthly VHNDs. Figure 13: VHND frequency in last six months, by treatment arm VHNDs held in last 6 months, by Arm Control 15.5% 84.5% Info + Facilitation 18.1% 81.9% Info Only 14.9% 85.1% 0% 20% 40% 60% 80% 100% Held consistently Not held consistently 39 4.9 Embedded studies on information provision and role of social networks In addition to enabling the analysis presented in this evaluation, the large-scale data collection and field presence in the SA intervention provided the IE team with further opportunities to undertake supplementary research. This additional research focused on informing the design of interventions as well as investigating underlying behavioral mechanisms that are central to collective action. The two sets of studies that the IE team undertook were: a. Rapid evaluation of information dissemination strategies. In the first two months of the SA intervention, as part of promoting information about SA meetings in the village, the IE team tested alternative models for disseminating information in the villages through “seeding� information to specific households or people in the village.9 b. Research on the effect of information on participation in collective action in social networks. Building on the high quality network data collected in the SA intervention, in months 3 and & 4 of the intervention, the IE team oversaw randomized information provision about upcoming meetings to households and provided small randomized incentives using scratch cards for cell phone credit to attend meetings. The IE team also tested the effect of common knowledge on participation within networks. Experiments within complete socio-centric networks on this scale are rare and this additional research represents cutting edge research on the effect of information on participation for collective action within networks and will inform the literature in sociology, political science and economics.10 4.9.1 Information Seeding Experiment The main objective of selectively “seeding� information was to track how (and how well) information about health-related activities in the village disseminates through the village network and to identify which seeding strategy was more effective. Villages were randomized into three intervention groups: 1. Broadcast messaging – Every household (that had a mobile phone) received an interactive voice response (IVR) message giving information regarding VHSNC roles and responsibilities. 2. Network central – Using the social network data collected in both the Information Only and information plus facilitation arms in 2016 the IE team selected households “central� in the village social network. In addition to all households receiving an IVR message as in (1), these network central households received a phone call from a Morsel field team representative relaying similar information but in more detail (as the IVR messages are more time constrained). The households were then asked to share that information with others in the village, such as their friends and neighbors. 9 Information dissemination interventions were funded by Sanford Faculty Research Funds awarded to the Principal Investigator, Manoj Mohanan and Bass Connections grant to Prof. Jim Moody and Prof. Manoj Mohanan. 10 Analysis and research on networks were supported by Bass Connections grant from Duke and funding from NIH - 2P2C-HD065563 . The IE team is also grateful for support from Duke Network Analysis Center for technical analysis. 40 3. Government officials – In addition to sending an IVR message to all households in these villages, local officials (Pradhan, AAA, and other officials who would be enlisted as VHSNC members) received a phone call from a Morsel field team representative relaying similar information regarding VHSNCs. They were then asked to spread this information throughout the village. The IE team then used IVR messages to all households in all three groups approximately 5-7 days after sending out the information asking them whether they had received this information and/or learned of it from their friends and neighbors. The IVR then prompted them to reply yes/no by pushing a button on their phone. The IE team then planned to use their responses as a proxy for estimating how well information traveled through the village social network when different types of information seeding strategies were used, and which strategy (network central, government officials, or broadcast messaging only) was most effective. The IVR data proved to be ineffective for analysis due to a combination of high non-response and illegible responses. As a result, the IET team relied on village-level attendance records for analysis and compared attendance across different strategies. Results from this analysis are largely inconclusive. It appears that using a consistent strategy (seeding information through network central households and/or government officials or IVR broadcast messaging) did relatively better than a mixed strategy (IVR broadcasting in one month and a seeding strategy in another month). This may suggest more expensive strategies, such as broadcast messaging and more invasive outreach, may be ineffective at generating more attendance at local meetings. However, the evaluation does not necessarily conclude that; the ineffectiveness of IVR data collection leads IE team to conclude that further research on this question is required. Table 17: Results from month-wise VHSNC attendance Month 1 Month 1 Month 2 Month 2 - Attendance Attendance Attendance Sultanpur - Sultanpur coef/se coef/se coef/se coef/se Month 1 - Network Central -9.774 -11.837 1.987 8.393 Seeding (10.387) (11.007) (10.897) (13.804) Number of HHs in the 0.051* 0.100*** 0.022 0.150*** village (0.029) (0.029) (0.032) (0.037) Month 2 - Local Officials 11.720 19.886 Seeding (10.386) (12.365) Constant 33.105*** 23.416* 29.608** -1.549 (12.680) (12.804) (14.022) (16.498) Observations 39 36 19 19 R2 0.146 0.398 0.029 0.587 Note: Statistically significant difference relative to the control group is noted at the 1% (***), 5%(**), or 10% (*) level. 41 4.9.2 The Network Study A key question in collective action efforts is how to encourage participation from community members for creation of public goods. The IE team leveraged data collection that was conducted on meeting participation during the first four months of SA interventions in Intervention Arm 2 to study this important question. The IE team also collected detailed social network data that created a unique socio-centric network dataset from over 20,000 households that enabled us to track participation within social networks. The specific questions that the evaluation aimed to answer were: • Are individuals more likely to participate when members of their social network participate in collective action? • Does individual participation in collective action increase with incentives? • Does individual participation in collective action increase when there is common knowledge across members of one’s network that they have all been invited to participate. Common knowledge, in the context of participation in a meeting for instance, is defined as when (1) a member i in a network knows about the meeting, (2) i's friends in the network know about the meeting, and (3) i's friends in the network know that i knows about the meeting. The IE team randomized 30-40 percent of households receive a reminder visit when they were told about the upcoming VHSNC meeting. Households were also reminded why it was important to attend and encourage friends and neighbors to attend. Each visited household given a scratch card for a phone credit top up to attend the meeting (half of which had value of INR 5 and the other half had INR 35). Using the scratch cards, which households were told to bring with them to the meeting, the IE team was able to monitor individual household attendance. The random assignment of share of households visited and households that received incentives to attend the meeting enables us to study households’ decision to participate in collective action (i.e. VHSNC meetings) as a function of information in their social networks. The IE team developed a simple theoretical model of public goods & civic action model that allows (a) social networks to play a role in participation, (b) Returns to participation is convex at least initially, and (c) shift in beliefs (about participation within networks) as a result of information being given to agents. Along with simple assumptions that private effort is costly and that there are complementarities in civic action (one individual showing up for community monitoring has less impact than many folks showing up), the IE team’s model yields the following four simple predictions (details are beyond the scope of this report): 1. Attendance by those invited should increase when there are incentives. 2. Attendance by those invited should increase when there is common knowledge that more of one’s neighborhood is informed as well. 42 3. Attendance by. those invited should increase more with incentives when there is common knowledge 4. A network that ceteris paribus has greater density than another network should exhibit greater attendance. The network graph below shows an example of village network with randomized assignment of households to receive a visit with information, combined with 5 Rupee or 35 Rupee incentives for participation (Figure 14). To test the effect of common knowledge, the IE team conducted the experiment twice. In month 3, the IE team introduced interventions to randomized households (i). In month 4, the team introduced similar information, but also told the visited household that surveyors will be visiting their friends A, B, and C, to provide the same information about meetings, and also told them that surveyors have visited them. This created Common Knowledge (CK) in month 4 but not in month 3. Figure 14: Sample blobbogram depicting social network data from a village No visit Info visit + 35 INR Info only (+ 5 INR) The main results of the network intervention study are shown in the following table. The coefficients (and SE in parentheses) are shown separately for months 3 and 4; a pooled regression yields identical results. 43 Table 18: Effect of network interventions on VHSNC attendance Attended in Month 3 Attended in Month 4 HH was incentivized .0474*** .0972*** (0.013) (0.015) Num. friends incentivized -.006 .0267* (0.010) (0.011) Num. friends visited but no incentivized .005 .0189** 0.010 0.007 Observations 3446 3330 Note: Statistically significant difference relative to the control group is noted at the 1% (***), 5%(**), or 10% (*) level. Marginal effects reported from probit model Consistent with the predictions of the IE team’s model, the evaluation finds that an additional INR 30 incentive increases individual participation. The effect of incentives is almost twice as much with common knowledge (CK) (~5 percentage points to 10 percentage points). Furthermore, incentives to friends within networks increases individual participation only in presence of CK (by about 2.7 pp), while information to friends within networks increases individual participation only in presence of CK (by about 1.8 percentage points). Finally, network characteristics such as higher network density is associated with higher attendance, especially in presence of CK. Policymakers may be able to leverage these findings to increase participation in collective action efforts by designing programs that create common knowledge within social networks. In the context of efforts such as SA and community monitoring, where participation is difficult to ensure, strategies that rely on mobilizing social networks hold great promise. 5. Conclusion and Policy Recommendations This evaluation of the SA interventions in Sultanpur and Fatehpur shows that the SA efforts led to consistent and large improvements in health and nutrition outcomes as well as improvements in indicators of health service delivery and utilization. (see Appendix 5 for full results on both intervention arms) While both intervention arms had positive effects that were significant. the information plus facilitation intervention arm demonstrated consistently larger effects than the information only intervention arm. The effects were seen for stunting and underweight prevalence, treatment patterns for childhood diarrhea, immunization rates, institutional delivery but not for childhood mortality or antenatal care utilization. • Both intervention arms had impacts on important nutrition outcomes. The information plus facilitation arm reduced stunting rates among children <2 years of age by almost 44 11 percent (5.6 percentage point reduction, relative to 51.3 percent stunting prevalence in the control arm). In the 2-5 year age group, the information only interventions reduced both stunting and underweight rates, by almost 11 and 13 percentage points, respectively. The information plus facilitation intervention arm had similar sized effects in reducing underweight among 2-5 year old children. • The significant improvements in nutritional status were accompanied by striking improvements in treatment patterns for childhood diarrhea (although there was no effect on prevalence). Among children aged <2 years, the information plus facilitation intervention increased likelihood that child was treated within the first day by almost 14 percentage points (an increase of more than half relative to 20.8 percent in control). The evaluation did not detect any effects on mortality rates in the study sample. • The SA interventions also led to dramatic improvements in vaccination rates. Among children aged 12-24 months, the share with full immunization in control is comparable to NFHS averages. Full immunization rates increased by 7.2 percentage points in the Info only arm, and by 11.8 percentage points in the information plus facilitation arm. Relative to 44.5 percent in the control arm, these are approximately 16 and 27 percent increases. • The SA interventions also improved indicators of maternal health care. Institutional delivery rates increased by over 5.6 percentage points in both arms. Improvements in rates of women receiving three or more ANC were relatively small, and not statistically significant. • Both interventions led to significant increases in awareness of entitlements in intervention villages. Households in information only arm were 9.3 percentage points more likely to be aware of their entitlements (6.7 percentage points in information plus facilitation arm). There were no effects on the likelihood of actual receipt of nutritional supplementation though, suggesting little to no improvement in supply of nutritional services. • Analysis of satisfaction with services from providers reveals that the intervention areas saw significant increases in satisfaction with ASHA workers’ performance, but not for other providers (ANM or AWW). The 6.4 percentage point increase in information plus facilitation areas represents a 9 percent increase, relative to 70.1 percent in control villages. • Finally, the evaluation finds that the information plus facilitation interventions led to a 1.2 percentage point increase in share of households that reported that VHNDs had been organized every month for the past six months. Taken together, the SA interventions in UP led to systematic improvements in health delivery and utilization for maternal and child health, improving key health and nutrition outcomes (See Summary Table). The large improvements in immunization rates suggest that further improvements might be expected over time. There have also been significant increases in frontline health worker satisfaction (for ASHAs), community health service delivery organization (VHNDs) as well as increased awareness of the community on their entitlements. The IE team also recognizes that gender may play a potentially important role in the effectiveness of the interventions. In the next round of analysis, the IE team will further analyze 45 data and conduct additional regression models to investigate the extent of gender’s importance in SA. However, sample size and statistical power may limit the IE team’s ability to make claims about impact by gender. With these data limitation in mind, contingent on additional funding become available, the IE team may conduct some qualitative data collection for further insight into this question. As impressive as these positive effects are, it is important to note that these effects are measures of effectiveness of the interventions when implemented in real world settings through government run implementation channels and all the challenges that accompany them. Without taking any credit away from UPHSSP and SIRD for the impressive work that these two institutions have done in the past few years, it is critical to recognize that they were operating in the extremely challenging environment of Uttar Pradesh. The implementation of the program had faced several critical delays and challenges – ranging from staffing of GPCs, recruitment of the implementing organization, technical problems in implementation of monitoring tools, contracting delays, payment delays to field staff, as well as a sizeable turn over in field staff due to these types of issues. The fact that this evaluation observes large program impacts despite all of these challenges a year after the GPC facilitation efforts implemented by UPHSSP-SIRD were halted due to contract renewal is a testament to the immense potential of SA mechanisms – including both information and facilitation – to successfully engage community members to participate in the monitoring of service delivery in their respective areas. 5.1 Recommendations Based on the evidence in the report, and summarized in the previous section, the following are the key recommendations: 1. Given the large improvements in various health and nutrition outcomes, health utilization and in indicators of service delivery, the Government of UP should consider scaling up these interventions across the state immediately. This could be done with the support of the UPHSSP as well as the NHM; 2. UPHSSP and the Government of UP should widely disseminate these findings at various national and state fora, so the other state health departments and the Ministry of Health, Government of India, can consider scale up at the national level; 3. While scaling up the interventions, the following points should be considered and incorporated into the design of these interventions: a. Continue to develop and implement SA interventions that feature both information dissemination along with GPC led facilitation of meetings of the VHSNCs. The main point to note here is that information provision combined with community facilitation produces greater effects than solely information provision; b. The SA interventions should modify its information dissemination component to use a combination of cell phone broadcast messages, social network based targeting, as well as household visits instead of the current method of delivering information using citizen charter documents and public dissemination methods; 46 c. The GPC can play a pivotal role in successful facilitation of VHSNC meetings. Recruitment and staffing processes of GPCs will need to be reviewed and monitored to ensure high quality staff working as facilitators; d. Successful VHNSC facilitation should focus on: i.) activation of dormant committees; ii.) assisting village-level representatives to organize meetings and taking necessary remedial steps; and iii.) encouraging active participation from community members; e. Additional training and hand holding components included in the two study districts were designed to be easily incorporated in the state-run training programs, and it is highly recommended that these training components also be incorporated in the scale up of SA interventions in the state; and, In addition to these immediate policy actions for consideration, the following may also be paid attention to: 4. The effects on health worker satisfaction, though mainly ASHAs, as well as the increased knowledge of available entitlements amongst communities shows this can be used as a viable option to strengthen frontline worker delivery and create positive linkages with an empowered community. 5. The study’s findings on the role of “common knowledge� within social networks on participation for VHSNC meetings suggests a potential role for these mechanisms to increase participation for collective action in a broad range of settings; 6. Given the vast range of positive effects, policymakers could consider using SA interventions across a broader range of diseases, both communicable and non- communicable diseases. This can be especially useful in countries like India, where the epidemiological transition towards more chronic non-communicable diseases, will need more active frontline delivery as well as health prevention and promotion in communities; and, 7. The ongoing implementation of SA interventions in the 10 districts in UP, randomized across a population of 23 million, is an excellent opportunity not only to improve health outcomes for the children in these areas but also evaluate its population-level impact. With appropriate design modifications and robust implementation, the program has the potential to achieve even larger effects than those observed in this evaluation. It is recommended that – in addition to strong and continued implementation – that the main stakeholders devote resources for a rigorous evaluation of the impact of SA when implemented at an even larger scale. In closing, more impact evaluations of the current type of study should be encouraged since they add to the global knowledge on implementation science. Policymakers not only benefit from the evaluation’s estimate of a particular intervention’s impact, but also its effects when implemented at scale and through the regular Government system. 47 Appendices The following section includes appendices containing additional information regarding the evaluation of SA interventions in Uttar Pradesh. Appendix 1 – VHSNC Description 1.1 VHSNC Composition As the village’s principal body for resolving village-specific public health problems, the VHSNC aims to bring together key stakeholders to implement relevant solutions. This includes members from local elected bodies, healthcare providers, and patients/local villagers [31]. Specifically, the composition of the VHSNC should include the following members: • Village Pradhan • Panchayati Raj Institution (PRI) members • Auxiliary Nurse Midwife (ANM) • Accredited Social Health Activist (ASHA) • Aanganwadi Worker (AWW) By including key elected officials together with healthcare providers, decision makers can directly and transparently address concerns raised by citizens. This is especially relevant in cases related to healthcare provider performance, absenteeism, and provision of free services. 1.2 VHSNC Activities and Responsibilities In order to solve local challenges related to health, sanitation, and nutrition, the VHSNC can carry out several activities: • Conduct awareness campaigns – The VHSNC can mobilize community members as well as use regular gatherings to highlight important issues that are affecting local health, sanitation, and nutrition. Chief among these, VHSNC members can provide information about best practices for maternal and child nutrition, including which local foods have high nutrient value. • Supervise Village Health and Nutrition Day (VHND) – This includes building awareness of the services provided during the monthly VHND, assisting with the scheduling and planning of the VHND, and managing complaints/concerns suggestions regarding the VHND. • Utilization of Village Untied Funds – Each Gram Panchayat (GP) is granted an annual fund from NRHM of INR 10,000 to help healthcare providers and villagers respond to local health and sanitation challenges. GPs may use these funds towards things such as subcenter maintenance, water provision, electrification, purchase of minor medical equipment/instruments, labor and supplies for vector control, and information/awareness materials. The VHSNC can provide crucial input on how the GP prioritizes the use of these funds and where they will be the most effective. • Provide Feedback to Healthcare Providers – The VHSNC meetings are a place where village members can report grievances or give feedback on the quality of local i providers and health services. The committee is thus tasked with management of such grievances and general feedback, which may include submission of Action Taken Reports to block-level officers. • Generate Action Taken Reports (ATRs) – The VHSNC is encouraged to generate ATRs and submit them to the Block Development Officer (BDO) for redressal. This is the official means for redressing the grievances and general feedback presented at VHSNC meetings. 1.3 VHSNC Challenges Although established in 2005 by NRHM, VHSNCs have faced significant implementation challenges. Data from the baseline survey conducted in April 2015, among 4,856 households in the 12 implementing districts, suggest that VHSNCs were largely inactive or had not yet been implemented in these areas. Only 8 percent of households were aware of a VHSNC existing in their village (AMS Final Report). ii Appendix 2 – Intervention Description 2.1 Intervention Arm 1: Information Only • Information provision on village-level health indicators for U5 children relative to the district average. • Information provision on roles and responsibilities of the VHSNC given through visits to all households in each village. • Information dissemination regarding VHSNCs through phone calls and Interactive Voice Response (IVR) calls. Information dissemination was conducted on behalf of the research team, (funded by Duke University grants) by Morsel Research & Development Pvt Ltd., a research consulting firm based in Lucknow.11 In coordination with the IE team, Morsel Research provided information interventions at the village level across both Intervention Arms 1 and 2. This information was provided by Morsel’s field teams through a combination of household visits, phone calls, and IVRs. Village members received information about the roles and responsibilities of VHSNCs, including details on the untied fund to which each VHSNC is entitled, and that the VHSNC was mandated by national government to be formed and functioning in their village. Additionally, village members were given information on child health indicators in their own village as well as the averages in their district for comparison (as estimated from household level data, collected at baseline from the same villages prior to the intervention). Figure 15 illustrates how such information was provided to households using one of the intervention villages, Andaraypur, in Sultanpur District. Figure 15: Participation rates of VHSNC members in activated VHSNCs (Jul ’16 – Feb ’17) 11 For details on Morsel, please visit http://morselindia.in/ iii Households were told that “Negative� indicators, or bad health outcomes, were displayed in red, while desirable health outcomes or “Positive� indicators were displayed in blue. The “negative� indicators included percent of households that had difficulty ob taining nutritional supplements in the year prior to baseline, percent of underweight children, and percent of stunted children. The “positive� indicators included were percent of households who had received nutritional supplements in the one month prior to baseline and percent of children who had received measles vaccination. Relative to district averages, the village of Andaraypur had performed worse on percent of households who had received nutritional supplements in the one month prior to baseline, percent of children who had received measles vaccination, and percent of stunted children, but better than the average in percent of households that had difficulty obtaining nutritional supplements in the year prior to baseline and percent of underweight children. All households in this intervention arm were visited by members of the Morsel field teams as part of the information dissemination effort at the start of the intervention and were shown these health indicator figures. Field team members were trained on explaining what the graphs meant – with taller red bars indicating worse performance, and taller blue bars indicating better relative performance. 2.2 Intervention Arm 2: Information plus Facilitation • Information provision as implemented in Intervention Arm 1. • Active engagement from a Gram Panchayat Coordinator (GPC), hired and trained by UPHSSP/SIRD to activate VHSNCs and to assist in facilitating monthly meetings and other functions of the committee. • Monthly VHSNC meetings conducted by key community stakeholders, where village members were encouraged to attend and discuss issues in the village related to health, sanitation, and nutrition. • Household visits from the Morsel field team. This included information regarding upcoming VHSNC meetings as well as small incentives for some households to attend the meetings. The information interventions described in Intervention Arm 1 were also implemented identically in Intervention Arm 2 villages in the first two months of SA interventions. In addition to Intervention Arm 1, the facilitation component of Intervention Arm 2 aimed to enhance the participation of the community in creating SA. This intervention provided trained facilitators, called Gram Panchayat Coordinators (GPCs) to help community members engage in a participatory process with VHSNCs and PRIs and identify key deficiencies for improvement in health services that most concerned community members. In order to carry out these interventions, UPHSSP collaborated with the Deen Dayal Upadhyaya State Institute of Rural Development (SIRD) to manage implementation of SA interventions in the selected villages.12 The SIRD is a state-run autonomous institution that falls under the Ministry of Rural Development. Located outside of Lucknow, SIRD has a long history conducting capacity building/training courses on a range of subjects - especially those related to improving local governance - through trainings for officials at the block and GP levels. To this 12 For details about Deen Dayal Upadhyaya State Institute of Rural Development please visit: http://www.sirdup.in/ iv end, SIRD recruited a project management (PMU) unit based at their state headquarters to direct intervention implementation. UPHSSP and SIRD jointly developed protocols and procedures for GPC activities on the ground, such as VHSNC activation forms, Action Taken Report documents, and other monitoring formats for GPCs to fill each month. GPCs were also provided a detailed checklist of activities that were to be undertaken prior to the meetings, such as informing the community representatives about the date, inviting the block level officers and ensuring that logistics requirements for VHNDs were conveyed to VHSNC members in advance of the meetings. The SIRD retained services of an HR firm to recruit 316 GPCs (including16 for the two IE districts of Sultanpur/Fatehpur and 300 for the other 10 districts), in April 2016. GPCs in the two IE districts were also given additional field support from representatives of Sambodhi Research & Communications Pvt. Ltd., contracted by the World Bank.13 The Sambodhi District Social Accountability Specialists (DSAS) assisted in training these 16 GPCs (conducting a supplementary training in June 2016), provided support to them during field work, and monitored their work. GPCs in the two IE districts were assigned to 2-3 villages each, and those in the other 10 districts were assigned to approximately 10 villages. A key component of the SA interventions involved activating VHSNCs in areas where they either did not exist previously or functioned irregularly (e.g. sporadic meetings, closed-door meetings, etc.). Gram Panchayat Coordinator Responsibilities In order to activate and facilitate the VHSNCs in assigned GPs, GPCs were responsible for the following tasks: 1. Identifying and contacting elected officials and PRI members to enlist and activate the VHSNC. 2. Setting the monthly VHSNC meeting date and assisting the committee in adhering to the set dates. In the two IE districts, the Sambodhi-trained GPCs were also further trained to help committees draft agendas and facilitate constructive meetings. 3. Assisting with the facilitation of the VHSNC meetings each month in their assigned villages. 4. Referral of active VHSNC members for SIRD training. This training was designed to help VHSNC members learn about the roles of the VHSNC, and how to respond to accountability initiatives in their respective villages. For instance, SA interventions in UPHSSP required committees to generate “Action Taken Reports� that summarize the outcome of meetings and subsequent actions taken. Members might have to navigate the VHSNC chain of command up through the block and district-level offices to address concerns of their village constituents. 5. Filling out timely formats and submitting them to SIRD for monitoring and evaluation. The facilitated meetings aimed to empower community members to demand better health services and quality, and to convey these demands more effectively to providers and officials. 13 For details about Sambodhi Research and Communications Pvt Ltd., please visit http://sambodhi.co.in/ v The three key health workers at the village level (ASHA, ANM, and AWW) report to the local (village level) elected representatives and block level authorities, who receive feedback from the community in the accountability interventions. Moreover, through repeated community meetings, village-level health workers were expected to respond to transparency and accountability innovations by improving quality of services delivered to their local constituents resulting in improvements in population health outcomes. vi Appendix 3 - Modified Theory of Change for SA Inputs Outputs Outcomes Goals • Villages receive Information provision localized information • Citizens have greater • Information on comparing their awareness of • Parents better local health village to nearby entitlements to understand performance is villages health & nutrition health/nutrition collected and • Citizens are services investments for compared against Child Health informed of • Citizens better themselves/their children peers health/nutrition understand how to • Healthcare providers are Lower under-5, • Interactive voice services more responsive to infant and response (IVR) access healthcare or • Villagers receive needs of community and neonatal mortality messages are demand access to calls about VHSNC more effective tested and healthcare rates. and health • Parents and children visit recorded. initiatives in village the healthcare provider Nutrition more frequently for Lower rates of preventive healthcare and/or in cases of illness stunting and Community • Citizens voice • Increased rates of wasting. Facilitation antenatal checkups complaints regarding Maternal Health • GPCs inform health, sanitation, (ANC), institutional • UPHSSP & SIRD Lower rates of villagers about delivery, and recruit and train and nutrition issues obstetric VHSNC meetings immunization among Gram Panchayat • VHSNCs generate complications • VHSNCs are children Coordinators action taken reports activated with • Improved nutrition and maternal (GPC) (ATR) and/or use behaviors among help of GPCs and untied funding to deaths. • GPCs are conduct regular children and parents assigned to rectify local (e.g. exclusive meetings villages and • Citizens attend problems breastfeeding & conduct visits to VHSNC meetings • Ineffective providers complementary feeding) activate VHSNCs are reprimanded and/or transferred Figure 16: Modified theory of change for social accountability interventions vii Appendix 4 – Implementation Overview The following section provides an overview of the intervention’s implementation including an understanding of how and when UPHSSP’s SA interventions took place. 4.1 Implementation Timeline Baseline Data Collection A baseline survey, funded by UPHSSP, was conducted in the first half of 2015 and covered 4,856 households in 240 villages. 120 villages were those selected for the impact evaluation in Fatehpur and Sultanpur, and the remaining 120 villages were sampled from the other 10 districts in which SA interventions were to be implemented. The baseline collected data on child anthropometrics, vaccination and other health seeking behavior, interaction and satisfaction with AAA, and other household characteristics. Baseline data used to create village matched-trios were then randomized to Treatments Arms 1 and 2 and to Control. In this report, the IE team focuses mainly on the endline data, funded by SIEF. The baseline data, collected by a firm contracted by UPHSSP, had data quality problems because of which the IE team does not use the data as extensively as the endline data in the evaluation analysis. However, in future analysis, IE team plans to augment its analysis with the baseline data to present difference-in-difference estimates to compare with the findings from the endline data alone. (Note that the randomized design ensures that the program impact estimated from endline alone will be an unbiased estimate of the impact. Including the baseline data for difference-in-difference could lead to increased precision, depending on the degree of measurement error in the baseline). Implementation of Intervention Arm 1: Information Only Morsel field teams first visited GPs and provided the first wave of information in September 2016. Information provision continued through December 2016 in Intervention Arm 1 covering the first four months of SA interventions. In Intervention Arm 2, the first four months of SA interventions continued into April 2017 to account for meeting cancellations/rescheduling happening across both districts, though at a more frequent degree in Fatehpur. Implementation of Intervention Arm 2: Information plus Facilitation A critical component of the community engagement intervention depended on the recruitment and training of GPCs. After the hiring of the SIRD-based PMU in August 2015, UPHSSP and SIRD hired and trained GPCs from March-June 2016. Shortly after training, GPCs were deployed to GPs to begin VHSNC activation and community engagement activities. Figure 15 demonstrates administrative data of meeting activity from June 2016 until March 2017. In both districts, meetings started taking place in the majority of GPs as of September 2016. In general, Sultanpur outperformed Fatehpur in terms of conducting VHSNC meetings regularly. In Sultanpur, meetings took place in the majority of GPs during 8 out of 10 months with an average 76 percent of GP’s conducting VHSNC meetings. During four of these months, all 20 GPs in Sultanpur conducted VHSNC meetings. In contrast, VHSNC meetings occurred in the majority of Fatehpur GP’s during 6 out of 8 month with an average 55.5 percent of GP’s conducting meetings. In the following section outlining implementation challenges the IE teams looks at administrative data to provide more in-depth measures of VHSNC activation. viii Figure 17: Social Accountability Intervention Timeline (2015-2018) June-July May Oct. – Nov. GPC GPCs not working; GPC intervention October Recruitment implementation on hold due to GPC contract & Training on hold contract renewed Q1 Q2 Q3 Q4 Q1 Q2 Q3 Q4 Q1 Q2 GPC Facilitation Spring 2015 Info. Provision April-May Baseline Endline conducted Info. Provision conducted September Information campaign begins in all areas Gram Panchayat Coordinator (GPC) facilitation of VHSNC Provision of information in GPC-facilitated villages (Sept. 2016 – Apr.2017) Provision of information in information-only villages (Sept. 2016 – Dec. 2016) ix Figure 18: Percentage of GP’s conducting VHSNC meetings (Jun ‘2016 - Mar ’17) GP 's Co n d u ct in g V HSN C M e et in gs in Su ltan p u r (%) 100% 100% 100% 100% 95% 100% 90% 90% 80% 65% 60% 40% 15% 20% 5% 0% GP 's Co n d u ct in g V HSN C M e et in gs in Fate h p u r (%) 100% 100% 85% 80% 75% 80% 70% 55% 60% 45% 40% 40% 20% 5% 0% 0% 4.2 Implementation Costs While understanding the importance of the program’s causal impact is this evaluation’s prime importance, the IE team recognizes the financial difficulty that state governments frequently encounter in scaling innovative health interventions. With that in mind, this evaluation attempts to calculate approximate cost estimates (in 2016 INR) for the SA intervention evaluated in this 2-district SIEF-funded evaluation. However, it is important to note that the estimated program costs represent only those costs borne by implementing partners/agencies (i.e. the State Institute of Rural Development and Morsel India). These estimated costs do not include significant investments of time provided from UPHSSP staff, the World Bank, and additional technical assistance from the IE team to operationalize the intervention. Information Provision x As the information intervention took place over a period of approximately 4 months in an overall 12-month program, this evaluation provides different cost estimates based on different time durations.14 In the first estimate, the IE team calculates the cost by dividing costs over a 4- month period that covers only the period of active information provision. In this model, the cost per GP per month was INR 12,460. However, if this cost is spread over a period of 12 months, the period of the overall intervention, then cost per GP per month decreases 75 percent to INR 3,115. Gram Panchayat Coordinator As the second intervention arm, which is exposed both information provision and community facilitation, relies heavily on the involvement of a GPC, the cost of recruiting and fielding a GPC is an important consideration for government partners. Two important factors in this cost estimation are 1) the ratio of GPs to each GPC and 2) the inclusion of general program costs. In the case of GP to GPC ratio, the general UPHSSP intervention implemented across 10 districts had a ratio of 10 GPs per one GPC. In this case, excluding general program costs, the cost per GP per month is approximately INR 800. However, in the 2-district SIEF-funded evaluation, this ratio decreases from 10 to 2.5 GPs per one GPC. As such, the cost per GP per month rises to INR 3,220. In addition to the ratio of GPs to each GPC, the program required significant overhead to ensure that GPC could work effectively. This includes i) a state management unit to lead coordination and strategy, ii) training and capacity building efforts among both the GPCs and VHSNCs, and iii) other coordination costs among district and block officials. When these costs are included, the approximate cost per GP per month in Sultanpur and Fatehpur almost doubles to INR 6,050. While this evaluation provides an initial estimate, further research could improve estimates with a better understanding of overall costs related to recruiting and fielding GPCs. 4.3 Implementation Challenges During the implementation of the proposed SA interventions, intervention stakeholders encountered several important challenges to program implementation. • Administration of GPC Recruitment and Training Despite the hiring of a state PMU in August 2015, the recruitment of GPCs took almost an entire year. This process arose in part from the large numbers of team members that UPHSSP and SIRD had to recruit and train.15 Other delays arose from the time taken to interview and hire intervention management teams at the district, and sub-district levels. To adequately handle the administrative burden of recruiting, paying, and replacing any attrition among a 316 strong team of GPCs, UPHSSP hired an HR firm to lead this process. After the completion of State PMU and GPC recruitment, team members still had to receive significant training to understand the local health system (i.e. AAA), the role of the VHSNC, and the responsibilities of the GPC. In the case of the 16 GPCs assigned to VHSNCs in 14 The information intervention provided by morsel India was divided into two parts: i) costs related to conducting phone calls and ii) costs related to the household information dissemination. 15 UPHSSP recruited four team members in the state PMU, 29 district and sub-district coordinators, and 316 Gram Panchayat Coordinators. GPC recruitment took place through an HR contractor. xi Sultanpur and Fatehpur, field observations noted that GPCs required frequent assistance in completing the various formats that had like community scorecards, VHND monitoring formats, and action taken reports [37]. This type of on-the-job training and support continued from the inception of the VHSNC intervention until the end of implementation monitoring in February 2017 [37]. • GPC Payment Delay and Personnel Strikes In villages where GPCs provided support to VHSNC members in setting up and conducting meetings, delays in payment sometimes led to GPCs refusing to work. Data collected in Fatehpur district provides a vivid example to administrative hurdles undermining the intervention. In the first month that the research team collected participation data (prior to the strike), the average number of community participants in Fatehpur was 50. By the third month (and after the strike had begun), the average number of community participants dropped to 31, a 38 percent decrease (Figure 16). However, the IE team cannot rule out that attendance fell only because of GPC work stoppage. Figure 19: Average number of VHSNC attendees in activated VHSNCs (Sep ’16 – Dec ’17) Average Number of VHSNC Attendees in Sultanpur (September-December 2016) 51.0 36.5 33.9 33.3 Sep-16 Oct-16 Nov-16 Dec-16 Average Number of VHSNC Attendees in Fatehpur (September-December 2016) 57.5 50.4 36.4 31.2 Sep-16 Oct-16 Nov-16 Dec-16 xii • Lack of VHSNC Participation among VHSNC members As the program continued in to late 2016 and more villages began to consistently hold VHSNC meetings, GPCs noticed reluctant participation in VHSNC among key stakeholders, especially the Pradhan and ANM [38]. Figure 17 demonstrates administrative data on the number of key VHSNC participants (i.e. Village Pradhans, PRI members, AAA, etc.) that attended meetings between July 2016 and February 2016. It is important to note that attendance of VHSNC members began to drop significantly in both Sultanpur and Fatehpur after October 2016. In December 2016, block level officials were informed about the drop off in participation and staff requested that block-level officials ask Pradhans and ANMs to regularly attend the VHSNC meeting. Additional requests for support from block level officials were made in January and February [37]. In many cases, requests for Pradhan and ANM attendance from the block office appeared to help increases attendance [38]. Despite block level support for attendance, VHSNC member attendance was at only 45 percent and 20 percent in Sultanpur and Fatehpur by February 2017, respectively. However, this drop in participation may have resulted in part from state elections taking place in February 2017 [38]. Figure 20: Participation rates of VHSNC members in activated VHSNCs (Jul ’16 – Feb ’17) Participation of VHSNC Members during VHSNCs in Sultanpur (figures in %) 97 96 100 91 92 76 75 80 57 60 45 40 20 0 Jul-16 Aug-16 Sep-16 Oct-16 Nov-16 Dec-16 Jan-17 Feb-17 Participation of VHSNC Members during VHSNCs in Fatehpur (figures in %) 100 73 80 66 57 52 60 44 42 40 29 20 20 0 Jul-16 Aug-16 Sep-16 Oct-16 Nov-16 Dec-16 Jan-17 Feb-17 xiii • Development and Delivery of GP Specific Information Although SIRD and UPHSSP were in charge of the development and delivery of GP-specific information interventions, their team’s capacity to develop and deliver these materials was relatively limited. Information that was provided to GP’s in the 80 intervention areas came largely from Morsel, which was hired to support information dissemination and data collection. Future implementation of information programs relating to SA should seriously consider the difficulty of expanding GP-specific information provision to a large sample. However, as technology and administrative data improve, this process could rely on relatively automated processes from an MIS system. • Coordination & Logistical Challenges in VHSNC Activation GPCs found difficulty in initially activating the VHSNC. Socialization of the program and initial awareness, as previously mentioned, were key issues in VHSNC start up. GPCs spent considerable amounts of time introducing the VHSNC to various community stakeholders and conducting activities to start regular meetings. This includes tasks related to coordination between officials at the block and GP level. In this case, letters of authorization from the district and block officials were helpful in moving tasks forward at the GP level [37]. In addition to coordination issues, GPCs frequently required some support in logistical issues like scheduling meetings and choosing meeting venues for VHSNCs in each GP [37]. xiv Appendix 5 – Full Regression Tables on Outcomes Nutritional Outcomes Table A1: Effect of Interventions on Rates of Stunting and Underweight Stunting outcomes Under 2 Full Sample years 2-5 years b/se b/se b/se Info Only -0.039 -0.009 -0.064** 0.021 0.028 0.023 Info + Facilitation -0.038 -0.056* -0.024 0.020 0.028 0.023 Age (or avg) of mother (or potential mother) -0.009*** -0.007** -0.010*** 0.002 0.002 0.002 Edu (or avg edu) of mother(or potential mother) -0.051*** -0.044*** -0.056*** 0.007 0.010 0.009 Child's gender 0.013 0.079*** -0.038* 0.014 0.024 0.018 Child's age in years=0 -0.022 0.054 Child's age in years=1 0.145** 0.055 Child's age in years=2 0.176** 0.163** 0.056 0.053 Child's age in years=3 0.153** 0.157** 0.052 0.051 Child's age in years=4 0.100 0.105* 0.052 0.050 Number of U5 kids 0.025 0.024 0.022 0.016 0.022 0.017 Household is SC/ST 0.049** 0.032 0.054** 0.018 0.029 0.020 Household is Hindu 0.021 0.108** -0.040 0.025 0.042 0.029 Child's age in months 0.014*** 0.002 Observations 4874 2000 2874 xv Underweight outcomes Under 2 Full Sample years 2-5 years b/se b/se b/se Info Only -0.034 0.013 -0.067** 0.019 0.025 0.025 Info + Facilitation -0.039* -0.006 -0.065** 0.018 0.025 0.023 Age (or avg) of mother (or potential mother) -0.005** -0.002 -0.007** 0.002 0.003 0.002 Edu (or avg edu) of mother(or potential mother) -0.045*** -0.040*** -0.050*** 0.007 0.010 0.008 Child's gender 0.034* 0.084*** -0.003 0.016 0.024 0.021 Child's age in years=0 -0.050 0.053 Child's age in years=1 0.054 0.052 Child's age in years=2 0.049 0.023 0.055 0.055 Child's age in years=3 0.050 0.040 0.052 0.054 Child's age in years=4 0.005 -0.006 0.051 0.053 Number of U5 kids 0.009 0.003 0.007 0.015 0.022 0.017 Household is SC/ST 0.049** 0.042 0.052** 0.017 0.028 0.020 Household is Hindu 0.001 0.019 -0.015 0.025 0.044 0.033 Child's age in months 0.007*** 0.002 Observations 4874 2000 2874 Marginal effects reported from probit; SEs clustered at village level; matched trio and district FEs xvi Treatment of Childhood Diseases Table A2: Effect of Interventions on Diarrhea Prevalence Full Sample Under 2 yrs 2-5 yrs b/se b/se b/se Info Only -0.002 0.002 -0.009 0.017 0.022 0.017 Info + Facilitation 0.004 0.007 -0.000 0.017 0.025 0.016 Age (or avg) of mother (or potential mother) 0.000 0.002 -0.001 0.001 0.002 0.001 Edu (or avg edu) of mother(or potential mother) -0.012* -0.010 -0.012* 0.005 0.008 0.006 Gender (0-female; 1-male) 0.000 -0.018 0.011 0.011 0.018 0.013 Child's age in years=0 0.105** 0.039 Child's age in years=1 0.047 0.038 Child's age in years=2 0.030 0.027 0.036 0.035 Child's age in years=3 -0.003 -0.006 0.036 0.036 Child's age in years=4 -0.028 -0.031 0.037 0.037 Number of U5 kids -0.013 -0.011 -0.016 0.010 0.016 0.012 Household is SC/ST 0.007 0.025 -0.003 0.011 0.017 0.014 Household is Hindu 0.001 0.008 -0.005 0.019 0.030 0.019 Child's age in months -0.003** 0.001 N 4916 2098 2818 Marginal effects reported for probit; SEs clustered at household-level; matched trio FEs xvii Table A3: Effect of Interventions on Diarrhea Duration Full Sample Under 2 yrs 2-5 yrs b/se b/se b/se Info Only 0.170 -0.317 0.665 0.263 0.278 0.508 Info + Facilitation -0.034 0.134 -0.317 0.252 0.291 0.323 Age (or avg) of mother (or potential mother) -0.007 -0.011 -0.001 0.026 0.052 0.026 Edu (or avg edu) of mother(or potential mother) 0.095 0.155 0.095 0.108 0.196 0.121 Gender (0-female; 1-male) 0.087 0.055 0.065 0.235 0.443 0.239 Child's age in years=0 1.752*** 0.425 Child's age in years=1 1.133** 0.370 Child's age in years=2 0.764* 0.727 0.336 0.491 Child's age in years=3 0.831* 0.760 0.337 0.504 Child's age in years=4 0.743* 0.776 0.311 0.450 Number of U5 kids -0.050 -0.065 -0.041 0.217 0.320 0.219 Household is SC/ST 0.388 0.172 0.721 0.276 0.434 0.371 Household is Hindu 0.090 0.340 0.108 0.405 0.697 0.454 Child's age in months -0.035 0.019 Constant 2.236 3.691 2.521 1.237 2.022 1.280 N 578 317 261 Coefficients reported from OLS; SEs clustered at village level; matched trio and district FEs xviii Table A4: Effect of Interventions on Diarrhea Treatment (Treatment sought within 1 day) Full Sample Under 2 2-5 years b/se b/se b/se Info Only -0.023 0.043 -0.065 0.040 0.054 0.065 Info + Facilitation 0.075 0.138* 0.041 0.042 0.063 0.050 Age (or avg) of mother (or potential mother) -0.004 -0.000 -0.006 0.003 0.006 0.005 Edu (or avg edu) of mother(or potential mother) -0.012 -0.003 -0.032 0.018 0.025 0.022 gender 0.023 -0.004 -0.006 0.036 0.051 0.059 Child's age in years=0 -0.056 0.126 Child's age in years=1 -0.082 0.121 Child's age in years=2 -0.010 -0.132 0.128 0.148 Child's age in years=3 -0.039 -0.136 0.123 0.140 Child's age in years=4 -0.055 -0.176 0.127 0.142 Number of U5 kids 0.001 0.078* -0.118* 0.028 0.036 0.049 Household is SC/ST -0.036 -0.029 -0.035 0.036 0.052 0.053 Household is Hindu -0.076 -0.014 -0.140 0.057 0.091 0.085 Child's age in months 0.001 0.003 N 661 337 285 Marginal effects reported for probit; SEs clustered at household-level; matched trio FEs xix Child Immunizations Table A5: Effect of Interventions on Immunization Rates for Children Aged 12-23 months Fully vaccinated Full DPT Full Polio BCG b/se b/se b/se b/se Info Only 0.072 0.109*** 0.057 0.021 0.045 0.028 0.045 0.017 Info + Facilitation 0.118** 0.110*** 0.104* -0.04 0.044 0.03 0.043 0.022 Age (or avg) of mother (or potential mother) 0.004 0.003 0.004 0.004 0.004 0.003 0.004 0.002 Edu (or avg edu) of mother(or potential mother) 0.015 -0.022* 0.016 0.004 0.014 0.011 0.014 0.009 gender (0-female; 1-male) 0.076* 0.035 0.072* 0.034 0.032 0.021 0.032 0.02 Number of U5 kids 0.011 -0.029 0.015 0.018 0.032 0.02 0.031 0.022 Household is SC/ST -0.057 -0.035 -0.071 -0.041* 0.036 0.023 0.037 0.019 Household is Hindu -0.026 0.037 -0.035 0.033 0.055 0.038 0.052 0.038 N 1005 1005 1005 531 Marginal effects reported for probit; SEs clustered at household-level; matched trio FEs xx Maternal Care Table A6: Effect of Interventions on Maternal Care At least 3 Number of Institutional ANC visits ANC visits delivery b/se b/se b/se Info Only -0.031 0.019 0.057** 0.032 0.062 0.021 Info + Facilitation 0.032 0.070 0.056** 0.033 0.063 0.020 Age (or avg) of mother (or potential mother) -0.005* -0.008 -0.000 0.003 0.004 0.002 Edu (or avg edu) of mother(or potential mother) -0.000 0.015 0.053*** 0.009 0.016 0.007 Gender (0-female; 1-male) 0.039* 0.044 -0.017 0.018 0.037 0.014 Child's age in months -0.001 -0.002 -0.002 0.002 0.003 0.001 Number of U5 kids -0.001 0.001 -0.053*** 0.017 0.031 0.013 Household is SC/ST -0.038 -0.020 -0.012 0.023 0.044 0.018 Household is Hindu 0.029 0.023 0.015 0.039 0.069 0.025 Constant 2.314*** 0.204 N 2114 2027 2079 Marginal effects for 'At least 3 ANC visits' and 'Institutional Delivery' reported from probit; Coefficients for No. of ANC visits reported from OLS; SEs clustered at household-level; matched trio FEs xxi Awareness and Access to Nutritional Supplement Table A7: Effect of Interventions on Awareness, Receipt, and Difficulty Accessing Supplementation Receipt of Difficulty obtaining Full awareness of nutritional nutritional nutritional supplementation supplementation entitlements in past 1 mo. in past 1 yr. b/se b/se b/se Info Only 0.093** -0.062 0.028 0.030 0.037 0.031 Info + Facilitation 0.067* -0.034 -0.014 0.030 0.039 0.027 Age (or avg) of mother (or potential mother) -0.002 -0.000 -0.001 0.002 0.002 0.002 Edu (or avg edu) of mother(or potential mother) 0.012 0.006 -0.001 0.008 0.009 0.007 Child's age in years=0 -0.153** -0.223** -0.009 0.054 0.069 0.052 Child's age in years=1 -0.157** -0.196** 0.020 0.050 0.066 0.052 Child's age in years=2 -0.138** -0.190** -0.012 0.051 0.065 0.053 Child's age in years=3 -0.146** -0.210*** 0.000 0.048 0.063 0.051 Child's age in years=4 -0.152*** -0.238*** 0.016 0.046 0.068 0.050 Number of U5 kids -0.019 0.002 -0.018 0.016 0.020 0.015 Household is SC/ST 0.028 0.038 0.047** 0.022 0.023 0.017 Household is Hindu -0.049 -0.004 -0.032 0.038 0.041 0.029 Household is related to AWW -0.045 0.233*** -0.104*** 0.036 0.039 0.030 Household is related to Pradhan 0.015 0.012 -0.041 0.026 0.032 0.025 N 4213 2733 4213 Marginal effects reported for probit; SEs clustered at household-level; matched trio FEs xxii Satisfaction with AAA and VHSNC Table A8: Effect of Interventions on Satisfaction with AAA (ASHA, ANM, and AWW) ASHA ANM AWW b/se b/se b/se Info Only 0.045* -0.017 -0.026 0.018 0.022 0.022 Info + Facilitation 0.064*** 0.026 -0.007 0.019 0.024 0.024 Age (or avg) of mother (or potential mother) -0.002 0.002 -0.001 0.001 0.002 0.002 Edu (or avg edu) of mother(or potential mother) -0.006 0.013 -0.002 0.006 0.007 0.008 Gender (0-female; 1-male) 0.002 0.007 0.024 0.011 0.013 0.014 Child's age in years=0 0.146** 0.176** 0.159** 0.052 0.058 0.053 Child's age in years=1 0.140** 0.168** 0.145** 0.051 0.053 0.050 Child's age in years=2 0.131** 0.134* 0.146** 0.047 0.052 0.052 Child's age in years=3 0.080 0.116* 0.108* 0.045 0.049 0.049 Child's age in years=4 0.109* 0.123* 0.143** 0.047 0.055 0.050 Number of U5 kids 0.003 -0.010 0.009 0.017 0.013 0.017 Household is SC/ST -0.010 -0.002 0.008 0.018 0.018 0.019 - Household is Hindu -0.078** 0.099*** -0.054 0.024 0.025 0.028 Household is related to ASHA -0.001 0.024 Household is related to ANM 0.166*** 0.039 Household is related to AWW -0.052 0.033 N 4048 4098 4213 xxiii Marginal effects for Satisfaction with AAA (1- score of 4+; 0-score of 1-3) reported from probits; SEs clustered at village level; matched trio and district FEs xxiv Appendix 6 - Power Calculation Estimates Prior to Intervention In order to properly assess program impact, the research team carried out sample power calculations prior to the experiment to estimate the minimum detectable effect for the listed outcomes of interest. Table A9 provides an overview of the evaluation’s relevant power to detect changes in different outcomes. This section was included in the original SIEF proposal and included for the reader’s convenience. Table A9: Estimates for minimum detectable effect size Diarrhoea WFH Z scores U-5 Mortality incidence Power ( ) Power ( ) Power ( ) 0.8 0.9 0.8 0.9 0.8 0.9 Number of clusters 40 x 3 40 X 3 40 x 3 40 x 3 40 x 3 40 x 3 Cluster size (households) 50 50 50 50 50 50 Intra-cluster correlation 0.1 0.1 0.0175 0.0175 0.03 0.03 Significance level ( ) 0.05 0.05 0.05 0.05 0.05 0.05 Sample size 6000 6000 6000 6000 6000 6000 Minimum Detectable Effect 0.19 0.225 0.3 0.45 0.05 0.06 The key outcomes of interest (listed in Section V) include weight-for-height Z scores (WHZ), under-5 mortality rates, Incidence of diarrhoea, and duration of diarrhoea illness. Under 5 mortality For U5 mortality, intra-cluster correlation is considerably lower than that for WHZ. Analysis of National Family Health Survey (NFHS) wave 3 data from India indicates that intra cluster correlation for U-5 mortality is 0.0175 (Mann, Veble et al., [2010]). Other perinatal outcomes are reported to have even far lower ICC: for example, neonatal mortality in rural India was estimated to have ICC of 0.00099 (Pagel, Prost et al., [2011]). • With a sample of 50 children under 5 years of age in each of the 120 clusters (a total sample size of 6000) and an ICC of 0.0175, IE team will have 80 percent power to detect a 30 percent reduction in U-5 mortality, an effect size consistent with the results reported by studies in Uganda (Bjorkman & Svensson [2009]), Nepal, (Manandhar, Osrin et al., [2004]) and India (Tripathy, Nair et al., [2010]). The minimum detectable effect size at 90 percent power for this sample size of 6000 is a 45 percent reduction in U-5 mortality, which would most likely be an unrealistic level of reduction in a 12-month time period. • The IE team will aim to collect mortality data during the houselisting, which will give us population level estimates of child mortality in the 120 clusters. However, to be conservative with our sample size calculations, IE team plans to rely on data from 6000 households who will be sampled for a full household interview. xxv Weight for Height Z scores • With 40 clusters (GPs) in each arm (in a 3 arm comparison) and 20 observations per village, (sample size of 2400) and an ICC of 0.1 (Fenn, Morris et al., [2004]) the IE team has 90 percent power to detect a change of a quarter of a standard deviation in the weight-for-height Z-score. While it would not be unusual to expect this difference between the intervention arm and the control arm, a key objective of our evaluation is to estimate the difference in impact between the two intervention arms. The difference between the two intervention arms is likely to be smaller and hence the IE team has powered the evaluation to test smaller differences. • With a sample size of 6000, the evaluation has 80 percent power to detect a difference of 0.19SD in Weight-for-Height Z score between the two intervention arms with 2000 observations in each intervention arm (50 observations in each cluster). With 6000 observations, the minimum detectable effect size at 90 percent power is 0.225. The team hypothesizes that this effect size or higher is plausible over the 12 month period of follow-up; it is a conservative estimate because other interventions aimed at improving child health have found effects of 0.50 or higher over even shorter durations of time (Graff Zivin, Thirumurthy, and Goldstein [2009]; Duflo [2000]). Diarrhoea • For Diarrhoea, with 6000 observations in our sample, the evaluation has 80 percent power to detect a reduction in diarrhoea incidence (over past two weeks) from 0.20 to 0.15, assuming an ICC of 0.03. (ICC estimated from ongoing evaluation in the adjoining state of Bihar (unpublished), with data from 23,000 households in 360 clusters). The minimum detectable effect for this sample size with this ICC assumption at 90 percent power is 0.06. • Data from our ongoing evaluation in Bihar (unpublished) indicate that the average duration of diarrhoea illness in a similar rural population is 3.5 days (SD = 2.3) with ICC = 0.07. With the proposed sample of 6000 observations, the evaluation has 90 percent power to detect a change of 12 percent of one SD – a reduction of diarrhoea reduction from 3.5 to 3 days. • In Bihar, which has a comparable population as the proposed evaluation sites, 35 percent of children who had diarrhoea in the past two-week period had received ORS or Zinc. With the SA interventions, while the IE team hypothesizes that the incidence of diarrhoea will reduce, the team also expects the share of children receiving ORS (and Zinc) conditional on getting sick will increase. Assuming 20 percent of children have diarrhoea (1200 out of 6000 sampled households), and an ICC of 0.05, the evaluation has 80 percent power to detect an increase in ORS consumption rate from 35 percent to 46 percent. With 90 percent power, for this sample size, the minimum detectable effect size increases to 12.5 percent points. xxvi References 1. International Institute for Population Studies, India Fact Sheet (National Family Health Survey-4). International Institute for Population Studies, 2017. 2. International Institute for Population Sciences. India National Family Health Survey (NFHS-3), 2005- 06. Vol. 1. International Institute for Population Sciences, 2007. 3. World Health Organization. 2015. Global Health Observatory Data Repository, “Proportion of child deaths by cause, by country, 2000-2015.� http://apps.who.int/gho/data/view.main.ghe3002015-IND?lang=en 4. International Food Policy Research Institute, Global Nutrition Report 2015: Actions and Accountability to Advance Nutrition and Sustainable Development. 2015, International Food Policy Research Institute: Washington, DC. 5. World Health Organization, UNFPA, World Bank Group, and United Nations Population Division Maternal Mortality Estimation Inter-Agency Group., Maternal mortality in 1990-2015., in http://www.who.int/gho/maternal_health/countries/ind.pdf?ua=1. 2015, World Health Organization: Geneva. 6. Census of India, MMR Bulletin 2011-2013 in http://www.censusindia.gov.in/vital_statistics/mmr_bulletin_2011-13.pdf. 2011, Government of India, Office of Registrar General and Census Commissioner: New Delhi. 7. Reddya H, Pradhan M, Ghosh R, and K. A., India’s progress towards the Millennium Development Goals 4 and 5 on infant and maternal mortality. WHO South East Asia J Public Health, 2012. 1: p. 279-89. 8. Government of India, N.H.M. Maternal Health: Background. 2016 [cited 2016 September 29]. 9. Lieberman, A., Challenges for maternal health efforts. The Lancet. 388(10050): p. 1146-1147. 10. Bredenkamp, C., M. Shekar, M. Gragnolati, M. Dasgupta, and Y.-K. Lee, ICDS and persistent undernutrition. Economics & Political Weekly, 2006. 41(12): p. 25. 11. Das, J., A. Holla, A. Mohpal, and K. Muralidharan, Quality and Accountability in Healthcare Delivery: Audit Evidence from Primary Care Providers in India. American Economic Review, 2016. 106(12): p. 376599. 12. Reddy, K.S., India's Aspirations for Universal Health Coverage. New England Journal of Medicine, 2015. 373(1): p. 1-5. 13. Das, J., A. Holla, V. Das, M. Mohanan, D. Tabak, and B. Chan, In Urban And Rural India, A Standardized Patient Study Showed Low Levels Of Provider Training And Huge Quality Gaps. Health Affairs, 2012. 31(12): p. 2774-2784. 14. Das, J., A. Kwan, B. Daniels, S. Satyanarayana, R. Subbaraman, S. Bergkvist, R.K. Das, V. Das, and M. Pai, Use of standardised patients to assess quality of tuberculosis care: a pilot, cross- sectional study. The Lancet Infectious Diseases, 2015. 15(11): p. 1305-1313. xxvii 15. Das, J. and J. Hammer, Quality of Primary Care in Low-Income Countries. Annual Review of Economics, 2014: p. 41. 16. Mohanan, M., M. Vera-Hernandez, V. Das, S. Giardili, J.D. Goldhaber-Fiebert, T.L. Rabin, S.S. Raj, J.I. Schwartz, and A. Seth, The know-do gap in quality of health care for childhood diarrhea and pneumonia in rural India. JAMA Pediatrics, 2015. 169(4): p. 349-57. 17. Scott , K.W. and A.K. Jha Putting Quality on the Global Health Agenda. New England Journal of Medicine, 2014. 371(1): p. 3-5. 18. Banerjee, A.V., R. Glennerster, and E. Duflo, Putting a Band-Aid on a Corpse: Incentives for Nurses in the Indian Public Health Care System. Journal of the European Economic Association, 2008. 6(2-3): p. 487-500. 19. Mohanan, M., K. Hay, and N. Mor, Quality of Health Care in India: Challenges, Priorities, and the Road Ahead. Health Affairs, 2016. 35(10): p. 1753-1758. 20. Ringold, D., A. Holla, M. Koziol, and S. Srinivasan, Citizens and Service Delivery: Assessing the Use of Social Accountability Approaches in Human Development Sectors. 2011, Washington, DC: The World Bank. 21. Rosato, M., G. Laverack, L.H. Grabman, P. Tripathy, N. Nair, C. Mwansambo, K. Azad, J. Morrison, Z. Bhutta, H. Perry, S. Rifkin, and A. Costello, Community participation: lessons for maternal, newborn, and child health. The Lancet, 2008. 372(9642): p. 962-971. 22. Björkman, M. and J. Svensson, Power to the People: Evidence from a Randomized Field Experiment on Community-Based Monitoring in Uganda. The Quarterly Journal of Economics, 2009. 124(2): p. 735769. 23. Manandhar, D.S., D. Osrin, B.P. Shrestha, N. Mesko, J. Morrison, K.M. Tumbahangphe, S. Tamang, S. Thapa, D. Shrestha, B. Thapa, J.R. Shrestha, A. Wade, J. Borghi, H. Standing, M. Manandhar, and A.M. de L Costello, Effect of a participatory intervention with women's groups on birth outcomes in Nepal: cluster-randomised controlled trial. The Lancet, 2004. 364(9438): p. 970-979. 24. Tripathy, P., N. Nair, S. Barnett, R. Mahapatra, J. Borghi, S. Rath, S. Rath, R. Gope, D. Mahto, R. Sinha, R. Lakshminarayana, V. Patel, C. Pagel, A. Prost, and A. Costello, Effect of a participatory intervention with women's groups on birth outcomes and maternal depression in Jharkhand and Orissa, India: a cluster-randomised controlled trial. The Lancet, 2010. 375(9721): p. 1182-1192. 25. Government of India, N.H.M. Village Health Sanitation & Nutrition Committee (VHSNC). Government of India, 27 Aug. 2014, www.nhm.gov.in/communitisation/village-health- sanitation-nutrition-committee.html. 26. Samuels, F., Holmes, R., Wiseman, K., & George, T. (2010, November). Convergence: UNICEF supported Integrated Districts - Integrated District Approach in India(Rep.). Retrieved ftp://ftp.solutionexchange-un.net.in/public/decn/comm_update/res-52-101210-04.pdf 27. UNICEF. (2013). C4D: Programme Results (2008-2012)(Publication). Retrieved https://www.unicef.org/cbsc/files/C4D_Result_booklet_Final_June_2013.pdf xxviii 28. Uttar Pradesh Health Systems Strengthening Project. (2011). Project Implementation Plan(pp. 44- 48, Rep.). Lucknow: Department of Medical Health & Family Welfare - Government of Uttar Pradesh. 29. World Bank Group. (2011). Project Appraisal Document for UPHSSP(p. 45-46, Rep. No. 64425-IN). World Bank Group. 30. Ray, Arunabh. Rapid Assessment of Maternal and Child Death Review in India. Access Health International, 2017. 31. Government of India, NHM. Guidlines for Community Processes, Ministry of Health and Family Welfare, 2013. 32. Imai K, King G, Nall C. The essential role of pair matching in cluster-randomized experiments, with application to the Mexican universal health insurance evaluation. Statistical Science. 2009;24(1):29-53. 33. King G, Gakidou E, Imai K, Lakin J, Moore RT, Nall C, Ravishankar N, Vargas M, Téllez-Rojo MM, �vila JE, �vila MH. Public policy for the poor? A randomised assessment of the Mexican universal health insurance programme. The lancet. 2009 Apr 25;373(9673):1447-54. 34. M. B. Nyqvist, D. D. Walque, and J. Svensson, “Information is Power: Experimental Evidence on the Long-Run Impact of Community Based Monitoring,� Policy Research Working Papers, 2014. 35 International Institute for Population Studies, Uttar Pradesh Fact Sheet (National Family Health Survey-4). International Institute for Population Studies, 2017. 36. Jayachandran S, Pande R. Why are Indian children so short? The role of birth order and son preference. American Economic Review. 2017 Sep;107(9):2600-29. 37. Sambodhi Research and Communications. Report on Technical Support Provided for Facilitation and Activation of VHSNCs in Fatehpur. Sambodhi, 2017, pp. 9–10. 38. Sambodhi Research and Communications. Social Accountability Initiative: End of Project Report. 2017, pp. 8–9. xxix