Report No: 43344-IN India Of Taps and Toilets: Evaluating Community- Demand-Driven Projects in Rural India Social, Environment & Water Resources Management Unit Sustainable Development Department South Asia Region June 30, 2008 World Bank Acknowledgements & Contributions The study was carried out by the World Bank and RTI International in collaboration with the Government of India’s Rajiv Gandhi National Drinking Water Mission, Maharashtra State Department of Water and Sanitation and the Orissa Water and Sanitation Mission. Project units, health officials, and other stakeholders in those states were also involved. The leadership, support and guidance by government officials and project staff in Orissa and Maharashtra have been critical for progress with this work. The authors are indebted to Aurobindo Behera, Pravin Darade, S.Dey, V.S. Dhumal, P.C. Khatua, Sanjeev Kumar, Ranjan Mallick, Chittaranjan Mohanty, Lalit Mohanty, Sabyasachi Nayak, R.K. Panigrahi, Rashmi Ranjan Pattanaik, S. Rath, Ratnakar Sial, Sudhir Thakre, Surendra Tripathy and Manoj Verma for their help during various stages of the study. The World Bank formed a multidisciplinary core study team of experts in water supply and sanitation, public health, epidemiology, environment, economic valuation, survey instruments, statistical analysis, and monitoring and evaluation. Research Triangle Institute (RTI International of North Carolina USA) was appointed as principal investigators and led the design, implementation, analysis and dissemination. TNS India collected the data and Knowledge Links (New Delhi, India) conducted the community mobilization program, in collaboration with the Orissa Water and Sanitation Mission. Staff from the World Bank, the joint UNDP-World Bank Water and Sanitation Program and project implementation units in the states supported the research. The study design was reviewed by an advisory group comprising experts from the World Health Organization, U.S. Centers for Disease Control and Prevention (CDC), Agency for International Development (USAID), and health institutions in India, including the Indian Council of Medical Research (ICMR), the National Institute of Epidemiology, and Christian Medical College. Overall, the study was led by Kseniya Lvovsky (World Bank) and Subhrendu Pattanayak (RTI). Outstanding support by Jeffrey Racki, former Sector Manager; Michael Carter, former Country Director for India; Fayez Omar, Senior Manager, India; and Rachid Benmessaoud, Operations Advisor, India, who have navigated the team through several difficult junctures, have made the completion of this research project possible. The principal authors of this report are Subhrendu K. Pattanayak, Jui-Chen Yang, Sumeet R. Patil, Christine Poulos, Katherine L. Dickinson, Kseniya Lvovsky and Priti Kumar. Ben Arnold, Stibniati Atmadja, Jonathan Blitstein, Jack Colford, Catherine G. Corey, Nitish Jha, Richard Kwok, Jagdish Krishnappa, Ranjan Mallick, Purujit Praharaj, Vijaya Rao, Erin Sills and Kelly J. Wendland made important contributions to various aspects of this study. The study team is grateful to, in alphabetical order, Ghanasham Abhyankar, Junaid Ahmad, Oscar Alvarado, Sushenjit Bandopadhyay, Massee Bateman, Peter Berman, Gene Brantley, Jeremy Bray, Kanchan Chopra, Jack Colford, Maureen Cropper, Anil Deolalikar, David Evans, Doug Evans, Jeff Hammer, Michael Hanemann, Kirsten Hommann, Rachel Kaufmann, Ranjiv Khush, Eckhard Kleinau, Peter Kolsky, Soma Ghosh Maulik, Ted Miguel, Smita Misra, Christine Moe, R. R. Mohan, Jack Molyneaux, Prasanta Pattnaik, Christophe Prevost, N. V. V. Raghava, Catherine Revels, Betty Sadoulet, Kurt Schwabe, Chris Timmins, Dale Whittington, and David Zilberman for their contributions to the design and/or implementation of the study and/or interpretation of results. June 30, 2008 i The study benefited from excellent comments by participants of various seminars at the Institute of Economic Growth (Delhi), Center for Excellence in Health Promotion Economics, Government of Maharashtra, Delhi School of Economics, University of California-Berkeley, University of California-Riverside, University of Minnesota (International Economic Development conference), Cornell University (Infectious Diseases in Poor Countries conference), Emory University, Duke University, the World Bank, and a joint workshop by the Indian Council for Medical Research & the U.S. Centers for Disease Control. Finally, we would like to gratefully acknowledge the contributions of the donor countries that have supported this study: United Kingdom via the Department for International Development (DfID) Trust Fund, Netherlands via The Bank Netherlands Partnership Program (BNPP), Norway and Finland via The Trust Fund for Environmentally and Socially Sustainable Development (TFESSD) and Spain via Spanish Trust Fund for Impact Evaluation (SIEF). June 30, 2008 ii Abbreviations and Acronyms APL Above Poverty line MIS Management Information System ARI Acute Respiratory Infections NIE National Institute of Epidemiology BPL Below Poverty line NGOs Non-Governmental Organizations CDD Community-demand-driven O&M Operation & Maintenance CFU Colony forming units RCT Randomized Community Trials CLTS Community-Led Total Sanitation RTI Research Triangle Institute CMC Christian Medical College RWSS Rural Water Supply and Sanitation DALYs Disability-Adjusted Life years SC/ST Schedule caste/Schedule tribe DID Difference-in-difference TB Tuberculosis DWSM District Water and Sanitation Mission TNS Taylor Nelson Sofres GoO Government of Orissa TSC Total Sanitation Campaign ICC Intracluster Correlation Coefficient UNDP United Nations Development Program ICMR Indian Council of Medical Research USAID United States Agency for International Development IEC Information, Education, and Communication VHWSC Village Health, Water, and Sanitation Committee IHL Individual Household Latrines W&S Water and sanitation KAP Knowledge, Attitudes and Practices WHO World Health Organization LPCD Liters per capita per day WSH Water, sanitation and hygiene M&E Monitoring and Evaluation WSS Water Supply and Sanitation June 30, 2008 iii Table of Contents Executive Summary ........................................................................................................................ 1 I. Study Methods ......................................................................................................................... 2 II. Intensified IEC Campaign in Orissa....................................................................................... 2 III. Jalswarajya in Maharashtra ................................................................................................... 3 IV. Concluding observations – A Way Forward ........................................................................ 5 1. Rationale and Objectives ............................................................................................................ 9 I. Why Another Study? ............................................................................................................... 9 II. The Importance of Impact Evaluations ................................................................................ 11 III. Outputs, Outcomes, and Impacts ........................................................................................ 14 IV. Genesis ................................................................................................................................ 15 2. Study Design: Evaluation Approach ......................................................................................... 17 I. Intervention-Based Design .................................................................................................... 17 II. Treatment and Control Groups ............................................................................................. 17 A. Randomization in Orissa .................................................................................................. 18 B. Propensity Score Pre-Matching in Maharashtra............................................................... 18 III. Sample Size ......................................................................................................................... 18 IV. Multiple Data Collection Tools .......................................................................................... 18 A. Survey Design .................................................................................................................. 19 B. Supplementary Data ......................................................................................................... 19 C. Enumerator Training ........................................................................................................ 20 D. Reflections on Data Collection ........................................................................................ 20 V. Analysis Plan: DID Estimation of Intention-to-Treat Parameter ......................................... 21 3. Orissa: Intensified Information, Education and Communication (IEC) Campaign .................. 23 I. The Intervention: Intensive IEC ............................................................................................ 23 A. Logic behind Intensified IEC Campaign ......................................................................... 24 B. What the Intervention Looked Like ................................................................................. 26 II. Site Selection ........................................................................................................................ 28 III Survey Implementation ........................................................................................................ 30 June 30, 2008 iv 4. Impacts of Intensive IEC in Bhadrak, Orissa............................................................................ 32 I. Baseline Results of the Intensified IEC Campaign ............................................................... 32 A. Comparison of Treatment and Control Villages .............................................................. 32 B. Using Baseline to Inform Intervention ............................................................................. 34 II Results of the Endline Survey ............................................................................................... 35 A. Latrine Adoption .............................................................................................................. 38 B. Child Diarrhea .................................................................................................................. 39 C. Child Anthropometrics ..................................................................................................... 43 D. Household Welfare .......................................................................................................... 43 III. Summary of Findings .......................................................................................................... 44 IV. Latrine Adoption: Lessons for Moving Forward ................................................................ 44 5. Maharashtra: Jalswarajya ......................................................................................................... 47 I. The Intervention: Jalswarajya ................................................................................................ 47 II. Evaluation Approach ............................................................................................................ 49 A. Sample Selection and Propensity Score ‘Pre-Matching’ to Identify Controls................. 49 III. Survey Implementation ....................................................................................................... 56 IV. Adequacy Assessment ........................................................................................................ 57 6. Impacts of Jalswarajya, Maharashtra ....................................................................................... 59 I. Descriptive Statistics and Bivariate Analysis ........................................................................ 59 A. Respondents Characteristics ............................................................................................ 59 B. Health Outcomes .............................................................................................................. 59 C. Water Sources ................................................................................................................. 61 D. Water Quantity ................................................................................................................. 62 E. Water Quality ................................................................................................................... 63 F. Personal Sanitation ........................................................................................................... 64 G. Environmental Sanitation ................................................................................................. 65 H. Household WSH Knowledge, Attitudes and Practices (KAP) ........................................ 65 I. Welfare Outcomes ............................................................................................................. 67 II. Multivariate (Regression) Analysis with Baseline Data ...................................................... 68 A. Open Defecation as Diarrhea Risk Factor........................................................................ 68 B. Behaviors Exposed ........................................................................................................... 68 June 30, 2008 v III. Analytic Statistics: Basic Differences-In-Differences (DID) Estimator ............................. 69 A. Inputs: Activities and Resources ...................................................................................... 69 B. Outputs ............................................................................................................................. 70 C. Outcomes.......................................................................................................................... 72 D. Impacts ............................................................................................................................. 73 IV. How Much Variation is There in the Sample? ................................................................... 74 V. Analytic Statistics Continued: Sub-Intervention Impacts to Uncover Heterogeneity in Interventions ............................................................................................................................. 76 A. Water and Sanitation Schemes – Community Leaders & Key Informants ...................... 76 B. Pay as You Go .................................................................................................................. 77 C. Tap and/or Toilet Villages................................................................................................ 78 VI. Summary of Findings in Maharashtra ................................................................................ 80 7. Conclusions: Moving Forward ................................................................................................. 82 I. In Search of Impacts: Is the Tap half-Full or Half Empty? ................................................... 82 II. Improving Design of Evaluations and “M&E� of WSH Projects ........................................ 84 III. Advancing RWSS Policies and Programs: Main Lessons from the Study ......................... 87 References ................................................................................................................................... 136 June 30, 2008 vi Tables Table 1-1. Generic Model of a Water-Sanitation-Hygiene Program ............................................ 15 Table 3-1. Summary of Benefits and Costs Associated with Latrine Adoption and Sanitation Improvements ............................................................................................................................... 24 Table 3-2. Number of Households Surveyed at Baseline and Endline in Orissa ......................... 31 Table 4-1. Differences between Control and Treatment Villages Prior to the Intensified IEC Campaign in Orissa ....................................................................................................................... 33 Table 4-2. Differences in Latrine Ownership and Diarrhea Prevalence between 2005 and 2006 in Treatment and Control Villages in Orissa (all numbers rounded) ................................................ 35 Table 5-1. Propensity Score Estimation of Participation in Jalswarajya ...................................... 52 Table 5-2. Testing covariate balance across treatment and ‘matched’ control villages using secondary data ............................................................................................................................... 54 Table 5-3. Testing balance across treatment and control villages using baseline survey data ..... 55 Table 5-4. Number of Completed Surveys in the Dry Season by Study District ......................... 56 Table 5-5. Number of Completed Surveys in the Rainy Season by Study District ...................... 56 Table 6-1. DID Analysis of Sub-Groups ...................................................................................... 75 Table A2-1. Comparison of Means for Outcome Variables ......................................................... 95 Table A2-2. Treatment Effects Analyses for Impact of Intervention on IHL Uptake (marginal effects from probit regressions) .................................................................................................... 96 Table A3-1. Results of Specification Tests for Impact of Sanitation Campaign on Other Diarrhea-related Behaviors ........................................................................................................... 99 Table A3-2. Estimates of IHL Adoption on Child Health ............................................................ 99 Table A4-1. Information-Gathering Methods and Topics for the Qualitative Study in Orissa .. 100 Table A4-2. Status of Treatment Villages at the Conclusion of the Intensified IEC Campaign in Orissa, As Reported in the Knowledge Links Report and the August 2006 Community Survey ..................................................................................................................................................... 102 Table A4-3. Attitudes Inimical to the Self-Help Concept in Treatment Villages in Orissa, According to the Knowledge Links Report ................................................................................ 104 Table A5-1. Summary Results of the Adequacy Assessment in Maharashtra: Water Supply ... 108 Table A5-2. Summary Results of the Adequacy Assessment in Maharashtra: Sanitation ......... 109 Table A5-3. Summary Results of the Adequacy Assessment in Maharashtra: Hygiene Education ..................................................................................................................................................... 110 Table A5-4. Results of the Adequacy Assessment in Maharashtra: Community Participation . 111 Table A6-1. Descriptive Statistics of Respondent and Household Characteristics .................... 114 Table A6-2. Descriptive Statistics of Health Outcomes ............................................................. 116 June 30, 2008 vii Table A6-3. Descriptive Statistics of Water Supply Services .................................................... 118 Table A6-4. Descriptive Statistics of Sanitation Facilities ......................................................... 120 Table A6-5. Descriptive Statistics of Household WSH Knowledge, Attitudes and Practices (KAP)†......................................................................................................................................... 122 Table A6-6. Descriptive Statistics of Welfare Outcomes and Benefits ...................................... 124 Table A6-7. DID Analysis of Program Inputs and Outputs........................................................ 125 Table A6-8. DID Analysis of Program Outputs, Outcomes and Impacts ................................... 126 Table A6-9A. DID Analysis of Village Categorization Based on Water and Sanitation Schemes by In-District Sub-Sample and Season ....................................................................................... 128 Table A6-9B. DID Analysis of Village Categorization Based on Water and Sanitation Schemes by Across-District Sub-Sample and Season ............................................................................... 129 Table A6-10A. DID Analysis of Village Categorization Based on Receiving Jalswarajya Addendum II Funds by In-District Sub-Sample and Season ...................................................... 130 Table A6-10B. DID Analysis of Village Categorization Based on Receiving Jalswarajya Addendum II Funds by Across-District Sub-Sample and Season .............................................. 131 Table A6-11A. DID Analysis of Tap or/and Toilet Villages by In-District Sub-Sample and Season ......................................................................................................................................... 132 Table A6-11B. DID Analysis of Tap or/and Toilet Villages by Across-District Sub-Sample and Season ......................................................................................................................................... 133 Table A7-1. Summary of Difference-in-Difference (DID) Estimates for Diarrhea in Children < 5 ..................................................................................................................................................... 135 June 30, 2008 viii Figures Figure 3-1. Logic Model of the Intensified IEC Campaign in Orissa .......................................... 25 Figures 3-2A and 3-2B. Intensive IEC activities (community meeting and defecation map) in the intervention villages. ..................................................................................................................... 27 Figure 3-3. Location of Bhadrak District within Orissa, India ..................................................... 29 Figure 3-4. Location of Treatment and Control Villages in Tihidi and Chandbali Blocks, Bhadrak, Orissa ............................................................................................................................. 30 Figure 4-1. Percent Owning and Using Toilets by Intervention and Year ................................... 36 Figure 4-2. Diarrhea Prevalence among Children under 5 by Intervention and Year .................. 36 Figure 4-3. Arm Circumference in centimeters (cm) of Children under 5 by Intervention and Year ............................................................................................................................................... 37 Figure 4-4. Time Spent Walking to Defecation Site (in minutes) by Intervention and Year ....... 37 Figure 4-5. Percentage of Households with Individual Household Latrines in Treatment Villages in Orissa Between 2004 and 2007 ................................................................................................ 41 Figure 4-6. Percentage of Households with Individual Household Latrines in Control Villages in Orissa Between 2004 and 2007..................................................................................................... 42 Figure 5-1. Logic Model of the Jalswarajya Program in Maharashtra ........................................ 48 Figure 6-1. Diarrhea Prevalence among Children under 5 by Intervention, Season, and Year .... 60 Figure 6-2. Arm Circumference in centimeters (cm) of Children under 5 by Intervention, Season, and Year ........................................................................................................................................ 60 Figure 6-3. Percentage of Households Reporting Use of Private or Public Tap as Main Water Source by Intervention, Season, and Year .................................................................................... 62 Figure 6-4. Household Water Consumption in LPCD by Intervention, Season, and Year .......... 63 Figure 6-5. E. Coli Levels* on log Scale in Household Drinking Water by Intervention, Season, and Year ........................................................................................................................................ 64 Figure 6-6. Percentage of Households Reporting Use of Toilet as Main Sanitation Practice by Intervention, Season, and Year ..................................................................................................... 65 Figure 6-7. Household Coping Costs in Rupees due to Inadequate Water Supply and Sanitation Services by Intervention, Season, and Year.................................................................................. 66 Figure 6-8. Household Cost of Illness in Rupees due to Diarrhea by Intervention, Season, and Year ............................................................................................................................................... 67 June 30, 2008 ix Annex Annex 1. Power Calculations ........................................................................................................ 90 Annex 2. Impact of Intensified IEC Campaign on Adoption of Individual Household Latrines in Orissa ............................................................................................................................................ 93 Annex 3. Impact of Intensified IEC Campaign on Child Diarrhea in Orissa ............................... 97 Annex 4. The Process Evaluation of Intensified IEC Campaign in Orissa ................................ 100 Annex 5: Adequacy Assessment ................................................................................................. 106 Annex 6. Maharashtra Results Tables ........................................................................................ 114 Annex 7. Summary of methods to adjust for additional covariates in the DID estimates for diarrhea in children under the age of five ................................................................................... 134 June 30, 2008 x Executive Summary 1. The Millennium Development Goals reflect the world’s collective hope and resolve to reverse a particularly pernicious, pervasive, and persistent set of problems in much of the world: high rates of diarrhea (the number one killer of small children), insufficient water and sanitation, and seemingly unsafe and myopic behaviors. Environmental health policies related to water and sanitation (W&S) must address the usual efficiency criteria (e.g., externalities), but also significant equity concerns. Health, time, and energy costs fall disproportionately on the poor, women and children. Furthermore, there is increasing emphasis in the public health community on the deployment of multi-pronged environmental strategies (e.g., water, sanitation, and hygiene) in concert with nutritional interventions to improve child survival (Gakidou et al., 2007). Yet, to date we have few or no rigorous scientific impact evaluations showing that WSS policies are effective in delivering many of the desired outcomes (Poulos et al., 2006). Recently there have also been calls for a closer look at the effectiveness of environmental health interventions (e.g., piped water supply), as contrasted with more conventional clinical or therapeutic interventions (e.g., oral re-hydration salts, vaccination) for combating diarrheal diseases (Kremer et al., 2008). 2. In response to these calls and gaps in the literature (summarized in Chapter 1), this paper reports the findings from empirical studies of the effectiveness of water and sanitation interventions in rural India. The goal is to inform water and sanitation policy and the practice of monitoring and evaluation in the sector. We describe intervention-based evaluations of two programs in rural India. The main objective of these studies in Maharashtra and Orissa is to determine whether water, sanitation and hygiene (WSH) interventions due to the Bank-funded Jalswarajya project in Maharashtra and a government social mobilization program in Orissa impact water and sanitation outcomes in villages that participate in these projects. To the extent water and sanitation outputs change, we evaluate the extent to which child health improves. A related objective is to identify individual and contextual factors that moderate the effectiveness of interventions, such as characteristics of individuals, communities and the bio-physical environment. 3. The study is conducted in two states with different geographic, public health and socio- economic characteristics, together spanning a range of conditions in rural India: Maharashtra and Orissa. In Maharashtra, we evaluate the impacts of the Bank-funded Jalswarajya project. In Orissa, we study education and communication aspects of the Total Sanitation Campaign (TSC). The study covers an experimental evaluation of information and communication elements of TSC in Orissa (2 surveys*1000 household each); and a quasi-experimental prospective cohort study of a rural water supply and sanitation project in Maharashtra (4 surveys *9,500 household each). 4. These studies have spanned almost five years – from identification of projects, development of the scientific protocols, implementation of large scale household surveys and extensive analyses. The studies have been undertaken by the World Bank and Research Triangle Institute (RTI International), in collaboration with the Orissa Water and Sanitation Mission, Maharashtra Water Supply and Sanitation Department and other stakeholders in the state. The Bank formed a technical advisory group of leading Bank and external experts in WSS and public health, representing Indian and international health institutions to clear the study protocols and analysis plan. June 21, 2008 1 5. The secondary objectives of the study were to develop proxy indicators for routine monitoring of the health impact of water and sanitation projects. A proxy is an easy-to-measure indicator for a hard-to-measure impact. Here the goal is to identify the intermediate outcomes (e.g., coverage of toilets and taps) that are causally linked to the policy or intervention as well as the outcome. The success of the analysis in establishing such links will determine the salience of the proxy indicators. Finally, we also evaluate the broader impacts of WSH interventions on rural livelihoods, such as savings in time, materials and money invested in coping activities; improvements in convenience and privacy; and indirect benefits to caregivers (e.g., gains in work efficiency, and time and work reallocation within the household). I. Study Methods 6. Chapter 2 describes the rigorous methods of the evaluation, which included:  Intervention-based design: Data were collected from the same set of households pre- and post-intervention to assess impact. Given the seasonality of diarrhea, baseline and endline data were collected during the same month of the year.  Treatment and control groups: Data were collected from households in villages where the intervention took place (treatment), as well as in those where it did not (control). The presence of controls helps assure that before-after changes are due to the intervention and not to some external factors. Thus, a panel based difference-in-difference (DID) estimation strategy was applied in both sites to ‘sweep out’ pre-existing differences in project and control communities.  Experimental and quasi-experimental assignment. In Orissa, villages in the study area were assigned randomly to treatment or control groups. In Maharashtra, ‘pre-intervention’ propensity score matching1 was used to identify control group villages that are comparable across multiple variables with the treatment group.  Multiple data collection tools: Extensive household and community questionnaires collected information on program outputs and impacts as well as on possible confounders – factors that could account for differences between treatment and control groups, such as education, family wealth, and community services. Drinking water quality samples were also taken at the source, distribution points, and inside households. Adequacy assessments based on interviews of key informants and rapid rural appraisals complemented the quantitative data. II. Intensified IEC Campaign in Orissa 7. A community-led, people-centered latrine construction program under the Total Sanitation banner was evaluated through a pilot study in Bhadrak District, Orissa. Chapter 3 describes both the intervention and the research design used in Orissa in detail. The intervention focused on ending open defecation and improving hygiene practices through an information, 1 Propensity score matching controls for observable selection bias by ensuring that treatment and control groups are comparable in all aspects except that they have not received the intervention. This method calculates the probability (i.e., propensity score) that participants and non-participants would participate in the intervention based on a set of observed characteristics, identified by the researcher. The statistical model allows the calculation of a score for every community, and then participants and non-participants are matched according to this propensity score. June 21, 2008 2 education, and communication (IEC) campaign and social mobilization. In treatment villages, the Total Sanitation approach was intensified through the addition of several features of Community- Led Total Sanitation (CLTS), a program initially developed and successfully applied in Bangladesh. CLTS tries to increase demand for sanitation through changing people’s knowledge and attitudes without direct hardware subsidies to households. In Orissa’s sanitation campaign, basic inputs were IEC on the costs and benefits of latrine use, technical assistance and subsidies (for the very poor) for latrine construction, and social mobilization to change the community norm from open defecation to the use of individual household latrines. The mobilization campaign used several unconventional approaches to spur social change: calculation of fecal materials in the village, walk of shame, and defecation mapping of unsanitary environments. The following major findings (detailed in Chapter 4) emerged:  Latrine adoption. The sanitation campaign had a substantial impact on latrine adoption and use. At baseline in 2005, there were no villages in which latrine ownership exceeded 50%. In 2006 following the sanitation campaign, latrine ownership increased significantly in many treatment villages. In one year, latrine ownership in treatment villages increased from 6.4% to 32%. These outcomes may be under-estimates of the effect because the qualitative assessment revealed that many villages had partially constructed latrines or planned to build latrines after the monsoon season, which is when the data were collected. Reports provided by the Government of Orissa in 2007 indicate that a year after our endline data were collected, 10 of the 20 treatment villages had achieved 100% latrine ownership.  Diarrhea prevention. There is some evidence that the increase in latrine ownership may have reduced child diarrhea (potentially by as much as 30%, although this finding needs additional robustness checks.). There are few possible reasons for the lack of a clear and clean result on child diarrhea impacts. First, diarrhea rates are substantially lower in both treatment and control villages in 2006 compared to 2005, decreasing our statistical ability to identify an impact of the campaign. Second, endline data were collected very shortly after the construction of latrines, allowing little time for the effects of improved sanitation to accumulate. Finally, despite the large increase in latrine adoption, overall levels of latrine use in treatment villages were still fairly low (32%) and may still have been below a critical “threshold� at which health effects are apparent.  While the short follow-up period of this study provides a useful snapshot indicating that the campaign was largely successful in increasing latrine adoption in the short term, it will be important to gather additional data on program outputs (latrine use) and impacts (child diarrhea) over time to assess the intervention’s long-term effects and sustainability. 8. Overall, the study confirms that community-wide social mobilization and intensive IEC can be successful in achieving a higher uptake of latrines that may improve health. This strengthens the case for scaling up such approaches in rural India. It also highlights capacity building needs and the importance of examining the long-term impacts and the sustainability of projects over time. III. Jalswarajya in Maharashtra 9. The Jalswarajya project (described in detail in Chapter 5) was launched by the Government of Maharashtra with support from the World Bank to improve the state’s current WSH conditions in rural areas. The project promotes community-led services and is based on the principles of the Government of India’s Swajaldhara approach. Jalswarajya’s main objectives June 21, 2008 3 are to increase access to rural drinking water and sanitation services and institutionalize decentralized delivery of water supply and sanitation services by local governments. With resources from the state and district governments, Panchayti Raj institutions, national and local organizations, and the World Bank, village residents organize to make improvements in their water and sanitation systems, selecting the package of interventions that best meets their needs and capabilities. Villages apply to the district governments to participate in the project and are selected based on three main criteria: (1) if they have poor quality drinking water and sanitation services; (2) if they have a high proportion of disadvantaged groups among their population; and (3) if they have institutional capacity to organize themselves and carry out community activities, such as collecting fees for water supply. 10. The Jalswarajya intervention began with selection of the villages and progressed through four stages: establishment of a Village Water and Sanitation Committee, selection and planning of water-sanitation-hygiene projects, implementation of the projects, and establishment of community-run operation and maintenance systems. Each village is expected to make improvements in water, sanitation, and hygiene and to end the practice of open defecation. The government financed 90% of the improvements; the villages 10%. O&M was the communities’ responsibility. 11. Three years after project initiation, which meant somewhere between three to twelve months after completion of project activities in some communities, the key findings (which are detailed in Chapter 6) are:  Communities with significant water and sanitation improvements (irrespective of whether they were participating in the Jalswarajya program or not) witnessed both water quality improvements and health improvements.  Overall, Jalswarajya has had a moderate, but significant impact on reported use of taps and toilets. The DID estimates show a doubling of dry season tap use in intervention villages (from 17% to 30%) and a 7 percentage point increase in toilet use (and a corresponding decrease in open defecation) to 35% of the population.  There are distinct seasonal differences in knowledge and self-reported safe WSH practices, and consequently on health outcomes. In the dry season, there was some increase in safe water handling, but no corresponding improvements in the potentially more contagious rainy season. Per capita consumption of water increases in the dry season as well. In general, prevention behaviors such as hand washing and a variety of safe water handling and storage do not change.  Consequently, there was some reduction in microbial contamination in the dry season in project villages. In contrast, control villages see a greater decrease in E.coli contamination in the rainy season (compared to the project villages), presumably because of reduced prevention behaviors and increase exposure due to incomplete toilet coverage in project villages.  Importantly, diarrhea incidence fell significantly during the evaluation period in both project and control villages. This general decline may reflect overall socio-economic development in rural Maharashtra, routine and general purpose water and health programs, and the fact that diarrhea prevalence rates tend to vary from year-to-year. Against this backdrop, the limited behavioral changes in project villages explain why, on average, the study found weak or no child health impacts as measured by diarrhea in these villages compared to control villages. June 21, 2008 4  Due to lack of natural sorting in communities, an exploratory analysis of the data examined different types of RWSS outputs (water focused, sanitation focused, and water & sanitation based on 50% threshold). In villages that opted for sanitation-only, and communities that attained a modest amount of tap and toilet coverage (less than 50%) improvements in E.coli contamination and child diarrhea were similar to those in control villages during the rainy season. Villages that achieved higher coverage rates (greater than 50% coverage with taps and toilets) show improvements relative to the control villages.  The Jalswarajya households at large have experienced significant reductions in time spent walking and waiting at the main water source and for defecation in project communities, which is one component of welfare effects. There were also savings in “cost-of-illness� (e.g., out-of-pocket expenditures for treatment plus the forgone economic productivity due to missed school and work by patients and their caregivers) in the dry season, albeit real gains could be attributed to the project in the rainy season only for communities with an advanced combination of interventions (50% of toilets and taps)  Overall impacts (as measured by all health and behavioral indicators mentioned so far) are more pronounced among poor and marginalized sub-groups. For example, in the dry season BPL households in project villages experience greater reductions in coping costs compared to their counterparts in control villages because more of them get water from taps. In contract, during the rainy season, BPL and SC/ST households in project villages experienced smaller reductions in diarrhea and illness costs compared to their counterparts in control villages, due to smaller improvements in E.coli levels and water quantity. The study did not observe such differences between non-poor (APL) households in project and control villages. Nor did it find such differences between open-caste households in project and control villages. IV. Concluding observations – A Way Forward 12. Chapter 7 summarizes the findings and discusses the lessons for future WSH impact evaluations and the lessons for rural water supply and sanitation (RWSS) programs:  As revealed in both sites (and consistent with broader literature), for both epidemiological and social reasons, an individual household’s payoff to behavior change depends in part on the decisions of other households in the community. For example, in Maharashtra high- coverage interventions yield health benefits that low-coverage interventions do not seem to bear. Thus community coordination is vital for supplying an environment that is free of microbial contamination. These significant externality dimensions need to be factored in the design of the RWSS programs and policy incentives, so that market signals and government laws influence community norms which are critical to achieving meaningful gains.  The Maharashtra study also demonstrates the decreases in dry season E.coli and diarrhea in communities with significant water and sanitation improvements (i.e., more than 50% of the community using taps and toilets). Importantly, while some earlier cross-sectional studies in India suggested that health benefits are realized only if a community achieves more than 50% coverage in private taps or wells only, there is some evidence in this study on the impact of piped water from private or public taps (the latter is the main method of providing water in rural India). This suggests that water and sanitation interventions, as currently provided in rural India and aiming at community-wide coverage, are important environmental health strategies because of potential externalities in prevention of and infection from water-borne and water washed diseases. June 21, 2008 5  The “externality effects� are more significant in the rainy season than in the dry season. Furthermore, the linkages between RWSS inputs, outcomes and impacts differ across seasons. To maximize the benefits, the design of the RWSS programs should address both types of interactions.  The lack of improvements in key behaviors such as hand washing and safe water handling provides a major lesson for a program implementation. The traditional over-emphasis on the hardware (of pipes and toilets) at the cost of “software� or behavior chance communication persists. This status quo can significantly limit potential health gains.  The capacity of state government departments/project units fall short of what is needed to implement behavior change communication, for example. Thus, one efficient solution is to provide RWSS departments/units with sufficient funds to contract out the design, implementation and monitoring of behavior change communication campaigns. 13. Incentives for latrine adoption: An evaluation of an intensification of the information component of the TSC program, which drew on social marketing tools, identified several key barriers to latrine adoption, as well as factors that help to overcome these barriers, summarized below (and detailed in Chapter 4):  Knowledge about the Advantages of Latrines: Beyond Health. Interestingly, awareness of the health linkages is already good but does not, by itself, lead to changing behaviors. Prior to the sanitation program, over 90% of households cited open defecation as a cause of diarrhea, yet this knowledge alone was not enough to generate widespread latrine use. It may be that specific information about water-borne diseases and ways to reduce those, rather than general knowledge, would influence people’s choices. What appears clearly is that attitudes about the importance of privacy and dignity can play a key role in determining households’ demand for latrines.  Ability to Obtain Latrines: Beyond Subsidy. Part of the campaign’s impact was almost certainly due to its role in increasing the supply of materials, along with the technical ability to construct latrines. Both below the poverty line (BPL) and above the poverty line (APL) households were just as likely to adopt latrines, though BPL households were eligible for a subsidy. Given the strong externality effects of sanitation (and even water supply improvements), there is clearly a case for rewarding communities for achieving village-wide level of service coverage rather than just subsidizing individual households2.  Collective Action to Change Social Norms: Beyond First Impulse. It is likely that a large part of the intensified IEC Campaign’s success was owing to the emphasis placed on addressing social norms and helping households to overcome collective action problems. By targeting whole communities rather than individuals, the intensive IEC harnessed the power of social pressure to conform to accepted practices. Some villages even began to develop systems of fines or punishment for households that did not comply with the new “universal latrine use� mandates. 14. These findings provide weak evidence that coverage of taps and toilets lead to less diarrhea and more time savings and could serve as proxy indicators of an effective intervention. 2 Note that the study did not evaluate the effectiveness of true CLTS interventions, where no direct household subsidies were given to households. June 21, 2008 6 Further work needs to be done to demonstrate the strength of these proxy indicators, particularly given the short follow-up period in this study (See Chapter 7 for a discussion). 15. It is important to note that the treatment effects measured in these studies are intention-to- treat (ITT) estimates that measure the effect of a community program to improve WSH services on child health outcomes. Since the health outcomes depend not only on the community level activities, but also on individual behaviors with respect to the use of improved services, the ITT parameter, which measures the difference between mean outcomes between communities with the program (whether individuals avail themselves of improved services or not) and those communities without the program, is an underestimate of the effect of the treatment on the treated. Despite this limitation, ITT estimates are widely used in the evaluation of community- based impact evaluations and can be highly policy-relevant since nearly all programs are administered to groups made up of individuals that must choose between “adopting� the improvements or not. 16. Lessons for future evaluations:  The study shows the importance of looking beyond the conventional methods like randomized community trials (RCT) to study complex but real projects, programs and policies. It demonstrates a complementary role for experimental and quasi-experimental evaluations as we scale up interventions whose efficacy and effectiveness have been demonstrated with RCTs. Since community participation is central to these projects, random assignment to a control or treatment group is not possible. In the evaluation in Maharashtra, propensity score matching was successfully used as a substitute.  A wide variety of double-difference (DID) results suggest that the evaluated projects have improved outputs and some outcomes, but these have not as yet fully translated into impacts. While behavior change and technology diffusion (even for mundane things like taps and toilets) take time, the chain is long and complex. Where the chain is shorter and the intervention more focused (Orissa), it is possible to see some significant improvements even three to six months after the intervention. In Maharashtra, the participatory and deliberative aspects of “decentralized delivery� of many interventions make it unreasonable to expect quick and definitive impacts.  Related to this point is the fact that these estimated study impacts may be considered early indicators of long-term impacts. Future follow-up studies of these study populations would permit the estimation of long-term impacts, realized upon maturation of the interventions in these communities. While the impact estimates reported here are promising, the long-term impacts are also important to understanding the project impacts and dynamics. In general, greater patience is needed in order to better understand the impacts of large, long and complex projects like Jalswarajya.  The study is also raising a fundamental question of what exactly can be evaluated in the conditions of a very dynamic economic and social transformation in India including many rural areas. Often, the “control� communities are the sites in which similar programs done by other agencies – or different programs affecting the same health and socio-economic outcomes (e.g. programs in health, education, poverty reduction). When so much is going on, the concept of “pure controls� from careful controlled trials, becomes a meaningless concept. As with most carefully conducted rigorous evaluations (compared to those that lack control groups or pre-post measurements) the additional gains from such real world programs are bound to be small. This is particularly true for complex intervention packages that take 2-3 June 21, 2008 7 years to complete on indicators that are affected by several multi-sectoral interventions against a backdrop of rapid growth, massive change, multiple programs, and active cross- learning.  This issue is compounded by the presence of a type of spillover effect described by Ravallion (2005 and 2008), among others. These spillovers occur because external aid spent on the program understudy displaces government and other sources of aid, which may be redirected to the control communities. Thus, control communities benefit from this reallocation of funds toward activities that improve their outcomes and the treatment effect is under-estimated. Disentangling these influences is difficult, thus before launching new major evaluations, it is important to be realistic in expectations. Limiting the majority of evaluations to very specific, confined initiatives, with a well attributed outcome impact, seems a reasonable way to go.  The study also highlights and reiterates unstated commandments of all applied policy work – engaging stakeholders early in the process, communicating frequently and substantively, retaining flexibility in design and analysis, building local capacity, and securing commitment and leadership of WSS project managers. June 21, 2008 8 1. Rationale and Objectives 1.1 The Millennium Development Goals reflect the world’s collective hope and resolve to reverse a particularly pernicious, pervasive, and persistent set of problems in much of the world: high rates of diarrhea (the number one killer of small children), insufficient water and sanitation, and seemingly unsafe and myopic behaviors. Environmental health policies related to water and sanitation (W&S) must address the usual efficiency criteria (e.g., externalities), but also significant equity concerns. Health, time, and energy costs fall disproportionately on the poor, women and children. Furthermore, there is increasing emphasis in the public health community on the deployment of multi-pronged environmental strategies (e.g., water, sanitation, and hygiene) in concert with nutritional interventions to improve child survival. Yet, to date we have few or no rigorous scientific impact evaluations showing that WSS policies are effective in delivering many of the desired outcomes (Poulos et al., 2006). Recently there have also been calls for a closer look at the effectiveness of environmental health interventions (e.g., piped water supply), as contrasted with more conventional clinical or therapeutic interventions (e.g., oral re-hydration salts, vaccination) for combating diarrheal diseases (Zwane and Kremer, 2007; Kremer et al., 2008). In response to these calls, this report summarizes two studies that produce empirical evidence on the effectiveness of water and sanitation interventions in rural India and generate recommendations for water and sanitation policy as well as the practice of monitoring and evaluation in the sector. I. Why Another Study? 1.2 Research has shown that inadequate water and sanitation and unsafe hygiene behaviors are responsible for this high rate of diarrhea, which is in turn a principal contributor to high infant and child mortality. Indeed, in 2004, less than 33% of the population of India had access to improved sanitation, and, while 86% overall had access to an improved water source, there were many gaps in rural areas (UNDP, Human Development Report 2006). The government of India’s Eleventh Five Year Plan (2007-2012) has set the goal of universal access to potable drinking water in 2009, with no slip-backs by 2012, and a reduction of infant mortality to 28 per 1,000 live births in 2012 (down from 72 per 1,000 in 1998-1999) (Government of India, Planning Commission, 2007 and 2001). These targets exceed the Millennium Development Goals, which aim to halve the number of people without access to safe water and basic sanitation by 2015, as compared to 1990. 1.3 If water-sanitation-hygiene efforts are to be increased in a cost-effective fashion, more information is needed on the specific kinds of interventions that yield the best health impacts and other positive outcomes. Yet there are few good studies. A recent review of over 2,100 published studies (Fewtrell et al. 2005) and previous similar reviews (Blum and Feachem 1983; Esry and Habicht 1986) found that half of the evaluations did not meet quality standards. For example, they did not take account of baseline diarrhea rates and pre-intervention hygiene behaviors; they failed to establish control groups; and they made no attempt to consider whether non-project events or conditions might have been the cause of the impact. 1.4 An examination of water supply and quality conditions in India by McKenzie and Ray (2005) concludes that there are few peer-reviewed studies evaluating drinking water interventions in rural India. There are studies using data from large-scale national surveys, but, as pointed out below, such surveys use broad definitions that fail to take into account all relevant June 21, 2008 9 factors that affect the linkages between water-sanitation-hygiene and heath. The evaluation of projects in Orissa and Maharashtra discussed here is one of the first rigorous impact evaluations of water supply and sanitation to be carried out in India. 1.5 Consider three potential reasons for the paucity of rigorous impact evaluations in the WSS sector (Poulos et al., 2006). First, mechanisms to achieve these goals are broad and varied in terms of the types of services (water supply, water quality, sanitation, sewerage, and hygiene); the setting (urban, peri-urban, rural); and the typology of delivery (public intervention, private interventions, decentralized delivery, expansion or rehabilitation). While these complex interventions call for carefully designed evaluation studies, most previous impact evaluations have had insufficient designs for measuring program impacts and/or for measuring the full range of impacts. 1.6 Second, the breadth of effects of WSS policies, which range from greater efficiency in the sector at the national level to improved health at the individual level, raises two challenges for impact evaluation. The first is that the engineering and fiscal outputs that are tracked in a Management Information System (MIS) by many projects yield little information on the effects of the program on poverty reduction – the underlying goal of development processes. The second challenge is that assessing these broad impacts requires a thorough and thoughtful approach to study design. Most impact evaluations of WSS programs focus primarily on health or a limited set of outcomes, and therefore, do not collect enough data to evaluate other impacts such as increased educational opportunities, improved rural livelihoods, or gender equity.3 1.7 Third, decentralized and community-level projects – particularly those that are community-demand-driven (CDD) or community participation based – are an important and growing class of development projects.4 The combination of voluntary participation in self- selected interventions by communities and targeted provision by program administrators increases the difficulty of identifying an appropriate control group. Early evaluations (Sara and Katz, 1998; Isham and Kahkonen, 2002) have lacked at least one or more features of rigorous evaluations: control group, baseline and follow up measurement, specific and sensitive indicators, credible program theory, multi-level multivariate data and large samples. 1.8 It is possible to design studies to meet these challenges. First, even with complex-multi- dimensional, multi-sectoral projects, researchers with a full grasp of the details of project implementation can design a study in which various dimensions are isolated so that a better understanding of their role in bringing about health improvements may be achieved. Second, statistical techniques are available for addressing the threat of selection bias. Third, qualitative as well as quantitative information can be collected at multiple levels. Structured surveys can be used to collect quantitative data; other techniques, such as rapid rural appraisals and focus groups, can be used to collect qualitative information. The two types of information are complementary: the analysis of quantitative data is augmented by insights from qualitative process evaluations or adequacy surveys. Chapters 3 and 5 describe how the impact evaluations in Orissa and Maharashtra met these challenges, using two different research designs. 3 For reviews of rigorous evaluations of health impacts of WSS, see Fewtrell et al. (2005), Esrey et al. (1991), Esrey (1996), Curtis and Cairncross (2003). 4 Mansuri and Rao (2004) define community-demand-driven development projects as those in which communities have direct control over key project decisions, including management of investment funds. June 21, 2008 10 Box 1.1 Why Are There So Few Impact Evaluations of Water-Sanitation-Hygiene Interventions? Source: Pattanayak et al. (2007a)  “One usually needs a remarkable combination of political will, a strong commitment to transparency, and a strong ethic of accountability to conduct a well-designed evaluation.  Second, water supply and sanitation program staff may be unaware of state-of-the-art empirical program evaluation techniques and the biases in current analyses.  Third, many correctly believe that rigorous evaluations are expensive but err in considering these evaluations as ‘non-essential’ investments.  Fourth, researchers and practitioners in other policy fields have demonstrated that randomized experimental methods can be implemented in the context of small pilot programs or policies that are phased in over time. However, the randomized methods are often fundamentally non-applicable to community demand-driven projects where self-selection and self-management are essential features of the interventions.  Fifth, many water supply and sanitation project implementation cycles are short but even the ‘immediate’ impacts of the project will largely be realized after the project ends. The evaluation ‘lessons’ from these immediate and other long-term impacts will accrue to the global community, not to the evaluated project per se.� II. The Importance of Impact Evaluations 1.9 Impact evaluations provide information that can support two important policy goals: improving water-sanitation-hygiene interventions, enabling the identification of reliable proxy indicators for project monitoring, and supporting cost-effectiveness and cost-benefit analysis. 1.10 To improve water-sanitation-hygiene programs. Health impact studies may be difficult to carry out, but the information they can provide is sorely needed. Programs enabling communities to gain access to a sufficient quantity of good quality water and to adopt good sanitation and hygiene practices are assumed to have a positive impact on health. Fewtrell et al. (2005) found in a meta-analysis of 46 relevant, quality studies that five types of water-sanitation- hygiene interventions reduce the prevalence of diarrhea in children under six from 15% to 50%. (The types of water-sanitation-hygiene interventions are hygiene education, water supply and water quality improvements, sanitation, and multiple interventions.) There is evidence, however, that health impacts vary greatly depending on the specific intervention or approach used. General information gives project designers little direction. Detailed information about which particular interventions bring about the greatest health improvements will enable better programming decisions at various levels. 1.11 The results of impact evaluations can be used to inform policy design and decision- making in a number of ways. First, by measuring the effectiveness of alternative of water- sanitation-hygiene interventions, the results can be inputs to cost-effectiveness analyses these packages. The results of these cost-effectiveness analyses would inform rural water and sanitation project teams who design projects. A second use of these effectiveness measures is in June 21, 2008 11 cost-effectiveness analyses of water-sanitation-hygiene interventions in improving health relative to other preventive health interventions (e.g., vaccines). 1.12 There are also several ways in which the results can be used to shape WSH projects and policies. For instance, armed with the information on effective interventions, project teams can use the information on the effectiveness of interventions to raise awareness and promote the most effective projects. These types of promotions may improve health by influencing households’ perceptions of project benefits, and steering communities’ choices toward more effective packages. Another way to use these results in policy is to harness, modify, and design rural water supply and sanitation incentives (financial and non-financial) provided by governments encourage the adoption of the most effective water-sanitation-hygiene interventions. Finally, the findings would help to define the level and type of access to water and sanitation, the kinds of hygiene behaviors, and the specific improvements in water quality that are characteristic of health-enhancing water-sanitation-hygiene interventions. 1.13 The need for studies of sanitation is particularly acute. According to Fewtrell et al. (2005) only four studies conducted between 1970 and 2003 examined the effect of sanitation interventions on health outcomes in developing countries. Only one of the four was considered to be of good quality. Box 1.2 How Research Could Enhance Health Benefits of Water-Sanitation-Hygiene Interventions Large-scale surveys in India, including the Indian National Family Health Surveys of 1992-1993 and 1998-1999, have shown that household water supply and sanitation has a strong positive impact on the mortality rate for children under five years of age, but this is not the case for community stand posts or wells. Results were similar for diarrhea prevalence (Wang 2002; Jalan and Ravallion 2003). This is an important piece of information because community stand posts or wells are the main mechanism for water delivery in rural India. A number of factors may account for the lack of health impact. Water, clean at the source, may be contaminated in the home where it is stored. Or perhaps, not having water in the house is a disincentive for hand washing and other hygiene practices. However, it is also possible that the National Family Health Survey cannot reveal the actual health benefits from public water supplies because their categorization of water supply types is too broad. Issues related to water quality at the point of use, maintenance and proximity of stand posts, among others, are not taken into consideration. Likewise, all relevant co-factors may not have been considered. In this context, a study to measure the health impacts of rural water supply and sanitation in a rigorous manner is important and of direct operational relevance. 1.14 To enable the identification of reliable proxy indicators for project monitoring. A proxy is an easy-to-measure indicator for a hard-to-measure impact. Projects generally evaluate their work by documenting their outputs: improvements in hand washing practices, increases in water use per capita, number of people using latrines, etc. They generally do not have the resources to document improvements in well-being that the outputs are intended to bring about. If research June 21, 2008 12 can establish a reliable association between a certain output and a health impact, then success in achieving the output can be assumed to indicate success in achieving the health improvement as well. That is why it is essential for impact evaluations to provide information on what particular water-sanitation-hygiene interventions (or combination of interventions) are associated with improvements in health. Particular proxy indicators that should be examined for correlation with health impacts include: water quality at the source, water quality at the point of use, quantity of water actually used for various household needs, use of latrines, condition of stand posts, retention of knowledge from hygiene education programs, etc. 1.15 To examine cost-effectiveness. Finally, a good impact evaluation permits an estimate of the averted burden of disease in terms of disability-adjusted life years (DALYs) and the comparison of various water-sanitation-hygiene intervention scenarios. (The DALYs are the sum of the years of life lost due to premature mortality and the number of years of life lived with disability, adjusted for the average severity of disability caused by a given disease (Murray and Lopez 1997). The measure is used in the WHO Global Burden of Disease Reports.)5 1.16 To provide the information to support such goals, the evaluation studies described in this report aim to:  determine whether WSH interventions due to the Jalswarajya project in Maharashtra and intensive public sanitation programs in Orissa cause differences in water, sanitation and health outcomes in villages that participate in the project.  evaluate the extent to which improvements in water supply and personal sanitation contribute towards improvements in child health outcomes, while identifying individual and contextual factors that moderate the effectiveness of interventions  develop proxy indicators for routine monitoring by identifying intermediate outcomes that are causally linked to the intervention as well as the outcome.  measure the broader impacts of WSH interventions on rural livelihoods, such as savings in time, materials and money invested in coping activities; improvements in convenience and privacy; and indirect benefits to caregivers. 1.17 By tying the evaluation to ongoing programs and projects in rural India, we can comment on effectiveness and plausibility, but not on efficacy per se. So moving away from controlled and textbook interventions forces a consideration of a longer and messier causal chain with less than ideal program implementation (see Victora et al. [2004]). Also, the focus on community interventions, not household or individual treatments, might seem like another form of imperfection. Yet, understanding how individual health improves because of collective prevention (e.g., percent of the community using private latrines) is a fundamental question for public policy. Any study that produces credible estimates of prevention and infection externalities of these sorts can test some of the main arguments for water and sanitation interventions. 5 Project cost information was not available to the research team, and so the calculations were not attempted. However, presumably this information exists with the project management team. Whittington et al. (2008) provide an example of how we could do these calculations if the information was available. June 21, 2008 13 III. Outputs, Outcomes, and Impacts 1.18 Impact evaluation measures the impacts of the program on individuals, households, and communities, and determines whether the program caused these impacts (Baker 2000; WB-OED 2004). An impact evaluation relies on comparison with control groups using a number of statistical techniques to determine what would have happened to outputs and outcomes in the absence of the program – this is known as the counterfactual. Box 1.3 Impacts, confounders and counterfactual “To measure final impact, an impact evaluation must determine what would have happened in the absence of the program – this is known as the counterfactual. This is complicated by the fact that the counterfactual is naturally unobservable – we can never know what change would have occurred in program participants (treatment group) if the program was not implemented....Impact evaluations must therefore rely on control (or comparison) groups, as well as a number of statistical and econometric techniques to estimate this counterfactual….These tools help the analyst control for factors or events (called confounders) that are correlated with the outcomes but are not caused by the project. Confounders are correlated with the intervention and may affect the outcomes, masking the intervention’s effect. Examples of confounders… include socio-cultural behaviors (e.g., collective action to improve access to community sources), institutional factors (e.g., others programs promoted by other government departments, non-governmental, or donor organizations), biophysical characteristics (e.g., water table and geology…). Failing to account for the influence of confounders introduces a source of bias – omitted variable bias. The identification and measurement of the counterfactual, comparison, or control and the careful consideration of confounders is the primary distinguishing feature between process evaluations and impact evaluations.� (Poulos et al. 2006, pp. 3-4) 1.19 A rigorous evaluation measures the changes in indicators of the desired outcomes using data collected before and after the intervention, from treatment and control or comparison groups, and then applies various statistical analysis techniques to determine whether the intervention has caused the changes. It is concerned with measuring outputs, outcomes, and impacts, but primarily outcomes and impacts. The generic model of how a water-sanitation- hygiene program’s resources and activities eventually lead to impacts is illustrated in Table 1-1. 1.20 Many evaluations do not attempt to measure impacts but more commonly look at outputs and outcomes and assume health impacts. Oftentimes, the selection of outputs and outcomes to track is not informed by evidence that these are valid proxies or indicators of impacts. Without conducting an impact evaluation, it is impossible to determine a priori what is a good proxy and what is not. June 21, 2008 14 Table 1-1. Generic Model of a Water-Sanitation-Hygiene Program Resources Activities Outputs Outcomes Impacts Water-Sanitation Policies Downstream Effects of Policies Type of Financial, products and Changes in Changes in human, Actions and levels of behaviors, well-being social, and processes service knowledge, experienced Definition institutional carried out by under the and actions by program capital to the direct as a result of beneficiaries support intervention. control of the as a result of activities. intervention intervention. the program. providers. Information- education- New communication People use Budget. latrines. latrines. Improvements Example: Project team. campaign. Operations Sanitation Technical People’s in health, Non- and education, campaign hygiene governmental guidance. maintenance behaviors income, etc. organizations. Funds systems put improve. distributed for in place. construction. 1.21 This model of program design clarifies the relationship among all program elements and helps to keep the focus on outcomes and impacts, achievement of which is the raison-d’être of a development program. While there is relatively greater tracking of activities and outputs in most monitoring and evaluation programs and project information systems, it is not enough to just measure outcomes and impacts. If we do not see an expected result, we will not be able to explain the lack of an impact without tracking the related activities and outputs. Measurement all along the causal chain is critical. IV. Genesis 1.22 The World Bank’s support to India’s rural water and sanitation program is motivated by the Government of India’s policy objectives: to accelerate service coverage and ensure sustainability. In keeping with those objectives, the World Bank’s India rural water and sanitation strategy, which undergirds this evaluation, consists of:  Promoting demand-responsive and decentralized service delivery and an integrated approach to water, sanitation, and hygiene.  Raising awareness of sanitation and the health impacts of improved sanitation and hygiene behavior.  Establishing satisfactory monitoring and evaluation arrangements (including indicators of safe water use and hygienic practices at the household level).  Filling knowledge gaps. June 21, 2008 15 1.23 As a result of World Bank consultations with the Rajiv Gandhi National Drinking Water Mission of the Government of India and interested state governments, an agreement was forged to use two ongoing Government of India programs – the community-led rural water supply Swajaldhara program and the Total Sanitation Campaign – in two selected states for a study to assess health impacts of various rural water and sanitation service packages. The study was designed to meet the need for a rigorous evaluation of the outputs and outcomes of rural water and sanitation interventions, including their health impacts. “Rigorous� implies that the study collects primary data, uses the most appropriate longitudinal investigation methods, and controls for factors that could muddy the results: cultural habits, geographic differences, and socioeconomic characteristics of the communities in question. 1.24 The study sites were representative of rural India and presented a range of service conditions and intervention packages so that results could be generalized to other states. The two states selected were Orissa (a state with a relatively adequate water supply but low sanitation coverage) and Maharashtra (where numerous water scarce areas are in need of both water supply and sanitation improvements). Government interest and support was a principal criterion for selecting a state for the study. Other key criteria for selection include: (a) current stage/timing of the project to allow sufficient time for study/survey design and undertaking a baseline assessment; and (b) project areas and the choice of interventions/services are (more or less) representative of rural India. 1.25 A technical advisory group of leading Bank and external experts in WSS and public health, representing premier Indian and international health institutions (such as WHO, ICMR, NIE, CMC, USAID)6 was established for the purposes of this study. The detailed study protocols and analysis plan were reviewed and cleared with the advisory group for each of the two states to ensure that the methodology is sound and the results will be defensible as guidance for policy and operational decisions. A number of stakeholder consultation workshops and meetings over the course of the study were held in both states involving the state counterpart departments, state project units (in the case of Maharashtra), other interested line departments and Bank project staff. 6 WHO- World Health Organization; ICMR – Indian Council of Medical Research; NIE – National Institute of Epidemiology; CMC – Christian Medical College; USAID – United States Agency for International Development June 21, 2008 16 2. Study Design: Evaluation Approach 2.1 This chapter summarizes the key features of the study in terms of evaluation design, sampling, and data collection. Most features are common across study sites; unique features (e.g., propensity score pre-matching in Maharashtra) are summarized briefly in this chapter, and described in greater detail in the following chapter. I. Intervention-Based Design 2.2 In simple terms, the study collected baseline data from households in “treatment� and “control� villages before the project activities began and then repeated the same survey in the same households after project interventions had been completed in treatment villages. Due to the high seasonality of diarrhea, the post-intervention, or endline, survey was conducted during the same month of the year as the baseline survey. The seasonality of health impacts also mandated that the surveys be undertaken around peak levels of diarrhea, which are during the monsoon season (July-September) and, to a lesser extent, the dry season (April-June). Data from the pre- and post- intervention rounds of surveys have been compared to estimate the impact of the project activities. II. Treatment and Control Groups Salient Features 2.3 An indispensable feature of the study was The study is designed to measure impacts inclusion of both treatment and control villages. using a before-after and with-without Treatment villages are those where the interventions and DID analysis. Thus, intervention was carried out; in the control inclusion of treatment and control villages, no intervention activities took place. communities and repeat measurements are indispensable features of the study The presence of a control group helps to assure that before-after changes are owing to the In Orissa, villages are randomly assigned to intervention, not just to general statewide treatment or control communities. In improvements or some other external factor. The Maharashtra, because communities self- methods used to identify control groups select, we find controls that have the same probability of selection into the project based increased the likelihood that the treatment and on propensity score matching. control were comparable in all observable and measurable variables. Control communities in this context represent “business as usual� government activity, rather than “doing nothing� (Ravallion, 2008). This is common in many evaluations of development impacts, and not unique to the water sector. While the program being evaluated is not implemented in control communities, other programs will be implemented in those communities during the study. These may be other programs affecting the same outputs (in this case, water supply, water quality, sanitation, or hygiene programs), or they may be different types of programs that affect the outcomes and impacts (e.g., health outcomes, safe water practices, access to water sources, sanitation options) being tracked in the evaluation. Thus, controls account for many other things that happen in villages, aside from the intervention: better or worse rainfall, income growth or decline, the introduction of other programs such as opening or staffing health posts among others. This is not unique to our studies or to the water sector and implies that the evaluation measures the effect of the program over and above other activities going on in the study population. Random assignment (Orissa) and a non-random quasi-experimental design (Maharashtra) represent two ways to include treatment and control groups, as described next. June 21, 2008 17 A. Randomization in Orissa 2.3.1 It is contended that ideally villages in the study area should be randomly assigned to treatment or control groups. Randomization offers the best chance that the two groups are comparable. In Orissa, random assignment to both groups was possible because a single intervention was being implemented and because of the way the intervention was designed, as explained later. B. Propensity Score Pre-Matching in Maharashtra 2.3.2 It is far more common, however, not to use random assignment in community-driven water and sanitation projects because of ethical, equity, and political issues. Most critically, in many projects, communities themselves decide if they wish to participate or not, and it is reasonable to assume that they differ in some unknown way from those villages that opt out or never show any interest. Differences may be observable (wealth, climate) or unobservable (motivation, leadership, social capital). In such cases, some method must be used to study comparable groups while leaving the self-selection feature intact. That was the case in Maharashtra, where villages had to apply to participate in the project and were chosen according to specific criteria. A statistical method called propensity score matching was used to select control villages that “matched� the treatment villages.7 Further details on the method are presented in Chapter 6 (“Jalswarajya�). III. Sample Size 2.4 How big a sample must be depends on the expected impact size, the baseline prevalence, the number of interventions, the unit of analysis and the significance and power of the test. Making decisions on the breadth and complication of the intervention involves striking a balance between the desirable and the feasible. Power calculations for the two study sites are described in Annex 1. Thus, the study in Maharashtra requires that about 40 households with children under 5 are sampled from each of 240 villages for a total of about 10,000 households to evaluate the impact of various intervention packages. In Orissa, sufficient statistical power would be attained with 1,000 households from 40 villages, where at least 25 eligible households per village were included. 2.5 However, it is important to recognize that the numbers produced by the power calculations are mere guides and not foolproof standards for any study (randomized trials as well as non-experimental studies) because of uncertainties related to the design effects, expected impact sizes and disease prevalence rates. Placed in the context of a fixed budget for collecting and analyzing the data and therefore a fixed overall sample size, this implies that we could evaluate fewer intervention packages if our parameters were different from their assumed values (for example, if diarrhea prevalence or the realized intervention impact was lower than assumed). IV. Multiple Data Collection Tools 2.6 To assess the campaign’s impacts, we required high quality measurement of key biological, socio-economic, behavioral, cultural, and environmental indicators. Data were 7 Propensity score matching controls for observable selection bias by ensuring that treatment and control groups are comparable in all aspects except that they have not received the intervention. This method calculates the probability (i.e., propensity score) that participants and non-participants would participate in the intervention based on a set of observed characteristics, identified by the researcher. The statistical model allows the calculation of a score for every community, and then participants and non-participants are matched according to this propensity score. June 21, 2008 18 collected using household and community surveys, water quality sampling, and qualitative assessments. Quality was assured through careful design and field testing of the survey instruments, rigorous training of the field enumerators and supervisors, and checking and verification efforts in the field and at the data entry stage. Collectively, such efforts can consume as much as 9-12 months of calendar time for a study of this scale. Indicators were measured at the individual (e.g., sex, age), household (e.g., class, caste, assets, education, and quality of community water supplies and water stored in households), and community levels (e.g., roads, clinics, schools, credit, and source water quality). A. Survey Design 2.7 The technical team designed the household and Field Basics community survey questionnaires based on existing In Orissa, the sample size is 1000 survey instruments, literature reviews of water, households from 20 treatment and 20 sanitation, and hygiene studies, and advice from local control communities. One measurement advisors. Preliminary versions of the questionnaires was made in August 2005 before the were reviewed in focus group discussions with intervention, and a second measurement selected individuals, key informants, and households. was made in August 2006 after the The questionnaires were revised and pretested in the intervention. field before they were finalized. In Maharashtra, the sample size is 2.8 The household questionnaires collected data on 10,000 households from 240 villages (95 outputs, outcomes, and impacts: treatment and 145 control) in each of the four rounds. Two measurements were  Program outputs and outcomes: Water, sanitation, made before the intervention in dry or hygiene intervention carried out; latrines built (summer) and monsoon months, and two and used. after the intervention in corresponding months.  Impacts: Child health as measured principally by diarrhea prevalence among children under five. A Data were collected using: child was classified as having diarrhea if, during � Household Questionnaires the two weeks prior to the survey, a household � Community Questionnaires caretaker reported that the child had had three or � Water Quality Assessments more loose stools in a 24 hour period. Other � Process Evaluation & Adequacy impacts, such as child growth, personal benefits, Assessments and cost savings, were also measured. 2.9 In addition, data were collected on a range of covariates (or possible confounders):  Individual level: sex, age, class, caste, religion.  Household level: family size and composition, education, housing conditions, asset holdings, occupation and expenditures, services, sanitation practices, water storage and treatment practices. The endline survey was virtually identical to the baseline, except for the addition of a number of questions on respondents’ awareness of the intervention. B. Supplementary Data 2.10 The community questionnaire asked for information on community-level infrastructure, such as roads, electricity, environmental sanitation, water sources, employment opportunities, clinics and health care facilities, schools, credit availability and markets. Information was also gathered on key governmental and nongovernmental programs and local government size and June 21, 2008 19 composition. Key informants were village heads, governing council members, and members of water and sanitation committees, if they existed. 2.11 Water quality sampling: In addition to questionnaires, water samples were collected and analyzed from both community sources and household storage containers in select households. The samples were tested for total coliform and fecal coliform (E. coli) counts. 2.12 Qualitative Assessments: In addition to the quantitative information provided by baseline and endline surveys, qualitative information was collected from stakeholders to assess perceptions of the campaign’s process and progress. Results from these assessments provide additional insight into the processes and reasons underlying the impacts of the campaign that we observe. In Orissa, qualitative information was collected in August 2006 through interviews with government officials, members of the Indian NGO Knowledge Links, which facilitated the sanitation intervention, and village members in project villages. 2.13 In Maharashtra, adequacy assessments were conducted to assess how Jalswarajya was progressing. These adequacy assessments triangulated “supply-side� data (from project management information systems) with qualitative and quantitative “demand-side� information (from community participants). First, project management and monitoring and evaluation data were collected from district water and sanitation committees, and qualitative insights on the implementation process were gleaned from the district staff during this exercise. Second, rapid rural appraisals were conducted in all project villages by interviewing key informants and observing field conditions. C. Enumerator Training 2.14 With the assistance of TNS Mode, an international survey organization with substantial local experience and presence, the RTI team recruited and trained local enumerators, supervisors, and water quality personnel. All had bachelor’s degrees at a minimum and were fluent in the local language. In two four-day sessions, trainees attended classes, conducted mock interviews, and learned to randomly sample households and to navigate the questionnaire, handle refusals, etc. Before the survey got underway, a listing and mapping team identified the eligible households within the villages (those with at least one child under five years of age). 2.15 As the survey forms were filled out, they were scrutinized by trained editors before the data was entered. Validity and accuracy were maintained through a number of checks. Baseline and endline data were entered using Microsoft FoxPro-based data entry template. D. Reflections on Data Collection 2.16 A study of this scale and scope posed significant coordination challenges, four of which are noteworthy.  In Maharashtra, 210 enumerators conducted approximately 10,000 household and 242 community surveys. Over 6,000 water samples were taken and transported to the lab within 24 hours. In Orissa, the totals were 1,086 households and 40 communities. It was challenging for the enumerators to carry the voluminous (95 page) survey questionnaires, the water sample bottles, the iceboxes for storing the samples, tape measures and weighing scale, and other paraphernalia.  Because the “cohort� design requires tracking the same child and his/her family members through time, it was critical to establish and maintain a family roster and panel identification system even as enumerators, supervisors, and data-entry personnel changed over time. For June 21, 2008 20 example, names of individuals were sometimes misspelled or written illegibly, requiring the field team to re-visit households and re-establish the identity of individuals.  Water quality samples had to be transported in sterile bottles secured in ice-boxes from inaccessible villages in the interior to Pune (in Maharashtra) or Calcutta (in Orissa) within 24 hours to prevent the bacterial culture from spoiling and to maintain the integrity of the match between the water sample and the household. Typically this resembled a relay race involving motor-cycle transport to a bus station, then on to a train station, where samples were assembled, before final shipment to the testing center.  Because diarrhea is a strongly seasonal disease, it was critical to complete full enumeration of all households in a short period of time to ensure the entire sample represented the same season. So, for example, in the dry season the enumerators were racing against time to finish before the monsoon arrived, whereas in the rainy season the race was to complete the surveys before the monsoon left. Needless to say, flooding and road damage caused by the monsoon complicated the fieldwork. V. Analysis Plan: DID Estimation of Intention-to-Treat Parameter 2.17 The pre-post data collection plan allows us to use a difference-in-difference estimator (also called double-difference or first difference) and measure the “treatment effect� by comparing the treatment and control units before and after the intervention (Heckman et al., 1998). The DID estimate is the mean difference in the change in the outcome across the intervention and control groups. That is, we can difference the outcome values for the intervention and their matched control units at post-intervention levels and then subtract any pre- existing differences in outcome values: DID  {E[Y1t | p(X)]  E[Y1c | p(X)]}  {E[Y0t | p(X)]  E[Y0c | p(X)]} where Y is the outcome with subscript 1 for post-treatment and 0 for pre-treatment levels, and subscripts t and c for intervention and control unit outcomes, respectively. E is the expectations operator suggesting that this is the expected treatment effect across all treatment units (individual subscripts have been suppressed to reduce notational clutter). For analysis of the Maharashtra data, the model is conditional on the propensity score of participation, p(X), which depends on all relevant covariates (X) included in the first stage estimation. DID estimators are often implemented in a regression framework by including an interaction variable for the study condition (d) and for the treatment period (T): Yijt  �  � Zijt  � Tjt  � d jt  � Tijt * d jt  Uijt The primary coefficient of interest κ measures the pre to post change in the outcome for the affected households relative to pre to post change in the outcome for the unaffected households. To be clear, most of the analysis in this report is implemented as a multi-level model. In the case of the health outcome (diarrhea), it is the individual child under 5 years of age. In the case of water and sanitation outputs, the unit is the household. In both cases, we study how the individual or household responded to the community-level intervention. Thus, as reported, these are intention-to-treat (ITT) estimates (see below). 2.18 Essentially this approach accounts for observable differences (by matching or random assignment) as well as time-invariant unobservable differences (by differencing) between affected and unaffected households (Heckman et al. 1998). The bias due to time-variant unobservables is likely to be negligible for many indicators because we conduct the pre- June 21, 2008 21 treatment and post-treatment surveys within a short time period and because control group members are likely to be drawn from very similar villages, based on the intervention probabilities estimated by us in the sample selection stage. 2.19 This analysis approach essentially measures the effect of a community program to improve WSH services on child health outcomes. Since the health outcomes depend not only on the community level activities, but also on individual behaviors with respect to the use of improved services, the average treatment effect estimated by the model above is referred to as the ITT parameter. It is the difference between mean outcomes between communities with the program (whether individuals avail themselves of improved services or not) and those communities without the program (Galasso, Ravallion, and Salvia 2001). Since there are individuals in the treatment group who do not use improved services, this ITT estimate of the treatment effect underestimates the effect of the treatment on the treated (Galasso and Umapathi 2007). Despite this limitation, ITT estimates are widely used in the evaluation of community- based impact evaluations. Further, these estimates are more policy-relevant than estimates of the effect of the treatment on the treated since development programs are often directed at communities and the challenge of getting individuals to “adopt� the improvements is nearly always present and beyond the scope of most programs. Recognizing that household behavior change is a major on-going challenge (e.g., particular households will continue to face major constraints or simply resist), the estimator captures the broad impact of a community intervention – averaging across adopters and non-adopters in the community. Thus, this coarser statistic is a conservative estimate and as such the key policy parameter. 2.20 Critically, under two assumptions presented in Annex 3, we can compute the treatment effect from a ITT estimate for a well-defined group of households (or individuals) by using a Wald estimator. The Wald statistic is essentially a ratio of the ITT and proportion of adopters, assuming the behavior change is the main mechanism for the health outcome. For example, we might find that an education campaign has an ITT estimate of 10% in children under 5, and we might find that the same campaign induced 30% more households to adopt toilets. Thus, the Wald statistic would suggest an average treatment effect of 30% (10/0.3) reduction in diarrhea because of greater use of toilets promoted by the campaign. This approach is less helpful if there is a wide range of behaviors (e.g., taps, toilets, hand washing, and water treatment) that might impact the outcome. 2.21 These studies estimate population average treatment effect (PATE), as well as the average treatment effect by key subgroups, including poor and non-poor. While the PATE may be an inaccurate measure of individual or household level treatment effects if the sample is heterogeneous, they are policy relevant because these estimates reflect the effect that policy makers can expect when the program is implemented in similar settings. While the conditional mean treatment effects are of interest when understanding the distribution of impacts within the population, they are of limited policy relevance because WSH programs and interventions are provided to diverse populations at the community level. These programs and interventions are rarely precise enough to target specific sub-groups. June 21, 2008 22 3. Orissa: Intensified Information, Education and Communication (IEC) Campaign 3.1 Less than 10% of Orissa’s population has access to both safe water and sanitation (Ministry of Rural Development, Department of Drinking Water Supply, 2004). Orissa also has higher infant mortality and childhood diarrhea rates as compared to the rest of India, which prompted the state government to establish a multi-sectoral Infant Mortality Reduction Taskforce in 2001. (The infant mortality rate in Orissa was 87 per 1,000 live births in 2002 [Government of Orissa 2003-2004].) Improvement in sanitation will likely lower childhood diarrhea rates and associated mortality. However, evidence about the impact of increasing access to sanitation coverage is currently lacking. According to a recent review of over 2,100 published studies Fewtrell et al. (2005) only four studies conducted between 1970 and 2003 examined the effect of sanitation interventions on health outcomes in developing countries. Only one of the four was considered to be of good quality. 3.2 This chapter outlines the design and results of a rigorous evaluation of a sanitation promotion campaign in Orissa. The Department of Rural Development supported the study as a worthwhile effort to obtain evidence of the health effects of rural sanitation. An intensive pilot intervention took place in 40 rural villages in Bhadrak District, Orissa, in 2006. This intervention employed an innovative community-led, people-centered approach that attempted to motivate lasting sanitation behavior change by shifting individual attitudes and collective social norms. Several design features were incorporated to facilitate estimation of the program’s impacts. First, the study location was chosen to provide an example of a rural area with little prior access to sanitation facilities and high baseline diarrhea rates. Second, an intervention- based design was employed in which data were collected before and after the sanitation program. Third, the study compared outcomes in villages that received the sanitation intervention with outcomes in similar “control� villages, and fourth, villages were randomly assigned to these treatment and control groups. Finally, multiple data collection methods were employed to provide a comprehensive dataset on key outcomes and impacts as well as potential confounders. Each of these design features is discussed in more detail below. I. The Intervention: Intensive IEC 3.3 The intervention that was applied in the 20 treatment villages in Bhadrak District represents an intensive (and randomly assigned) version of the Total Sanitation Campaign of the Government of India. The basic elements of the intervention are shown in the Logic Model of the Program (Figure 3-1). Basic inputs are IEC on the costs and benefits of latrine use – such as health, dignity and privacy; technical assistance and targeted subsidies for latrine construction (for the poorest households); and social mobilization to change the norm from open defecation to use of individual household latrines. The inputs are designed to create demand for household latrines and to change people’s behaviors. Among key desired impacts is improvement in child health, as measured by diarrhea prevalence of children under five years of age. June 21, 2008 23 A. Logic behind Intensified IEC Campaign 3.4 The intensified IEC campaign draws many ideas from a model of Community Led Total What is intensive IEC? Sanitation (CLTS) developed by Kamal Kar in The intervention adopted several elements of Bangladesh (Kar, 2003), and subsequently the CLTS approach to promote collective employed in Indian states like Maharashtra (Sanan behavioral change to achieve total sanitation outcomes. CLTS focuses on creating demand & Moulik, 2007). The CLTS approach focuses on for sanitation after creating awareness and “empowering local people to analyze the extent need through several participatory measures. and risk of environmental pollution caused by open IEC and social mobilization are two important defecation� (Kar, 2003). CLTS seeks to reduce or components of CLTS to address: eliminate open defecation village by village � Lack of Knowledge about the through an IEC campaign and social mobilization Advantages of Latrines. leading to latrine construction and an end open � Lack of Collective Action to Change defecation by a community-defined target date. Social Norms. 3.5 The theory that underpins CLTS is that The techniques used in this study were: demand-driven sanitation programs are more � Calculation of fecal materials effective and sustainable than those that are � Walk of shame supply-driven. Demand-driven programs seek to � Defecation mapping create and intensify a demand for latrines, which District officials and NGOs helped with the households then pay for themselves, while in process and materials. Desired effect of supply-driven programs, latrines may be supplied intervention was resolution of the villages to through government largesse with no community end open defecation. contribution. Thousands of latrines supplied in this manner stand unused. Demand-driven efforts assume that behavior change strategies increase program adoption and that community empowerment increases sustainability. They seek to change the structure of costs and benefits from the end user’s perspective. 3.6 To effectively increase demand for latrines, the CLTS approach must take as a starting point an understanding of the costs and benefits of latrine use that households perceive. A review of the sanitation-related literature, along with conversations with village members and government officials in Bhadrak District, highlight a number of different factors that may influence a household’s sanitation decision. A key insight is that the decision to use a latrine involves health as well as non-health costs and benefits. Furthermore, sanitation “payoffs� can be private (i.e., independent of other households’ decisions) as well as collective. Table 3-1 presents a summary of costs and benefits broken down according to these two dimensions. Table 3-1. Summary of Benefits and Costs Associated with Latrine Adoption and Sanitation Improvements Health Non-Health BENEFIT: Avoiding exposure to BENEFIT: Reducing time spent pathogens in the act of open defecation walking to open defecation site; privacy Private COST: Increase in fecal matter near COST: Time and money spent building home from poorly constructed latrine a latrine; bad smell BENEFIT: Cleaner environment; BENEFIT: Prestige; social pressure to reduction in diarrhea and other diseases improve sanitation Collective COST: Decreased community water COST: Breaking with tradition and quality from poorly constructed latrines existing social norms June 21, 2008 24 Figure 3-1. Logic Model of the Intensified IEC Campaign in Orissa PROGRAM INPUTS PROGRAM OUTPUTS PROGRAM OUTCOMES PROGRAM IMPACTS Improvements in child health Spending on education and hygiene messages Use of IHL Number of IHL Personal benefits constructed Spending on Satisfaction with IHL physical infrastructure Broader welfare impacts MEDIATING FACTORS Knowledge Attitudes Practices Intensified IEC Campaign June 21, 2008 25 3.7 This discussion highlights a few potential barriers to latrine construction and use. In order to effectively increase demand for latrines, the intensified IEC campaign addresses each of these factors. Barriers include:  Lack of Knowledge about the Advantages of Latrines. Increased awareness of the links between open defecation and disease, along with strengthened perceptions of the non-health benefits of latrine use (e.g., privacy, convenience), may increase the perceived benefits of latrine use.  Lack of Ability to Obtain Latrines. Difficulty in obtaining materials for latrine construction, lack of technical expertise, and high prices may all contribute to high perceived costs and low demand for latrines. Addressing these supply and cost constraints is one component of an effective sanitation intervention.  Lack of Collective Action to Change Social Norms. For both social and epidemiological reasons, the benefits (and costs) to any individual household from adopting a latrine will depend in part on the adoption decisions of other households in the village. Without a mechanism to motivate collective action, individual households may not have an incentive to adopt latrines on their own. By focusing on whole communities rather than individuals and shifting social norms, the intensive IEC approach may help address this problem. B. What the Intervention Looked Like 3.8 The intervention began with a workshop for personnel from the District Water and Sanitation Mission (DWSM) and from local non-governmental organizations (NGOs) on the development of the IEC plan and training of Social Mobilization Teams that would work in the villages. Delhi-based Knowledge Links, Ltd., a company with extensive experience in Total Sanitation, conducted the training. The original intent was for the Social Mobilization Teams from DWSM to work independently in the villages; however, in Orissa most of the personnel from the DWSM were engineers who knew well the hardware of sanitation but had little familiarity with IEC or behavior change techniques. Accordingly, Knowledge Links stayed on throughout the intervention and played an active role in the villages. Social Mobilization Teams that were supposed to be made up predominantly of district personnel consisted instead of four or five people from Knowledge Links, at least one member of the DWSM, and the village-level motivators. However, the DWSM did conduct follow-up visits independently. 3.9 The first task of the Social Mobilization Teams was to assist village residents to form a Village Health, Water, and Sanitation Committee (VHWSC). The VHWSC, whose members were drawn from all walks of life (for example, the village sarpanch [chairman of the village council], aanganwadi workers [child-care providers], self-help group members, and other village leaders), was registered with the tahasil (a local administrative subdivision similar to a county) as a legal entity and the entry point for intensive IEC activities in the village. In addition, Village Production Centers were established to fabricate latrine construction materials and provide technical know-how. Normally, an NGO was the implementing agency for construction, although sometimes that role was played by the VHWSC. 3.10 The Social Mobilization Teams (with their Knowledge Links advisors) visited each village two or three times and conducted the IEC activities through focus group meetings and village gatherings. Information was provided about latrine design, cost, and the subsidy program for households below the poverty line, as well as about the health and economic costs of illness June 21, 2008 26 due to diarrhea and other water-washed diseases. Also discussed were non-health benefits of latrines, such as dignity and privacy for women. The heart of the effort, however, was the use of various unconventional techniques to persuade the village to make a firm commitment to stop open defecation and build and use individual household latrines. If one strategy did not work, another was tried during the next visit. Here are the techniques in the intensive IEC toolbox.  Calculation of fecal materials. Groups of villagers develop a core fecal count: the amount of fecal materials that accumulate in a village each day.  Walk of shame. Groups of villagers walk around the village to identify current environmental sanitation conditions.  Defecation mapping. Groups of villagers map the village and major defecation sites to reach an understanding of the spatial distribution of feces. 3.11 Obviously, these activities have a shock value and are meant to cause revulsion and spur action to change the status quo. The approach is mainly visceral, not intellectual, as traditional IEC campaigns tend to be. It is based on the premise that an intellectual understanding of the relationship between poor sanitation and diarrhea does not necessarily prompt people to build latrines, any more than knowledge of the relationship between smoking and lung cancer prompts people to put out their cigarettes for good. It takes a strong emotional response for a community to change a well-established norm. The intensive IEC project asked villages to make a formal commitment to foreswear open defecation and to name a date certain when open defecation would end. Figure 3-2A shows villagers engaged in an intensified IEC campaign meeting, and Figure 3-2B shows a “defecation mapping� exercise. Figures 3-2A and 3-2B. Intensive IEC activities (community meeting and defecation map) in the intervention villages. June 21, 2008 27 3.12 The model of latrine promoted in Orissa was a single off-pit latrine costing Rs.1,500 (the equivalent of about US $ 35). For households below the poverty line, the government provides Rs.1,200 as a subsidy, leaving the individual households to pay Rs. 300. Subsidies are administered through the implementing NGOs or VHWSCs. The commitment to end open defecation is made at the village level, while the decision to build a latrine is made by the household. Presumably social pressure is a factor in a household’s decision. In some cases, the village devised punishments for those defecating in the open – for example, their neighbors would mock them or throw stones at them. II. Site Selection 3.13 Bhadrak District, shown in Figure 3-3, was selected as the location for the study for several reasons. This district had a large number of blocks and villages in which no previous Total Sanitation Campaign-related interventions had been carried out. Further, use and maintenance of latrines was unsatisfactory in Bhadrak despite the availability of adequate water (a prerequisite for adequate sanitation). Finally, no additional water, sanitation, or hygiene programs would be carried out there during the study. 3.14 Within Bhadrak, the 40 study villages were selected from an initial sampling frame (or universe) of 1,112 villages using a multi-stage process. Two blocks, Tihidi and Chandbali, were selected within Bhadrak due to their accessibility and particularly low prior exposure to Total Sanitation campaigns. Villages with fewer than 70 or more than 500 households were excluded to ensure that those selected would be similarly rural and would have enough households with at least one child under five. To reduce the possibility that intervention activities would spill over to adjacent or nearby villages, only one village per panchayat was selected (a panchayat is a group of three to six villages linked administratively and closely connected). Likewise, contiguous villages were removed from the initial sampling frame. To facilitate the survey work, villages inaccessible by road or with political issues that could affect the survey work were excluded. From the final sample frame, 40 villages were randomly selected for inclusion in the study as shown in Figure 3-4. June 21, 2008 28 Figure 3-3. Location of Bhadrak District within Orissa, India June 21, 2008 29 Figure 3-4. Location of Treatment and Control Villages in Tihidi and Chandbali Blocks, Bhadrak, Orissa LEGEND: Treatment villages Control villages Other villages III Survey Implementation 3.15 Over 1000 household surveys and 40 community surveys were conducted in 2005 and 2006. Field Highlights The survey-takers worked in three teams, each comprising five enumerators and a supervisor. One 28 households with U5 children per field executive supervised the teams. Interviews of village selected randomly from a list of more than 200 households. households were carried out by the enumerators. In each village they randomly selected 28 households The household survey took 120 from the list prepared by the mapping team. Whenever minutes to administer; the community possible, the principal caregiver of the child under five survey took 150 minutes; and the was selected as the interviewee so that information on water quality tests, 10 minutes. child health and sanitation practices would be as Attrition between 2005 and 2006 was accurate as possible. In half of the households, the negligible (from 1086 to 1050 enumerators also collected water samples. The households). community interviews were conducted by the team supervisor, who also collected samples from up to ten in-use water sources. June 21, 2008 30 3.16 Because diarrhea is a strongly seasonal disease, it was critical to complete full enumeration of all households in a short period of time to ensure the entire sample represented the same season.. Since surveys in Orissa were timed to coincide with the rainy season, the race was to complete the surveys before the monsoon ended. Needless to say, flooding and road damage caused by the monsoon complicated the fieldwork. The household survey took on average 120 minutes to administer; the community survey, 150 minutes; and the water quality tests, 10 minutes. Table 3-2 provides an estimate of the number of completed surveys. Table 3-2. Number of Households Surveyed at Baseline and Endline in Orissa Baseline 2005 Endline 2006 Treatment group 534 521 Control group 552 529 TOTAL 1086 1050 Box 3.1 Read More about the Orissa Study This report provides an overview of the Orissa study, including its methods and main results. The following paper gives a more complete description of the details of the study’s design and baseline findings.  TVs or toilets? Design and baseline findings from a community mobilization campaign to promote toilet use in Bhadrak, Orissa by Subhrendu K. Pattanayak, Jonathan L. Blitstein, Jui-Chen Yang, Katherine L. Dickinson, Sumeet R. Patil, Christine Poulos, Purujit Praharaj, Ranjan Mallick and Kelly J. Wendland. RTI International Working Paper 06-05, August 2006. June 21, 2008 31 4. Impacts of Intensive IEC in Bhadrak, Orissa 4.1 The intervention was completed in April 2006. In its final report, Knowledge Links stated that 9 of the 20 villages resolved to end open defecation and 4 were unable to reach a consensus. Between these two extremes were 2 village that had made the decision “in principle,� and 5 that had scheduled or planned to schedule a meeting to make a final decision. I. Baseline Results of the Intensified IEC Campaign 4.2 Baseline data present a snap shot of the general characteristics of the study villages. A few observations are worth noting.  Both the treatment and control households are located in rural areas that lack access to services, such as private water connections, roads, and village dispensaries. Over half of the households are below the poverty line.  About 95% of the respondents were female, as might be expected, since the enumerators were asked to seek out the primary care giver of children under five.  A variety of water sources are available in the villages, with most households using public wells and surface water sources for their daily supply. Water quality results show that household water sources are significantly more contaminated than community water sources. Most houses do not treat their water but they do cover their stored water.  Most adults reportedly wash their hands and their children’s hands at appropriate times but rarely use soap or ash.  Waste disposal options are limited. Most people dispose of their waste directly outside the house. Likewise, there is limited access to latrines or community toilets in the villages; over 90% of the households defecate in the open. These and other practices contribute to the perception of households that their villages are dirty. Respondents rated their villages as “very dirty� (an average of 40%) or “somewhat dirty� (35%), and they expressed “complete� dissatisfaction with current sanitation practices (67%).  The majority of households (59%) say it is the family’s responsibility to improve household hygiene, health, and sanitation, with 33% saying it is the government’s responsibility. However, when the question is asked about who should pay for the costs of hygiene and sanitation improvements, the results are just the opposite: 52% say the government should pay, and 28% say the family should pay.  Diarrhea rates are high in the area with just about 30% of households having a child under five with diarrhea in the past two weeks. Most households have some knowledge of the causes, symptoms, and proper treatments for diarrhea.  Few people are involved in community activities such as sweeping streets, cleaning schools, and planting trees. A. Comparison of Treatment and Control Villages 4.3 While baseline data confirm that treatment and control villages were similar according to most observable characteristics, the data did include a few surprises. Through simple luck of the draw, treatment and control villages differed significantly on some variables, particularly latrine June 21, 2008 32 ownership. Taken together, these differences add up to worse conditions in the treatment villages. As shown in Table 4-1, in treatment villages, compared to controls, fewer households owned latrines and more of them practiced open defecation (as might reasonably be expected). Table 4-1. Differences between Control and Treatment Villages Prior to the Intensified IEC Campaign in Orissa Treatment Village Control Village Variable p-value* (N=534) (N=552) Currently have IHL 5.99% 12.68% 0.030 Respondent information Female 93.26% 93.84% 0.870 Age 29.25 29.52 0.661 Government assigned caste Open / general 35.77% 44.02% 0.293 Scheduled Castes (SC) 27.72% 26.27% 0.796 Other Backward Classes (OBC) 29.03% 23.91% 0.387 Scheduled Tribes (ST) 1.50% 0.72% 0.243 Open defection 94.94% 87.68% 0.008 Waste Safety Practices Use pots to store water 56.18% 50.18% 0.414 Use buckets/vessels to store water 68.91% 74.28% 0.435 Boil or treat drinking water 9.36% 13.04% 0.192 Adult hand washing practices: 2.98 2.86 0.578 Report of 5 critical times** Child hand washing practices: 1.42 1.43 0.709 Reported of 2 critical times** % households reported a case of diarrhea in past 2 weeks Adults 10.75% 11.23% 0.710 Children under 5 32.40% 29.24% 0.401 Household drinking water quality Count of total coliform 101 112 0.302 Count of E. coli 10.4 11.2 0.805 * Statistical tests assessed the difference between treatment conditions, measured among households nested within villages. ** The critical hand washing times for adults are before preparing food or cooking, before eating, before feeding children, after changing baby/handling child's feces, and after defecation. The critical hand washing times for children under 5 are before eating and after defecation. June 21, 2008 33 4.4 Ideally, random assignment of villages to treatment and control groups would result in a more equal distribution of characteristics across the two groups. However, the fact that we ended up with a treatment group that was initially “worse off� due to simple luck of the draw does not pose a major problem for our estimation of the sanitation intervention’s impacts since we can account for these pre-existing differences using a “difference-in-difference� estimator. In other words, rather than simply comparing sanitation levels following the campaign, we can compare changes in sanitation conditions across the two groups. These comparisons are only possible because of the care we have taken in collecting baseline data, highlighting the importance of controlling for initial conditions in estimating a program’s impacts. B. Using Baseline to Inform Intervention 4.5 Baseline findings also have implications for how the intervention should be designed and carried out. With baseline data in hand, planners can make necessary adjustments in the planned interventions. For example, baseline findings provided insights on the barriers that had to be overcome if the intensified IEC campaign was to be successful. While the majority of the households said they were dissatisfied with their current sanitation system (open defecation) and said they wanted to make changes, their practices were at odds with their stated desire for change. Few used latrines (even when they possessed them); only 11% treated or boiled their drinking water; and a mere 2% used soap or ash in hand washing. 4.6 Households’ erroneous ideas about the cost of latrines appeared to be a major constraint to improving coverage. Most households that defecate openly reported that they had no alternative because latrines were too costly. Households that said they would like to build a latrine but had not done so gave cost as an inhibiting factor. We also found that households may have over-estimated the true cost of a latrine by as much as Rs. 4,000. Furthermore, even if they had a realistic idea of what a latrine cost, many households might not have built one without financial assistance, as they believed sanitation was the government’s responsibility. 4.7 Only a quarter of the households that used a latrine noted privacy and dignity benefits, yet usage rates were higher for women and lower at night for all groups – both findings suggest that privacy is a motivating factor. Health benefits of latrine use were recognized by about half the households that used them. Similarly, awareness that open defecation is related to diarrhea was high (91% said it was a cause of diarrhea). Yet children, whose feces are especially dangerous and who are especially at risk for diarrhea, rarely or never use latrines. 4.8 Results suggested that it could be advantageous for the IEC campaign to focus on disseminating accurate information to men about the costs of latrines and government subsidies. Messages about the privacy and dignity of latrine use would likely result in more demand from women. 4.9 Finally, baseline data indicate that water and sanitation were not highly rated by households when asked what they judged to be the most important community improvements in the next ten years. Eighty percent would have liked a village health dispensary and roads were deemed most important by 59%, compared with 9% for household water supply and 7% for sanitation and hygiene. These preferences point to the difficulty of creating demand for latrines. June 21, 2008 34 II Results of the Endline Survey 4.10 Following the sanitation campaign, we find few significant changes in most of the covariates. However, looking at our outcome variables, two main findings are apparent:  The campaign had a substantial and statistically significant impact on latrine adoption and use in the villages that experienced the intensive IEC. Within these treatment villages, the percent of households owning a latrine increased from about 6% to more than 30%. In control villages, the percent of households owning latrines remained constant at about 13%. Consequently, one impact of household toilet use is the time savings (from walking to and from the open defecation site). On average, households in treatment villages save 5 minutes per person per trip compared to households in control villages.  Results also indicate that diarrhea rates fell significantly following the sanitation campaign. However, it is unclear whether or not the sanitation campaign and the resulting increase in latrine use are responsible for these reductions. A second indicator of child health, arm circumference, shows statistically significant improvements because of the campaign. Table 4-2. Differences in Latrine Ownership and Diarrhea Prevalence between 2005 and 2006 in Treatment and Control Villages in Orissa (all numbers rounded) Year Overall Treatment Control T-C DID† 2005 10% 6% 13% -7%*** % owning IHL 25%*** 2006 23% 32% 13% 19%*** % of children <5 with diarrhea 2005 26% 28% 23% 5% -6%* (2 week recall) 2006 15% 15% 16% -1% Arm circumference of children 2005 13.77 13.73 13.83 -0.10 0.24** <5 (cm) 2006 13.91 13.98 13.84 0.14 Time spent walking to place 2005 6.9 7.2 6.8 0.4 -4.6*** for defecation (minutes) 2006 4.4 0.8 5.1 -4.2*** *=significant at 10% level, **=significant at 5% level, ***significant at <1% level Difference (T-C) in 2006 minus in Difference (T-C) in 2005 June 21, 2008 35 Table 4-2 summarizes the baseline and endline data on our four main indicators: latrine ownership, child diarrhea rates, child arm circumference, and time savings. Figures 4-1, 4-2, 4-3, and 4-4 illustrate these impacts. We discuss each of these results in turn. Figure 4-1. Percent Owning and Using Toilets by Intervention and Year 40% 32% 30% 20% 13% 13% 6% 10% 0% 2005 2006 2005 2006 Treatment Villages Control Villages I indicates the 95 confidence interval. Figure 4-2. Diarrhea Prevalence among Children under 5 by Intervention and Year 35% 28% 30% 23% 25% 15% 16% 20% 15% 10% 5% 0% 2005 2006 2005 2006 Treatment Villages Control Villages I indicates the 95 confidence interval. June 21, 2008 36 Figure 4-3. Arm Circumference in centimeters (cm) of Children under 5 by Intervention and Year 16 13.73 13.98 13.83 13.84 12 8 4 0 2005 2006 2005 2006 Treatment Villages Control Villages I indicates the 95 confidence interval. Figure 4-4. Time Spent Walking to Defecation Site (in minutes) by Intervention and Year 10 7.2 8 6.8 6 5.1 4 2 0.8 0 2005 2006 2005 2006 Treatment Villages Control Villages I indicates the 95 confidence interval. June 21, 2008 37 A. Latrine Adoption 4.11 At endline, there was a substantial increase in latrine ownership in the treatment villages, with no change in the control villages. In a single year, latrine ownership among households in treatment villages increased from 6.4% to 32%. In addition to simple comparisons of mean adoption levels over time, analyses were conducted using several different statistical models to estimate the impact of the sanitation campaign on latrine uptake. These are discussed in detail in Annex 1. All of the models confirmed that the sanitation campaign had a substantial and statistically significant impact on latrine adoption. Estimated impacts ranged from a 29% to 36% increase in latrine ownership. Using a difference-in-difference estimator, which accounts for the fact that latrine use was initially lower in the treatment villages relative to the controls, it was found that the intervention increased latrine uptake by about 30%. 4.12 There may be some concern that imbalance in key variables in the baseline might invalidate the use of the DID estimator (see Ravallion, 2005)? First, we might be concerned that the baseline imbalance influences the likelihood of program placement. This was clearly not the case because villages received the treatment as a result of a randomized assignment. Second, we might worry that differences in initial coverage would have slowed or hastened uptake/adoption. Only one important variable (toilet ownership) was somewhat imbalanced: toilet ownership. This had no bearing on baseline diarrhea rates, other disease rates, water quantity, water quality, hygiene practices, demographics, opinions, knowledge, education levels, distance to public infrastructure, community participation, and civic capital. So can a small difference in toilet coverage (substantially higher only in 2 out of 20 control villages) alter adoption rates? We might suspect that it would be hard to induce change in communities that had no toilets because there must have been a reason they had few toilets after all these years (in the baseline). If this is true, our estimate is a lower bound of the true impact. We implement two robustness checks. First, we include baseline toilet ownership as a linear control in a impact evaluation model of “change� (i.e., a probit model of whether a household adopted a toilet in 2006 as a function of treatment assignment and several controls, including 2005 ownership). The treatment effect is still significant and the impact estimate does not change. Second, we follow the literature (Abadie, 2005; Hirano et al., 2003) are estimate a semi-parametric DID model, which essentially uses inverse probability weights that are function of covariates that we might be concerned about – e.g., toilet coverage. Again, the size and significance of the impact estimate is virtually identical. Thus, we are reasonably reassured that the campaign did cause at least most of the observed toilet use. 4.13 This significant increase in latrine adoption may actually be an underestimate of the campaign’s full impact. Endline data were collected within six months of the conclusion of the campaign, and the process of building latrines may still have been ongoing. Indeed, our qualitative results (discussed in more detail below) indicated that latrine construction had been put on hold when the monsoon started in June. In addition to the 157 households (142 treatment and 15 control) who said that they had built a latrine in the past year, a total of 44 households, all in treatment villages, claimed to have a partially constructed latrine, and 43 treatment households and 2 control households stated that they were planning to build a latrine within the next year. It is thus highly likely that the full impact of the campaign had not materialized as of the endline survey. 4.14 This claim is furthered substantiated by 2007 data on latrine adoption provided by the Government of Orissa. Figure 4-5 summarizes various data sources on latrine ownership in the June 21, 2008 38 40 study villages between 2004 and 2007. Data sources include surveys conducted by the Government of Orissa as well as our household surveys. While these data sources are not directly comparable, they provide a general picture of how the distribution of latrine ownership has changed in the treatment and control villages over time. At the time of our baseline in 2005, there were no villages in which latrine ownership exceeded 50%; in fact, in treatment villages the rate was below 20%. Following the sanitation campaign, latrine ownership increased significantly in many treatment villages. At endline, latrine ownership was about 20% in 12 of the 20 treatment villages, above 50% in 5, and above 80% in 3. Finally, according to the most recent data provided by the Government of India, by 2007 ten of the 20 treatment villages had achieved 100% latrine coverage. Meanwhile, there is very little change in latrine ownership across the 20 control villages.8 4.15 The purpose of the intensive IEC was to move communities from one social norm to another: from open defecation to universal latrine use. Results suggest that several treatment villages are moving in that direction, although the campaign has not yet reached its full impact. B. Child Diarrhea 4.15 Have increases in latrine ownership resulted in improved health impacts, as would be expected from an epidemiologic model that links open defecation to diseases like diarrhea? Answering this question is more difficult than measuring the effect of the intensive IEC more broadly because, unlike the sanitation campaign, which was assigned randomly to half of the villages, purchase of a latrine is a matter of choice. Households decide whether or not to build and use a latrine, and it is possible that factors influencing this decision may also be correlated with diarrhea outcomes. For example, if households that adopt latrines are more health and sanitation-conscious, we might expect that they would have lower rates of diarrhea even without latrines. A number of statistical models can be used to deal with this potential bias. They are explained in detail in Annex 3. 4.16 Across the different models used, the estimated effect of latrine adoption and use on childhood diarrhea outcomes is consistently negative: adoption may have decreased diarrhea rates. The estimated effect is significant in two cases. As suggested in the theoretical model, the ITT will naturally underestimate the impact (and serve as a lower bound) because it includes the impacts (or lack of impacts) in the sample of non-adopters. The IV method is prone to identify insignificant coefficients because it is a two stage model employing a noisy imputed variable. Thus, we are willing to use a weaker than optimal result to infer that there is some evidence of 8 There are some measurement issues with one control village, Agiria, which displays relatively high and increasing levels of latrine ownership. As previously mentioned, the selection process was intended to exclude from the sample villages with any prior exposure to the Government of Orissa’s Total Sanitation Campaign. Although Agiria was not listed as one of the villages where TSC had been implemented, a visit to this village in 2006 and conversations with village leaders indicated that some sort of government sanitation campaign had been conducted in the village prior to the 2005 round of household surveys conducted by this study. As part of this intervention, households were reportedly provided latrine construction materials (rings and a pan), and were required to pay a nominal fee toward labor charges. It is unclear why this village did not appear on the list of TSC-implemented villages provided to the study team during the initial selection of villages. To address the concerns that this village does not represent a valid “control,� subsequent treatment effects analyses were replicated on samples excluding Agiria. Since these results are virtually identical to results on the full sample of villages, it does not appear that the inclusion of this village substantially “contaminates� the results. June 21, 2008 39 the campaign on diarrhea outcomes. Interestingly, the Wald statistic (described in chapter 2) suggests an average treatment effect of 30% (10/0.3) reduction in diarrhea because of greater use of toilets promoted by the campaign. This impact is equivalent to the 31% impact estimated from the IV model. 4.17 There are a few factors which may contribute to our inability to identify a much larger or cleaner impact of latrine adoption on child diarrhea given the available data. First, overall diarrhea rates were lower in 2006 than they were in 2005 in both treatment and control villages. Between the two survey rounds, the rate dropped over 10% (from 26% to 15%) for children under five and 15% (from 31% to 16%) for those under three. It appears that some external factor (perhaps the strength of the monsoon around the time of the survey) led to a lower overall diarrhea rate across both treatment and control villages in 2006. The large impact of this external factor makes it more difficult to identify any effect of latrine use on diarrhea rates. Second, the endline survey was conducted only a few months after the conclusion of the campaign, allowing little time for the effects of improved sanitation to accumulate. Further, even in treatment villages in 2006, the average percentage of households per village with latrines was still fairly low at 32%. Open defecation, with its attendant health risks, is still the dominant practice throughout the study area. It is possible that a critical mass of households must switch to using latrines before the fecal load in the environment is low enough to have an impact on diarrhea rates. If latrine use has indeed increased since the collection of our endline data, as the 2007 GoO data suggest, collecting follow-up data on both latrine use and child diarrhea in a future round of surveys would help to resolve some of these questions. June 21, 2008 40 Figure 4-5. Percentage of Households with Individual Household Latrines in Treatment Villages in Orissa Between 2004 and 2007 2004 GoO Data Percent of Households Owning a Latrine, 2005 HH Survey Treatment Villages 2006 HH Survey 2007 GoO Data 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% as edi r ad la ha la r a N ur r ou lipu ah r s uk r M pur hi B a B ur B ia ba r ap la P ahi pu pu ap pu am ga Ta kha ira u uk ad pa p sa un ad bo as av rp Ta nd B mp ra hu ub ha as jna m a ik a s ip im ia hi eg pr m ua at ba an ar du hi ad al hr d al ad an a ra A H B o la R B B S G B D S June 21, 2008 41 Figure 4-6. Percentage of Households with Individual Household Latrines in Control Villages in Orissa Between 2004 and 2007 2004 GoO Data Percent of Households Owning a Latrine, 2005 HH Survey Control Villages 2006 HH Survey 2007 GoO Data 100% 90% 80% 70% 60% 50% 40% 30% 20% 10% 0% an ana ua ur a r ur a D uti la ati nb r a B a Te iut i an pu ap r ri a B u en ur hu i nl as ali dh rju pu B ah rg pu id nd jp h ah gp ua br at Ja up ip nc lig ul gi ar an Or at ra ar in is du hi na in B in ar nt A G g dh S ai al eu ga m ya as H A Ja J H N A M S June 21, 2008 42 C. Child Anthropometrics 4.19 We also considered the impacts on the Summary of Intensive IEC Impacts arm circumference of children, widely acknowledged as a reliable indicator of child Endline Results show that the Campaign had a substantial and statistically significant health and nutritional status (Alderman, 2000). impact on latrine adoption and use. Additionally, because this indicator is a continuous variable, our sample of 2000 On average, households in treatment villages observations is powered to detect a statistically save 5 minutes per person per trip compared significant impact, compared to a binary to households in control villages, which indicator such as child diarrhea. The ITT translates to saving of about $9.50 per month per household, a very significant amount to analyses discussed in the previous sub-section Indian households. are also applied to the arm circumference of children under the age of 5 (measured in Diarrhea rates fell significantly following the centimeters). intensive IEC, but it is unclear whether this effect is due to latrine use and the intensified 4.20 In terms of raw statistics, on average IEC campaign. children in the treatment villages had arm Arm circumference of children in treatment circumference that measured 0.1 cm less than villages is statistically significantly higher children in the control villages in the baseline than that in control villages. survey in 2005. This result was reversed by the time of our follow-up survey in 2006. Children The process and impact evaluations can help us understand barriers to latrine adoption. in treatment villages recorded arm circumference � lack of knowledge of the “germ theory� that measured 0.14 cm greater than their is not the most important barrier. counterparts in control villages. It is no surprise � know-how, availability of materials, and then that an intention-to-treat analysis using a costs loomed large as barriers. double-difference estimator (see Annex 2) shows � the power of social pressure to conform a statistically significant treatment effect of 0.25 with accepted practices could be cm in children under 5. This translates into a 2% powerful motivator. gain in terms of child anthropometrics. D. Household Welfare 4.21 Child diarrhea imposes significant coping and averting costs on households. We present some ‘reduced form’ assessment of this by comparing means across treatment and control households across 5 variables – days that the child was unproductive, nights spent in hospital, overall medical costs (including on medicines and doctor fees), work days lost by primary care giver, and the time spent walking to the open defecation site. We cannot detect a statistically significant impact of the toilet use on any of these variables – except the averting costs avoided, expressed in terms of time spent walking one way for defecation. A quick-&-dirty benefit assessment can help put the estimated one-way time savings of 4.5 minutes per person in perspective. The average daily wage in rural Bhadrak is about $1/day and the typical family has 7 members (Pattanayak et al., 2006). If we assume that the time savings is valued at 30% of the daily wage, then benefits amount to $0.05 per day per person or about $9.50 per month per June 21, 2008 43 household.9 Given that such a household spends (and earns) about $50 per month, these calculations indicate that latrine use can significantly improve economic welfare. III. Summary of Findings 4.22 The key conclusions from this study can be summarized as follows (see also Box 4.1 for further reading about the Orissa study):  The pilot social mobilization campaign conducted by the Orissa Water and Sanitation Mission and Knowledge Links demonstrated very impressive results in increasing latrine coverage, pointing to the potential of the intensive IEC approach in the Orissa context.  The study confirms that intensive IEC combined with community-wide social-mobilization and targeted financial assistance can be successful in achieving behavior change. In the treatment communities, latrine coverage increased from about 5 % to about 30% within six months of the campaign’s conclusion, and indicators suggest that coverage has continued to increase since the completion of our endline surveys.  There is some evidence that child diarrhea rates declined (and anthropometrics improved) as a result of this increase in latrine use, and we think it is likely that this effect will become more substantial as latrine construction continues and the habit of latrine use becomes engrained.  A benefit of the short period of time between the end of the intervention and the endline study is that it permits a rapid identification of the short-term impacts and gives us an initial sense of whether or not the intervention worked. However, it is also important to monitor outcomes over a longer period. Many impacts will be apparent only if latrine use is maintained, and hopefully, increased. 4.23 Based on the evaluation results, the Government of Orissa intends to expand the social mobilization approach piloted in the study to other areas of Orissa. Follow-on data collection, including additional questions aimed at understanding the reasons for changes (or no changes) in hygienic behaviors and sustainability of the impact, will help answer key policy questions and will inform the next steps in sanitation promotion in Orissa. IV. Latrine Adoption: Lessons for Moving Forward 4.24 In addition to the impact evaluation results, which tell us the quantitative effects of the intensified IEC campaign on latrine adoption and health outcomes, our qualitative data can shed more light on the processes generating these results and assists us to interpret them better. Beginning in early August 2006, about four months after the completion of the intensive IEC, we conducted a series of discussions with major stake holders – Knowledge Links, DWSM personnel, and village-level leaders and individual household members – to get their views on the conduct and outputs of the intervention (see Annex 3). 4.25 The qualitative analyses highlight the fact that while the sanitation campaign was successful in achieving a large increase in latrine construction on average across the study villages, there was significant variation both within and across villages in how the campaign was 9 See Pattanayak et al. (2005b) for a discussion of how to value time inputs, including a review of the drinking water literature. June 21, 2008 44 perceived and responded to. While it is hard to identify a single factor or set of factors that caused the campaign to be more successful in some villages, the level of trust established between village members and other campaign actors, particularly the implementing NGOs, appears to be an important factor. 4.26 These results also have implications for the longer term impacts and sustainability of the intensified IEC campaign that was conducted in Bhadrak. On the positive side, many stakeholders at all levels, from Knowledge Links and DWSM to individual village members, recognized the importance of improved sanitation and expressed a commitment to working toward better sanitation outcomes. While some village members expressed the opinion that open defecation was a long-standing and acceptable practice, the majority of respondents expressed a desire to build and use latrines. Thus, we find some evidence that social norms in these villages are shifting, an important step in the process of sustainable behavior change. 4.27 Discussions with project stakeholders also uncover a few areas of concern, however. The first problem identified by all three groups had to do with the subsidy scheme. Village members often felt that the subsidies were insufficient, DWSM personnel highlighted the fact that the subsidy scheme may have created an incentive for NGOs to “cut corners� and produce lower quality latrines, and Knowledge Links was concerned that subsidies in general defeat the sense of self-reliance that the intensified IEC campaign tries to promote. Clearly, this is not an easy issue to resolve, as eliminating subsidies entirely may address Knowledge Links’ concerns and potentially lead to better incentives for NGOs, while being an unacceptable alternative for poor households that have come to rely on these financial supports. Nonetheless, finding a way to address some of the issues created by the subsidy scheme may be important for the long-term sustainability of this intervention. Furthermore, in scaling up the intensive IEC approach to different areas in Bhadrak and Orissa, the issue of generating local capacity to carry out social mobilization must be addressed. 4.28 The logic model for the intensified IEC campaign identified several key barriers to latrine adoption, each of which was addressed through program inputs. Stakeholder discussions helped confirm the barriers to latrine adoption. What does the analysis of the baseline and endline results, along with the qualitative analysis, have to say about overcoming these barriers?  Knowledge about the Advantages of Latrines: Beyond Health. It is assumed that knowledge of the health as well as the non-health benefits of latrines should increase demand for them. However, lack of knowledge of the “germ theory� linking open defecation to diarrhea is not the most important barrier to latrine adoption. Prior to the sanitation program, over 90% of households cited open defecation as a cause of diarrhea, yet this knowledge alone was not enough to generate widespread latrine use. It may be that specific information about water- borne diseases (infectious people, vectors, parasites, and pathways), rather than general knowledge, would influence people’s choices. Furthermore, it does appear that attitudes about the importance of privacy and dignity can play a key role in determining households’ demand for latrines.  Ability to Obtain Latrines: Beyond Subsidy. Ability (defined to include wealth, ability to pay, technical know how) can be increased through subsidized labor and materials and technical assistance. The study showed that know-how, availability of materials, and costs loomed large as barriers. Part of the campaign’s impact was almost certainly due to its role in increasing the supply of materials, along with the technical ability to construct latrines. It also June 21, 2008 45 appears that the existence of subsidies might have increased the uptake of latrines. However, other Total Sanitation campaigns in Orissa, which also offer subsidies, have not equally good results, and the qualitative results highlight the fact that the particular subsidy scheme employed in Bhadrak created problems of its own. These problems, along with concerns about the quality of latrines being constructed under this campaign, should be addressed.  Collective Action to Change Social Norms: Beyond First Impulse. It is likely that a large part of the IEC campaign’s success was owed to the emphasis placed on addressing social norms and helping households to overcome collective action problems. By targeting whole communities rather than individuals, the intensified IEC campaign harnessed the power of social pressure to conform with accepted practices. As noted in the Knowledge Links report, some villages even began to develop systems of fines or punishment for households that did not comply with the new “universal latrine use� mandates. Over the longer term, villages that succeed in establishing such systems, either formally or informally, stand a better chance of ensuring that open defecation becomes a thing of the past. Box 4.1 Read More about the Orissa Study This report provides an overview of the Orissa study, including its methods and main results. The following are references to additional papers and reports that provide more detail on this study.  Nature's Call: Can a (randomized) social mobilization campaign lead households to use toilets and reduce diarrhea? by Subhrendu K. Pattanayak, Katherine L. Dickinson, Jui- Chen Yang, Sumeet R. Patil, Purujit Praharaj, Ranjan Mallick, and Christine Poulos. RTI International Working Paper 07-02, March 2007.  Open-sky latrines: Understanding the role of social interactions in households’ decisions to adopt latrines in Orissa, India by Katherine L. Dickinson (with Subhrendu K. Pattanayak). Ph.D. dissertation chapter. Duke University. May, 2008. June 21, 2008 46 5. Maharashtra: Jalswarajya 5.1 The western Indian state of Maharashtra is among the largest Indian states, with a population of approximately 100 million living in 44,000 villages. The state is also among the most developed and prosperous in India with a variety of economic activities, relatively high literacy and per capita income, and only about half the population engaged in agriculture. 5.2 Statewide, the infant mortality rate is 44 per 1,000 live births and for children under five it is 58 per 1,000. The picture is different in rural areas. There the infant mortality rate is 51 per 1,000, and, for children under five, it is 68 per 1,000. According to the 1998-1999 National Family Health Survey, 23% of children under three suffer from diarrhea. In rural areas, according to the same survey, 85% have no sanitation, only 23% have a household water connection, and there is little or no treatment of water in the home. We hypothesize that these poor water and sanitation conditions contribute to the high rate of water-related diseases such as diarrhea and consequent socio-economic outcomes. I. The Intervention: Jalswarajya 5.3 Jalswarajya’s main objectives are to increase access to rural drinking water and sanitation services, institutionalize decentralized delivery of water supply and sanitation services by local governments, and improve rural livelihoods. With resources from the state and district governments, Panchayti Raj institutions, national and local organizations, and the World Bank, village residents organize to make improvements in their water and sanitation systems, selecting the package of interventions that best meets their needs and capabilities. Villages apply to the state government to participate in the project and are selected based on three main criteria; they have poor quality drinking water and sanitation services and a high proportion of disadvantaged groups among their population, but at the same time they have sufficient institutional capacity to organize themselves and carry out community activities, such as collecting fees for water supply. 5.4 The flow of project resources, outputs, outcomes, and impacts is shown in the logic model (Figure 5-1). The assumption to be tested in this evaluation is whether the program outputs – which are the water-sanitation-hygiene packages – will bring about improvements in child health, as measured principally by the prevalence of diarrhea among children under five. The evaluation is designed to rule out the impact that mediating and intervening factors could have on diarrhea prevalence so that the true impact of project activities can be assessed. Of primary interest in determining impact are the program outcomes – improved water quantity and quality, more latrines to reduce open defecation, improved hand washing and water handling, etc. – and measures of child health – diarrhea prevalence and anthropometrics. 5.5 Jalswarajya is being implemented by the Government of Maharashtra from 2003 to 2009 in approximately 2,800 villages in 26 of the state’s 33 districts. This very extensive effort has been designed to address the shortcomings of previous programs. Four principles guide the program – all highly correlated with sustainability:  It is community-demand driven. Villages must apply to participate.  Communities must share the cost of projects: 10% of capital costs and 100% of operation and maintenance costs. June 21, 2008 47 Figure 5-1. Logic Model of the Jalswarajya Program in Maharashtra PROGRAM GOALS Decentralized RWSS delivery HYPOTHESIZED CAUSALITY Improved rural livelihoods PROGRAM OUTPUTS PROGRAM OUTCOMES Piped water Water quantity PROGRAM RESOURCES PROGRAM ELEMENTS (liters per capita per day) IMPACTS Panchayti Raj institutions Community mobilization Individual household latrines In-house water quality Cost of illness (child diarrhea) National NGOs and local SOs Pilot projects Drains and garbage dumps Safe water management Savings in coping costs WB funding Village planning Health and hygiene training Improvements in handwashing Source water quality TARGET POPULATION MEDIATING FACTORS INTERVENING FACTORS Households in 9 districts in Physiographic Regional epidemics Maharashtra Environmental WSH initiatives by others Socio-cultural Biological and SES factors Institutional June 21, 2008 48  Decision-making is decentralized – made by the community not by any government or political institution.  Capacity building of local institutions is a focus – not only traditional institutions (for example the Gram Panchayats [GP] ), but also new ones (Village Water and Sanitation Committees and Social Audit Committees). 5.6 Jalswarajya is to be implemented in four phases (following the pilot phase), each with two batches of villages. The pilot phase comprised 30 villages in 3 districts; Phase I, Batch 1, comprised 225 villages in 9 districts; the remaining 17 districts will be covered in subsequent phases and batches until the target number of 2,800 villages is met. The villages included in the study are a subset of the Phase I villages. There are four districts in the study area: Buldana, Nashik, Osmanabad, and Sangli (Figure2). The study total is 242 villages, with approximately 50 households per village. 5.7 The intervention begins with selection of the villages and then progresses through three additional stages:  Preplanning and community mobilization, including the establishment of a Village Water and Sanitation Committee.  Selection and planning of project activities, subject to review and approval Implementation of the projects, with due attention to procurement, construction, and financial management.  Establishment of ongoing, continuous operation and maintenance procedures and systems by the Village Water and Sanitation Committee. 5.8 It is up to each village to customize its package of interventions, but the packages tend to fall naturally into several clusters. In practical terms, each community is expected to make improvements in all three basic components: water, sanitation, and hygiene. Communities were also expected to end the practice of open defecation. II. Evaluation Approach A. Sample Selection and Propensity Score ‘Pre-Matching’ to Identify Controls 5.9 A key feature of the study was inclusion of both treatment and control villages. Treatment villages are those where the intervention was carried out; in the control villages, no intervention activities took place. The presence of a control group helps to assure that before- after changes are owing to the intervention, not just to general statewide improvements or some other external factors. Ideally, the two groups would be comparable in all observable and measurable variables. Random assignment (Pattanayak et al., 2006) and a non-random quasi- experimental design (i.e., this case study in Maharashtra) represent two ways to include treatment and control groups, as described next. 5.10 It is far more common, however, not to use random assignment in community-driven water and sanitation projects because of ethical, equity, and political issues. Most critically, in many projects, communities themselves decide if they wish to participate or not, and it is reasonable to assume that they differ in some unknown way from those villages that opt out or never show any interest. Differences may be observable (wealth, climate) or unobservable (motivation, leadership, social capital). In such cases, some method must be used to study June 21, 2008 49 comparable groups while leaving the self-selection feature intact. That was the case in Maharashtra, where villages had to apply to participate in the project and were chosen according to specific criteria. Propensity score matching was used to select control villages that “matched� the treatment villages.10 5.11 To select the villages to be included in the study, several steps were taken. First, the urban or coastal Jalswarajya districts were eliminated from the analysis. This was done to keep the focus on rural and dry or drought-prone villages. One district was chosen from each of four geographically different regions: Osmanabad from the Marathwada region, Nashik (Near Mumbai Region), Sangli (Western Maharashtra), and Buldana (Vidarbha). 5.12 Second, from this sample of districts, each project (treatment) village was matched with an observationally similar non-project (control) village, using propensity score matching. These involve the following steps (see Pattanayak et al., [2007] for a detailed description of the process):  Village level data were drawn from the 2001 and 1991 Censuses, the 1999 Habitat Data, and the Jalswarajya (Table 6-1 lists the 30 variables used in this model). Data on these variables were available for two samples: (a) 6200 villages with data on all 30 variables, and (b) 7200 villages with data on all 2001 census variables and water supply coverage. These two data sets are used to construct various matched samples and select the final sample.11  A logit model of project participation was estimated on a pooled sample of selected project villages and all non-project villages. Our choice of pre-determined variables to be included in the estimation of the model for propensity score estimation relied on factors that might influence (or proxy for) the three eligibility criteria for being a Jalswarajya participant: (a) poor quality and quantity of drinking water and sanitation services, (b) high proportion of socially disadvantaged groups, and (c) institutional capacity for fiscal responsibility, community participation and governance. Table 5-1 shows that the estimated logit models are statistically significant, but only explained about 10% of the variation in the data (i.e., pseudo-RSquared of 0.1). The pseudo-RSquared was closer to 0.35 for the sub-sample of matched project and control villages. The statistically significant variables had the appropriate signs – that is, they are consistent with the criteria that the Jalswarajya program is supposed to use to select project villages.  The propensity score for participating in the Jalswarajya project was calculated in the pooled sample. Given this score, for each project village, we find the non-project village with the closest propensity score. Note, restricting matches to lie within the region of common support (Augurzky and Schmidt, 2001) and trimming 5% of the distribution of 10 To our knowledge, this strategy of pre-matching based on propensity scores has been suggested only in a handful of evaluations that are outside the WSS sector (see Almus et al. [2001], Ho et al. [2007], Priesser et al. [2003] and Sills et al. [forthcoming]). 11 We account for the fact that many of the program and administrative decisions are made at the district level through two tasks. First, we include a district dummy variable in the estimation of propensity scores to account for all remaining characteristics unique to the district. Second, we consider two restrictions to the matching process – (a) match within the district (from model with larger K, smaller N), and (b) unrestricted match with any other village in the pool (from Model with larger N, smaller K) June 21, 2008 50 estimated propensity scores did not significantly change the set of potential project villages because of the large pool of non-project villages available to provide a match.  A short-list of the non-project villages was created to serve as the ‘matched control’ villages. The matching strategy is evaluated by checking the balance in covariates (the variables in 5-2) across the project and control villages. We check to see if the matching process reduced (a) the bias (or difference) averaged across Table 5-1 variables, (b) the bias for key individual variables such as those representing water supply conditions and proportion of socially disadvantaged populations. Table 5-2 confirms that these criteria were satisfied for all samples retained for further evaluation. Matching reduced bias between 7 and 99 percent, depending on the variable. With one exception, matching eliminated statistically significant differences in the means between treatment and control villages. After the balance test, we eliminated any matched pairs that were statistically different in terms of percent scheduled castes in the village, percent scheduled tribes in the village, and water supply level (lpcd) in the village.  A conservative strategy was applied in picking the final list of sample villages by over- sampling control villages. The final sample of 242 villages comprised 95 treatment villages (2 pilot villages and 93 Phase I villages) and 147 control villages. The control villages were ones that had applied for participation in Jalswarajya but whose participation was postponed for later project batches. The matching process accounts for many changes affecting child health and makes certain the villages are comparable as to the criteria for selection. 5.13 Third, households were selected before the start of the baseline surveys in May 2005 using two steps. We listed and mapped all households in each of the selected villages. Following that, all households with at least one child under five years of age were identified. Because there were no pre-existing data on households with children under five, house-to-house visits had to be made to identify them. It took 20 teams of 4 people each 15 days to complete this labor-intensive process. 5.14 Finally, in villages with 50 or fewer eligible households, all were interviewed. In villages with more than 50, a random sub-sample of 50 was interviewed. If a household was not available for an interview, then it was visited the same day or the next day at different times for up to three follow-up visits. If a household was not found or if an interview was refused, the next household was selected according to a pre-established procedure. June 21, 2008 51 Table 5-1. Propensity Score Estimation of Participation in Jalswarajya Logit Model using 2001 Census, Logit Model Description 1991 Census, and 1999 Habitat Data using 2001 Census Dependent Variable Dummy variable indicating whether village is in Jalswarajya Project -8.24 -5.33 % males in village (2001) (4.36)* (3.97) 1.56 0.75 % children in village (2001) (3.29) (3.22) 1.58 1.56 % scheduled castes in village (2001) (0.80)** (0.74)** 1.99 1.91 % scheduled tribes in village (2001) (0.44)*** (0.41)*** -0.19 -0.64 % female workers in village (2001) (1.59) (1.44) -0.71 -0.84 % cultivators in village (2001) (0.74) (0.68) 0.82 0.82 % agricultural labors in village (2001) (0.78) (0.71) 0.71 0.53 % marginal workers in village (2001) (0.58) (0.54) -0.00 0.00 Households in village (2001) (0.00) (0.00) 0.35 0.31 Average household size in village (2001) (0.16)** (0.15)** 0.01 0.00 Female literacy rate in village (2001) (0.01) (0.01) -0.87 -1.01 % permanent houses in block (2001) (0.82) (0.77) % households with private tap in block 0.00 -0.67 (2001) (1.06) (0.99) -5.57 -3.71 % households without toilets in block (2001) (1.83)*** (1.69)** % households with electricity in block -1.27 0.13 (2001) (1.46) (1.29) % households who use firewood / crop -1.41 -1.39 residue / cowdung as cooking fuel in block (1.58) (1.45) (2001) -0.05 -0.04 Water supply level (lpcd) in village (1999) (0.01)*** (0.01)*** 0.00 Distance to nearest water in village (1999) (0.03) 0.86 Arsenic in village water (1999) (0.73) 0.65 Fluorides in village water (1999) (0.69) 1.04 Nitrate in village water (1999) (0.73) -0.43 Salinity in village water (1999) (1.04) June 21, 2008 52 Logit Model using 2001 Census, Logit Model Description 1991 Census, and 1999 Habitat Data using 2001 Census Dependent Variable Dummy variable indicating whether village is in Jalswarajya Project 0.89 Odor in village water (1999) (0.62) -1.02 Market facility in village (1991) (0.22)*** -0.11 Bus stop in village (1991) (0.24) -1.27 Railway station in village (1991) (1.04) 0.24 Pucca roads in village (1991) (0.21) -0.01 Kutcha roads in village (1991) (0.01)* 0.00 Village area (1991) (0.00) 0.72 0.82 Buldana District dummy (0.38)* (0.33)*** -0.08 -0.46 Nashik District dummy (0.43) (0.41) 0.87 0.64 Osmanabad District dummy (0.50)* (0.47) 1.13 1.12 Sangli District dummy (0.41)*** (0.36)*** 7.19 2.33 Constant (3.96)* (3.61) Number of observations 6201 7181 Pseudo R2 0.1091 0.0799 † Standard errors are in brackets and noted *** if significant at 1%; ** if significant at 5%; * if significant at 10%. June 21, 2008 53 Table 5-2. Testing covariate balance across treatment and ‘matched’ control villages using secondary data Covariate of Interest % bias reduced† t-statistic % males in village (2001) 20 0.29 % children in village (2001) 68 0.74 % scheduled castes in village (2001) -749 1.78 % scheduled tribes in village (2001) 79 -0.36 % female workers in village (2001) 83 -0.05 % cultivators in village (2001) 9 -1.18 % agricultural labors in village (2001) 37 1.55 % marginal workers in village (2001) 49 -0.37 Households in village (2001) 98 -0.03 Average household size in village (2001) 99 0.03 Female literacy rate in village (2001) 55 -1.05 % permanent houses in block (2001) 56 -0.57 % households with private tap in block (2001) 98 0.07 % households without toilets in block (2001) 66 -0.49 % households with electricity in block (2001) 77 0.82 % households who use firewood / crop residue / 85 0.4 cowdung as cooking fuel in block (2001) Water supply level (lpcd) in village (1999) 7 2.33 † Reduction in bias when comparing mean difference between treatment and unmatched controls to mean difference between treatment and matched controls. Bias is the difference in standardized means between JS and control (non-JS) villages.  For mean difference between treatment & matched control villages 5.15 After the completion of the baseline household surveys, which is described in the next section, we were able to re-check the central premise of our evaluation design – that propensity score matching reduced differences between treatment and matched control villages. 5.16 A comprehensive analysis of key covariates of interest reveals that treatment and control villages were balanced across many health outcomes, water and sanitation conditions, personal hygiene behaviors, and perceptions of local health and environmental problems (See Table 5-3 for the details). However, the use of 2001 census data for matching did not eliminate all differences. We find that treatment villages are more exposed to key hygiene messages (delivered through non-electronic media), in their perceptions of main day-to-day problems, and participation in community activities (e.g., sweeping streets, cleaning drains) and organizations (e.g., village water and sanitation committees). This is not surprising because the baseline surveys were conducted several months after the projects had been initiated in some of the treatment villages, and presumably the early project activities focused on social mobilization and June 21, 2008 54 awareness that would result in these differences. Therefore, in baseline treatment villages were different in three factors: (a) exposure to public health messages, (b) identification of the main problem, and (c) community participation. This finding further underscores the importance of our difference-in-difference modeling plans to remove any residual time-invariant selection bias. Table 5-3. Testing balance across treatment and control villages using baseline survey data Treatment Control Covariate of Interest z-value‡ Mean Mean % under 5 children with diarrhea 11% 10% 1.62 % under 5 children with ARI 21% 22% -0.71 % households using private tap 18% 24% -1.55 % households using private toilet 13% 10% 0.96 # of critical times a caregiver washes hands 2.3 2.4 -0.51 # of critical times a child washes hands 1.1 1.2 -0.44 % households treating drinking water 64% 63% 0.11 % households stating roads are ‘main’ problem 19% 21% -0.84 % households stating water supply is ‘main problem’*** 54% 42% 3.27 % households stating sanitation is ‘main problem’* 11% 14% -1.72 % households stating public well water quality is bad* 19% 24% -1.77 % households stating public tap water quality is bad 24% 22% 0.44 Household water consumption (liters per capita per day 29.9 30.5 0.36 [LPCD]) E. coli levels on log10 scale in household drinking water 0.63 0.53 1.32 % households stating village water-sanitation committee 20% 12% 2.71 (VWSC) is active*** % households participating in VWSC*** 5% 3% 3.35 † Differences are noted *** if statistically significant at 1%; ** if significant at 5%; * if significant at 10%. ‡ For mean differences after adjusting standard errors to account for clustering at the village level. 5.17 By virtue of having to estimate a statistical model of participation in the CDD project, the first stage of the matching strategy also allows us to uncover some of the political economy factors that influence targeting and self-selection (Bardhan and Mookherjee, 2000). We see greater participation by communities that had more socially marginal sub-populations and poorer sanitation and water conditions. We also see that villages with larger size households, in districts with a proactive administration, and that are more remote (no market facilities or unpaved roads) and were more likely to participate. While these results are all suggestive of targeting, it is difficult to rule out self-selection because many of these covariates would also increase community demand. June 21, 2008 55 III. Survey Implementation 5.18 Baseline data were collected in two phases in 2005, before the monsoon (May-June) and after (August-September). The same sample was used in both seasons. Over 10,000 household surveys and 240 community surveys were conducted in 2005 and 2007. The survey-takers worked in several teams (ranging from 13 to 21 team members), each comprising five to six enumerators, an editor or scrutinizer, a water sample collector, and a supervisor. At least 6 TNS researchers, field executives, managers were involved in managing the surveys, conducting the training, and other field activities. We collected water samples in half of the households and from up to ten in-use water sources and transported them to a lab in Pune within 24 hours. The lab tested the samples using a substrate method with Chromagar media for total coliform and e. coli counts. The household survey took on average 120 minutes to administer; the community survey, 150 minutes; and the water quality tests, 10 minutes. Tables 5-4 and 5-5 report the achieved sample size in each round of the survey. Table 5-4. Number of Completed Surveys in the Dry Season by Study District Number of Household Surveys Water Samples villages 2005 2007 2005 2007 Sangli 57 2,424 2237 951 Nashik 91 3,324 3006 1,273 Buldana 55 2,145 1948 954 Osmanabad 39 2,312 2166 1,046 Total 242 10,205 9357* 4,224 6434# * Represents a loss of 8 percent compared to summer 2005 baseline. # Represents total water samples collected and includes approximately 5,500 household samples and more than 900 source samples. Table 5-5. Number of Completed Surveys in the Rainy Season by Study District Number of Household Surveys Water Samples villages 2005 2007 2005 2007 Sangli 57 2,271 2,230 1,294 1,424 Nashik 91 2,970 2,986 1,747 2,065 Buldana 55 2,020 2,005 1,269 1,335 Osmanabad 39 2,087 2,139 1,261 1,281 Total 242 9,348 9,360* 5,571 6,105# * The sample size can increase between Round 2 and 4 because of different household migration pattern as well as better field implementation in year 2007. Also, different set of households might be missing in both years. # Represents 5232 household samples and 873 community source samples. The sample size in round 4 is higher due to negligible loss of samples in transport and proper sample identification + coding on questionnaires. June 21, 2008 56 IV. Adequacy Assessment 5.19 The endline survey for Jalswarajya was originally scheduled to coincide with the project cycle, the 18-month period from project inception to completion plus an interval long enough for the project to reasonably show some results. Following that schedule, endline data collection would have begun in September 2006. In a project of this size and scope, it is vitally important that the expensive and time-consuming process of data collection not be undertaken unless the project has actually reached a stage where endline assessment makes sense. To determine whether or not that was the case, an adequacy assessment was conducted. 5.20 The adequacy assessment was executed in three waves comprising 25, 45 and 50 villages each. It included rapid rural appraisals which consisted of interviewing key informants in each community, verifying physical progress, and conducting transect walks through the community. Then, it integrated M&E information from DWSC staff members as well as conducted informal yet detailed interviews with them to understand the process of Jalswarajya. The team triangulated information from these sources to determine the adequacy of interventions. The data from three waves of data collection (with 2 to 4 months between waves) were used in a simple trend analysis to predict adequacy of interventions in the first quarter of 2007. 5.21 The adequacy assessment helped to determine that the endline surveys had to be postponed given inadequate intervention progress in 2006. It further revealed that the interventions had a potential to become adequate in 2007 provided that project implementation was accelerated, and that some specific interventions progressed faster than others. Under these conditions, villages would be adequate in at least some of the interventions planned under Jalswarajya by 2007. Therefore, in consultation with the government’s project team, it was decided to conduct the endline surveys in Maharashtra in May-June and August-September, 2007. More details about adequacy assessment are included as Annex 5. June 21, 2008 57 Box 5.1 Read More about the Maharashtra Study This report provides an overview of the Maharashtra study, including its methods and main results. The following are references to additional papers and reports that provide more detail on this study.  Of taps and toilets: Quasi-experimental approaches for evaluating community-demand- driven projects by Subhrendu K. Pattanayak, Christine Poulos, Jui-Chen Yang, and Sumeet R. Patil. RTI International Working Paper 06-04, June 2006.  Seasonal variation in risk factors associated with diarrheal diseases, rural Maharashtra, India by Catherine G, Corey, Jui-Chen Yang, Subhrendu K. Pattanayak, and Richard K. Kwok. RTI International Working Paper 07-03, June 2007.  The In-House Water Quality Management: Uncovering the Complex Web of Household Behaviors, and Water and Sanitation Interventions by Sumeet R. Patil and Subhrendu K. Pattanayak. RTI International Working Paper 07-04, December 2007.  Gauging Adequacy of Community Water Supply and Sanitation Projects in Maharashtra: Methodological Triangulation by Sumeet R. Patil, Subhrendu .K. Pattanayak, and Suresh Vinerkar. RTI International Working Paper 06-06, December 2006.  Hydraulic Self-Rule: Establishing Community Water Supply and Sanitation Schemes in Rural Mahashtra, India by Nitish Jha and Subhrendu K. Pattanayak. Paper presented at the 11th Biennial Conference of the International Association for the Study of Common Property, Ubud, Bali, Indonesia, 19-23 June, 2006. June 21, 2008 58 6. Impacts of Jalswarajya, Maharashtra 6.1 This chapter summarizes the results of pre- and post-intervention surveys conducted in the dry and rainy seasons of 2005 and 2007. It reports descriptive statistics and bivariate analysis for 5 categories of variables that include (a) respondent and household characteristics, (b) health outcomes, (c) water supply and sanitation conditions, (d) household water supply, sanitation and hygiene (WSH) knowledge, attitudes and practices (KAP), and (e) welfare outcomes. We then discuss the results of key variables of interest using difference-in-difference (DID) estimation. I. Descriptive Statistics and Bivariate Analysis 6.2 In general, living conditions improve in both treatment and control villages across survey rounds. However, in some cases, there are no major changes in the covariates between survey years (i.e., 2005 to 2007), between dry and rainy seasons, or across treatment and control villages following Jalswarajya. In the following sections, our discussion focuses on the changes between survey years and seasons as well as those across treatment and control villages. A. Respondents Characteristics 6.3 Almost all respondents were women of childbearing age. While half of the respondents had received primary education, one third of them had never attended school. As shown in Table A6-1 in Annex 6 (and t-tests for differences across survey rounds that are not reported here), there were no major changes in respondent and household characteristics across the survey rounds as expected. 6.4 One third of the sample households classified themselves as scheduled castes (SC) or scheduled tribes (ST). About half of the households lived below the poverty line (BPL).12 The average household had 6 members in 2005 and 7 members in 2007. When asked about the most important improvement that they would like to see in their village over the next ten years, household water supply improvements were most frequently mentioned, followed by road improvements and sanitation and hygiene improvements. When questioned about perceptions of disease, ARI, diarrhea and malaria were most frequently stated concerns. There is no apparent difference between treatment and control villages across survey rounds. B. Health Outcomes 6.5 Table A6-2 reports statistics on the prevalence of various diseases, including diarrhea, acute respiratory infections (ARI), malaria, tuberculosis (TB), typhoid fever, and cholera, and arm circumference of children under the age of five. There were no major changes in these health outcomes across treatment and control villages or survey rounds, except for diarrhea, ARI (indicated by cough and cold), and arm circumference. Thus, we focus our discussion on these outcomes, which are also illustrated in Figures 6-1 and 6-2. 12 We asked respondents to identify themselves as living in a below poverty line (BPL) or above poverty line (APL) household. According to the 62nd round of National Sample Survey (NSS) of India conducted in 2005 and 2006, the national poverty line was defined as monthly per capital consumption expenditure to be less than 365 Rupees; that is, spending less than 12 Rupees per person per day. Less than this amount will not allow you to buy 2200 calories of food per day. June 21, 2008 59 Figure 6-1. Diarrhea Prevalence among Children under 5 by Intervention, Season, and Year 15% 12% 13% 11% 12% 9% 10% 9% 8% 9% 7% 6% 3% 0% 2005 2007 2005 2007 2005 2007 2005 2007 Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages Dry Season Rainy Season I indicates the 95 confidence interval. Figure 6-2. Arm Circumference in centimeters (cm) of Children under 5 by Intervention, Season, and Year 16 13.78 14.20 13.82 14.04 12.97 13.89 13.04 13.95 12 8 4 0 2005 2007 2005 2007 2005 2007 2005 2007 Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages Dry Season Rainy Season I indicates the 95 confidence interval. June 21, 2008 60 6.6 Dry season: In 2005, 27% of households had a diarrhea case in the two weeks prior to the survey. In 2007, this number dropped to 22%. Among children under five, the prevalence rates of diarrhea were 10% in 2005 and 9% in 2007. Compared to control villages, treatment villages had a higher child diarrhea rate in both years. The majority of the households did not treat diarrhea. Among those who did treat, pill and syrup and injections were the two most popular treatments. 6.7 The prevalence rates of ARI among children under five were 22% in 2005 and 13% in 2007. However, there was no noticeable difference in treatment and control villages. In terms of arm circumference, in 2007 children in treatment villages had an average arm size that was almost 0.20 centimeters bigger than those in control villages although they were about the same size in 2005. Though these are small changes, the literature suggests that small gains in anthropometric measurements in this age group translate into significant developmental outcomes in the long run (Alderman, 2000). 6.8 Rainy season: Twenty-eight percent of households had a diarrhea case in 2005, and the number dropped to 17% in 2007. Among children under five, the prevalence rates of diarrhea were 13% in 2005 and 8% in 2007. Treatment villages had a slightly lower child diarrhea rates before the intervention and a slightly higher diarrhea rates after the intervention, compared to control villages. Like in the dry season, most households did not treat diarrhea and those who did generally relied on pill and syrup or injections. 6.9 The prevalence of ARI among children under five was about 30% in 2005 and 17% in 2007. Again, no clear difference was observed between treatment and control villages. Turning to arm circumference, children in treatment villages had relatively smaller arm size than those in control villages in both years. C. Water Sources 6.10 Table A6-3 shows that on average households had access to two water sources and used only one source in their village. Public wells, private taps, public taps, and private wells were the four major water sources for households. Overall, more households used an improved water source in 2007, a stronger trend in control villages. (Figure 6-3) 6.10.1 Dry Season: In general, more households used private taps and fewer households relied on private wells in 2007 than in 2005. While the use of private taps remained similar in control villages, treatment villages experienced an increase from 17% in 2005 to 30% in 2007. In terms of the use of private wells, a decline was evident in both treatment and control villages. 6.10.2 Rainy season: Households relied on similar water sources between the survey years. In general, more households used private taps and fewer households relied on public wells in 2007 than in 2005. Households in treatment villages reduced their reliance on public taps and private wells in 2007 while households in control villages did the opposite. Again, the uptake of private taps was very pronounced in treatment villages (19% in 2005 and 32% in 2007), compared to control villages June 21, 2008 61 Figure 6-3. Percentage of Households Reporting Use of Private or Public Tap as Main Water Source by Intervention, Season, and Year 60% 51% 47% 45% 44% 50% 41% 42% 40% 35% 40% 30% 20% 10% 0% 2005 2007 2005 2007 2005 2007 2005 2007 Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages Dry Season Rainy Season I indicates the 95 confidence interval. D. Water Quantity 6.11 The quantity of water consumed was measured based on the size of a typical collection vessel, the number of trips per day, and storage capacity. The findings are summarized in Figure 6-4. 6.11.1 Dry Season: Households in both treatment and control villages reported collecting approximately180 liters per day in the pre-intervention survey, which is equivalent to 30 liters per capita per day (LPCD). Following the intervention, households in treatment villages reported collecting more water than those in control villages. 6.11.2 Rainy Season: In both treatment and control villages, on average, a total of 170 liters was collected each day in the pre-intervention survey, which is equivalent to 27 LPCD. Following the intervention, households in treatment villages reported collecting more water than those in control villages. June 21, 2008 62 Figure 6-4. Household Water Consumption in LPCD by Intervention, Season, and Year 45 41 39 38 38 30 31 27 27 30 15 0 2005 2007 2005 2007 2005 2007 2005 2007 Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages Dry Season Rainy Season I indicates the 95 confidence interval. E. Water Quality 6.12 Household water samples were tested for fecal and total coliform levels and evaluated in terms of India’s safe water standards. The standard for fecal coliform (E. coli) is 0 colifom per 100 ml, and the standard for total coliform is 10 total coliform per 100 ml. Total coliform is not necessarily a risk factor for diarrheal diseases.13 (Figure 6-5) 6.12.1 Dry Season: Table A6-3 shows that water quality worsened in the post-intervention survey. Thirty percent of water samples contained fecal coliform in 2007 while only 20 percent did in 2005. About 80% contained more than 10 CFU (colony forming units) in 2007, compared to 50% in 2005. In addition, households in treatment villages experienced worse water quality than those in control villages. 6.12.2. Rainy season: Water quality improved in the post-intervention survey as shown in Table 6-3. While about 40% of water samples contained fecal coliform in 2005, only 26 percent did in 2007. About 80% contained more than 10 CFU (colony forming units) in 2005, compared to about 70% in 2007. In addition, households in treatment villages seemed to have better water quality in 2005 and worse water quality in 2007 when compared to those in control villages. 13 Based on personal communication with Professor Christine Moe (Emory University’s Rollins School of Public Health). June 21, 2008 63 Figure 6-5. E. Coli Levels* on log Scale in Household Drinking Water by Intervention, Season, and Year 1.5 1.2 1.1 1.2 0.8 0.8 0.7 0.9 0.7 0.6 0.5 0.6 0.3 0.0 2005 2007 2005 2007 2005 2007 2005 2007 Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages Dry Season Rainy Season * E. coli tests were done with 1 mL volume, but results are reported on per 100 mL basis. I indicates the 95 confidence interval. F. Personal Sanitation 6.13.1 Dry season: Table A6-4 shows that in 2005 the majority of households did not have a sanitation facility near their house while 12% used individual household latrines (IHL) and about 2% used community toilets. In 2007, we see that 13% fewer households defecate in the open and IHL use increases to 24%. These changes were mostly contributed by treatment villages. (Figure 6-6) 6.13.2 Rainy season: In 2005, most households did not have a sanitation facility near their house while 15% used IHL and 2% used community toilets. In 2007, we see that 11% fewer households defecate in the open and IHL use increases to 24%. These changes were more evident in treatment villages. June 21, 2008 64 Figure 6-6. Percentage of Households Reporting Use of Toilet as Main Sanitation Practice by Intervention, Season, and Year 40% 34% 34% 30% 22% 21% 19% 16% 20% 12% 14% 10% 0% 2005 2007 2005 2007 2005 2007 2005 2007 Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages Dry Season Rainy Season I indicates the 95 confidence interval. G. Environmental Sanitation 6.14.1 Dry season: Most respondents reported dumping their garbage immediately outside the house: 77% in 2005 and 86% in 2007. Only a few households reported dumping garbage in the fields (15% in 2005 and 8% in 2007) and in the community garbage dump (18% in 2005 and 6% in 2007). About 30% each disposed of the household wastewater in the backyard, outside the house but not in a drain, or outside the house in a drain. The remaining 10% disposed of it within the house. As shown in Table A6-4, there were changes in these behaviors across survey rounds, but not across treatment and control villages. 6.14.2 Rainy season: The majority of the respondents reported dumping their garbage immediately outside the house: 86% in both 2005 and 2007. Only a few households report composting garbage in the field or yard (19% in 2005 and 14% in 2007) and in the community garbage dump (16% in 2005 and 5% in 2007). Around one third each disposed of the household wastewater in the backyard, outside the house but not in a drain, or outside the house in a drain. The remaining disposed of it within the house. There were changes in these behaviors across survey rounds, but not across treatment and control villages (Table A6-4). H. Household WSH Knowledge, Attitudes and Practices (KAP) 6.15.1 Knowledge: As reported in Table A6-5, caregiver knowledge of diarrhea improved across time. Caregivers were able to correctly identified 2 out of the 3 key diarrhea symptoms (i.e., loose stool, vomiting, and weight loss) in the post-intervention surveys, compared to just 1 in the pre-intervention surveys. The same caregivers also correctly identified 4 out of the 8 key diarrhea causes (including eating stale foods, eating food touched by flies, dirty/smelly food, drinking contaminated water, using unhygienic latrines/OD, not washing hands, household June 21, 2008 65 dirtiness, and village dirtiness) in the post-intervention surveys, compared to 3 in the pre- intervention surveys. There were no differences between dry and rainy season. 6.15.2 Averting behaviors: Over 60% of the households treated their water before drinking in all survey rounds. Households are more likely to treat their water in the rainy season. More than 50% of respondents reported that they filtered water, less than 5% reported using chemicals, and about 2% reported boiling water. 6.15.3 A large majority of the households stored their drinking water and always covered their drinking water containers. However, fewer than 20% used only narrow mouth storage and less than 30% practiced good water transferring techniques (i.e., only pouring or dipping with ladle, no dipping with hands). These estimates are lower for the rainy season – less than 15% use narrow mouth storage and less than 20% safely handle water. Both treatment and control villages showed similar results. Figure 6-7. Household Coping Costs in Rupees due to Inadequate Water Supply and Sanitation Services by Intervention, Season, and Year 1200 1064 926 836 869 1000 800 605 562 513 600 458 400 200 0 2005 2007 2005 2007 2005 2007 2005 2007 Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages Dry Season Rainy Season I indicates the 95 confidence interval. June 21, 2008 66 Figure 6-8. Household Cost of Illness in Rupees due to Diarrhea by Intervention, Season, and Year 500 391 349 356 400 289 305 268 300 160 200 135 100 0 2005 2007 2005 2007 2005 2007 2005 2007 Jalswarajya Villages Control Villages Jalswarajya Villages Control Villages Dry Season Rainy Season I indicates the 95 confidence interval. 6.15.4 Hand washing: Hand washing improved from 2005 to 2007. In 2007, households were more likely to wash hands at critical times than in 2005. For adults, these critical hand washing times include before preparing food or cooking, before eating, before feeding children, after changing baby/handling child's feces, and after defecation. For children under 5, before eating and after defecation are the critical times. However, there was not much difference between treatment and control villages. In 2007, more households reported the availability of water and soap at the place for hand washing than in 2005. There were no observable differences in hand washing between the dry and rainy seasons. I. Welfare Outcomes 6.15.5 In Table A6-6, we present some evidence of improvements to household welfare (such as costs of illness due to diarrhea, coping costs and averting behaviors) across the survey rounds. In 2007, we see reductions in unproductive days, work days lost, and hospital nights and medical costs compared to 2005. However, it is unclear if there is a statistically significant difference between treatment and control villages. 6.15.6 In addition, we estimated monthly household averting costs due to inadequate WSS services in Rupee terms (Figure 6-7). The averting costs were calculated as the sum of the value of time and money spent on water collection (i.e., walking to and waiting at water sources and investments in water collection systems), water treatment (i.e., boiling, filtering and the use of chemicals), water storage (i.e., investments in storage containers at home), and the use of sanitation facilities outside the house (i.e., walking to the open defecation site). The average averting costs are lower in 2007 than in 2005. Households in treatment villages experience the most reduction in the costs between survey rounds. This can be attributed to the fact that more June 21, 2008 67 households started using private taps and IHLs at home in 2007, compared to 2005; thus, the reduction in time traveling back and forth between water sources and defecation sites. 6.15.7 We also imputed the cost of illness due to the most recent episode of diarrhea. It was estimated by converting unproductive days and work days lost into Rupee terms based on an average wage rate in each village, plus the additional medical expenditure (Figure 6-8). In general, households incurred lower average costs of illness in the post-intervention surveys than in the pre-intervention surveys. II. Multivariate (Regression) Analysis with Baseline Data 6.16 This sub-section briefly summarizes results from two masters’ theses conducted with the baseline data. The first used a case-control approach to examine if household open defecation caused diarrhea (Corey et al., 2007). The second utilized a household production framework to examine the interaction among household behavior, WSS conditions, demographic and knowledge variables, and exposure to disease-causing microbes (Patil and Pattanayak, 2007). A. Open Defecation as Diarrhea Risk Factor 6.17 Corey et al. (2007) reveals significant seasonal variations in the association between diarrheal morbidity and household sanitation and hygiene practices. Not surprisingly, diarrhea is statistically more prevalent among children under five years of age during the rainy season, as compared to the dry. This analysis also shows that the impact of open-defecation could be modified by several factors. These include the availability of sufficient water quantity, hand washing at critical points (e.g., before eating and after defecating), child vaccination history, diarrhea knowledge, and household wealth. 6.18 During the dry season, joint behaviors of open defecation with less frequent child hand washing and water quality and consumption were positively associated with diarrhea, underscoring the interdependence of sanitation, water and hygiene factors. In contrast, these interactions were not observed or less pronounced during the rainy season. Aside from environmental factors, younger age, shorter exclusive breastfeeding, and large number of under 5 children in the household were consistently associated with disease. The variation in risk factors by season and the interaction of sanitation with other environmental factors suggest that more research is needed into how and why these interactions vary by season and how development efforts can best intervene to affect key variables. B. Behaviors Exposed 6.19 Patil and Pattanayak (2007) rely on a household production model in which the quality of water is assumed to depend on households’ water handling behaviors (e.g., using a dipper, covering the storage container, etc.) This analysis finds the following. First, source water E. coli contamination clearly influences the likelihood of in-house water contamination. Second, having a piped water connection reduces E. coli contamination in-house by more than 40%. Third, organized drainage in the village can reduce total coliform contamination of household water by 30% but has no effect on E. coli. Fourth, water quantity has no impact on water quality. Fifth, each additional averting behavior reduces E. coli contamination by 10%. And lastly, households that use simple cloth filters may in fact be increasing contamination of drinking water. 6.20 This preliminary analysis suggests that piped water and an organized drainage system in a village can substantially decrease household exposure to disease causing pathogens through June 21, 2008 68 drinking water. Additionally, the adoption of several household averting behaviors can reduce microbial exposure. Finally, E. coli tracks closely with community water and sanitation behaviors. These findings must be considered tentative because of the cross-sectional nature of the data. However, they point in an interesting direction to interventions with high potential health impacts. III. Analytic Statistics: Basic Differences-In-Differences (DID) Estimator 6.21 The evidence we have presented indicates that changes are visible in some disease outcomes, water supply and sanitation, household KAP and welfare outcomes. As discussed in Chapter 2 DID analysis forms the cornerstone of our plan to determine if the changes were caused by Jalswarajya program or just due to secular trends. A statistically significant improvement in outcomes in project villages, relative to control villages, is indicated by a probability value of less than 0.15 on the coefficient for the “treat*post� variable (Chapter 2). We take a more liberal p-value because of the smaller samples used in these village level assessments. 6.22 Previously, we have also contended that it is vital to measure impacts along the causal chain from indicators that describe the RWSS policies (e.g., resources, activities, inputs) to the outcomes and impacts of interest (e.g., outputs, outcomes and indicators), not just one or the other (see Table1-1 in Chapter 1). Thus, we short list 25 indicators that correspond to input, output, outcome and impact indicators and discuss these in sequence. 6.23 As suggested earlier, the decentralized delivery philosophy of Jalswarajya implied that many of the program and administrative decisions are made at the district level. Beyond the inclusion of a district fixed-effect in the estimation of propensity scores, we also attempted two restrictions to the matching process – (a) match within the district, and (b) unrestricted match with any other village in the participating districts. The choice between in-district and out-of- district matches essentially boils down to a tradeoff between bias and efficiency: in-district matching reduces bias in unobservable institutional drivers, whereas out-of-district matching permits searching over a larger pool of matches. Thus, the results of all these analyses are reported for these two sub-samples. For most parts, we see no significant difference in the in- district or out-of-district matches. A. Inputs: Activities and Resources 6.24 Addendum-II funding: “Addendum II funds� released by the Jalswarajya project are for the civil works (or infrastructure) activities related to WSS delivery (Patil et al., 2006). The project staff condition payment on progress and release it in three installments. Installment 1 (45% of costs) is paid to start the work. Installment 2 (50% of costs) is paid when the actual construction is completed, and the district staff determines ‘open-defecation-free’ status. Field visits and rapid rural appraisals suggest that it is not clear if the latter condition is truly satisfied. Finally, installment 3 (5% of costs) is paid when the gram panchayat has collected O&M expenses for 6 months and there is evidence of project completion. We consider the receipt of the 2nd installment of Addendum II as evidence of significant resource availability. Because no community – in the Jalswarajya program or outside it – had received this funding at the time of our baseline survey, we cannot estimate double-differences and must use a single-difference estimator for this particular indicator. In simple terms, this is a probit model in which we test the significance of the ‘Jalswarajya (JS)’ condition dummy variable. As shown in Table A6-7, being June 21, 2008 69 in the project raises the probability of receiving the 2nd installment by about 45%. Note, this probability increases by about 12% if we use an updated definition of ‘project participation’ based on the adequacy survey reported in Patil et al. (2006). The fact that we do not have perfect prediction confirms our field observations that some communities (29 villages) had dropped out of the project, and others (including 13 of our original controls) have joined the project since its initiation in 2003. 6.25 For the remainder of the analysis, we re-define the Jalswarajya condition from the intention-to-treat indication to include villages that have received at least installment 1 (for civil construction) of the Addendum II funding. The drop outs are treated as controls because treatments and controls were chosen on equal probability of participation in Jalswarajya and a variety of covariate balance tests confirmed that these communities were observationally similar. 6.26 Water and Sanitation schemes: The community survey polled key informants about the existence of significant projects and schemes related to water and sanitation in their community. These water (sanitation) scheme indicators were scored as 1 if a significant water (and/or sanitation) program/scheme was ongoing and as a 0 otherwise. Here we can apply a DID estimator because such schemes may have been ongoing in the baseline (i.e., we are not subtracting 0 from 0). We find that Jalswarajya participation raises the probability of a water scheme/program by 21%. We find no such impact of Jalswarajya participation on sanitation schemes. 6.27 Facilitation visits by district officials: The community survey also polled key informants about the frequent help by district officials with evaluation, technical guidance, funding and monitoring. This indicator was scored as 1 if district staff helped with one or more of these activities by visiting at least once a month. We find weak evidence that district officials were more likely to visit Jalswarajya villages to help. The marginal effect is statistically significant at the 10% level only in the pooled sample of in-district and x-district matches. 6.28 Community contribution to capital and maintenance costs: The community survey also generated data on whether the key informant believed that the community had completed its own obligations and contributed 10% towards the capital costs of the water and sanitation scheme. Operations and maintenance costs are expected to be borne fully by the community. There is some non-response on this variable. Nevertheless, the DID estimators suggest that communities participating in the Jalswarajya program were 50% more likely to meet their financial obligations. 6.29 Village water and sanitation committee (VWSC): The key informant also reported on the activity of the village water and sanitation committee. Formation of this committee constitutes a key first step in capacity building at the village level and successful operation signals likelihood of WSS activity. This indicator was scored as 1 if an active VWSC exists in the village (and 0 otherwise). Again, the DID estimators suggest that communities participating in the Jalswarajya program were 25% more likely to have an active VWSC in their village. B. Outputs 6.30 Hygiene IEC: From the household survey, we were able to glean the extent to which households remembered receiving four key hygiene messages from project controlled media. The four key messages are about hand washing, water storage and treatment, food handling, and personal sanitation (defecation). Project medium included posters and billboards, local village June 21, 2008 70 health worker (auxiliary-nurse-midwife), village school and wall paintings. The indicator measures the percent of households in the village who remembered receiving messages about all four hygiene behaviors from any of the four sources. The DID estimator suggests that if there was a concerted information, education and communication effort, it certainly did not seem to have registered among the people living in Jalswarajya villages. Modifying the definition to be more inclusive and including any message from any media suggest that Jalswarajya participation had a negative impact: more people in control villages remembered receiving hygiene related communication compared to households in Jalswarajya villages. 6.31 Sufficiency water (40 lpcd): From the household survey, we are able to compute the amount of water collected and consumed by the households. This indicator measures the percent of the village households who consume more than 40 liters per capita per day (an international standard for water sufficiency). The DID estimators suggest that the program helped meet water sufficiency to a modest extent: on average, 5% more households are likely to consume more than 40 lpcd in Jalswarajya villages compared to their counterparts in control villages. 6.32 Water quality (no e. coli in community sources): The testing of water quality from community sources allows us to measure the extent of e.coli. The indicator used here is the percent of community sources for which we were not able to detect any e. coli. The DID estimators suggest that the program had no impact on water quality. If anything, there is weak evidence (p-value of 0.18) that villages in the program are 8% more likely to have contaminated sources. This is not a surprising outcome, given the lack of emphasis on water quality in program design. Nevertheless, this does not bode well for potential health impacts related to community water contamination (or lack thereof). 6.33 Garbage dumps: The household surveys also allowed us to compute an indicator of the percent of the households in the village whose solid waste is dumped in community garbage dumps. Per Table A6-7, we see a modest impact on village cleanliness. On average, households in Jalswarajya villages are 3% more likely to dump their solid waste in garbage dumps at the village outskirts. 6.34 Organized drainage: Similar to the garbage indicator, we can also compute an indicator of the percent of the households in the village whose wastewater flows into organized drains. Per the DID estimation reported in Table A6-7, we find no impact on wastewater management in villages participating in the Jalswarajya program. 6.35 Use of toilet: Based on household reported sanitation practices, we computed an indicator of the percent of the households in the village that used any type of toilet as their main defecation site (i.e., IHL, community toilet, neighbor’s toilet). The DID estimators suggest that the program increased the use of toilet by 6-10% in Jalswarajya villages compared to that in control villages. 6.36 Use of tap water: From the household surveys, we are able to compute an indicator of the percent of the households in the village that used either private or public tap as their main water source. The DID estimators suggest that in the dry seasons the program increased the use of tap water by 8-13% in Jalswarajya villages compared to that in control villages while no impact was observed in the rainy reason. June 21, 2008 71 C. Outcomes 6.37. Dry Season: Although there were various improvements over time as reported in Tables A6-1 to A6-6, we see that Jalswarajya had statistically significant impacts on household water treatment, safe water handling, and diarrhea knowledge (Table A6-8):  Water quality and quantity: The indicator of water quality is whether or not E. coli was present in the household water. The DID estimators suggest that Jalswarajya had no impact the likelihood of detecting e.coli in the household’s primary drinking water container. From the household surveys, we are able to compute the amount of water collected and consumed by the households. This indicator measures water consumption per capita per day in liters (LPCD). The DID estimators suggest that the program had no discernible impacts on water quantity. There is weak evidence that households in Jalswarajya villages consumed 3 or 4 liters more per capita per day compared to those in control villages.  Hygiene behaviors: This indicator measures the self-reported number of critical times caregivers and children under five washed their hands (see Section 6.15.4 for the definition of critical hand washing times). The DID estimators suggest that households in Jalswarajya villages increased the frequency of hand washing to some extent, but the impacts are not statistically significant. In the household surveys, we asked the households whether or not they treated water before drinking. The DID estimators suggest that on average 6% more households are likely to treat their water in Jalswarajya villages compared to their counterparts in control villages. Finally, we measured the number of ways households safely handled their drinking water at home, including the exclusive use of narrow mouth storage for drinking water, covered drinking water storage at all times, and practice of good water transferring techniques (i.e., no direct hand contact with drinking water). The DID estimators suggest that Jalswarajya encouraged safe water handling practices.  Diarrhea knowledge and treatment: The indicator of diarrhea knowledge is measured by the number of key symptoms and causes of diarrhea caregivers could identify. Diarrhea treatment is computed based on whether or not households treated diarrhea at home or sought medical attention. The DID estimators show no significant difference in diarrhea knowledge or diarrhea treatment between Jalswarajya and control villages. 6.38 Rainy Season: Looking at the same outcomes described above, we see that Jalswarajya had statistically significant impacts on water quality, hand washing, and safe water handling in the rainy season.  Water quality and quantity: The DID estimators suggest that households in Jalswarajya villages experienced worse water quality compared those in control villages. On average, 10 more fecal coliform colonies were present in Jalswarajya villages than in control villages. The DID estimators suggest that the Jalswarajya had no statistically significant impact on water quantity. We find weak evidence that households in Jalswarajya villages consumed about 3 liters more per capita per day compared to those in control villages.  Hygiene behaviors: The DID estimators suggest that households in Jalswarajya villages reduced the frequency of hand washing to some extent. We find no evidence that households in Jalswarajya villages increased water treatment. However, there is some June 21, 2008 72 evidence that households in Jalswarajya villages handled their drinking water more safely.  Diarrhea knowledge and treatment: The DID estimators suggest that, similarly to the dry season, Jalswarajya had no additional impact on the improvements in diarrhea knowledge or diarrhea treatment when compared to control villages. D. Impacts 6.39 Dry (pre-monsoon) season: Table A6-8 reports on some indicators of impacts of the Jalswarajya program such as child anthropometrics and cost of illness.  Child diarrhea: The indicator of child diarrhea was based on whether or not a child under five experienced diarrhea 2 weeks prior to the survey. In general, the DID estimators suggest that child diarrhea was reduced in Jalswarajya villages but the impact, as a whole, was no greater than that in control villages. The binary nature of the diarrhea indicator reduces the statistical power of the inference, compared to a continuous indicator such as arm circumference (see next).  Child arm circumference: The household surveys also measured and recorded the arm size of children under five, which generally is a good indicator of child growth. As suggested by the DID estimators, Jalswarajya improved arm circumference of children under five by 0.26 centimeters when compared to control villages.  Averting costs: Using the household data, we estimated averting costs due to inadequate WSS services by adding up the time and money that households spent on water collection, water treatment, water storage, and use of sanitation facilities outside the house. The DID estimators offer weak evidence that households in Jalswarajya villages incurred fewer costs than their counterparts in control villages.  Cost of illness: We also computed the cost of illness due to the most recent episode of diarrhea. It was estimated by converting unproductive days and work days lost into Rupee terms based on an average wage rate in each village, plus the additional medical expenditure. As the DID estimators show, households in Jalswarajya villages had a lower cost of illness compared to those in control villages, a result that is consistent with the improvements in indicators such as arm circumference. The difference is as much as 107 Rupees per household. 6.40 Rainy (post-monsoon) season: As shown in Table A6-8, the following effects were observed between Jalswarajya and control villages.  Child diarrhea: While child diarrhea prevalence was noticeably reduced compared to the baseline, the DID estimators show that control villages experienced a 3% decrease in child diarrhea prevalence, compared to their counterparts in Jalswarjaya. This could possibly be explained by the fact that we did not see any statistically significant improvements in water quality, water quantity, and personal hygiene behaviors. Sub- group analysis discussed in the next section offer some insights on this finding.  Child arm circumference: The DID estimators suggest that Jalswarajya’s impact on arm circumference of children under five in the rainy season was no different from that in the control villages. June 21, 2008 73  Averting costs: There was no statistically significant difference in household averting costs between Jalswarajya and control villages  Cost of illness: The DID estimators suggest that a reduction in the cost of illness for households in Jalswarajya villages was less than in control villages. The difference is as significant as 404 Rupees per household. IV. How Much Variation is There in the Sample? 6.41 Ravallion (2008) has emphasized the importance of “opening the black box of the conditional mean impacts�. One of the first steps towards this is to look at the impacts of the Jalswarajya program in sub-groups of the population sampled to see if there are winners and losers because of the project implementation. Two indicators of sub-groups are households above and below India’s poverty line (APL vs. BPL), and households who are classified as scheduled castes, scheduled tribes, and other tribes (“SCST� vs. “open� caste). Here we report the results of conducting the DID analysis for each of these sub-groups by season and by type of control (i.e., whether we are comparing Jalswarajya communities to matched control communities in-district or outside-district). The next section considers another way of examining heterogeneous program impacts – by potentially different intervention categories. 6.42 Furthermore, to keep things manageable, we focus on a core sub-set of the welfare indicators reported previously, including under 5 diarrhea, under 5 arm circumference, household cost-of-illness, household monthly coping costs, liters of water consumption per capita per day, count of E.coli in household drinking water container, toilet use, and tap water use. The results of the sub-group analyses by season and by type of matched control are reported in Table 6-1. The following general trends emerge:  In the rainy season, BPL households in Jalswarajya villages have higher incidence of diarrhea compared to BPL households in control villages. A similar trend holds for SCST households in Jalswarajya villages compared to other SCST households in control communities. We find weak evidence on improvements in child anthropometrics for APL household in Jalswarajya villages compared to other APL households in control communities.  SCST households in Jalswarajya villages experience a reduction in coping costs (as much as Rs 200 per month) compared to SCST households in control communities. These gains are ten times larger in the dry season.  Because of higher diarrhea, we see higher costs-of-illnesses for SCST and BPL households in the Jalswarajya villages in the rainy season compared to their counterparts in the control villages. There is weak evidence of some reductions in cost-of-illness in the dry season.  In the dry season, BPL and SCST households in the Jalswarajya villages experience an increase in water consumption (of as much as 5 liters per capita per day) compared to their counterparts in the control villages.  In the rainy season, BPL households in Jalswarajya villages experience an increase in microbial contamination of the water stored inside their house compared to similar households in control communities. Similar trends are observed for SCST households in Jalswarajya villages. In dry season, by contrast, microbial contamination declines for both BPL and SCST households in Jalswarajya villages compared to the control villages. June 21, 2008 74 Table 6-1. DID Analysis of Sub-Groups IN-DISTRICT X-DISTRICT POST-MONSOON (rainy) PRE-MONSOON (dry) POST-MONSOON (rainy) PRE-MONSOON (dry) BPL APL BPL APL BPL APL BPL APL Poverty mfx sig mfx sig mfx sig mfx sig mfx sig mfx sig mfx sig mfx sig diarrhea 0.049 ** n.s. n.s. n.s. 0.043 ** n.s. n.s. n.s. armsize n.s. n.s. n.s. 0.341 † n.s. n.s. n.s. n.s. coping costs n.s. n.s. -176 † -244 * n.s. n.s. -178 † n.s. cost of illness 142 * n.s. -208 *** n.s. n.s. n.s. n.s. n.s. lpcd n.s. n.s. n.s. n.s. n.s. n.s. 5.331 ** n.s. log10(E. coli) 0.314 † n.s. n.s. n.s. n.s. n.s. n.s. n.s. no-OD 0.079 ** 0.101 *** 0.087 ** 0.092 *** n.s. 0.065 ** 0.086 ** n.s. tap: n.s. n.s. 0.133 ** n.s. n.s. n.s. 0.156 *** 0.109 * private/public SCST open SCST open SCST open SCST open Caste mfx sig mfx sig mfx sig mfx sig mfx sig mfx sig mfx sig mfx sig diarrhea 0.039 † 0.025 † 0.033 † n.s. n.s. n.s. n.s. n.s. - armsize * n.s. n.s. 0.521 ** n.s. n.s. n.s. n.s. 0.469 coping costs -142 † n.s. -282 ** -179 † -191 ** n.s. -267 ** n.s. cost of illness n.s. n.s. n.s. -176 ** n.s. 177 † n.s. n.s. lpcd n.s. n.s. n.s. n.s. n.s. n.s. n.s. n.s. log10(E. coli) 0.548 ** n.s. n.s. n.s. n.s. n.s. n.s. n.s. no-OD n.s. 0.09 ** 0.064 † 0.107 *** n.s. 0.058 † n.s. 0.093 ** tap: n.s. n.s. n.s. n.s. n.s. n.s. n.s. 0.167 *** private/public *** significant at 1%;** significant at 5%; * significant at 10%; † significant at 15%. June 21, 2008 75  Across both seasons and all sub-groups, toilet use increases in Jalswarajya villages compared to households in control villages. However, this increase is 2-4% greater in APL and open-caste households. There is weak evidence of increase in tap use (and only in the dry season) in BPL households in Jalswarajya villages compared to similar households in control villages. 6.43 Overall impacts (as measured by all health and behavioral indicators mentioned so far) are more pronounced among poor and marginalized sub-groups. For example, in the rainy season, compared to their counterparts in control villages, BPL and SC/ST households in project villages experience lower reductions in diarrhea and illness costs because they experience less reduction in E.coli and lower increases in water quantity. In the dry season in contrast, BPL households in project villages experience lower coping costs compared to their counterparts in control villages because more of them get water from taps. We do not see such differences between non-poor (APL) households in project and control villages. Nor do we see such differences between open-caste households in project and control villages. V. Analytic Statistics Continued: Sub-Intervention Impacts to Uncover Heterogeneity in Interventions 6.44 Next we conduct further analysis to understand and confirm the impacts of a multi- dimensional program like Jalswarajya and the associated heterogeneity. A re-analysis of the household data, corroborated by project records on intervention activities, can provide evidence on the presence or lack of differences in interventions. We employ three different ways to identify sub-categories of RWSS activities and measure potentially heterogeneous impacts. Each is discussed below. As in the previous section, we only look at a small subset of 8 key indicators. A. Water and Sanitation Schemes – Community Leaders & Key Informants 6.45 One way to understand what happened in each village (irrespective of whether the village is participating in the program or not) is to ask village leaders and key informants. The community survey gathered this information from key informants through a module of the survey that was focused on major ongoing water and sanitation schemes. Based on the community survey, the following 4 sub-groups of villages emerged (see Tables A6-9A and A6- 9B)  36 villages in the control group (not participating in the Jalswarajya program) which have at least one significant water or sanitation scheme that is currently operational.  23 villages in the Jalswarajya program that do not have any significant water or sanitation scheme that is currently operational.  33 ‘water’ villages in the Jalswarajya program that have at least one significant water scheme, which is currently operational.  Finally, 23 ‘sanitation or water-&-sanitation’ villages in the Jalswarajya program that have at least one sanitation or water and sanitation scheme which are currently operational. 6.46 As before, these groups were created to capture the local knowledge and perceptions of schemes and programs on the ground. Furthermore, for practical reasons we ensured that no group had too few villages. For example, per community surveys, there were too few (less than 5) villages that were purely ‘sanitation’ only villages and so these were combined with the last June 21, 2008 76 group. In the DID estimation, all 4 groups are compared to a control group of non-Jalswarajya villages that have no significant water or sanitation schemes, which are currently operational. 6.47 DID analyses are conducted for these four “treatment� types using the full sample of 242, with re-classifications for those that had switched their status either from controls to treatment (having gone through all the planning phases) or from treatment to controls (having dropped out). Note, the distribution of villages’ likelihood of participating in the Jalswarajya program is similar in the treatment and control groups, as verified by the PSM model (see Chapter 5). We also tested the robustness of this analysis by only including the sub-sample of 198 villages that have retained their original treatment status (i.e., not switched status). Because we find no discernible differences between the analysis using 242 villages and the analysis using 198 villages, we do not discuss or report the results of the smaller sample analysis. 6.48 The following results emerge:  In the dry season, more households in Jalswarajya villages use toilets. However, this analysis also suggests that tap use increases for households in the 23 Jalswarajya villages that do not have any significant and operational water or sanitation scheme. This finding is difficult to interpret and places some doubts on the quality of perceptions of these informants.  The ‘water’ and ‘sanitation or water-&-sanitation’ villages show significant improvements in arm circumference of children under five in the dry season. These improvements could be possibly explained by the increased toilet use.  Across the groups and seasons, we do not see any change in terms of under 5 diarrhea, household cost-of-illness, averting costs, water consumption, and count of E.coli in household drinking water.  Compared to dry season, the DID estimators suggest that no improvements in any of the indicators mentioned in the previous point. 6.49 In general, the lack of consistent finding of increased tap and toilet use in the more advanced communities – those with ongoing water and/or sanitation scheme suggests some problems with this classification of communities. There may have been some misunderstanding of what constitutes a major ongoing scheme, and the consistency of this answer across all four rounds of surveys. Thus, we treat this as an exploratory exercise and turn to some other more practical and objective indicators that are discussed next. B. Pay as You Go 6.50 Behrman et al. (2005) make a strong case for examining heterogeneity based on duration of the program. Some Jalswarajya villages may not have advanced as much as the others or sufficiently enough to have ‘adequate’ interventions. The Jalswarajya administration releases funds to villages on the basis of progress, paying in three installments (see discussion in sub- section titled “adequacy assessment� for further details). Thus, the “Addendum II funds� data provides another way to create sub-groups of Jalswarajya villages:  Installment 1 is paid to start the work. 29 ‘original’ Jalswarajya villages that did not receive this fund have dropped out and constitute the first group (labeled in the Tables 6-10A and 6- 10B as pay0). June 21, 2008 77  20 villages received this money to start civil work, but not the next installment. They constitute the second group and are labeled in the Tables 6-10A and 6-10B as pay1.  Installment 2 is paid when the actual construction is completed. In total, this group includes 50 villages (labeled in the Tables 6-10A and 6-10B as pay2).  Finally, installment 3 is paid when the local village government (gram panchayat) has collected O&M expenses for 6 months and there is evidence of project completion. This group includes 7 villages (labeled as pay3). 6.51 These four groups do not correspond to different types of interventions – e.g., water and sanitation (see next sub-section). Nevertheless, they offer one way to examine intervention heterogeneity. The DID estimation reported in Tables A6-11A and A6-11B are summarized as follows:  In dry season, there is strong evidence of the increased use of tap water and toilets among pay1, pay2 and pay3 villages, but not in pay0.  In the rainy season, we see less pronounced impacts on the use of toilets and tap water and no impacts on other indicators.  In both dry and rainy seasons, water quantity increases in pay1 villages, and water quality worsens in pay2 and pay3 villages.  Households in pay1 villages experience a reduction in child diarrhea in the dry season. Similar pattern is also observed among households in pay0 villages. 6.52 The lack of expected impacts on tap use and water quality suggests that data on funding allocation has low or no correlation with intervention types. Thus we turn to a more direct measurement and analysis next. C. Tap and/or Toilet Villages 6.53 It is possible that different Jalswarajya villages focused on different interventions, e.g., some focused on water supply, others focused on sanitation, and others on both. For water supply in Jalswarajya villages, we identified two groups: (1) villages that had more than 50% of households using improved water sources (i.e., private or public tap), and (2) villages that had less than 50% households using taps. Similarly, for sanitation, we identified two groups of Jalswarajya villages: (1) villages that had more than 50% of households using a toilet (mostly private latrines, and some community latrines and neighbor’s toilets), and (2) villages with less than the 50% threshold.14 Note, there are some villages currently not participating in the Jalswarjya programs which have at least 50% of the households using taps, toilets or both. When we combined the water and sanitation village types and eliminated types with very few villages, five types of village groups emerged: 14 As discussed extensively with two peer reviewers (Jack Colford and Peter Kolsky), this represents an exploratory attempt to tease out threshold effects and impacts of different potential interventions. Additional work is needed to determine the appropriate threshold (50:50 or 60:60 or what) and to present an evaluation of the comparability of the 50:50 villages and the 0:0 villages. This stop-gap analysis was conducted in response to the fact that communities did not naturally sort into different intervention clusters as was suggested and recommended in the initial concept motivation. June 21, 2008 78  Group 1 comprises 80 villages in the control group (not participating in the Jalswarjya program) which have more than 50% of the households using either taps, or toilets, or both.  Group 2 comprises 21 villages in the Jalswarjya program which have not seen much progress. Less than 50% of the households in these villages use taps or toilets.  Group 3 comprises 10 ‘sanitation’ villages in the Jalswarjya program which have at least 50% of the households using toilets, but less than 50% using taps.  Group 4 comprises of 31 ‘water’ villages in the Jalswarjya program which have at least 50% of the households using taps, but less than 50% using toilets.  Finally, Group 5 comprises of 17 ‘water-&-sanitation’ villages in the Jalswarjya program which have 50% of the households using taps and toilets. 6.54 In the DID estimation, all 5 groups are compared to a control group of 83 villages not in the program which have less than 50% using taps or toilets. 6.55 While there is some arbitrariness involved in these groupings, they were created to capture the policy relevant variation in water supply and to ensure that no group had too few villages. Unlike controlled trials, we do not have the luxury of working with pre-designed groupings. Instead we have non-standard, different sized intervention clusters that reflect some mix of community-demand and local implementation capacity. This connection to realistic scaling up of WSS delivery is the hallmark of the current study. As in the previous two cases, we also tested the robustness of this analysis by only including the sub-sample of 198 villages that have retained their original treatment status (i.e., not switched status). We do not discuss or report the results of the smaller sample analysis because there are no discernible differences. 6.56 Several interesting findings emerge from the results reported in Tables A6-11A and A6- 11B and we highlight only those features that are distinctly different from the general results described previously.  In general, an increased use of taps and toilets is evident in all groups when compared to control villages. We see weak indication in groups 1 and 2 and strong indicator in groups 3-5 that constitute the most advanced groups. This result essentially confirms the definition of the group. The dry season statistics are similar to the rainy season ones. Interestingly, tap use declines in sanitation villages suggesting some displacement or fungibility of public funds and government staff.  In the dry season, households in the ‘water-&-sanitation’ groups (groups 1 and 5) have less E. coli in their household drinking water and households in the sanitation focus villages experience gains in water quantity. While the latter result is puzzling at first, note that the other treatment categories relate to tap use that might relate to quality (not quantity per se). It is possible that quantity would increase in group 3 if households in these groups of villages tend to have more numbers of water sources than the control (no program, no progress) villages.  In the dry season, household cost of illness declines in water-&-sanitation groups 1 and 5 (1 is outside the program, and 5 is inside the program), which is attributed to the reduction in child diarrhea in the same groups.  In the rainy season, in-house water quality becomes worse in relative terms in groups 1, 3, and 4. The decline in groups 1 & 3 could be due to households in these communities having June 21, 2008 79 more toilets in these villages, without a corresponding increase in improved water sources that might be seen as necessary to maintain suitable sanitation.  Overall, coping costs decline in the rainy season, especially in groups 1 and 5. This is presumably because of time savings related to tap and toilet use.  In the rainy season, households in group 3 incur higher cost of illness, higher diarrhea rates and smaller arm circumference. Group 2 also experiences higher diarrhea in rainy season. These maybe because of decline in water quality for group 3. 6.57 Overall, communities from “water and sanitation� groups with significant community- wide advances in water supply improvements and sanitation coverage do appear to enjoy the benefits of reduced E. coli contamination, better child health indicators, and lower cost of illness and coping costs. Importantly, this applies to villages both within and outside the Jalswarjya program, pointing out that the outcome (having good water and sanitation services) rather than the process of getting to it (Jalswarjya or another program) is what really matters. VI. Summary of Findings in Maharashtra 6.58 Participatory community-demand driven projects are by definition slow moving processes for a variety of reasons discussed in the previous sections. Nonetheless, six to twelve months after many program activities have been completed the following kinds of impacts have emerged:  Jalswarajya has had a moderate, but significant impact on reported use of taps and toilets. The DID estimates show a 5 to 15 percent increase in intervention villages in the use of taps as water sources and the use of toilets (and corresponding decrease in open defecation).  There are some seasonal differences in knowledge and practice, and consequently on health outcomes. For example, in the dry season, there is some increase in safe water handling in project villages compared to controls. Yet, there are no corresponding improvements in the potentially more contagious rainy season. Perhaps more tellingly for health outcomes, the analysis does not detect any significant differences in a key hygiene practice - hand washing - across project and control villages, irrespective of the season.  Consequently, while overall microbial contamination is reduced, the reduction is less in project villages (compared to control villages) in the rainy season in project villages. This is presumably because of some combination of reduced prevention behaviors and increase exposure because of incomplete toilet coverage. In the dry season, the opposite holds true – overall trends suggest increased contamination, but the increase is smaller in project villages compared to controls. Water quantity (lpcd) increases in the dry season as well.  Thus, it is not surprising that there are no discernible improvements in child health as measured by diarrhea compared to control villages, while there is significant improvement compared to the baseline. As we might expect, in the rainy season diarrhea does not improve for some project villages (relative to controls) and sub-groups within these villages. The study also has less power for a binary indicator such as diarrhea because the baseline rates are lower than initially estimated. Consequently, continuous indicators such as arm circumference and cost-of-illness are better able to signal changes in health outcomes.  The lack of improvements in key behaviors such as hand washing and safe water handling provides a major lesson for a program implementation. The traditional over-emphasis on the June 21, 2008 80 hardware (of pipes and toilets) at the cost of “software� or behavior chance communication persists. This status quo can significantly limit potential health gains.  Furthermore, there are significant gains to households in terms of the amount of time households spent producing these services (e.g., walking to and waiting at water source), which is one component of welfare effects. 6.59 To explore the heterogeneity of Jalswarajya impacts, we analyzed the data by sub-groups of the population and by sub-categories of RWSS activities. These findings can be summarized as follows:  Jalswarajya’s impacts are more pronounced among poor and marginalized sub-groups (e.g., BPL and SC/ST). Compared to those who are better-off, these groups tend to have more diarrhea and higher costs of illness in the rainy season. They also have lower coping costs. However, these groups also experienced smaller improvements in toilets and taps than the non-poor or open caste groups.  Analysis of villages based on types of RWSS outputs showed that E.coli contamination and child diarrhea worsened during the rainy season in communities that opted for sanitation- only, and communities that attained a modest amount of tap and toilet coverage (less than 50%).  Another source of heterogeneity in the impacts may derive from the fact some villages have a higher propensity to participate in Jalswarajya than others. To explore this source of heterogeneity, we can block the sample by the propensity scores and estimate mean impacts conditional on the propensity score. However, as discussed in section 2.20, such conditional mean treatment effect estimates have limited policy relevance. This is especially true when the conditioning variable is the propensity score, which is unobservable to policy makers and implementers.  The silver lining is that communities with significant water and sanitation improvements (irrespective of whether they were participating in the Jalswarajya program or not, and irrespective of whether they received piped water from private or public taps) witnessed both water quality improvements and health improvements. Thus, increasing coverage of improved water supply and sanitation to the extent when the majority of households in a village benefit from these services – an approach promoted in by the Government of Maharashtra and many other states - is an effective environmental health strategy. Box 6.1 Read More about the Maharashtra Study This report provides an overview of the Maharashtra study, including its methods and main results. The following are references to additional papers and reports that provide more detail on this study.  How valuable are environmental health interventions? Evidence from a quasi-experimental evaluation of community water projects by Subhrendu K. Pattanayak, Christine Poulos, Jui- Chen Yang, and Sumeet R. Patil. RTI International Working Paper 08-01, June 2008.  Health and behavioral impacts of a community-demand-driven water and sanitation program in rural India by Subhrendu K. Pattanayak, Jui-Chen Yang, Benjamin Arnold and Jack Colford. RTI International Working Paper 08-02, June 2008. June 21, 2008 81 7. Conclusions: Moving Forward 7.1 The studies in Orissa and Maharashtra, summarized together, provide an ample opportunity to learn broader lessons about how to conduct evaluations in the water and sanitation sector, and what those evaluations can tell decision-makers about the types of policies and programs that would improve performance in the sector. This chapter offers some insights and lessons on each of these in the sections that follow. I. In Search of Impacts: Is the Tap half-Full or Half Empty? 7.2 To assess the lessons of this exercise for future evaluations, it is useful to directly compare the Orissa and Maharashtra approaches. Briefly, the Total Sanitation Campaign evaluated in Orissa was a community-led, people-centered program, which sought to empower communities and individuals to set their own sanitation goals and carry them out. It emphasized information, education, and communication (IEC) activities to change knowledge, attitudes, and practices (KAP) regarding water, sanitation, hygiene and health (WSH). The desired intervention output was construction of individual household latrines, the desired outcome was regular use of the latrines, and the desired impact was improvement in child health. Small subsidies to support latrine construction were offered to poorer segments of the population. 7.3 As mentioned, the study design for Orissa was a social experiment in which treatment and control groups were randomly assigned. Several features of the intervention made it suitable for randomization. First, the intervention – intensified IEC campaign – was homogeneous across villages. While each village developed its own sanitation goals, these goals were developed within the CLTS framework, which was operationalized similarly in all treatment villages. This homogeneity was possible because the intervention was implemented by a single social mobilization organization. Second, this intervention had a relatively short and simple causal chain, or pathway from activities to impacts. There were a small set of IEC activities undertaken in each treatment village (i.e., community meetings, walk of shame, technical assistance). These behavior change communications are expected to change KAP, which lead to impacts. As we discuss later in this chapter, these features make the Orissa intervention more amenable to a randomized study. 7.4 Typically, it is challenging and difficult to assign scalable water and/or sanitation interventions randomly. Thus, the Orissa study offered a rare opportunity for a field experiment. The villages were randomly assigned to treatment or control groups, while individuals remained the primary unit of observation. This is appropriate for evaluating a program like the Total Sanitation Campaign, which operates at the group level. 7.5 The Jalswarajya program in Maharashtra invited villages to apply to participate in the design and implementation of water supply and sanitation of their own choosing. Selected villages went through a community capacity-building process. Ninety percent of the up-front capital costs were borne by the government, but the villages also paid their ten percent share and were responsible for all costs of operations and maintenance. The desired outputs were improvements in water supply and quality, excreta management, environmental sanitation, and health and hygiene training. In principle, the desired impacts were improved child health and other social indicators such as better school attendance, time savings, and lower heath care expenditures. June 21, 2008 82 7.6 In contrast to the intervention in Orissa, Jalswarajya is a large intervention that is heterogeneous across villages. Since villages select from a range of hardware and software that best meet their needs, the activities and stakeholders (most notably, the implementers) vary across villages. As a consequence, the intervention and exposure to the intervention are likely to vary across villages as well. Furthermore, software outputs relating to behavior change communication were no different in Jalswarajya villages compared to control villages in Maharashtra. This is in direct contrast to Orissa, where knowledge and attitude change was emphasized, while ensuring supply-side factors (masons, raw materials, technical know how) was sufficient. 7.7 On a related point, the causal chain for Jalswarajya is longer and more complex than that in Orissa’s intervention. Because of these features, the study in Maharashtra was quasi- experimental: a non-randomized method, using propensity score matching to identify comparison groups. For every treatment village, a matching village was found from among the villages not participating in the program. The matched community having an equal probability of participation on a set of observed characteristics. Additionally, checks were conducted to ensure no additional observable significant differences between treatment and control. 7.8 Both studies estimate intention-to-treat estimates of treatment effect as the difference between mean outcomes between communities with the program (whether individuals avail themselves of improved services or not) and those communities without the program. As discussed in section 2.19, since there are individuals in the treatment group who do not use improved services, this ITT estimate of the treatment effect underestimates the effect of the treatment on the treated (Galasso and Umapathi 2007). Despite this limitation, ITT estimates are widely used in the evaluation of community-based impact evaluations and can be highly policy- relevant since nearly all programs are administered to groups made up of individuals that must choose between “adopting� the improvements or not. 7.9 Futher, these studies estimate population average treatment effect (PATE), as well as the average treatment effect by key subgroups. While the PATE may be an inaccurate measure of individual or household level treatment effects if the sample is heterogeneous, they are policy relevant because these estimates reflect the effect that policy makers can expect when the program is implemented in similar settings. Programs and interventions precisely targeted enough to target specific sub-groups in order to achieve conditional mean treatment effects. 7.10 The studies’ findings that diarrhea rates decrease over time in both treatment and control villages underscores three important messages. First, the use of controls in impact evaluations is central to estimating the impacts of programs. Had these studies not included control villages, the findings would have falsely attributed the decline in diarrhea to the programs. Second, as discussed in section 2.3 (and below), there are likely to be other activities and programs in the control communities that affect the indicators. On the one hand, controlling for these activities ensures that the treatment effect is not over estimated. On the other hand, if these other activities are a result of spillovers (see section 7.22), these treatment effect estimates are under-estimates. Third, diarrhea rates are highly variable from year-to-year and affected by several unobservables. Other studies of diarrhea have observed that control units experience a reduction in diarrhea (personal communication with Jack Colford; Kremer et al., 2008). 7.11 Overall, the differences and similarities between the two studies study are hardly surprising. A wide variety of double-difference (DID) results suggest that both projects have June 21, 2008 83 improved outputs and some outcomes, but these have not as yet translated into impacts at scale. Where the chain is shorter and the intervention more focused (Orissa), we see significant improvements even three-six months after the intervention. In Maharashtra, the participatory and deliberative aspects of “decentralized delivery� make it unreasonable to expect quick and definitive impacts. Additionally, behavior change communication was the focus in Orissa. In contrast, information and communication was much more diffused in Maharashtra. II. Improving Design of Evaluations and “M&E� of WSH Projects 7.12 The study applications of a randomized community trial in Orissa and quasi-experimental design in Maharashtra are among the first rigorous impact evaluations in the water and sanitation sector to focus on broad set of impacts including not only health, but also socioeconomic and other impacts. While these studies are novel in their application of established study design features to the sector, their findings have reinforced what is known about the strengths and limitations of these designs. 7.13 In particular, randomized controlled trials, which are considered the gold standard in providing evidence to support decision-making in public health (Victora et al., 2004) and other policy domains (Duflo and Kremer, 2003), provide internally valid measures of impact when the intervention is implemented as it was in the study. However, as Victora et al. (2004) assert, for these impacts to be valid, they should also reflect how the intervention would be implemented in the real world under less than ideal conditions. By their nature, randomization frequently precludes routine implementation practices in order to satisfy the conditions of the study. Thus, when randomized evaluations are implemented, they may answer only a narrow set of policy questions, limited by institutional constraints and be subject to randomization bias and substitution bias (Heckman and Smith, 1995). Further, there are many projects of relevance to sectoral decision-making for which randomization is not ethically or politically feasible. Thus, Victora et al. (2004) argue that they are never sufficient to support decision-making by themselves because they do not account for multiple interventions, long and complex pathways (or chains) to impacts and both behavioral and biological effect modification. When these conditions are present, they argue that either randomized trials should be strengthened by adding quasi-experimental study features or replaced with quasi-experimental evaluations (referred to as “plausibility evaluations� in their paper). 7.14 This study reinforces Victora et al.’s (2004) arguments. Jalswarajya illustrates the types of complexity that are rife in the water and sanitation sector – complexity that limits the usefulness of controlled trials and increases the usefulness of quasi-experimental designs.  First, mechanisms to achieve sector goals are broad and varied in terms of the types of services (water supply, water quality, sanitation, sewerage, and hygiene); the setting (urban, peri-urban, rural); and the typology of delivery (public intervention, private interventions, decentralized delivery, expansion or rehabilitation).  Second, decentralized and community-demand-driven (CDD) programs are an important and growing class of development projects that are not amenable to randomized trials. For example, there is an inherent contradiction in randomly assigning one community to demand interventions and another to not.  Third, many important impacts of these poverty reduction programs lie well outside the sector due to the long and complex pathways from resources and activities to impacts. For June 21, 2008 84 example, most rigorous evaluations of large-scale water and sanitation programs are done ‘outside’ the sector by evaluators either focusing on health outcomes (e.g., Galiani et al., 2005) or considering water and sanitation outcomes as part of a broader development package (e.g., Social Investment Funds summarized by Rawlings et al., 2004). 7.15 Given this context, it is critical to look beyond randomized community trials (RCT) to study these messy, complex but real projects, programs and policies. Clearly, there is a complementary role for experimental and quasi-experimental evaluations. 7.16 While these complexities underscore the need for quasi-experimental design features in WSS evaluations, randomized controlled trials are critical for understanding what impacts are possible under ideal conditions. A randomized trial like the one in Orissa has the advantage of both controlling what the intervention includes and controlling how it is administered. By limiting the intervention to a concise set of activities and controlling participants’ exposure to the intervention, a randomized study helps us understand whether an intervention is capable of causing impacts. This is a prerequisite for achieving impacts under more complex and realistic conditions. On the other hand, quasi-experimental studies can accommodate the complexity of routine implementation, but they may do so at the expense of control over the intervention and participants’ exposure to the intervention. Despite this, quasi-experimental studies are capable of a rigorous and internally valid (i.e., controls for selection and other biases and rules out confounding) analysis of impacts. However, when the interventions are too ill-defined in that the activities and the participants’ exposure levels are broad and variable, it is difficult to identify appropriate comparisons, which limits the impacts that even the most rigorous statistical methods can detect. 7.17 The studies described in this report also reinforce the use of many “best practices� in both impact evaluations and monitoring and evaluation. We highlight several in this section.  First, the evaluation team should allow sufficient time to meet with both clients and key stakeholders in the program being evaluated. This study began with a scoping trip by PI and the core research team, which entailed meeting with in-country World Bank and government staff to start a two-way dialogue about the content and structure of the programs under evaluation and the structure and importance of the impact evaluation. Participant communities were visited to learn and observe the conditions under which the programs were implemented. These interactions with stakeholders were critical for building the political and institutional support early in the evaluation. Regular visits by the project team (two or more times a year) throughout the four-year study period served to boost engagement, which tends to wane over time, by re-engaging original stakeholders and/or by engaging stakeholders new to the process due to turnover in implementing agencies. The purpose of these visits varied from meetings to exchange information on status of the implementation and evaluation to workshops for local government to build capacity in evaluation. Evaluations require discipline in order to maintain coherent and consistent interventions over time. Regular interactions with stakeholders serve to remind them of the importance of fidelity to program plans and encourage them to stay the course.  Second, it is important to develop a program theory or logic model. The model serves to articulate the causal chain and the expected relationships among activities, outputs, outcomes, and impacts. The model is an important tool in the design and planning of the evaluation and can be used to communicate with the stakeholders about their role and their knowledge of program operations. It should be refined during the planning stages and used to June 21, 2008 85 inform the selection of indicators of each step in the chain. Finally, the length and complexity of the final causal chain will determine the number of indicators that are required, and it will inform the selection of appropriate study design and data collection methods, with longer and more complex chains indicating the use of quasi-experimental methods as the main design, or as a supplement to randomized designs.  Third, evaluations should get off to a quick start. Baseline measurements should reflect conditions without the intervention. In fact, it is difficult to obtain such measures because early planning of interventions may budge indicators in anticipation of the intervention. For example, implementing agencies may make subtle changes in processes in preparation for the project. Or, in the process of becoming aware of possible community-driven projects communities may become more aware of unsafe water and sanitation conditions and change behaviors and conditions (e.g., changes in perceptions in Jalswarajya villages by the time of the 2005 ‘baseline’ measurement). Because indicators are expected to continue to change throughout program implementation, it is important to obtain measures as early as possible in order to have the best chance of showing the effects of the project.  Fourth, evaluations should make mid-course corrections when possible. Programs and interventions are rarely, if ever, implemented precisely as planned. And this is especially true for complex programs trying to effect change in a long and complex pathway to impacts. For instance, delays in implementation are commonplace and even more likely in community- demand-driven interventions which rely on beneficiary participation in several stages. The endline surveys in Maharashtra were postponed for one year when the qualitative adequacy study found that the implementation of Jalswarajya was behind schedule. By adjusting the evaluation design, we reduced the likelihood of a finding of “no significant program impact� when, in fact, the program had not been implemented. Similarly, our recommendations to collect another round of data in Orissa is based on our findings to date and secondary data that project outcomes (IHL ownership and use) have continued to improve since our endline study. By adjusting the original study design to include another wave of data collection, this study increases the likelihood of both finding significant health and socioeconomic impacts and our understanding of the pathway that leads from the intervention to these impacts. The capacity to make these corrections depends on both the design of the study and its flexibility, as well as the types of deviations from program design that occur. Bamberger et al. (2004; 2006) have developed a modified impact evaluation framework specifically for those cases in which analysts must conduct impact evaluations under budget, time, and data constraints.15  Fifth, in communicating with key stakeholders the evaluation team should cultivate a learning and participation mindset. While evaluations can serve an important role of ensuring accountability by focusing stakeholders on evidence of program effectiveness, they have an equally important role in contributing to publicly available information on what works and what does not work. A narrow focus on accountability can undermine contributions to the public domain by limiting the scope of the evaluation, cutting the evaluation short when key accountability lessons are clear, limiting the scope of the data collected to that which bears on questions concerning accountability. For example, impacts may take years to materialize 15 These may occur when the evaluation is begun well after the program design and implementation or when baseline data is unavailable because of budget or political realities. Their framework offers a structured approach to addressing the constraints in order to ensure the highest quality evaluation possible. June 21, 2008 86 and may peak well after the project cycle is complete. Implementing a study that measures indicators after the project cycle is complete requires an evaluation team (including implementing agencies and, to some extent, beneficiaries) who are committed to a longer- term study. Also, evaluations should attempt to open the “black box� in order understand the activities and processes that give rise to outcomes and impacts. To learn which interventions and features are effective, it is important to use a variety of data collection approaches. The studies reported here relied on qualitative assessments of the adequacy of the intervention to understand the status of implementation. These assessments also provided different types of data on how well the intervention was working.  Finally, many of these recommendations for “best practice� evaluations require the commitment and leadership of WSS project managers. Given their central role in both the implementation of the intervention and the evaluation, it is critical that they are advocates for these practices, are knowledgeable about the evaluation, and encourage and lead other stakeholders to commit to the evaluation process. In particular, WSS project managers should understand the counterfactual, why it is important, and how it is best measured. This entails a willingness to collect baseline and covariate data in control or comparison sites near the project sites. Many M&E frameworks are actually what Victora et al. (2004) call “adequacy evaluations� and they document time trends in key indicators in treatment sites only. To establish causality, it is necessary to expand measurement to include control or comparison sites that will represent the counterfactual (or, “without intervention�) situation. This may be counterintuitive to some WSS project managers, as well as their staff and other stakeholders. Finally, the leadership of the WSS project managers is critical for encouraging other stakeholders to stay the course complete the evaluation. As mentioned above, interest and engagement wane over time, yet the endline data is crucial. 7.18 For large, long and complex projects like Maharashtra, “in short, we probably need to scale up something that is in short supply, namely, patience� (Devarajan & Kanbur, 2004). Another fundamental issue is what exactly can be evaluated in the conditions of a very dynamic economic and social transformation in India including many rural areas. III. Advancing RWSS Policies and Programs: Main Lessons from the Study 7.19 As revealed in both sites (and consistent with broader literature), for both epidemiological and social reasons, an individual household’s payoff to behavior change depends in part on the decisions of other households in the community. For example, in Maharashtra high-coverage interventions yield health benefits that low-coverage interventions do not seem to bear. Thus community coordination is vital for supplying an environment that is free of microbial contamination. These significant externality dimensions need to be factored in the design of the RWSS programs and policy incentives, so that market signals and government laws influence community norms which hare critical to achieving meaningful gains. 7.20 The Maharashtra study also demonstrates the decreases in dry season E.coli and diarrhea in communities with significant water and sanitation improvements (i.e., more than 50% of the community using taps and toilets). Importantly, while some earlier cross-sectional studies in India suggested that health benefits are realized only if a community achieves more than 50% coverage in private taps or wells, this study records the impact of piped water from private or public taps (the latter is the main method of providing water in rural India). This suggests that water and sanitation interventions, as currently provided in rural India and aiming at community- June 21, 2008 87 wide coverage, are important environmental health strategies because of potential externalities in prevention of and infection from water-borne and water washed diseases. 7.21 The “externality effects� are more significant in the rainy season than in the dry season. Furthermore, the linkages between RWSS inputs, outcomes and impacts differ across seasons. To maximize the benefits, the design of the RWSS programs should address both types of interactions. 7.22 An evaluation of the intensified IEC campaign identified several key barriers to latrine adoption, as well as factors that help to overcome these barriers, summarized below:  Knowledge about the Advantages of Latrines: Beyond Health. Interestingly, awareness of the health linkages is already good but does not, by itself, lead to changing behaviors. Prior to the sanitation program, over 90% of households cited open defecation as a cause of diarrhea, yet this knowledge alone was not enough to generate widespread latrine use. It may be that specific information about water-borne diseases and ways to reduce those, rather than general knowledge, would influence people’s choices. What appears clearly is that attitudes about the importance of privacy and dignity can play a key role in determining households’ demand for latrines.  Ability to Obtain Latrines: Beyond Subsidy. Part of the campaign’s impact was almost certainly due to its role in increasing the supply of materials, along with the technical ability to construct latrines. With respect to latrine subsidies, there is no evidence that targeted subsidies for latrines to BPL households impede adoption – both BPL and APL households were just as likely to adopt latrines. However, given strong externality effects of sanitation (and even water supply improvements), there is clearly a case for rewarding communities for achieving village-wide level of service coverage rather than just subsidizing individual households.  Collective Action to Change Social Norms: Beyond First Impulse. It is likely that a large part of the IEC campaign’s success was owing to the emphasis placed on addressing social norms and helping households to overcome collective action problems. By targeting whole communities rather than individuals, the intensive IEC harnessed the power of social pressure to conform to accepted practices. Some villages even began to develop systems of fines or punishment for households that did not comply with the new “universal latrine use� mandates. 7.23 These findings provide weak evidence that coverage of taps and toilets lead to less diarrhea and more time savings and could serve as proxy indicators of an effective intervention. Further work needs to be done to demonstrate the strength of these proxy indicators, particularly given the short follow-up period in this study (See Chapter 7 for a discussion). 7.24 Both sites focus on community environmental interventions not individual/household level interventions because water-borne and water-washed are caused by local prevention and infection externalities. Positive health impacts in high-coverage communities (e.g., Maharashtra) provide empirical support this logic. These externality effects justify rewarding communities for achieving village-wide level of service coverage rather than just subsidizing individual households. Thus, incentives such as the GoI’s Nirmal Gram Puraskar (or clean village award) should be expanded. June 21, 2008 88 7.25 Often, “control� communities are subject to similar programs done by other agencies – or different programs affecting the same health and socio-economic outcomes (e.g. programs in health, education, poverty-reduction). This allows our double-difference estimates of treatment effect estimates to be realistic in the sense that they measure the contribution of the program being studied over and above the other programs and interventions being implemented in the study population. As with most carefully conducted rigorous evaluations (compared to those that lack control groups or pre-post measurements), the additional gains from such real world programs are bound to be small. This is particularly true for complex intervention packages that take 2-3 years to complete on indicators that are affected by several multi-sectoral interventions against a backdrop of rapid growth, massive change, multiple programs, and active cross- learning. On the other hand, this issue may be compounded by the presence of a type of spillover effect described by Ravallion (2005 and 2008), among others. These spillovers occur because external aid spent on the program understudy displaces government and other sources of aid, which may be redirected to the control communities. Thus, control communities benefit from this reallocation of funds toward activities that improve their outcomes and the treatment effect is under-estimated. Disentangling these influences is difficult, so before launching new major evaluations, it is important to be realistic in expectations given the dynamic policy and socio- economic environment in which interventions are administered. In most cases, limiting the majority of evaluations to very specific, confined initiatives, with a well attributed outcome impact seems a reasonable way to go. Large-scale exercises like the one reported here should be undertaken very selectively. 7.26 It is also clear that tap or toilet (“hardware�) subsidy is not a substitute for a very intensive IEC campaign triggering a mass behavior shift. An intensive model of community mobilization like the one studied in Orissa has proved impressively effective. Before going to scale, it is important to recognize that the capacity of state government departments/project units may fall short of what is needed to implement such initiatives as village-level, intensified IEC campaigns. Thus, one efficient solution is to provide RWSS departments/units with sufficient funds to contract out the design, implementation and monitoring of such interventions. June 21, 2008 89 Annex 1. Power Calculations How big a sample is needed depends on the number of service packages to be tested, outcome indicators and control variables, the unit of analysis, etc. The goal of sample size calculations is to identify the minimum efficient number of observations needed to ensure adequate statistical power. So long as these assumptions are consistent with conditions in the field, the results of the sample size calculations are valid. Since the primary research question focuses on health outcomes – in particular, diarrhea rates in children five years of age and under, we compute the sample size necessary to measure these health effects. As described in Donner et al. (1981) and Diggle et al. (1994), the size of the sample necessary for a health treatment study of this nature can be calculated using Equation [1] �1  � �11  �1  � 1  �2   2  z  z�  2 1  �2 a1  a a 1  a   [1] n   r  �1  �2      where:  π1 = p + d; π2 = p; and π = a*π1 + (1 − a)*π2  d is the smallest difference (i.e., health impact) that it is important to be able to detect or the desired treatment effect  a represents the proportion of the total sample that will be allocated to the intervention group  α is the significance level to be used in the statistical tests  1−β is the power of the study;  p indicates the approximate proportion of positive outcomes in the control group or the baseline prevalence rate  r is the response rate, which depends on attrition of the intervention and non-intervention sub-groups during the course of the study, due to treatment spillovers, and individual or household non-compliance among other things. Because the intervention includes the allocation of villages to intervention packages along with measurements on primary outcomes of interest taken from individuals nested within the villages, we will employ sample size estimation procedures appropriate for group-randomized trials (Blitstein et al. 2005). These calculations account for the multiple sources of random variation June 21, 2008 90 that arise in the evaluation of correlated data, ensure adequate statistical power to assess the effect of the intervention, and protect the nominal Type I error rate. Sample size calculations are relatively more complex when the units of analysis (i.e., respondents) and nested within the units of assignment (i.e., villages). First, the number of villages and the number of respondents per village must be balanced. Consideration must be given to statistical, logistical, and financial matters. Statistically, it is usually advantageous to maximize the number of villages as power is most directly effect by this number. At the same time, a sufficient number of respondents per village must be interviewed to ensure a representative sample is included for each village. Logistically and financially, it is usually more difficult and more expensive to recruit and survey more villages than it is to interview more respondents per village. Second, individuals’ data will be correlated to an unknown degree. This correlation arises from the fact that respondents within the same village will have shared histories and common experiences that make them more alike to each other than they are to respondents in another village and introduces a component of random variation that is attributable to the village over and above the random variation associated with the individual respondents. The degree of within village correlation can be expressed as the intracluster correlation coefficient (ICC) which indexes the proportion of total variation in the study attributable to the village.16 Sample size estimation involves a number of parameters and assumptions. These include: (a) the Type I and Type II error rates; (b) the anticipated effect of the intervention, often referred to as the effect size estimate; and (c) an expression for the anticipated ICC. To begin, we establish our Type I and Type II error rates. These reflect the evaluator’s willingness to reject a true null hypothesis and to accept a false null hypothesis, respectively. We set the Type I error rate at 0.10, and the Type II error rate at 0.20 to provide a test of the intervention effect with 80% power to identify statistically meaningful differences between intervention condition. Further, we employ a two-tailed test when we assess the effect of the intervention. This last assumption is conservative in the sense that it places a heavier onus on the evaluation, but is appropriate in field trials where it would be important to observe intervention effects that are not in the desired direction. The effect size measures the expected change in the diarrhea rate among individuals exposed to the treatment. The effect size estimate is determined through a review of the literature. Previous studies that have examined similar outcomes provide a reasonable expectation from the impact of the planned intervention. Based on Fewtrell et al.’s (2005) meta-analysis of the health impacts of water interventions, we assume an estimated effect size of 30%. When an effect size estimate is based on a percent change, it is also important to understand and incorporate information on 2 �g . Where �g:c indicates village-level variation and �e :c 2 2 16 The ICC can be expressed as ICC  �g2  � 2 :c e indicates respondent-level variation. The ICC is the critical factor in the design effect (DEFF) which describes the magnitude of additional variation found in a GRT relative to a study that employed simple random sampling. The DEFF is expressed as DEFF=1+(m-1)ICC. Here, m indicates the number of respondents per village. If the ICC is very small and the number of respondents per village is also small, the DEFF would be close to 1, indicating little additional variation. However, as either of these factors increases, so too does the DEFF and study level variation. June 21, 2008 91 the current prevalence rates as this provides an estimate of the baseline diarrhea rates in the study population. It is also instrumental in determining the realized value of the intervention. Based on National Family Health Survey-II, the diarrhea prevalence rate in children five years and under is about 20% in rural Maharashtra. In rural Orissa, the prevalence rate is about 30%. An estimate of the ICC is also obtained from a review of the published literature. Katz and colleagues (Katz et al. 1993) examined the clustering of diarrhea rates at the village-level in several developing countries and provide measures of the DEFF. With cluster sizes standardized to 50 households the DEFF ranged from 1.38 to 4.73; we calculated ICCs that ranged of 0.008 to 0.076. Hence, for our sample size calculations employed an ICC of 0.05, a conservative estimate within the range indicated by Katz et al. (1993). Our previous work in the region suggests that we can expect 10 percent loss to follow-up or non- compliance. Based on the assumptions and parameters noted, our sample size calculations indicate that sampling 40 households with children five years of age or younger in each village in Maharashtra will generate a sample that is sufficient to provide 80% power to detect an intervention impact of 30% or greater in a population with a baseline diarrhea prevalence of 20%. This implies we need an overall sample of 3,000 individuals per intervention or a total of 9,000 individuals to evaluate potentially 3 interventions (as matched pairs of treatment and control). In Orissa, our power calculations indicate that sampling 25 households with children five years of age or younger in each village will generate a sample that is sufficient to provide 80% power to detect an intervention impact of 35% or greater in a population with a baseline diarrhea prevalence of 30%. This implies we need an overall sample of 1,000 households. It is important to recognize that these calculations are based on best available information and buffered by a number of reasonable assumptions to help us protect the desired goals regarding statistical power and the planned tests of intervention effectiveness. For example, we incorporate conservative assumptions regarding the reduction in study level variation associated with taking repeated measures on respondents and villages. Further, the addition of covariates related to the outcome can further reduce random variation. These factors can improve statistical power and their place in the final evaluation will help to protect our analysis in the event that our parameters are very different from their assumed values. In addition, the sample size and sampling plan are based on our best information and judgment concerning other study conditions, including project implementation schedules and expected uptake rates. In these calculations, there is an inverse relationship between the number of villages and the number of households required from each village. In selecting to work in 50 villages, we balanced several technical and budgetary factors. On the one hand, by increasing the number of villages and reducing the number of households per village, we can increase the amount of independent data and distribute the potential bias more evenly across intervention packages. This reduces overall sample size, but increases costs of transportation during data collection. On the other hand, while the intervention takes place at the community level, the decision to use, water from the improved source for example, is made by individual households. Consequently, the proportion of the population and the sample that would be using the intervention at the endline survey is uncertain. Sampling a larger number of households in each village increases the likelihood of interviewing users, which permits investigation of factors affecting usage. This increases the sample size and the study is more vulnerable to statistical problems if projects are not completed on time and villages need to be removed from the sample. June 21, 2008 92 Annex 2. Impact of Intensified IEC Campaign on Adoption of Individual Household Latrines in Orissa [Excerpted from Pattanayak et al. 2007b, pp. 8-10, 28, & 30-31.] We estimated the impact of the sanitation campaign on latrine uptake using several different models. Given that treatment was randomly assigned, the comparison of village-level means provides the most simple treatment effect estimate. Random assignment assures us that treatment conditions are uncorrelated with potential outcomes in expectation. Based on the means comparison presented in Table 1, the estimated impact of the sanitation campaign on IHL adoption is the difference in means in 2006, or 19%. However, since we know that treatment and control villages were somewhat different prior to the intervention, and that, in particular, IHL ownership was significantly lower in treatment villages in 2005 compared to controls, we can obtain more precise estimates by controlling for these differences in a number of different ways. Moreover, we are ultimately interested in understanding outcomes (IHL adoption and use and health outcomes) at the household level. By analyzing outcomes at the village level, we miss heterogeneity in household characteristics that may play a key role in determining sanitation and health outcomes. Thus, we conduct a number of analyses at the household level, controlling for various combinations of household- and village-level covariates. Table 2 presents results of various model specifications, where the dependent variable is household ownership of IHL: Yijt =1 if household i in village j owns a latrine in year t, and Yijt =0 otherwise. Since this outcome is binary, the models were estimated using probit regressions. Models 1-4 are variations of the model: Yijt  E [ Yijt | Tit , Xijt ]  uij E [ Yijt | Tit , Xijt ]  P ( Yijt  1 | Tit , Xijt )  1  F (  �1Tit  �2 Xijt ) Tit =1 if village i was in the treatment group in year t, Xijt is a vector of village and household characteristics, and F is the standard normal cumulative density function. Standard errors are clustered at the village level. In models 1 and 2, the dependent variable is latrine ownership in 2006, and covariates include a number of village and household characteristics. Model 2 is a difference-in-differences model, where the right-hand variables are Ti=1 if the village was assigned to the treatment group, a if the village was assigned to the treatment group, a Yeart variable that takes a value of 1 in 2006 (after the intervention), and the interaction term, Ti *Yeart, which will take a value of 1 for treatment villages in 2006. The coefficient on the interaction term gives the difference-in-differences (DID) estimate of the sanitation campaign’s impacts. Model 3 uses the same DID setup, and also includes both socioeconomic controls. Model 4 is similar to model 3, except now we include village level fixed effects and estimate this as a linear model. Not surprisingly, all of these models confirm that the sanitation campaign had a substantial and statistically significant impact on latrine adoption. Estimated impacts range from a 29% to a 36% increase in IHL ownership. These estimates also confirm that, because of the initially lower level June 21, 2008 93 of latrine ownership in treatment villages, the simple means comparison for 2006 underestimates the impact of the sanitation campaign on IHL adoption. There may be some concern that imbalance in key variables in the baseline might invalidate the use of the DID estimator (see Ravallion, 2005)? First, we might be concerned that the baseline imbalance influences the likelihood of program placement. This was clearly not the case because villages received the treatment as a result of a randomized assignment. Second, we might worry that differences in initial coverage would have slowed or hastened uptake/adoption. Only one important variable (toilet ownership) was somewhat imbalanced: toilet ownership. This had no bearing on baseline diarrhea rates, other disease rates, water quantity, water quality, hygiene practices, demographics, opinions, knowledge, education levels, distance to public infrastructure, community participation, and civic capital. So can a small difference in toilet coverage (substantially higher only in 2 out of 20 control villages) alter adoption rates? We might suspect that it would be hard to induce change in communities that had no toilets because there must have been a reason they had few toilets after all these years (in the baseline). If this is true, our estimate is a lower bound of the true impact. We implement two robustness checks. First, we include baseline toilet ownership as a linear control in a impact evaluation model of “change� (i.e., a probit model of whether a household adopted a toilet in 2006 as a function of treatment assignment and several controls, including 2005 ownership). The treatment effect is still significant and the impact estimate does not change. Second, we follow the literature (Abadie, 2005; Hirano et al., 2003) are estimate a semi-parametric DID model, which essentially uses inverse probability weights that are function of covariates that we might be concerned about – e.g., toilet coverage. Again, the size and significance of the impact estimate is virtually identical. Thus, we are reasonably reassured that the campaign did cause at least most of the observed toilet use. June 21, 2008 94 Table A2-1. Comparison of Means for Outcome Variables† Variable Year Overall Treatment Control T-C 9.7% 6.4% 13% -6.4%*** 2005 (.9%) (1.1%) (1.4%) (1.8%) % owning IHL 23% 32% 13% 19%*** 2006 (1.3%) (2.0%) (1.5%) (2.6%) 26% 28% 23% 4.9% 2005 % of children<5 suffering from (1.2%) (1.8%) (1.7%) (2.4%) diarrhea in past 2 weeks 15% 15% 16% .8% 2006 (1.1%) (1.5%) (1.7%) (2.3%) 13.77 13.73 13.83 -0.10 2005 Arm circumference of children (0.04) (0.05) (0.05) (0.07) <5 (cm) 13.91 13.98 13.84 0.14* 2006 (0.04) (0.06) (0.06) (0.08) 107 101 112 -10.7 2005 Total coliform level in household (5.17) (7.26) (7.34) (10.3) drinking water†† 130 126 134 -8.59 2006 (5.08) (7.23) (7.14) (10.2) 10.8 10.4 11.2 -.80 2005 E. coli level in household (1.61) (2.26) (2.31) (3.23) drinking water†† 14.4 14.7 14.1 .56 2006 (2.00) (2.94) (2.73) (4.00) † Standard errors are in parentheses †† Water quality testing was performed for a sub-panel of 50% of surveyed households in 2005 and 2006. Sample sizes for water quality measures are thus 553 (270 treatment, 283 control) in 2005 and 529 (263 treatment, 266 control) in 2006. * = significant at 10% level, **= significant at 5% level, ***= significant at <1% level June 21, 2008 95 Table A2-2. Treatment Effects Analyses for Impact of Intervention on IHL Uptake (marginal effects from probit regressions) Model 1: Model 2: Model 3: Model 4: Probit (2006) & Probit DID‡ Probit DID & Linear DID, Controls ‡ Controls ‡ Village FE & Y = household has toilet Controls Intensive IEC (‘treat’) 0.293*** -0.092** 0.043~ Post 0.003 -0.182** 0.019 Intensive IEC (‘treat’) x Post 0.305*** 0.303*** 0.247*** IHL-2005 Yes SES-2005 Controls† Yes Yes Yes KAP-2005 Controls†† Yes Yes Yes N 1043 2136 2122 2122 Pseudo R.Sq 0.275 0.075 0.25 0.202 ‡ errors are clustered at village level † includes household’s religion (Hindu), caste (open caste), land ownership, TV ownership, and electricity connection in 2005 †† includes whether household knowledge, attitudes and practices. Attitudes include if they think their village is very dirty, if they completely dissatisfied with their current sanitation situation. Knowledge includes ability to correctly identify symptoms and causes of diarrhea and exposure by 2005 to TV and radio campaigns that focused on toilet and sanitation. Practices include whether they treat or boil their drinking water, how often adults and children wash hands (after critical daily activities – eating, defecating), and their participation in community activities such as sweeping streets and cleaning drains. *= significant at 10% level, **= significant at 5% level, ***= significant at <1% level June 21, 2008 96 Annex 3. Impact of Intensified IEC Campaign on Child Diarrhea in Orissa [Excerpted from Pattanayak et al. 2007b, pp. 11-14 & 32-34.] The evidence we have presented to date indicates that the sanitation campaign had the intended impact on latrine construction and use in our study villages. In a single year, latrine ownership among households in treatment village increased from 6.4% to 32%, while the percentage of households in control villages owning IHL remained constant at about 13%. We also observed that a handful of villages in the treatment group now have more than 50% of households owning IHL. The next question is whether or not these increases in latrine ownership have resulted in improved health outcomes, as we would expect from an epidemiological model that links open defecation to diseases like diarrhea. Answering this question is more difficult than measuring the effect of the IEC campaign more broadly because unlike the sanitation campaign treatment, IHL adoption itself is not randomly assigned. Ultimately, households choose whether or not to build and use IHL, and it is quite possible that some of the factors that influence this decision will also be correlated with diarrhea outcomes. For example, if households that adopt IHL are naturally more health- and sanitation-conscious, we might expect these households to have lower diarrhea rates even in the absence of the IHL “treatment.� Thus, simply regressing diarrhea outcomes on IHL ownership will likely result in a biased estimate of the treatment effect in this case. We can interpret this problem as a case of partial compliance (Duflo et al., 2006) with assignment to receive an IHL. In this framework, the sanitation campaign acts as a randomly assigned “encouragement� to adopt the IHL “treatment.� However, not all household that are subject to the campaign actually build an IHL, and there may be households in the comparison group that decide to build IHL on their own. Nonetheless, we can use the random variation in the campaign, which influences the IHL adoption decision, to identify the impact IHL adoption on diarrhea outcomes. If we denote Yij as the individual or household level diarrhea outcome and Zi as the sanitation campaign (Zi =1 if village i received the campaign, and Zi =0 otherwise), then random assignment implies that E[Yij(0)| Zi =1]=E[Yij(0)| Zi =0] (where Yij is the individual or household level diarrhea outcome). We can thus measure the Intention to Treat effect as the realized outcome: E[Yij| Zi=1]-E[Yij| Zi=0]. Furthermore, under two assumptions, the following Wald estimator will give us a valid estimate of the impact of IHL adoption (T) on diarrhea outcomes for a well-defined group of households (or individuals): E[ Yij | Zi 1 ]  E[ Yij | Zi  0 ] ��  E[ Tij | Zi 1 ]  E[ Tij | Zi  0 ] The numerator is the difference in diarrhea outcomes between households (or individuals) in treatment villages and diarrhea outcomes in control villages—i.e., the ITT. The denominator is the difference in IHL uptake between households exposed to the campaign and those that were not exposed. Under the assumptions discussed below, this estimator (or, equivalently, an instrumental variables estimator) will give us the effect of IHL adoption on diarrhea outcomes for the group of households that were induced by the campaign to adopt IHL. This is the local average treatment effect, or LATE (see Imbens and Angrist 1994). June 21, 2008 97 The assumptions that are required to identify the LATE are: 1. Independence: (YijC, YijT, Ti(1), Ti(0)) is independent of Z; 2. Monotonicity: Either Ti(1)≥Ti(0) for all i, or Ti(1)≤Ti(0) for all i. The monotonicity assumption is fairly straightforward: it requires that households who are exposed to the sanitation campaign are (weakly) more likely to adopt a latrine than they would have been in the absence of the campaign. Given the large, positive, and statistically significant average effect of the campaign on IHL uptake that we found in the previous section, this assumption seems plausible. The independence assumption is somewhat more complex. This assumption essentially requires that the only way the instrument (the sanitation campaign) affects diarrhea outcomes is through its effect on IHL uptake. While there is no way to test this assumption directly, we ran two specification tests to examine whether the campaign had any effect on two behaviors that may also influence diarrhea outcomes: hand washing behavior (for both mothers and children under 5) and whether or not households treat their drinking water. Difference-in-differences estimates of the impact of the sanitation campaign on these two behaviors are presented in Table 1. Furthermore, we see no difference in disposal of garbage or waste water, or in safe handling and storage of water. Because these tests do not suggest that the sanitation campaign had a significant effect on these other diarrhea-related behaviors, we have some reason to believe that the independence assumption may be valid in this context. Rather than implement the Wald estimator directly (which does not allow calculation of standard errors), we use several instrumental variables estimators where the sanitation campaign serves as an instrument for IHL adoption. Table 2 presents results of these different models for child health. In several of these regressions, the dependent variable is measured at the individual level: did that child experience a diarrhea episode within a two week recall period. We test this for two age groups – under 3 kids and under 5 kids. To control for pre-existing differences among households and trends over time, we include various household and village characteristics. Columns 1 and 2 present intention to treat effects for under 3 and under 5 respectively. Columns 3 and 4 presents two-stage IV estimates, where the first stage is a probit regression of IHL adoption on treatment (and other covariates), and the results of this regression are used to generate predicted variables, IHL_hat, which are used in the second stage regression to estimate effects on diarrhea outcomes, including corrections for the standard errors because we are using a predicted value. We also estimate the ITT model for arm circumference of children under 5. We explored one additional method, fairly similar to the IV approach, for identifying the impact of IHL ownership on child diarrhea rates. This method uses a propensity score matching technique to compare diarrhea outcomes among IHL adopters in the treatment village to outcomes for non-adopters in control villages who would have adopted IHL if their village had been exposed to the campaign. In the first stage, we conduct a probit regression of IHL adoption on household characteristics using only treatment villages. (This corresponds to the first model presented in Table 3.) We then used the estimated coefficients from this regression to predict the likelihood of adoption for households in control villages. Finally, the sample was limited to adopters in treatment villagers and non-adopters in control villages, and households were matched based on predicted likelihood of adoption. Each treatment household was matched to four control households, and diarrhea rates were compared. This model does not show a significant effect. June 21, 2008 98 Across these different models, we observe that the estimated effect of IHL adoption and use on child diarrhea outcomes is consistently negative, indicating that IHL adoption may have decreased diarrhea rates. However, ITT result is only significant in 1 model (under 3) and the IV estimate is significant in 1 model (under 5). Thus, we are cautious about over-interpreting or misinterpreting these results due to concerns about the validity of our instruments, exclusion restrictions, and functional form across the different models. Table A3-1. Results of Specification Tests for Impact of Sanitation Campaign on Other Diarrhea-related Behaviors Y-variable: Y-variable: Y-variable: Number of times Number of times HH treats mother washes children under 5 drinking water hands have their hands washed Explanatory variables: Treatment .331 -.026 -.038 Post 2.00*** -.038 -.002 Treatment*Post -.056 .147 .035 † Standard errors are in parentheses * = significant at 10% level, **= significant at 5% level, ***= significant at <1% level Table A3-2. Estimates of IHL Adoption on Child Health Child has diarrhea Arm circumference Intention to Treat Instrumental Variable ITT Under3 ‡ Under5‡ Under3‡ Under5‡ Under 5 Treatment 0.098*** 0.051** -0.084 Post -0.042 -0.051 -0.182** -0.321** 0.12 Treatment x Post -0.098* -0.051 0.25* IHL hat -0.191 -0.318* Community WQ 0.003 (total coliform) 0.001*** 0.001* 0.004* 0.001* SES-2005 Controls † Yes Yes as IV as IV as IV KAP-2005 Controls † Yes Yes as IV as IV as IV N 1440 2720 1440 2720 2513 Pseudo R.Sq 0.05 0.03 ‡ errors are clustered at village level † same as in latrine adoption model – see Table 12. *= significant at 10% level, **= significant at 5% level, ***= significant at <1% level June 21, 2008 99 Annex 4. The Process Evaluation of Intensified IEC Campaign in Orissa In addition to the impact evaluation results, which tell us the quantitative effects of the intervention on latrine adoption and health outcomes, our qualitative data can shed more light on the processes generating these results. Three sets of major stakeholders were contacted: Knowledge Links, DWSM personnel, and village-level leaders and individual household members. Table A4-1 outlines the methods used and the topics addressed. The information obtained from this process evaluation paints a detailed picture of what happened and why. It gives us a deeper understanding of the campaign and assists us to interpret the analysis of the hard data. Table A4-1. Information-Gathering Methods and Topics for the Qualitative Study in Orissa Stakeholder Information gathering methods Topics addressed Knowledge Personal interviews by RTI teams. Timing and sequence of Links intensified IEC campaign. Actors involved. Review of Knowledge Links final report to the World Bank17. Evaluation of effectiveness. Reasons for success and shortcomings. DWSM Review of DWSM documents on five Personnel key actors: 1. Production centers 2. Self-help groups 3. Aanganwadi workers 4. Village Health, Water and Sanitation Committees 5. NGOs. Village leaders RTI team field observations and semi- Knowledge and attitudes and households structured interviews in 7 treatment and about sanitation. 3 control villages (villages selected to Level of sanitation. be representative of the results obtained from the intensified IEC campaign). State of latrine construction. Defecation practices. Perception of campaign and its effects. Time constraints prevented visits to all 40 project villages to collect qualitative data from village leaders and household members. Instead, a subset of villages were selected using data from the 17 “Technical Assistance for Implementation of an Information, Education, and Communications Program in Bhadrak, Orissa,� Final Report to the World Bank, New Delhi. April, 2006. June 21, 2008 100 Knowledge Links report and the community surveys, which were completed in early August, 2006. The Knowledge Links report records how receptive village members were to the intensified IEC campaign activities in each treatment village. In particular, the report states whether village members achieved consensus to end open defecation, whether plans were made to work toward consensus, or whether no consensus could be reached. Meanwhile, data from the community survey which had recently been completed provided a preliminary indication of the state of latrine construction in each village. Thus, putting these two data sources together allowed us to categorize each treatment village according to two indicators: Knowledge Links’ perception of the campaign’s outcomes, and the number of latrines that were subsequently constructed according to the community surveys. The resulting village grouping is shown in Table A4-2. As shown, control villages were also separated into two categories reflecting high and low levels of latrine ownership. The Knowledge Links report stated that of the 9 treatment villages that resolved to end open defecation, many established specific dates for achieving this goal. It also pointed out that of the 11 villages that did not reach consensus, many nonetheless had responded favorably to the campaign and 7 were making plans to work towards consensus. Where the campaign was not favorably received, two issues that may have impeded progress are mentioned: lack of trust of the implementing NGO and unhappiness with the baseline survey teams. The community survey data indicated that the campaign had a substantial impact on the program output (latrines constructed) and outcome (use of latrines). Referring again to Table A4-2, note that 12 of the 19 treatment villages surveyed reported that new latrines were constructed in the first six months following the campaign. Latrine building reportedly took place, not just in villages that had reached no-defecation consensus, but also in those that did not. Conversely, consensus did not necessarily lead to latrine construction. Using this table, villages (indicated with *s) were selected for further study to represent a variety of different “cells.� The first category included clear “successes� in which village members resolved to end open defecation and followed through by constructing a substantial number of latrines. Rajnagar and Hatapur were selected from this category. Additional villages were selected in which no clear consensus was reached, yet community survey data indicated that some latrine construction had occurred. These included Bodhapur, Bhimpur, and Padhisahi. Another village, Amarpur, was selected to represent the case in which a consensus on ending open defecation was reached, yet no latrines had been constructed. In the “failure� category, Sasankhas provided an example of a village in which no consensus was reached, and no latrines were constructed. In total, seven treatment villages were visited for qualitative data collection. In addition, three control villages were also selected. In two of these villages (Agiria and Mangarajpur), the community survey data indicated that latrine ownership was high despite the fact that no intervention had taken place. The last village, Amarpur, represented a more typical control village in which latrine ownership remained quite low. June 21, 2008 101 Table A4-2. Status of Treatment Villages at the Conclusion of the Intensified IEC Campaign in Orissa, As Reported in the Knowledge Links Report and the August 2006 Community Survey Intensive IEC Villages: Outcomes Reported in KL Control Report villages Reached Made plans to Did not reach consensus to end work toward consensus open defecation consensus Villages that Rajnagar* Nuasahi Bhimpur* Agiria* built latrines (120; 100%) (81; 73%) (143; 100%) (0; 100%) (No. of Gouraprasad Bodhapur* Padhisahi* Nuananda latrines built (120; 100%) (75; 67%) (30; 15%) (0; 100%) March-August Hatapur* Balipada Mangarajpur* 2006; % of households (80; 89%) (50; 37%) (12; 35%) with a latrine) Begunia Badapimpala Baincha (80; 67%) (18; 53%) (5; 29%) Dhrubapahalipur (25; 20%) Barikpur (2; 13%) Villages that Ambola* Birabarapur Sanasamukavedi Remaining did not build (2%) (1%) (0%) villages, latrines Baliarpur Talabandha Sasankhas* including (% with (0%) (3%) (4%) Amarpur* latrines) (2%) Madhupur (0%) No survey Taladumuka available * Villages selected for qualitative data collection (village visits and interviews) How the Campaign Was Perceived and Evaluated by Stakeholders Views of DWSM Personnel. The DWSM assigned letter grades on effectiveness to the various actors in the treatment villages (Village Production Centers, self-help groups, aanganwadi workers, the VHWSC, and implementing NGOs) and then gave each village an average score. There is considerable variation: the high score was 21, the low, 6; the median was 12. Of the actors, the aanganwadi workers received the lowest grades. There is some correlation between the effectiveness of the actors and the success of the villages in reaching consensus and building latrines, but only at the highest and lowest levels. The relationship breaks down in the mid-range. Key informants from the DWSM told the RTI team that the sanitation intervention led by Knowledge Links was a major improvement over previous Total Sanitation campaigns in Orissa. They pointed to the energetic pace of implementation, the effective “arsenal� of techniques, and the overall social mobilization strategy. June 21, 2008 102 The subsidy system was viewed as a problem. The government pays the implementing NGOs Rs. 1,200 per latrine (the full subsidy) for “below the poverty line� households. It is up to the NGOs to collect the household’s contribution of Rs. 300. The government subsidy is fixed, but the actual cost of the latrines is not – in some areas a latrine may cost as much as Rs. 2,000. That discrepancy and the difficulty of collecting the Rs. 300 from the households force the NGOs to look for ways to cut costs, for example, by using less cement. Cost cutting drags down the quality of the latrine, which in turn leads to less latrine use and less willingness to pay on the part of the households. Another problem with subsidies is that only those below the official poverty line are eligible for them; households above the line, who are also poor in absolute terms, must pay the full cost themselves. The baseline surveys played both a negative and positive role in the villages, according to DWSM personnel. Discussing sanitation as part of the survey raised awareness and itself contributed to the demand for latrines, but the survey teams also created a sense of “euphoria� in some villages – a belief that all kinds of improvements were on the way. People consequently reacted negatively to the sanitation campaign; they would have preferred roads, electricity, schools, etc. The DWSM informants acknowledged that the DWSM does not possess the capacity to carry out an intensified IEC campaign on its own, even after working with Knowledge Links. As mentioned, DWSM personnel have technical know-how but lack social mobilization skills. They recommend that organizations like Knowledge Links be involved in follow-up activities. Views of Knowledge Links Personnel. Key informants from Knowledge Links agreed that the DWSM is not yet ready to implement intensive IEC on its own. They were of the view, however, that while the involvement of Knowledge Links certainly contributed to the generally positive results of the campaign in Orissa, the need for continued outside assistance is not conducive to wider replication of intensified IEC campaigns in Bhadrak District. Knowledge Links also had strong views on the subsidy issue, asserting that subsidies are inimical to the concept of community-led, demand-creation. Ideally, according to Knowledge Links, intensified IEC campaigns should be carried out in areas with no pre-existing sanitation policies. Where a system of subsidies has been established, people develop a sense of entitlement and a feeling that sanitation is the government’s job, not theirs. In villages where such attitudes were prevalent there was no commitment to end open defecation (see Table A4-3). Similarly, promotion of just one type of latrine runs counter to the spirit of CLTS model; making choices about what type of latrine to obtain increases a sense of ownership and personal responsibility. Apparently the NGOs hesitated to offer other options because they were more costly and might take longer to construct. Yet some households said that the poor quality of latrines was a reason for continuing the practice of open defecation. June 21, 2008 103 Table A4-3. Attitudes Inimical to the Self-Help Concept in Treatment Villages in Orissa, According to the Knowledge Links Report Village Outcome Re Issues Open Defecation Sanasamukavedi No community Villagers are not happy with the survey team that resolve conducted the baseline survey, as they made false promises about infrastructure during their visits. Talabandha Will hold village- Women in one hamlet of the village were wide meeting annoyed due to the non-fulfillment of a promise made by the NGO regarding road construction. These women were not ready even to conduct a meeting about sanitation. Bhimpur Village will hold There was a lot of annoyance about the false another meeting promises made by the survey team. The villagers complained about the lukewarm and delayed response of the Production Center to supply latrine construction material. Sasankhas No consensus Lack of trust between the community members and the NGO. Community members annoyed with the NGO for delays in production of latrine construction materials. Rajnagar Consensus Community members were annoyed with the reached NGO because of lack of transparency in financial dealings. Begunia Consensus Community reported that the survey team made reached false promises such as provision of roads, electricity, and uniforms for school children. Badapimpala Will meet Community members were annoyed with the NGO for not keeping a promise to establish a school and to set up a meeting with the local road- building agency. Views of Village Members. Observations and interviews of village members in 10 of the 40 study villages revealed a range of contrasting views but no consistent theme or village perspective. The following topics were raised repeatedly:  While people talked about the links between sanitation and serious health problems, knowledge of the links did not seem to be a motivating factor as to latrine use. Other motivations, such as dignity, privacy, and safety for women, were mentioned.  In many villages, women claimed that they desired sanitation improvements more than men, and that latrines that had been constructed were being used primarily by women. Attitudes were not clearly defined along gender lines, however. In one village women saw sanitation as June 21, 2008 104 less important compared to other problems in the village, while men seemed to be the main proponents of improved sanitation.  Some people said they built a latrine because a subsidy was available; others argued that the subsidy was insufficient. In one village in particular, people claimed that individuals in neighboring villages had had to pay less for their latrines (Rs. 125 vs. Rs. 300 in the intensified IEC Campaign), and had also received higher-quality latrines with septic tank systems rather than the very basic on-pit latrine they had been offered. They refused to pay more than Rs. 125.  Several informants raised the issue of space. Some said they did not have space to build a latrine; others said that open defecation was not a problem because there was plenty of space for it. Since latrines are considered unclean, people do not want them to be too near their houses.  Many people pointed out that open defecation – one person called it “open sky latrines� – is a long-standing tradition. One man said, “If open defection was good enough for the Maharajas, it’s good enough for me.� June 21, 2008 105 Annex 5: Adequacy Assessment The endline survey for Jalswarajya was originally scheduled to coincide with the project cycle, the 18-month period from project inception to completion plus an interval long enough for the project to reasonably show some results. Following that schedule endline data collection would have begun in September 2006. In a project of this size and scope, it is vitally important that the expensive and time-consuming process of data collection not be undertaken unless the project has actually reached a stage where endline assessment makes sense. To determine whether or not that was the case, we conducted an adequacy assessment. Mixed Methods Approach A mixed-methods approach was used in this assessment. First a rapid rural appraisal was carried out in selected villages. The RTI team stayed in each village for two weeks during which it observed field conditions and interviewed key informants. Second, the team collected and reviewed official data from District Water and Sanitation Committees. Third, district staff were interviewed for their insights on project implementation. To estimate when various aspects of the project would reach adequacy, the team linearly extrapolated the findings from the three activities, using trend analysis. To answer the narrow question (Is this intervention far enough along to fairly be evaluated?), the RTI team assessed the physical progress and level of access or use of Jalswarajya outputs: water supply, sanitation infrastructure, and hygiene and sanitation information. Since there is a hypothesized connection between the outputs and the hoped for impacts, it would not make sense to collect endline data unless enough outputs were in place for their effect to be felt. The team also evaluated the process of Jalswarajya project implementation: community participation, the project implementation schedule, and availability of funds. This aspect of the assessment can reveal what is holding up achievement of the outputs so that adjustments can be made. Criteria had to be set for deciding whether or not a village could be judged “adequate� or not on a given variable. The RTI team set the criteria on the basis of the team’s familiarity and first- hand experience with Jalswarajya, using the project implementation plan as a guide. For example, the project implementation plan expects all participating villages to become open- defecation free, but the criteria was set at 50% household latrine coverage as a demonstration of sufficient progress. On the other hand, the RTI team retained the project implementation plan standard for water quantity of 40 liters per person per day, since water supply is the most important component of Jalswarajya. Rapid Rural Appraisals Rapid rural appraisals were conducted in all 95 Phase I Batch 1 Jalswarajya villages in the study sample in three waves in 2006: Wave I: March 1-4. 26 villages, all randomly selected Wave II: June 8-13. 54 villages, 50 randomly selected and 4 hand-picked from the first wave Wave III: August 19-25. 45 villages, 23 randomly selected and all villages in Wave I June 21, 2008 106 The interview instrument designed for the rapid rural appraisals was based on the baseline data questionnaire and the Jalswarajya project implementation plan. The instrument was field tested and revised during a visit to Sangli District in February 2006. Twelve investigators (many of whom had worked as supervisors on the baseline study) attended a three-day session in rapid rural appraisal techniques just prior to the first wave. For subsequent waves, refresher training was given. In each village, the investigators sought information from key informants drawn mainly from Village Water and Sanitation Committees (30%) or from other leadership groups. These informants had direct knowledge of Jalswarajya implementation in their village. They were well educated as a group and most had lived in their village for 20 years or more. The investigators also visited various sites in the village on a “transect walk� (a key aspect of rapid rural appraisal) to verify the information obtained from the key informants. At times, the interview findings had to be tempered by on-the-ground findings. Project Monitoring Information System The RTI team reviewed the monthly progress reports of the District Water and Sanitation Committees that were submitted to the Reform Support and Project Management Unit of the Government of Maharashtra. The District Water and Sanitation Committees gather information from the support organization and consortiums working in the villages and from visits to the villages. In addition to perusing these reports, the RTI team met with district staff to discuss the progress of Jalswarajya. Analyzing adequacy data Adequacy was determined mainly through the results of the rapid rural appraisal data, including the results of the transect walk. We computed the percent of villages in a given district or in the entire study area that satisfied the adequacy criteria. Following that, the team used the percentages from three points in time (March, June, and August) to forecast what the percentages would be in September 2006 (the scheduled completion date) and March 2007 (completion plus 6 additional months). This “trend analysis� was “triangulated� with the qualitative insights from personal discussions and communications and review of district monthly reports. Triangulation attempts to neutralize the biases of rapid rural appraisal investigators and informants. For example, respondents may present an overly pessimistic assessment of the intervention because they are discouraged by delays in project implementation. Or informants may have faulty memories or lack information. The overarching finding of the adequacy assessment, which was completed in August 2006, was that the interventions were inadequate at that time but probably would reach a sufficient level of adequacy for the endline data to be collected in May-September 2007.  Hygiene interventions were reasonably adequate.  Sanitation was mixed (two sanitation schemes were complete in most villages but latrine coverage was inadequate).  Water supply was inadequate in inputs (funding, participation, planning), outputs (physical infrastructure, trained personnel), and outcomes (minimum liters per person per day). It was not wise to plan to collect endline data as soon as the project was officially over in a village. There should be a gap of time for the project to stabilize in the communities. It is not June 21, 2008 107 reasonable to expect an immediate change in household behavior and ultimately health and other outcomes of interest (quality of life). After building a water supply and sanitation system, there is a break-in period during which the system is tested and fine-tuned. Then a month should be allotted for households to start using the new system. Since the main health indicator is two- week recall of diarrhea, an additional two weeks must be added. These factors suggest that the endline survey in Maharashtra should be conducted well after completion of the interventions. It would have been unwise to have conducted the surveys in September 2006, as originally planned. Major findings of the adequacy assessment are given in Tables A5-1, A5-2, A5-3, and A5-4: three on outputs and one on inputs. Table A5-1. Summary Results of the Adequacy Assessment in Maharashtra: Water Supply Output/Finding Adequacy Criteria Status August 2006 Water Supply Water supply has 6 of 7 components **28% of villages met criteria. completed or in use [Trend analysis: 38% in Finding: Water supply *source September and 60% in October is inadequate; major *extraction mechanism 2006; 100% in March 2007] features of water supply *storage were observed in less *treatment than 60% of villages; *distribution minimum lpcd not met. *delivery access points *source strengthening. Water availability is 40 liters per capita per **29% met criteria day (lpcd). **56% have 21-40 lpcd [Trend analysis: 40% by March 2007] Water sources serve at least 80% of **76% met criteria households. [Trend analysis: 100% by March 2007] Note that the table above uses data collected by the investigators from key informants unless otherwise indicated. The findings are corroborated by the analysis of monitoring and information reports and discussions with district personnel. Differences among the districts are not given here, although in some cases they were marked. Concerning water supply, the trend analysis is not able to capture the acceleration in implementation noted in August-September 2006. In some cases, improvements suggested by the adequacy assessment were carried out right away; they may have accelerated progress. June 21, 2008 108 Table A5-2. Summary Results of the Adequacy Assessment in Maharashtra: Sanitation Output/Finding Adequacy Criteria Status August 2006 Sanitation 50% of households have individual **60% of villages met criteria household latrines (IHL). (83% reached level of 30% IHL Finding: Sanitation coverage) situation is mixed: [Trend analysis: 75% by March though 2 sanitation 2007] schemes are complete At least one population group (men, **84% met criteria in most villages, IHL women, or children) do not openly [Trend analysis: 100% by March coverage is inadequate. defecate during the day or night. 2007] (16% of villages reported being open-defecation free) [Trend analysis: 23% by March 2007] At least 2 sanitation projects completed. **83% met criteria Concerning sanitation, several of the schemes counted in the survey were actually implemented under the aegis of other projects (for example, the Total Sanitation Campaign pays for IEC related to latrines and provides a subsidy for latrine construction as well; the Prime Minister’s Rural Roads Project shares the cost of drainage systems with Jalswarajya, etc.). Typically, Jalswarajya does not fund any project in excess of Rs. 3,000,000. The hygiene education output is the most difficult to assess because there are few or no physical indictors to observe and measure. Also, hygiene education is focused on households, not on villages. The finding that “almost all villages changed 4 of 7 behaviors� is not very reliable. Key informants are basically making guesses about the behaviors of all households in the village. Without conducting a house-to-house survey, behavior change can only be guessed. June 21, 2008 109 Table A5-3. Summary Results of the Adequacy Assessment in Maharashtra: Hygiene Education Output/Finding Adequacy Criteria Status August 2006 Hygiene Education Village is observed to be clean on a five- **In transect walk 45-54% of point scale. villages seen to have good or Finding: Hygiene and excellent sanitation condition. sanitation information (72-92% rated average or better) and behavior were Presence of IEC message in village. **In transect walk, 75% of reasonably adequate in villages seen to have excellent August 2006 and will advertisements of Jalswarajya definitely be adequate and water and sanitation by March 2007. information. Some change in 4 of 7 behaviors **Almost all villages changed 4 corresponding to key messages of 7 behaviors (based on *poor hygiene causes disease investigators’ estimates). *use tap water *stop open defecation/use IHL *wash hands with soap regularly *safely handle and store drinking water *boil, filter, or treat drinking water *safely handle and prepare food. It is encouraging that hygiene interventions appeared adequate as early as August 2006. There will have been adequate time for the messages to impact behaviors and ultimately improve child health by the time of the follow-up surveys during the summer of 2007. June 21, 2008 110 Table A5-4. Results of the Adequacy Assessment in Maharashtra: Community Participation Inputs/Findings Adequacy Criteria Status August 2006 Community Village Water and Sanitation Committees **92% of the villages met the Participation (VWSC) meet once a month. criterion Finding: The general state of community VWSC carries out 3 of 7 key duties **89% met criterion participation and *maintenance of water sources involvement of *purification/chlorination of water sources implementing and *construction of latrines support agencies is *clearing of drains/garbage dumps encouraging. *cleaning of roads *hygiene education and training *school sanitation programs. District Water and Sanitation Committee, ** 70% met criterion NGOs, and/or support organizations visit once a month. Project Schedule VWSC formed within 6 months from date of **58% met criterion application. Finding: In general the project schedule is Village action plan submitted within 10 months **39% met criterion lagging by several of VWSC formation. months; in a few villages the lag is Started civil construction within 3 months of **54% met criterion more than a year. submission of village action plan. Funding Villages have made their contribution to capital **80% met or plan to meet this costs or will make it one month after the criterion Finding: The funding adequacy assessment survey. for capacity building Villages have received their second installment and institutional of… development is *funds for capacity building and village-level **64% met criterion adequate, but IEC (Addendum I) construction funds *funds for empowerment of women **16% met criterion have not been (Addendum III) dispersed beyond the first of three *funds for construction of water and sanitation **36% met criterion installments (in other facilities (Addendum II) (only 7% have received the 3rd and words, construction is [Note: 2nd installments are given when last installment) still ongoing). considerable progress in spending the 1st installment has been made and when key administrative/financial requirements have [Trend analysis: 37% will not have been met. 3rd installments suggest near received 2nd installment of completion of the activity.] construction funds by March 2007 in Nashik and Sangli Districts] June 21, 2008 111 Tracking the process indicators for Jalswarajya helps to explain some of the factors that may have contributed to the delay and inadequacy of the water, sanitation, and hygiene interventions. Delays in the project implementation schedule were anticipated because the study villages are from the first batch and some kinks in the provision of inputs are to be expected. Given the delays documented, it is more realistic to expect construction to start within approximately 18 months after selection, rather than expecting the system to be operational within 18 months. Various reasons for the delays were identified by district staff: natural calamities, lack of source water testing (a criterion for selection into Jalswarajya), unavailability of the funds, more than expected time to stimulate demand, and others. Another reason for delay may have been the turnover of project villages. Many villages from the original batch opted out due to internal problems or preference for other projects, of they may have been dropped because of poor performance. From the original sample of 95 villages, 82 continue to participate. The slow disbursement of funds for construction presents a “chicken and egg� situation. Lack of funds can lead to lack of physical progress, but funds will not be disbursed until physical progress has been verified. Perhaps due to this paradoxical situation, project disbursement rules have been revised, and the new rules have contributed to the acceleration of project activity. Importantly, 80% of villages reported either contributing or planning to contribute their 10% share of the capital cost of the projects, which is an clear manifestation of ownership and self- rule. Better understanding and revised approaches are needed for the remaining 20% of the villages, which apparently had some problems with contributing their share. Insights  Project timelines should be adjusted to accommodate the tricky process of demand creation. It is tricky because of the complex social structure of Indian village communities, their lack of information, and the local political context. Demand creation will not fit in with a year- long project cycle.  More focus should be placed on the supporting role of the supply side in a demand-driven project. Supply side activities, if they unfold at a snail’s pace with bureaucratic delays and staff turnover, can negatively affect growth in demand for water and sanitation technologies. Communities get tired and lose interest. As a field staff person put it, what is needed is “demand-driven, supply-pushed.�  The monsoon seriously affects a project like Jalswarajya. Funding and other decisions are still controlled to some extent by staff in Mumbai, a city that is severely impacted by the rains. Water source testing, a vital part of the selection process for villages that have applied to participate in Jalswarajya, must take place before the monsoon. Any community that is not ready to do this in March or April, automatically falls behind by almost 11 months. In some parts of Maharashtra, rains are so heavy that no civil construction can take place. These realities must be recognized and the project cycle must be revised accordingly.  Monitoring and evaluation systems should be uniform and village-focused. Currently, monitoring and evaluation formats differ from district to district and region to region. Instead of changing the forms to meet ad hoc requirements, there should be a basic form used by all districts with additional information appended. The information should be collected and June 21, 2008 112 recorded only at the village level. Village reports can be used to create aggregated reports for the districts or states. Bottom line: The endline surveys were postponed and conducted in Maharashtra in May-June and August-September, 2007. June 21, 2008 113 Annex 6. Maharashtra Results Tables Table A6-1. Descriptive Statistics of Respondent and Household Characteristics† Dry Season Rainy Season Respondent and Household Survey Non-JS Villages Non-JS Villages Characteristics Round JS JS Overall In- X- Overall In- X- Villages Villages District District District District 2005 91 92 91 89 84 85 83 83 Female respondent 2007 95 95 96 94 94 95 95 94 2005 28.5 28.6 28.5 28.5 31.3 31.2 31.2 31.4 Respondent's age (# of years) 2007 32.1 31.9 32.2 32.2 32.6 32.4 32.8 32.4 Respondent's education 2005 37 36 39 36 36 36 38 37 No education 2007 40 39 42 40 37 37 39 37 2005 14 14 13 15 12 12 12 13 Primary school (1-4 years) 2007 12 12 11 12 12 12 12 12 2005 38 39 37 37 41 42 40 41 Secondary school (5-10 years) 2007 40 40 40 40 43 42 42 44 % households belong to scheduled castes 2005 29 29 30 29 28 28 30 28 (SC) or scheduled tribes (ST) 2007 35 37 35 33 37 37 37 37 2005 51 50 53 53 47 48 48 47 % households below poverty line (BPL) 2007 51 50 53 53 47 47 48 47 2005 6.4 6.4 6.4 6.4 6.6 6.6 6.6 6.6 Household size (# of persons) 2007 7.0 7.0 7.0 7.1 7.1 7.1 7.1 7.1 # of under 5 children in the household 2005 1.4 1.4 1.4 1.4 1.4 1.4 1.4 1.4 (# of persons) 2007 1.1 1.0 1.1 1.1 1.0 1.0 1.0 1.0 Main problem 2005 20 18 20 23 20 20 17 22 Roads 2007 15 15 14 15 21 22 23 18 Household water supply 2005 46 56 43 41 29 33 30 25 June 21, 2008 114 Dry Season Rainy Season Respondent and Household Survey Non-JS Villages Non-JS Villages Characteristics Round JS JS Overall In- X- Overall In- X- Villages Villages District District District District 2007 38 39 39 37 25 21 30 28 2005 13 11 14 13 33 30 33 35 Sanitation & hygiene 2007 20 19 20 20 27 25 26 29 Main disease 2005 17 18 17 16 11 12 12 10 Diarrhea 2007 17 18 18 17 8.2 10 7.7 7.0 2005 29 27 29 31 56 52 55 60 Cough, cold, pneumonia (ARI) 2007 47 45 45 49 57 53 56 60 2005 9.2 9.1 10 8.4 15 16 16 13 Malaria 2007 12 10 14 12 19 18 20 18 † All statistics are mean values reported in percentage terms unless otherwise noted. June 21, 2008 115 Table A6-2. Descriptive Statistics of Health Outcomes† Dry Season Rainy Season Survey Non-JS Villages Non-JS Villages Health Outcomes JS JS Round Overall In- X- Overall In- X- Villages Villages District District District District 2005 10 11 11 10 13 12 14 14 Diarrhea prevalence among under 5 children 2007 9.1 9.8 8.0 9.2 7.5 7.8 6.9 7.5 2005 27 27 27 26 28 27 29 30 % households had at least 1 diarrhea case 2007 22 22 20 22 17 18 16 17 2005 6.1 6.3 5.8 6.2 3.7 3.7 3.4 3.7 % households that treated diarrhea 2007 4.8 5.5 4.0 4.5 3.3 4.0 3.1 2.9 How diarrhea was treated 2005 78 79 80 77 87 86 89 88 No treatment 2007 79 75 82 81 81 78 81 83 2005 3.4 3.2 3.5 3.2 0.6 0.4 0.2 1.1 Homemade sugar & salt water 2007 3.9 4.4 3.6 3.7 1.8 2.4 1.5 1.5 2005 0.8 0.4 0.8 1.3 0.2 0.1 0.3 0.0 Traditional remedies 2007 0.6 0.8 0.1 0.7 0.5 0.5 0.4 0.8 2005 2.4 2.2 2.7 2.7 1.5 1.3 1.4 1.6 ORS 2007 1.9 2.1 1.6 1.9 3.4 3.7 3.4 2.8 2005 20 21 19 22 13 14 11 12 Pill & syrup 2007 20 23 18 18 19 21 19 17 2005 13 13 11 14 7.6 8.9 6.8 6.8 Injection 2007 11 12 10 10 11 12 10 10 2005 2.9 2.9 2.3 3.5 2.0 3.1 1.9 1.0 Intravenous fluid (I.V.) 2007 3.3 3.3 3.6 2.9 3.7 4.7 3.8 2.5 2005 22 21 21 22 29 28 29 29 ARI prevalence among under 5 children 2007 13 13 12 13 17 17 16 16 2005 0.9 1.0 0.8 0.8 1.1 1.0 1.0 1.2 Malaria prevalence among under 5 children 2007 0.7 0.6 0.9 0.6 0.8 1.1 0.5 0.7 TB prevalence among under 5 children 2005 0.39 0.08 0.77 0.38 0.02 0.02 0.02 0.02 June 21, 2008 116 Dry Season Rainy Season Survey Non-JS Villages Non-JS Villages Health Outcomes JS JS Round Overall In- X- Overall In- X- Villages Villages District District District District 2007 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 Typhoid fever prevalence among under 5 2005 0.71 0.42 1.06 0.71 0.28 0.39 0.20 0.23 children 2007 0.37 0.31 0.46 0.41 0.10 0.12 0.06 0.08 2005 0.17 0.15 0.26 0.21 0.04 0.04 0.04 0.04 Cholera prevalence among under 5 children 2007 0.05 0.08 0.0 0.05 0.0 0.0 0.0 0.0 Arm circumference of under 5 children 2005 13.81 13.78 13.75 13.86 13.01 12.97 13.05 13.01 (in cm) 2007 14.10 14.20 14.02 14.01 13.93 13.89 13.93 13.96 † All statistics are mean values reported in percentage terms unless otherwise noted. June 21, 2008 117 Table A6-3. Descriptive Statistics of Water Supply Services† Dry Season Rainy Season Survey Non-JS Villages Non-JS Villages Water Supply Services JS JS Round Overall In- X- Overall In- X- Villages Villages District District District District 2005 2.2 2.2 2.1 2.1 2.2 2.1 2.1 2.2 # of water sources an average household accesses 2007 2.1 2.1 2.0 2.2 2.2 2.2 2.0 2.3 2005 1.4 1.5 1.4 1.4 2.2 2.1 2.1 2.2 # of water sources an average household uses 2007 1.4 1.4 1.4 1.4 1.2 1.2 1.2 1.3 Main water source 2005 21 17 22 22 22 19 23 22 Private tap 2007 27 30 25 24 29 32 27 25 2005 18 18 15 18 19 25 13 17 Public tap 2007 17 19 14 16 20 23 15 21 2005 16 18 17 13 7.4 7.6 6.5 7.3 Private well 2007 8.2 6.5 8.9 8.5 6.5 4.6 7.1 7.4 2005 38 36 39 40 46 42 52 48 Public well 2007 40 37 41 44 38 34 44 39 2005 2.0 1.7 2.5 1.3 2.1 1.5 2.8 1.6 Neighbors 2007 5.3 4.7 7.0 4.4 3.7 3.1 4.0 4.0 Surface water 2005 3.1 4.9 3.4 2.5 3.1 4.3 2.1 3.8 (river/stream/spring/lake/pond/dam) 2007 2.4 2.6 3.4 2.5 3.2 3.6 2.5 4.2 2005 66 61 67 68 41 45 36 39 % households use improved water source 2007 72 74 70 72 76 77 74 76 Time spent walking to main water source 2005 11 12 11 10 9 9 10 9 (in minutes) 2007 9 8 10 9 7 7 8 8 Time spent waiting at main water source 2005 23 25 24 21 14 13 16 13 (in minutes) 2007 13 13 15 14 9 8 11 9 2005 180 183 179 172 169 174 164 163 Total water collected per day (in liters) 2007 262 283 253 247 252 258 244 247 LPCD (based on household size) 2005 30 31 30 29 27 28 27 26 June 21, 2008 118 Dry Season Rainy Season Survey Non-JS Villages Non-JS Villages Water Supply Services JS JS Round Overall In- X- Overall In- X- Villages Villages District District District District 2007 40 43 38 37 38 39 37 37 Water quality tests E. coli levels in household drinking water (in 2005 0.58 0.67 0.54 0.53 1.17 1.11 1.23 1.20 log10) 2007 0.79 0.90 0.71 0.77 0.73 0.82 0.69 0.68 Whether E. coli is present in household drinking 2005 22 25 21 21 40 38 42 41 water 2007 30 33 28 29 26 28 25 24 Count of total coliforms present in household 2005 68 70 64 70 170 160 177 176 drinking water 2007 142 149 138 140 136 143 135 133 Whether total coliforms are present in household 2005 50 51 49 50 76 74 79 77 drinking water 2007 79 82 77 79 68 70 67 66 † All statistics are mean values reported in percentage terms unless otherwise noted. June 21, 2008 119 Table A6-4. Descriptive Statistics of Sanitation Facilities† Dry Season Rainy Season Survey Non-JS Villages Non-JS Villages Sanitation Facilities JS JS Round Overall In- X- Overall In- X- Villages Villages District District District District # of sanitation facilities that an average 2005 1.04 1.04 1.04 1.03 1.05 1.06 1.05 1.04 household uses 2007 1.08 1.10 1.06 1.07 1.08 1.12 1.06 1.06 Main sanitation facility 2005 86 83 89 90 84 79 87 88 No facility/open defecation 2007 73 62 81 80 73 62 81 81 2005 1.7 2.0 1.9 0.9 1.3 1.4 1.8 0.3 Community toilet 2007 2.6 2.7 2.5 2.0 2.3 2.1 2.8 1.9 2005 0.4 0.4 0.3 0.5 0.2 0.2 0.1 0.2 Neighbor's toilet 2007 0.5 0.5 0.2 0.5 0.3 0.3 0.1 0.4 2005 12 15 8.7 9.1 15 20 11 11 Private toilet 2007 24 35 16 17 24 36 16 17 Time spent walking to main sanitation site 2005 8.3 8.1 8.6 8.5 8.2 7.7 8.8 8.7 (in minutes) 2007 7.8 6.6 8.6 8.8 7.8 6.4 8.9 8.6 Garbage disposal Garbage dump immediately outside the 2005 77 74 78 77 86 84 86 89 house 2007 86 82 88 88 86 82 89 88 2005 6.8 6.2 7.2 7.0 7.2 7.8 6.1 6.8 Roadside community garbage dump 2007 3.0 4.2 2.4 2.0 2.9 4.4 2.1 1.4 Community garbage dump on village 2005 11 9.0 12 11 9.2 10 8.4 8.9 periphery 2007 2.6 3.4 2.7 2.0 1.8 2.8 1.4 1.3 2005 15 15 15 15 5.8 6.6 5.1 5.5 Garbage dumped in the fields 2007 8.3 10.2 8.3 6.6 5.8 5.9 6.3 6.1 2005 0.0 0.0 0.0 0.0 19 20 17 19 Composted in field or yard 2007 12 13 12 12 14 16 13 13 Wastewater disposal Thrown within the house 2005 10 10 10 8.9 13 11 14 14 June 21, 2008 120 Dry Season Rainy Season Survey Non-JS Villages Non-JS Villages Sanitation Facilities JS JS Round Overall In- X- Overall In- X- Villages Villages District District District District 2007 14 14 16 14 20 19 19 21 2005 42 38 43 45 41 38 41 45 Thrown into the backyard 2007 24 24 26 26 30 27 32 33 2005 35 36 35 32 36 40 34 32 Flows into drain outside the house 2007 25 28 22 21 30 34 27 26 2005 12 15 11 10 9.2 11 8.7 7.3 In a soak pit 2007 7.1 9.3 5.3 5.6 5.3 6.3 4.2 5.0 Flows outside house but not into any 2005 37 34 39 39 34 31 38 36 organized drainage 2007 27 25 27 28 27 28 29 25 † All statistics are mean values reported in percentage terms unless otherwise noted. June 21, 2008 121 Table A6-5. Descriptive Statistics of Household WSH Knowledge, Attitudes and Practices (KAP)† Dry Season Rainy Season Survey Non-JS Villages Non-JS Villages Household WSH KAP JS JS Round Overall In- X- Overall In- X- Villages Villages District District District District Public health messages 2005 48 57 44 43 50 53 45 49 Wash hands with soap regularly 2007 59 62 55 58 70 73 69 68 Use tap water & correctly treat, store & 2005 45 55 40 38 67 69 65 65 transfer drinking water 2007 76 78 75 74 83 86 82 81 2005 40 49 36 34 21 23 21 20 Safely handle & prepare food 2007 37 40 36 35 48 54 45 43 Stop open defecation/use IHL & poor 2005 48 56 44 41 42 46 43 39 hygiene causes disease 2007 51 56 50 46 60 61 59 58 Diarrhea knowledge Out of 3 key symptoms, # of symptoms a 2005 1.5 1.6 1.5 1.4 1.4 1.4 1.4 1.4 caregiver can correctly identify‡ 2007 1.7 1.7 1.6 1.7 1.6 1.6 1.6 1.6 Out of 8 key causes, # of causes a 2005 3.0 3.1 2.8 3.0 3.4 3.4 3.3 3.3 caregiver can correctly identify# 2007 3.7 3.7 3.8 3.7 4.1 4.0 4.2 4.2 % households treat their water before 2005 63 64 62 63 75 75 75 76 drinking 2007 68 71 66 65 68 70 66 67 2005 2.9 3.0 3.2 2.5 2.4 2.9 1.7 2.4 Boil 2007 1.0 1.7 0.7 0.5 1.5 1.8 1.5 1.2 Filter 2005 57 57 57 57 66 64 68 68 (e.g., sieve it through cloth, ZeroB) 2007 64 68 63 62 65 67 63 64 Use chemicals 2005 3.6 3.5 2.9 3.8 0.0 0.0 0.0 0.0 (bleach/chlorine/alum/potash) 2007 3.5 4.1 3.1 2.7 5.1 5.3 3.7 5.4 2005 94 94 95 94 98 98 98 98 % households store their water 2007 98 99 98 98 99 99 99 99 % households only use narrow mouth storage 2005 17 17 17 18 15 15 16 14 for drinking water 2007 15 15 14 16 10 10 11 10 % households always cover drinking water 2005 91 91 90 90 90 91 88 90 storage 2007 92 92 90 92 88 91 89 85 % households have good water-transferring 2005 29 30 28 29 18 18 19 18 June 21, 2008 122 Dry Season Rainy Season Survey Non-JS Villages Non-JS Villages Household WSH KAP JS JS Round Overall In- X- Overall In- X- Villages Villages District District District District practice* 2007 19 21 17 18 19 23 17 17 # of safe water handling techniques 2005 1.4 1.4 1.4 1.4 1.2 1.2 1.2 1.2 households practiced* 2007 1.3 1.3 1.2 1.3 1.2 1.2 1.2 1.1 # of critical times a caregiver washes her/his 2005 2.3 2.3 2.3 2.4 2.9 2.9 2.8 2.9 hands** 2007 3.4 3.5 3.4 3.5 3.4 3.3 3.4 3.4 # of critical times a child washes her/his 2005 1.1 1.1 1.1 1.2 1.7 1.7 1.6 1.7 hands*** 2007 1.3 1.3 1.3 1.4 1.2 1.2 1.2 1.2 Things present at the place for hand washing 2005 48 47 49 46 67 66 67 68 Flies 2007 48 47 49 48 45 48 42 44 2005 57 57 56 58 68 68 67 67 Water from tap or container 2007 70 70 67 72 72 71 73 72 2005 58 58 55 59 57 62 53 54 Soap or detergent 2007 62 65 59 60 65 68 62 64 2005 36 36 35 38 38 39 36 38 Ash 2007 37 41 35 33 37 37 37 37 2005 26 24 25 29 19 19 19 19 Towel or cloth 2007 18 20 16 16 21 19 20 21 2005 3.0 3.0 2.7 3.2 1.8 2.3 1.6 1.4 Basin or sink 2007 2.2 2.3 1.5 2.3 1.9 1.7 2.0 1.7 † All statistics are mean values reported in percentage terms unless otherwise noted. ‡ The 3 key diarrhea symptoms include loose stool, vomiting, and loss of weight. # The 8 key diarrhea causes include eating stale foods, eating food touched by flies, uncleaned/smelly food, drinking contaminated water, using unhygienic latrines/OD, not washing hands, household uncleanliness, and village uncleanliness. * The safe water handling practices include the exclusive use of narrow mouth storage for drinking water, covered drinking water storage at all times, and practice of good water transferring techniques (i.e., no direct hand contact with drinking water) ** The critical hand washing times for adults are before preparing food or cooking, before eating, before feeding children, after changing baby/handling child's feces, and after defecation. *** The critical hand washing times for children under 5 are before eating and after defecation. June 21, 2008 123 Table A6-6. Descriptive Statistics of Welfare Outcomes and Benefits† Dry Season Rainy Season Survey Non-JS Villages Non-JS Villages Welfare Outcomes and Benefits JS JS Round Overall In- X- Overall In- X- Villages Villages District District District District Days unproductive due to diarrhea 2005 1.7 1.9 1.6 1.7 1.8 1.8 1.9 1.8 (in days) 2007 1.4 1.4 1.5 1.4 0.9 1.0 0.9 0.9 Hospital nights due to diarrhea 2005 0.2 0.2 0.2 0.2 0.1 0.1 0.1 0.1 (# of nights) 2007 0.1 0.1 0.1 0.1 0.0 0.0 0.0 0.1 Medical costs due to diarrhea 2005 245 258 198 263 214 181 191 277 (in Rupees) 2007 181 194 198 157 80 88 68 80 Work loss due to diarrhea 2005 1.4 1.5 1.3 1.4 1.6 1.6 1.7 1.6 (in days) 2007 1.0 1.1 1.0 1.0 0.6 0.7 0.6 0.6 Monthly household averting costs due to 2005 976 1064 950 914 857 836 884 849 inadequate water supply and sanitation services (in Rupees) 2007 590 562 644 596 493 458 526 526 Household cost of illness due to diarrhea 2005 369 391 310 386 333 305 307 395 (in Rupees) 2007 275 289 299 248 144 160 124 142 † All statistics are mean values reported in percentage terms unless otherwise noted. June 21, 2008 124 Table A6-7. DID Analysis of Program Inputs and Outputs IN-DISTRICT X-DISTRICT mfx pval N Ps.RSq mfx pval N Ps.RSq Resources & Activities Received installment 2 (complete construction) 0.47 0.000 306 23% 0.47 0.000 346 23% of Add II Water scheme 0.21 0.019 608 18% 0.21 0.011 692 21% Sanitation scheme -0.03 0.647 608 14% -0.05 0.350 692 12% District help > 1/mo 0.06 0.355 608 18% 0.09 0.170 692 19% Community contribution 0.56 0.000 481 18% 0.44 0.000 537 21% VWSC is active in village 0.24 0.001 608 5% 0.23 0.002 692 4% Outputs Key hygiene message from 4 key sources -0.01 0.593 608 15% -0.03 0.25 692 13% Key hygiene message from any sources -0.06 0.001 608 4% -0.05 0.003 692 4% % of households with > 40 lpcd 0.05 0.149 608 21% 0.06 0.068 692 20% % of community sources with no e.coli -0.08 0.184 608 1% -0.04 0.453 692 0% Extent of garbage bins 0.03 0.048 608 10% 0.02 0.122 692 15% Extent of organized drains 0.02 0.454 608 12% -0.001 0.965 692 11% Whether a household used a toilet (i.e., IHL, community toilet, and neighbor’s toilet) as main 0.097 0.001 24763 0.055 0.064 0.028 27435 0.053 defecation site Whether a household relied on private and public 0.055 0.183 24683 0.008 0.084 0.035 27346 0.008 taps as main water source June 21, 2008 125 Table A6-8. DID Analysis of Program Outputs, Outcomes and Impacts IN-DISTRICT X-DISTRICT Dry Season Rainy Season Dry Season Rainy Season mfx pval mfx pval mfx pval mfx pval Outputs Whether a household used a toilet (i.e., IHL, community toilet, and neighbor’s toilet) as main 0.101 0.002*** 0.092 0.004*** 0.074 0.020** 0.055 0.076* defecation site Whether a household relied on private and public 0.083 0.084* 0.024 0.611 0.132 0.008*** 0.032 0.44 taps as main water source Outcomes E. coli levels in household drinking water (in 0.034 0.778 0.258 0.113† 0.052 0.661 0.170 0.234 log10) LPCD (based on household size) 3.164 0.202 2.591 0.164 3.601 0.117† 1.03 0.539 # of critical times a caregiver washes her/his 0.073 0.69 -0.233 0.047** 0.077 0.666 -0.008 0.944 hands # of critical times a child washes her/his hands 0.086 0.432 -0.048 0.403 0.052 0.617 0.025 0.639 Whether a household treated its water before 0.016 0.631 0.03 0.406 0.055 0.080* 0.039 0.242 drinking # of safe water handling techniques households 0.074 0.106† 0.056 0.288 0.001 0.983 0.088 0.092* practiced$ diarrhea knowledge Out of 3 key symptoms, # of symptoms a -0.072 0.14† 0.028 0.604 -0.127 0.012** 0.033 0.573 caregiver can correctly identify‡ Out of 8 key causes, # of causes a caregiver -0.328 0.152 -0.333 0.22 -0.058 0.776 -0.267 0.23 can correctly identify# Whether a household that treated diarrhea 0.006 0.558 0.004 0.652 0.012 0.315 0.01 0.246 Impacts Diarrhea prevalence among under 5 children 0.017 0.232 0.028 0.049** -0.001 0.951 0.021 0.138† Arm circumference of under 5 children 0.262 0.104† -0.063 0.688 0.167 0.389 -0.056 0.73 June 21, 2008 126 IN-DISTRICT X-DISTRICT Dry Season Rainy Season Dry Season Rainy Season mfx pval mfx pval mfx pval mfx pval Monthly household averting costs due to inadequate water supply and sanitation services -202.37 0.069* -20.22 0.797 -146.50 0.130† -41.56 0.513 (in Rupees) Household cost of illness due to diarrhea -150.17 0.004*** 40.05 0.415 34.13 0.635 107.07 0.208 (in Rupees) ‡ The 3 key diarrhea symptoms include loose stool, vomiting, and loss of weight. # The 8 key diarrhea causes include eating stale foods, eating food touched by flies, uncleaned/smelly food, drinking contaminated water, using unhygienic latrines/OD, not washing hands, household uncleanliness, and village uncleanliness. $ The safe water handling practices include the exclusive use of narrow mouth storage for drinking water, covered drinking water storage at all times, and practice of good water transferring techniques (i.e., no direct hand contact with drinking water) *** significant at 1%;** significant at 5%; * significant at 10%; † significant at 15%. June 21, 2008 127 Table A6-9A. DID Analysis of Village Categorization Based on Water and Sanitation Schemes by In-District Sub-Sample and Season IN-DISTRICT POSTWS1 POSTWS2 POSTWS3 POSTWS4 Dry Season mfx sig mfx sig mfx sig mfx sig † diarrhea -0.031 0.118 0.004 0.846 0 0.991 0.032 0.158 armsize 0.586 0.009*** 0.134 0.449 0.481 0.068* 0.471 0.061* HHcoping 84 0.490 6 0.967 -447 0.016** 17 0.884 HHcoi -7 0.952 -114 0.158 -117 0.074* -247 0.003*** lpcd -2.653 0.513 6.504 0.075* -0.158 0.965 2.976 0.424 log10(E. coli) 0.006 0.965 -0.087 0.602 0.219 0.233 -0.114 0.539 † no-OD 0.051 0.274 0.084 0.108 0.09 0.032** 0.194 0.004*** † † tap: private/public -0.088 0.120 0.121 0.100 0.034 0.630 0.061 0.413 Rainy Season mfx sig mfx sig mfx sig mfx sig † diarrhea -0.006 0.821 0.005 0.772 0.03 0.108 0.047 0.073* armsize -0.012 0.956 -0.281 0.324 0.128 0.558 -0.104 0.661 HHcoping -88 0.550 103 0.338 -124 0.287 -45 0.672 † HHcoi 85 0.130 44 0.469 45 0.412 82 0.429 lpcd 1.856 0.599 3.462 0.178 0.528 0.812 5.943 0.067* log10(E. coli) 0.399 0.188 0.518 0.028** 0.159 0.426 0.386 0.204 no-OD -0.002 0.948 0.031 0.526 0.091 0.030** 0.169 0.010*** tap: private/public 0.011 0.847 0.035 0.637 0.031 0.659 0.015 0.804 *** significant at 1%;** significant at 5%; * significant at 10%; † significant at 15%. June 21, 2008 128 Table A6-9B. DID Analysis of Village Categorization Based on Water and Sanitation Schemes by Across-District Sub-Sample and Season X-DISTRICT POSTWS1 POSTWS2 POSTWS3 POSTWS4 Dry Season mfx sig mfx sig mfx sig mfx sig diarrhea -0.008 0.681 -0.002 0.908 -0.005 0.814 -0.002 0.931 armsize 0.861 0.000*** 0.346 0.074* 0.301 0.386 0.618 0.017** HHcoping -314 0.086* -85 0.440 -433 0.004*** -75 0.409 HHcoi 107 0.202 44 0.578 178 0.168 -98 0.218 lpcd 1.652 0.657 6.865 0.043** 1.825 0.591 4.347 0.197 log10(E. coli) -0.015 0.927 0.003 0.990 0.184 0.268 -0.098 0.605 no-OD 0.026 0.622 0.038 0.407 0.051 0.207 0.163 0.010*** tap: private/public 0.058 0.267 0.189 0.010*** 0.149 0.080* 0.101 0.143† Rainy Season mfx sig mfx sig mfx sig mfx sig diarrhea 0.028 0.227 0.002 0.922 0.043 0.032** 0.042 0.089* armsize 0.221 0.333 -0.309 0.269 0.232 0.316 0.015 0.949 HHcoping 68 0.486 108 0.252 -58 0.528 -106 0.289 HHcoi 297 0.012** 139 0.227 204 0.061* 202 0.134† lpcd -3.22 0.172 1.747 0.461 -1.881 0.347 1.607 0.631 † † log10(E. coli) 0.278 0.125 0.36 0.110 0.14 0.467 0.265 0.307 † no-OD 0.062 0.342 -0.01 0.799 0.063 0.121 0.148 0.014** tap: private/public 0.077 0.186 0.046 0.494 0.07 0.288 0.033 0.531 *** significant at 1%;** significant at 5%; * significant at 10%; † significant at 15%. June 21, 2008 129 Table A6-10A. DID Analysis of Village Categorization Based on Receiving Jalswarajya Addendum II Funds by In-District Sub- Sample and Season IN-DISTRICT JSPAY0POST JSPAY1POST JSPAY2POST JSPAY3POST Dry Season mfx sig mfx sig mfx sig mfx sig diarrhea -0.008 0.764 -0.006 0.792 0.019 0.247 0.03 0.242 armsize -0.098 0.629 0.153 0.428 0.263 0.182 0.155 0.772 HHcoping 98 0.445 85 0.465 -128 0.335 -865 0.008*** HHcoi -21 0.803 -173 0.052* -186 0.003*** 23 0.822 lpcd 2.271 0.562 8.214 0.077* 1.781 0.530 6.32 0.236 log10(E. coli) 0.113 0.399 0.168 0.460 0.003 0.984 0.207 0.587 no-OD 0.034 0.516 0.239 0.001*** 0.082 0.022** 0.11 0.253 tap: private/public -0.006 0.931 0.05 0.617 0.082 0.157 0.142 0.129† Rainy Season mfx sig mfx sig mfx sig mfx sig diarrhea 0.011 0.600 0.037 0.188 0.036 0.033** -0.008 0.799 † armsize 0.03 0.881 -0.555 0.108 0.073 0.675 0.249 0.415 HHcoping 180 0.091* 88 0.488 18 0.860 -6 0.969 HHcoi -99 0.400 73 0.098* 24 0.623 -157 0.082* lpcd -0.354 0.914 7.464 0.003*** 1.841 0.413 -3.014 0.175 log10(E. coli) -0.171 0.371 0.132 0.605 0.243 0.264 0.19 0.481 no-OD 0.035 0.268 0.222 0.010*** 0.077 0.023** 0.03 0.596 tap: private/public -0.004 0.964 -0.048 0.424 0.065 0.273 -0.073 0.211 *** significant at 1%;** significant at 5%; * significant at 10%; † significant at 15%. June 21, 2008 130 Table A6-10B. DID Analysis of Village Categorization Based on Receiving Jalswarajya Addendum II Funds by Across-District Sub-Sample and Season X-DISTRICT JSPAY0POST JSPAY1POST JSPAY2POST JSPAY3POST Dry Season mfx sig mfx sig mfx sig mfx sig diarrhea -0.04 0.057* -0.032 0.057* -0.01 0.544 0.008 0.690 armsize -0.003 0.989 0.201 0.376 0.173 0.506 -0.03 0.952 HHcoping 86 0.468 51 0.625 -108 0.372 -617 0.024** † HHcoi 124 0.130 -12 0.886 75 0.440 215 0.059* lpcd -0.332 0.930 7.315 0.076* 1.594 0.538 7.285 0.180 log10(E. coli) -0.056 0.676 0.151 0.519 -0.044 0.751 0.293 0.460 no-OD 0.008 0.836 0.14 0.028** 0.066 0.076* 0.08 0.428 tap: private/public 0.029 0.659 0.066 0.438 0.158 0.011** 0.189 0.066* Rainy Season mfx sig mfx sig mfx sig mfx sig † diarrhea 0.009 0.652 0.036 0.147 0.024 0.152 -0.015 0.664 † armsize 0.073 0.710 -0.529 0.109 0.116 0.542 0.219 0.528 HHcoping 64 0.498 15 0.886 -36 0.638 -26 0.856 † HHcoi 27 0.850 171 0.124 124 0.259 -84 0.544 lpcd -2.445 0.425 5.67 0.008*** -0.712 0.717 -4.871 0.028** log10(E. coli) -0.095 0.547 0.147 0.508 0.116 0.537 0.337 0.176 † no-OD 0.022 0.483 0.12 0.091* 0.052 0.149 0.011 0.858 tap: private/public -0.041 0.584 -0.045 0.376 0.054 0.293 -0.049 0.173 *** significant at 1%;** significant at 5%; * significant at 10%; † significant at 15%. June 21, 2008 131 Table A6-11A. DID Analysis of Tap or/and Toilet Villages by In-District Sub-Sample and Season IN-DISTRICT JSPACK5POST JSPACK4POST JSPACK3POST JSPACK2POST JSPACK1POST Dry Season mfx sig mfx sig mfx sig mfx sig mfx sig diarrhea -0.022 0.366 0.002 0.916 0.013 0.749 0.013 0.545 -0.029 0.122† armsize 0.068 0.757 -0.074 0.668 0.45 0.207 0.366 0.327 -0.294 0.137† HHcoping -89 0.472 -115 0.431 196 0.255 -568 0.049** 26 0.846 † HHcoi -266 0.010** -163 0.052* -93 0.353 -115 0.147 -26 0.716 lpcd -0.593 0.900 -0.603 0.861 20.809 0.000*** 2.457 0.434 -1.108 0.732 log10(E. coli) -0.326 0.050* 0.068 0.717 0.401 0.255 -0.128 0.518 -0.152 0.248 no-OD 0.295 0.000*** 0.042 0.160 0.523 0.003*** 0.097 0.070* 0.055 0.098* tap: 0.169 0.123† 0.213 0.006*** -0.094 0.690 0.132 0.179 0.105 0.120† private/public Rainy Season mfx sig mfx sig mfx sig mfx sig mfx sig † diarrhea 0.047 0.143 0.011 0.573 0.086 0.008*** 0.053 0.052* 0.013 0.498 armsize 0.013 0.968 -0.035 0.886 -0.988 0.000*** 0.314 0.241 0.036 0.832 HHcoping -471.416 0.000*** -208.161 0.080* -92.435 0.496 49.037 0.619 -332.692 0.002*** HHcoi -40.728 0.672 -5.345 0.901 173.486 0.199 -31.347 0.548 -77.117 0.291 lpcd -1.545 0.667 1.532 0.546 5.242 0.082* 1.5 0.584 -2.546 0.349 † † log10(E. coli) 0.063 0.821 0.341 0.126 0.909 0.007*** 0.507 0.111 0.274 0.173 no-OD 0.25 0.000*** 0.007 0.831 0.294 0.029** 0.07 0.231 -0.005 0.845 tap: 0.214 0.042** 0.294 0.001*** -0.187 0.023** 0.213 0.113† 0.293 0.002*** private/public *** significant at 1%;** significant at 5%; * significant at 10%; † significant at 15%. June 21, 2008 132 Table A6-11B. DID Analysis of Tap or/and Toilet Villages by Across-District Sub-Sample and Season X-DISTRICT JSPACK5POST JSPACK4POST JSPACK3POST JSPACK2POST JSPACK1POST Dry Season mfx sig mfx sig mfx sig mfx sig mfx sig diarrhea -0.04 0.027** -0.015 0.502 -0.017 0.659 -0.001 0.953 -0.031 0.102† armsize 0.018 0.947 -0.116 0.709 0.581 0.143† 0.363 0.316 -0.1 0.671 HHcoping -173 0.079* -247 0.048** 39 0.806 -411 0.084* -175 0.141† HHcoi -197 0.036** 37 0.798 -33 0.760 6 0.943 -115 0.132† lpcd -1.505 0.708 -0.019 0.996 17.901 0.000*** 2.968 0.324 -2.368 0.403 log10(E. coli) -0.244 0.194 0.086 0.638 0.339 0.346 0.193 0.327 0.012 0.932 no-OD 0.217 0.002*** 0.017 0.661 0.48 0.008*** 0.089 0.186 0.039 0.339 tap: 0.154 0.147† 0.27 0.005*** -0.093 0.700 0.212 0.083* 0.057 0.436 private/public Rainy Season mfx sig mfx sig mfx sig mfx sig mfx sig † diarrhea 0.035 0.212 0.022 0.303 0.074 0.037** 0.038 0.122 0.023 0.247 armsize 0.144 0.619 0.087 0.742 -0.806 0.005*** 0.223 0.426 0.186 0.346 HHcoping -574 0.000*** -307 0.000*** -228 0.074* -70 0.283 -455 0.000*** HHcoi -94 0.282 -8 0.879 123 0.391 -79 0.278 -246 0.077* lpcd -4.562 0.185 0.192 0.932 3.224 0.300 -0.711 0.779 -3.183 0.161 log10(E. coli) 0.089 0.714 0.356 0.087* 0.829 0.016** 0.229 0.429 0.257 0.118† no-OD 0.185 0.001*** -0.006 0.866 0.253 0.052* 0.082 0.171 0.012 0.718 tap: 0.109 0.217 0.233 0.004*** -0.264 0.000*** 0.19 0.124† 0.141 0.063* private/public *** significant at 1%;** significant at 5%; * significant at 10%; † significant at 15%. June 21, 2008 133 Annex 7. Summary of methods to adjust for additional covariates in the DID estimates for diarrhea in children under the age of five The difference-in-differences (DID) estimator that we have used in our analyses assumes that the change in diarrhea prevalence in the intervention and control group would be the same in the absence of the Jalswarajya program. This assumption is reasonable in this study design due to the careful selection of control villages and the comparability of the two groups at baseline across a wide range of potentially confounding characteristics (Table XX cite baseline comparison). To test the robustness of this assumption, we have conducted additional analyses that control for variables that were associated with diarrhea at baseline and could potentially lead to a different trajectory in the diarrhea prevalence over time. Specifically, we considered the following baseline covariates: Village-level diarrhea prevalence (0-1) Household has private tap (yes/no) Household treats their water (yes/no) Household does not practice open defecation (yes/no) Household liters per capita per day (LPCD) (continuous) Household practices safe water handling-narrow neck, covered, good transfer (yes/no) Count of number of critical times caregiver reports washing hands (0-5) Count of number of caregiver’s correct responses to diarrhea causes (-2 - 8) We estimated the univariate association of each covariate with child diarrhea status at baseline. All covariates except for (i) having a private tap in the home and (ii) treating their water had associations with baseline diarrhea with a p-value less than 0.2. Using marginal probit estimation identical to our unadjusted analyses, we estimated the DID with the additional covariates added as main effect linear terms. We also estimated the DID using household-level fixed effects, which is highly conservative and controls for all household level differences. Additionally, we re-estimated the models using two treatment definitions: Actual Treatment, where the villages are analyzed according to their actual participation status in Jalswarajya, and Assigned Treatment, where the villages are analyzed according to their treatment assignment at enrollment into the study. At enrollment, 95 villages were expected to receive the intervention and 142 study villages were not included in the program. By 2007, our records indicate that 29 of these villages did not participate in Jalswarajya, but 13 control villages actually participated in the program (for a net count of 79 intervention villages and 163 control villages). Our adjusted analyses suggest the DID estimates are not confounded by the covariates that we have included in the extended specification (Table XX.1). We obtained very similar results with household-level fixed effects. We conclude based on these analyses that there is little detectible confounding bias in the unadjusted DID estimates. Our treatment definition is consequential: under the Actual Treatment definition Jalswarajya has lower effectiveness during the rainy season but relatively little effect in the dry season; under the Assigned Treatment definition, Jalswarajya again has lower effectiveness in the wet season, but greater effectiveness in the dry season in the cross-district (X-District) sample and in the Full June 21, 2008 134 Sample (including all study villages). Overall, the full sample results are similar to the cross- district results but the gains in the dry season are smaller under the Assigned Treatment definition. Table A7-1. Summary of Difference-in-Difference (DID) Estimates for Diarrhea in Children <5 IN DISTRICT X DISTRICT FULL SAMPLE Treatment Definition* / Dry Season Rainy Season Dry Season Rainy Season Dry Season Rainy Season Model DID pval DID pval DID pval DID pval DID pval DID pval Actual Treatment Unadjusted † 0.017 0.232 0.028 0.049 -0.001 0.951 0.021 0.138 0.004 0.757 0.023 0.061 Adjusted ‡ 0.016 0.243 0.026 0.064 0.000 0.979 0.021 0.136 0.004 0.749 0.022 0.067 Household- level fixed 0.015 0.349 0.027 0.062 -0.003 0.859 0.015 0.251 0.000 0.977 0.018 0.129 effects Assigned Treatment Unadjusted † 0.011 0.454 0.019 0.172 -0.028 0.038 0.015 0.286 -0.013 0.264 0.018 0.129 Adjusted ‡ 0.011 0.423 0.018 0.194 -0.026 0.047 0.016 0.264 -0.012 0.303 0.017 0.135 Household- level fixed 0.009 0.598 0.01 0.475 -0.038 0.019 0.009 0.489 -0.018 0.199 0.010 0.369 effects * The Actual Treatment analysis analyzes villages according to the treatment they actually received. The Assigned Treatment analysis analyzes villages according to the treatment they were assigned at enrollment. † These estimates are identical those reported in the WB report (draft 4/17/2008), and were used as a benchmark. ‡ Adjusted models include linear main effects (no interactions) for baseline values of variables that have univariate associations with diarrhea at baseline (p < 0.2): Village-level diarrhea prevalence (0-1) No open defecation (yes/no) LPCD (continuous) Household practices safe water handling-narrow neck, covered, good transfer (yes/no) Count of number of critical times caregiver washes hands (0-5) Count of number of correct responses to diarrhea causes (-2 - 8) June 21, 2008 135 References Abadie, A., 2005. Semiparametric Difference-in-Differences. Review of Economic Studies. Alderman, 2000. LSMS Almus, M., Lechner, M., Pfeiffer, F., & Spengler, H. 2001. The Impact of Non-profit Temping Agencies on Individual Labour Market Success in the West German State of Rhineland- Palatinate. In Lechner, M., & Pfeiffer, F. (Eds.), Econometric Evaluation of Labour Market Policies. Heidelberg: Physica-Verlag, pages 211-242. Augurzky and Schmidt, 2001 Baker, J.L. 2000. Evaluating the Impact of Development Projects on Poverty: A Handbook for Practitioners. The World Bank, Washington. Bamberger, M. 2006. Conducting Quality Impact Evaluations under Budget, Time, and Data Constraints. World Bank Evaluation Group Evaluation Capacity Development Publication. Bamberger, M., J. Rugh, M. Church, and L. Fort. 2004. Shoestring Evaluation: Designing Impact Evaluations under Budget, Time, and Data Constraints. American Journal of Evaluation 25 (1): 5-37. Bardhan and Mookherjee, 2000 Behrman et al. (2005) Blitstein, J.L., Murray, D.M., Hannan, P.J., & Shadish, W.R. (2005). Increasing the degrees of freedom in future group randomized trials: The df* approach. Evaluation Review 29(3), 268- 286. Blum, D., and R.G. Feachem. 1983. “Measuring the Impact of Water Supply and Sanitation Investments on Diarrhoeal Diseases: Problems of Methodology,� International Journal of Epidemiology 12 (3): 357-365. Corey, C.G., J.-C. Yang, S.K. Pattanayak, and R.K. Kwok. 2007. Seasonal variation in risk factors associated with diarrheal diseases, rural Maharashtra, India. RTI International Working Paper 07-03, June 2007. Curtis, V., and S. Cairncross. 2003. Effect of washing hands with soap on diarrhea risk in the community, a systematic review. Lancet Infectious Disease. 3:275-281. Devarajan & Kanbur, 2004 Diggle et al. (1994) Dickinson, K.L (with S.K. Pattanayak). 2008. Open-sky latrines: Understanding the role of social interactions in households’ decisions to adopt latrines in Orissa, India. Ph.D. dissertation chapter. Duke University. May, 2008. Donner et al. (1981) June 21, 2008 136 Duflo, E., A. Banerjee, R. Glennerster, and M. Kremer. 2006. “Randomization in Development Economics: A Toolkit,� in Handbook of Development Economics, Vol. 4. Duflo and Kremer, 2003 Esrey, S. 1996. Water, Waste and Well-being: A Multi-Country Study. American Journal of Epidemiology 143(6):608-623. Esrey, S., J. Potash, L. Roberts, and C. Shiff. 1991. Effects of Improved Water Supply and Sanitation on Ascariasis, Diarrhea, Dracunculiasis, Hookworm Infection, Schistosomiasis, and Trachoma. Bulletin of the World Health Organization 69(5): 609-621. Esrey, S.A., and J-P Habicht. 1986. “Epidemiologic Evidence for Health Benefits from Improved Water and Sanitation in Developing Countries,� Epidemiologic Reviews 8:117-128. Fewtrell, L., R.B. Kaufmann, D. Kay, W. Enanoria, L. Haller, and J.M. Colford, Jr.. 2005. Water, Sanitation, and Hygiene Interventions to Reduce Diarrhoea in Less Developed Countries: A Systematic Review and Meta-Analysis. The Lancet Infectious Diseases 5(1): 42–52. Galasso, E., M. Ravallion, and A. Salvia. 2001. World Bank Policy Research Working Paper 2738. The World Bank. Galasso, E. and N. Umapathi. 2007. Improving Nutritional Status through Behavioral Change: Lessons from Madagascar. World Bank Policy Research Working Paper 3625. The World Bank. Galiani, S., P. Gertler, and E. Schargrodsky. 2005. Water for Life: The Impact of the Privatization of Water Services on Child Mortality. Journal of Political Economy 113: 83- 120. Gakidou et al., 2007 Heckman et al., 1998 Heckman and Smith, 1995 Hirano et al., 2003 Ho, D., K. Imai, G. King, and E. Stuart. 2007. Matching as Nonparametric Preprocessing for Reducing Model Dependence in Parametric Causal Inference. Political Analysis.15: 199-236. Imbens, G., and J. Angrist. 1994. “Identification and Estimation of Local Treatment Effects,� Econometrica 61 (4): 655-680. India, Government of, Ministry of Rural Development, Department of Drinking Water Supply. 2004. Guidelines on the Central Rural Sanitation Programme. India, Government of, Planning Commission. 2001. National Human Development Report. India, Government of, Planning Commission. 2007. Towards More Inclusive Growth: An Approach to the Eleventh Five Year Plan 2007- 2012. Isham and Kahkonen, 2002 June 21, 2008 137 Jalan, J. and M. Ravallion. 2003. Does Piped Water Reduce Diarrhea for Children in Rural India? Journal of Econometrics Vol. 112(1): 153-173. Jha, N. and S.K. Pattanayak. “Hydraulic Self-Rule: Establishing Community Water Supply and Sanitation Schemes in Rural Mahashtra , India.� Paper presented at the 11th Biennial Conference of the International Association for the Study of Common Property, Ubud, Bali, Indonesia, 19-23 June, 2006. Kar, Kamal. 2003. Subsidy or Self-Respect? Participatory Total Community Sanitation in Bangladesh. Institute of Development Studies, Working Paper Series, Sussex, England. Katz, J., V.J. Cary, S.L. Zeger, and A. Sommer. 1993. Estimation of design effects and diarrhea clustering within households and villages. American Journal of Epidemiology 138(11): 994- 1006. Kremer, M., J. Leino, E. Miguel, and A. Peterson-Zwane. 2008. Spring Cleaning: A Randomized Evaluation of Source Water Quality Improvement. Working Paper. Harvard University. Mansuri, G., and V. Rao. 2004. Community–Based and –Driven Development: A Critical Review. The World Bank Research Observer 19(1):1–39 McKenzie, D. and I. Ray. (forthcoming). Urban Water Supply in India: Status, Reform Options, and Possible Lessons. Water Policy. McKenzie, D. and I. Ray. 2005. Household Water Delivery Options in Urban and Rural India. Stanford Center for International Development Working Paper no. 224. Stanford University: Stanford, CA. Orissa State (India), Government of, 2003-2004. Activities of the Health and Family Welfare Department in 2003-2004 and Plan for 2004-2005. Patil, S.R., and S.K. Pattanayak. 2007. The In-House Water Quality Management: Uncovering the Complex Web of Household Behaviors, and Water and Sanitation Interventions. RTI Working Paper 07-04, Research Triangle Institute, RTP, NC. Patil, S.R., S.K. Pattanayak and S. Vinerkar. 2006. Gauging Adequacy of Community Water Supply and Sanitation Projects in Maharashtra: Methodological Triangulation. RTI Working Paper 06-06, Research Triangle Institute, RTP, NC. Pattanayak, S.K., C. Poulos, J.-C. Yang, and S.R. Patil. 2007. Of taps and toilets: Quasi- experimental approaches for evaluating community-demand-driven projects. RTI International Working Paper 06-04, May 2007. Pattanayak, S.K., J.-C. Yang, D. Whittington, and K.C. Bal Kumar. 2005b. Coping with Unreliable Public Water Supplies: Averting Expenditures by Households in Kathmandu, Nepal. Water Resources Research 41, W02012, doi: 10.1029/2003WR002443. June 21, 2008 138 Pattanayak, S.K., J.L. Blitstein, J.-C. Yang, S.R. Patil, K.M. Jones, C. Poulos, and K.L. Dickinson. 2006. Evaluating Information and Communication Strategies to Promote Latrine Use and Improve Child Health: Design and Baseline Findings from a Community Randomized Trial in Bhadrak, Orissa. RTI Working Paper 06-05, Research Triangle Institute, RTP, NC. Pattanayak, S.K., K.L. Dickinson, J.-C. Yang, S.R. Patil, P. Praharaj, R. Mallick, and C. Poulos. 2007b. Nature's Call: Can a (randomized) social mobilization campaign lead households to use toilets and reduce diarrhea? RTI International Working Paper 07-02, March 2007. Pattanayak, S.K., J.L. Blitstein, J.-C. Yang, K.L. Dickinson, S.R. Patil, C. Poulos, and K.J. Wendland. 2006. TVs or toilets? Design and baseline findings from a community mobilization campaign to promote toilet use in Bhadrak, Orissa. RTI International Working Paper 06-05, August 2006. Pattanayak, S.K., C. Poulos, J.-C. Yang, and S.R. Patil. 2008. How valuable are environmental health interventions? Evidence from a quasi-experimental evaluation of community water projects. RTI International Working Paper 08-01, June 2008. Pattanayak, S.K., J.-C. Yang, B. Arnold, and J. Colford. 2008. Health and behavioral impacts of a community-demand-driven water and sanitation program in rural India. RTI International Working Paper 08-02, June 2008. Poulos, C., S. K. Pattanayak, and K. Jones. 2006. A Guide to Water and Sanitation Sector Impact Evaluations. Doing Impact Evaluation Series No. 4. World Bank, Washington DC. Preisser, J.S., M. L. Young, D. J. Zaccaro and M. Wolfson. 2003. An integrated population- averaged approach to the design, analysis and sample size determination of cluster-unit trials. Statistics in Medicine. 22:1235–1254. Rawlings, L.B., L. Sherburne-Benz, and J.V. Domelen. 2004. Evaluating Social Funds: A Cross- Country Analysis of Community Investments. The World Bank, Washington, D.C. Ravallion, M. 2008. Evaluation in the Practice of Development. World Bank Policy Research Working Paper 4547. The World Bank. Ravallion, M. 2005. Evaluating Anti-Poverty Programs. World Bank Policy Research Working Paper 3625. The World Bank. Sara and Katz, 1998 Sanan, D., & Moulik, S. G. 2007. Community-Led Total Sanitation in rural areas: An approach that works: Water and Sanitation Program -South Asia, The World Bank. Sills, E., R. Arriagada, S. K. Pattanayak, P. Ferraro, L. Carrasco and S. Cordero. Forthcoming. “Private Provision of Public Goods: Applying Program Evaluation to Evaluate ‘Payments for Ecosystem Services’ in Costa Rica�. Chapter 10 in Ecomarket: Costa Rica’s Experience with Payments for Environmental Services. G. Platais and S. Pagiola (eds.). World Bank, Washington D.C. June 21, 2008 139 The United Nations Development Programme (UNDP) 2006. Human Development Report 2006 – Beyond scarcity: Power, poverty and the global water crisis. UNDP: New York. Accessed at < http://hdr.undp.org/hdr2006/report.cfm> on 31 January 2007. The World Bank – Operations Evaluation Department. 2004. Monitoring and Evaluation: Some tools, methods, and approaches. The World Bank, Washington, D.C. Victora, C.G., J.P. Habitch, and J.Bryce. 2004. Evidence-based public health: Moving beyond randomized trials. American Journal of Public Health. 94(3): 400-405. Walker et al. 1999 Wang, L. 2002. Health Outcomes in Low-income Countries and Policy Implications: Empirical Findings from Demographic and Health Surveys. Policy Research Working Paper #2831, The World Bank. Whittington et al. (2008) Zwane, A.P., and M. Kremer. 2007. “What Works in Fighting Diarrheal Diseases in Developing Countries? A Critical Review� World Bank Research Observer 22: 1-24. June 21, 2008 140