WPS6042 Policy Research Working Paper 6042 Impact of Social Fund on the Welfare of Rural Households Evidence from the Nepal Poverty Alleviation Fund Dilip Parajuli Gayatri Acharya Nazmul Chaudhury Bishnu Bahadur Thapa The World Bank South Asia Region Sustainable Development Unit and Human Development Unit April 2012 Policy Research Working Paper 6042 Abstract The Nepal Poverty Alleviation Fund is a World Bank program on key welfare outcomes. The treatment-on- supported community-driven development program. the-treated estimate on real per capita consumption is 19 Its objective is to improve rural welfare, particularly for percent growth. Other impacts include a 19 percentage groups that have traditionally been excluded for reasons points decline on incidence of food insecurity (defined of gender, ethnicity, caste, and location. Since its launch as food sufficiency for six months or less) and a 15 in 2004, the Fund has covered the 40 poorest districts percentage points increase in the school enrollment rate of the country, supported some 15,000 community among 6–15 year-olds. Impacts (positive or negative) organizations, and benefited more than 2.5 million are yet to be detected on indicators associated with child people. This paper attempts to estimate the impact of malnutrition, social capital, and empowerment. The this large-scale program using a randomized phase- policy implications of these results should be of interest in approach, in which certain localities are randomly to the government and to development partners in selected for earlier intervention than others. Using two determining what may be effective instruments to deliver rounds of survey data and a difference-in-difference services to marginalized communities in what remains a combined with instrumental variable estimation method, fragile and difficult political environment. it finds statistically significant causal impact of the This paper is a product of the Sustainable Development Unit and the Human Development Unit, South Asia Region. It is part of a larger effort by the World Bank to provide open access to its research and make a contribution to development policy discussions around the world. Policy Research Working Papers are also posted on the Web at http://econ.worldbank. org. The authors may be contacted at dparajuli@worldbank.org. The Policy Research Working Paper Series disseminates the findings of work in progress to encourage the exchange of ideas about development issues. An objective of the series is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. Produced by the Research Support Team Impact of Social Fund on the Welfare of Rural Households: Evidence from the Nepal Poverty Alleviation Fund Dilip Parajuli, Gayatri Acharya, Nazmul Chaudhury and Bishnu Bahadur Thapa 1 World Bank JEL: H53, I38 Keywords: Poverty Alleviation Fund, Impact Evaluation, Instrumental Variables, Nepal 1 Dilip Parajuli is a Senior Economist in the South Asia Education Unit; Gayatri Acharya is a Country Sector Coordinator in the South Asia Agriculture & Rural Development Unit, Nazmul Chaudhury is a Country Sector Coordinator in the East Asia Social Protection Unit, and Bishnu Bahadur Thapa is a Junior Professional Associate in the South Asia Agriculture & Rural Development Unit. The authors would like to acknowledge collaboration from Poverty Alleviation Fund (PAF) Secretariat of the Government of Nepal in the implementation of this Impact Evaluation initiative, and from Tribhuvan University (TU) in the survey data collection. We would like to thank Jeffery Hammer, Atul Pokharel and Geeta Sethi for their significant support during the early stages of the Impact Evaluation design and survey activities. We would also like to thank participants at presentations made at the PAF, the National Planning Commission, Simeon Ehui, Amit Dar, Madhur Gautam, Elliot Mghenyi, Juan Carlos Osario, Venkatesh Sundararaman, and participants at a number of seminars and presentations for helpful comments on presentations and earlier versions of the paper. The findings, interpretations and conclusions expressed in the paper are entirely those of the authors and do not necessarily represent the views of the World Bank or its affiliates. Please send correspondence to dparajuli@worldbank.org. 1. Introduction Developing countries are increasingly promoting social fund type programs where the focus is on community driven development (CDD) approaches to identify livelihood and income enhancing activities which are then implemented and financed through these funds. In conflict or post-conflict countries, particularly where the root causes of strife are often related to wide disparities and inadequate public sector support to the needs of lagging populations, governments and development partners often respond to this need through community-based poverty alleviation funds (World Bank, 2006). Some examples include Afghanistan National Solidarity Program (NSP), Angola Social Action Fund, Colombia Peace and Development Project, Indonesia Program Nasional Pemberdayaan Masyarakat (PNPM), and Nepal Poverty Alleviation Fund. However, there is no consensus on the extent of their effectiveness, in large part since rigorous impact evaluations of CDD-type social funds programs, particularly in conflict-affected settings, are far less common given and the complexity of operations and practical constraints in designing and implementing robust impact assessments. Afghanistan NSP (Beath et. al, 2008) and Indonesia PNPM (Olken et. al, 2010) are among the few successfully implemented examples. Nepal Poverty Alleviation Fund (PAF) is a semi-autonomous government agency, created by the Government of Nepal by an Act of Parliament to function as a targeted program of poverty alleviation for marginalized and poor households. The World Bank provides financial and technical support to this targeted instrument, in particular to improve living conditions, livelihoods and empowerment among the rural poor, with particular attention to groups that have traditionally been excluded by reasons of gender, ethnicity, caste and location. Since its launch in 2004, PAF has covered 40 poorest districts of the country, supporting some 15,000 community organizations and benefiting more than 500,000 households, over 2.5 million people, roughly 10% of the population. The two main interventions in PAF are: (i) income generating activities (IG); and (ii) small-scale village and community infrastructure (INF). IG interventions are easily the more prevalent ones, taking up more than 80% of resources and more than 90% of sub- projects. In this paper we present the design and results of an impact evaluation of the Nepal PAF program. The design involves a randomized phase-in approach, in which certain localities were randomly assigned for earlier intervention than others. As expected, there is no perfect compliance between the actual treatment and the planned treatment but sufficiently strong for us 2 to identify the causal impact of PAF program on outcomes of interest. Using two rounds of survey data and difference-in-difference combined with instrumental variable estimation method, we find that there is a positive and significant “PAF effect� on key welfare indicators. The estimated net program impact on real per capita consumption growth is 19 percent for PAF participants. Other impacts include a 19 percentage points decline on incidence of food insecurity (as defined as self-reported food sufficiency for six months or less) and 15 percentage points increase in school enrollment rate among 6-15 year olds. While improvements in nutrition and social empowerment indicators are seen for all groups (treated/untreated), the impact of the PAF program on indicators associated with child malnutrition and indicators of social capital (trust, respect, relationships between different ethnic groups within the community) is yet to be detected. The policy implications of these results should be of interest to the government and to development partners in determining what may be effective instruments to deliver services to marginalized communities in what remains a fragile and difficult political environment. This study contributes to the existing literature on impact evaluation of social fund in a few ways. First, we present results of a large-scale intervention using an experimental impact evaluation design. Second, we estimate the causal impact even when there is no perfect compliance with randomized assignment. Imperfections in randomized evaluations are quite reasonable in real world settings of large-scale programs and allowing for such in fact can maintain external validity of the impact estimates. Third, in addition to using per capita household consumption, we also report impact on food security and access to education which are important indicators of vulnerability in conflict settings. 2. Evaluation Design PAF Setting: As mentioned earlier, PAF interventions are predominantly income generating activities (IGAs), particularly in the initial rounds, since the social cohesion required to motivate investments in public goods often takes longer to form, particularly in communities affected by conflict. The interventions typically have the following sequencing: - PAF and its Partner Organization (PO) – local NGO- select the village in a targeted district (PO’s village selection depends on qualitative and quantitative assessment of need and feasibility) - PO carries our community mobilization in the selected village on possible PAF interventions and household participation - Community Organizations (CO) are formed with 25-30 households as CO members 3 - CO submits a proposal for an IGA for each household in the CO - Each IGA proposal is then evaluated and, if endorsed, is funded by PAF through a grant to the community and the member household implements the IGA - IGA composition: 75% are livestock (goats, cows, buffaloes, etc.), 15% small trading/retail business (tea shops etc.), 5% vegetable farming, and 5% service sector - On average, PAF provides US$ 185 per IGA/household (about $30 per capita). Communities typically establish revolving funds from which households borrow for their own IGAs and self-regulate. Randomized Experimental Design: A pure randomized control trial (RCT) experiment is considered ideal for evaluations. Given the operational setting where PAF interventions are community-demand projects and targeted to poorer communities and poorer households within those communities, a pure RCT is a difficult proposition. However, budget restrictions for any particular year and implementation capacity constraints mean that one could use a randomized phase-in approach that randomly assigns certain communities for early phase-in (Gertler et. al, 2011). At the time of this evaluation, PAF was being scaled up from its pilot phase to a wider geographic coverage. This presented an opportunity to carry out the impact evaluation in communities that were going to be intervened in the near future. A two-stage stratified sampling procedure was adopted. First, six districts2 representing different geographic regions of the country were randomly selected from 25 PAF targeted districts. Second, the sampling frame consisted of those wards/villages (Primary Sampling Units (PSUs)) in these six districts that were not yet intervened but were in a potential pool to be intervened sometime in the future because of their poverty ranking. From a list of approximately 1000 such potential villages, 200 villages/PSUs were randomly selected for the evaluation sample. Allocation across each district was based on the district size (number of wards). Of the 200 evaluation villages, 100 were randomly assigned for early phase-in and the remaining as control villages. This randomization was stratified by district so as to maintain equal proportion of treatment and control PSUs in each district. The randomized list, prepared together with the PAF implementing unit in the government, was to be provided to the district Partner Organizations (NGOs) which partner with local communities/villages to identify PAF intervention proposals. 2 Districts: Rautahat (Terai), Rolpa, Dailekh, Doti (Hills) and Humla, Jumla (Mountains). 4 Survey design: A pre-intervention baseline survey was carried out in 20073. The baseline involved conducting a census of all households in the selected 200 villages, and the administration of a multi-module detailed household questionnaire to randomly sampled 15 households from each village (total of 3,000 households). The survey questionnaire was adapted from the Nepal Living Standards Survey4 (NLSS) and included detailed information on consumption and income, socio-economic and demographic issues, including education, health and nutrition, physical assets, migration and remittances, employment, social environment, community relationship, voice and participation. For comparability with the national household survey based welfare measures, the PAF survey included a similar consumption module and followed the same consumption aggregation method. The follow-up survey, carried out more than two years after the baseline, included the same questionnaires from the baseline survey and also gathered basic information on the actual treatment status (PAF intervention) and non-treatment (control) at both household level and at the village/PSU level. Randomized assignment to early phase-in was not followed perfectly, but sufficiently strongly to allow us to identify a suitable empirical strategy to report causal impact estimates. Compliance with Randomization: Table 1 below describes planned treatment status and actual treatment status of the sampled villages at the time of the follow-up survey. We note two patterns: actual status does not follow random assignment perfectly; however, random assignment strongly predicts actual treatment status (72 of the 100 randomly treatment-assigned villages are treated at the follow-up survey compared to only 39 from the control-assigned). Imperfect compliance with randomized assignment in this evaluation was to be expected given the nature of the PAF interventions. As mentioned earlier, PAF makes use of NGOs to initiate community- based interventions in the rural villages in Nepal. The decision to select a village over another one for early phase-in cannot all be enforced through lottery alone. Implementation readiness, geography, socio-economic conditions and other factors may contribute towards inability to comply with random selection. Allowing the NGOs to implement their work may have resulted in imperfect compliance, but it may have helped maintain a certain degree of external validity (the extent to which the process and results of the interventions are generalized and replicable elsewhere). For the purpose of evaluation, we consider a village in the treatment status if there is a CO in that village supported by the PAF. We also note that not all 15 households (which were 3 Piloting for the impact evaluation was initiated in 2006 in a small sample. The design and implementation of the main impact evaluation benefited from lessons learnt from the pilot in term of sampling procedure, survey questionnaire, and accessibility of sampled villages for reasons related to weather seasonality and remoteness. 4 For details on national household surveys, see Nepal Living Standards Surveys (NLSS 1995/96, 2003/04, 2010/11). The reports can be found at www.cbs.gov.np 5 randomly selected at the baseline) in any treated village may be direct PAF beneficiaries. We follow the approach suggested by Duflo, Glennerster, and Kremer (2007). Within-village5 spillover effects (on outcomes of interests) are plausible for non-beneficiary households in the treated villages, and thus consider all households (regardless of their benefit status) in the treated village as part of the treatment unit. Table 1: Actual and Planned treatment status (number of villages) Randomly assigned Randomly assigned treatment control Actual control 61 28 Actual treatment 39 72 Total 100 100 Outcome of interest: The main welfare indicator of PAF impact evaluation is per capita consumption. Consumption is considered to be a better measure of welfare than income since it captures the family’s expectations of a more permanent change in future income than current income alone6. Food insecurity is another outcome indicator – given it is one of the key outcome indicators for PAF program- and is measured by number of months of food sufficiency for a household. We also examine effects on child level opportunities in education and health/nutrition: school enrollments rates for 6-15 year olds and malnutrition rates for younger children (Table 2). Our analysis also looks at many other dimensions and indicators, most notably those on social capital and empowerment. Table 2: Outcome Indicators for PAF Impact Evaluation Dimension Indicators Sample size Income/Consumption Real Per capita Household Consumption 3,000 HHs (2,774 panel households) Proportion of households with food sufficiency of Food insecurity 3,000 HHs (2,774 panel households) 6 months or less Education School enrollment rate of 6-15 year olds 5,500 children Child Malnutrition (underweight) among children Health/Nutrition 2,300 children under 5 years 3. Empirical Strategy We start with the “intent-to-treat� estimation specification using the following regression: (1) Yijt  �  � Aj  �Tt  � Aj * Tt  �X j  eijt 5 Cross-village spillovers are less likely given villages are far from each other. 6 The PAF monitoring system also collects data on income and other socio-economic characteristics at the household level to report on income changes for households assisted by the fund. 6 where Yi denotes outcome of interest for household i, located in village j, at time t. Change between baseline and follow-up survey periods in per capita consumption (or some other outcome indicator) for household i, A (0,1) denotes random assignment, T (0,1) follow-up time-period. X denotes a vector of village-specific geographic and socio-economic characteristics at baseline. The coefficient β measures the “intent-to-treat� (ITT) impact of the PAF program, while the standard error of the coefficient would provide the statistical significance. Because the treatment unit is at the village level, standard errors are clustered at the PSU level. We add X control variables to improve statistical power of our estimates (to reduce variance). In sample sizes such as ours, this is an important consideration. Data availability at baseline and at follow-up allows us to use the difference-in-difference regression as shown in (1). We note that the same regression can be run for observations for follow-up period only and still identify the net impact as long as the random assignment A ensures that the characteristics and outcomes at the baseline are between the two groups. Intent to treat estimator, however, is less relevant as a parameter of interest for us. One, ITT gives a smaller (in magnitude) impact estimate because in the presence of crossover or imperfect compliance, not all originally treatment-assigned villages are actually treated and some originally control-assigned villages are in fact treated. Two, our primary interest is not so much on the impact of random phase-in program but rather the impact of the actual PAF treatment itself. We cannot simply compare outcomes in actually treated and not-treated villages since actual treatment is endogenous and likely affected by characteristics (such as targeting criteria observable or unobservable) that may themselves influence outcomes, leading to selection bias (Ravallion, 2005). Treatment-on-the-treated (TOT) impact estimator is one solution to this evaluation problem. TOT estimates the impact of the intervention on those whom it was offered and who actually received the intervention. In econometric terms, this is an instrumental variable (IV) estimation technique, whereby random assignment A of treatment status is used to instrument actual treatment status (Angrist and Pischke, 2009). In the first-stage, actual treatment P is regressed with random assignment A and other X variables (2). (2) Pj  �1  �2 Aj  �X j  � j   (3) Yijt  �  � P j  �Tt  � iv P j * Tt  �X j  � ijt 7  Second-stage regression (3) uses P j predicted value of the actual PAF treatment that is obtained through a first-stage regression in (2). The coefficient � iv from the second-stage IV regression is the TOT impact estimate on the outcome of interest. Cluster-corrected standard error associated with this coefficient indicates statistical significance. We also correct for the standard errors to account for the fact that predicted values, and not the actual values of the P, are used in the second-stage regression. To generate consistent estimates of the parameter coefficients and their standard errors, we implement the empirical strategy using “ivreg2� command in STATA (Baum et al. 2007), with “cluster� option and appropriate specification to account for the instrumentation of the interaction term in the model when applicable. The validity of IV estimation strategy requires that two conditions are met: the instrument is exogenous and relevant. Exogeneity means that the instrument should not be correlated with outcomes (except through actual intervention) or with unobserved variables. Relevancy implies that the instrument should be correlated with the actual treatment status. The first condition is met by construction in our evaluation, since the instrument A is randomly assigned. The second condition is met if there is a strong correlation between random assignment A and actual treatment status P and this can be tested using the results from the first-stage regression. 4. Results We start with comparison of baseline outcome variables as well other characteristics between the two randomly assigned treatment and control groups. As seen in annex table 1, the mean differences between the two types of villages are not statistically different, suggesting that the randomization was generally successful. As described above in section 2 and table 1, actual PAF treatment status for a village did not follow random assignment perfectly. However, random assignment strongly predicted actual treatment status (72% PAF intervention from the treatment assignment, compared to only 39% from the control-assignment, thus a 33% differential take-up). We note that the actual treatment status may have additionally been driven by implementation readiness, pressure from a particular village, geography, socio-economic conditions and other factors. The first stage regression estimates, as presented in annex table 2, confirm that this is indeed the case. Probability of actual PAF intervention is positively associated with random treatment assignment (instrument is relevant), “poorest� third of villages as ranked by the NGOs, higher elevation, further distance from district headquarter, higher proportion of uneducated households heads, higher proportion of 8 dalit and other disadvantaged households, and higher proportion of landless households. On the other hand, it is negatively correlated with higher proportion of households owning radios and with higher average age of first marriage for females in the village. These findings are consistent with PAF’s objective of targeting the villages that have been traditionally excluded due to geography and socio-economic features. The results of the impact estimation are summarized in table 3 below (detailed results and specifications in Annex tables 3-6). Consumption: As seen in table 3, the estimated net PAF impact on per capita consumption (in real terms adjusted for price inflation) growth is 7 percent for intent-to-treat estimate and 19 percent for treatment-on-the-treated estimate from instrumental variable regression. The results are statistically significant and robust across alternate specifications (such as that using second round of data only7). In terms of magnitude, 19 percent change translates to about US$40 (Rupees 3,000) absolute change in real per capita consumption. In other words, the initial per capita investment of $30 from PAF appears to provide more than 130% return in just over two years of time. The finding that PAF IG intervention is associated with significant welfare increase provides a strong signal to policy makers on the program’s ability to influence development objective to reduce poverty. Table 3: Summary of Impact Estimates Intent-To-Treat Instrumental Variable Outcome Indicator Specification (ITT) (IV) Per capita consumption, Difference-in-Difference 0.071** 0.187** Log Follow-up Round 0.075** 0.196** Food Insecurity, food Difference-in-Difference -0.071* -0.187* sufficient < 6 months Follow-up Round -0.084*** -0.219*** School Enrollment, 6-15 Difference-in-Difference, all 0.048** 0.139** year olds Difference-in-Difference, girls 0.072** 0.213** Underweight status, 0-4 Difference-in-Difference, all -0.018 -0.056 year olds Difference-in-Difference, girls -0.016 -0.050 Notes: 1. * p<0.10, ** p<0.05, *** p<0.01; 2. Statistical significance is derived from standard errors that are corrected for clustering at PSU level; 3. There are 200 PSUs in the sample, 3000 Households, 2774 Panel HHs, 5500 6-15 year olds, and 2300 0-4 year olds; 4. On panel and non-panel households, there are no statistical differences in baseline values of both the outcome variables and other key correlates, suggesting attrition is not a problem in our analysis. 7 Because randomization was successful in ensuring comparability in the outcome variables at baseline, one can easily estimate the impact using second round observations only for outcome data. 9 Food Insecurity: The analysis estimates that the net PAF impact on incidence of food insecurity (as defined as self-reported food sufficiency for six months or less) is a decline of 19 percentage points when using a combined IV with difference-and-difference method, and a decline of 22 percentage points when using IV on post-treatment data only. The effect is stronger for disadvantaged households, such as those from dalit and janajati communities8. The results are significant since chronic food insecurity is a particularly important concern in Nepal and substantial amounts are spent per year on public works programs aimed at alleviating hardship for food insecure households. Sustained food price inflation remains a concern and an estimated 3.7 million people are currently food insecure. The impact of high prices and food insecurity is most severe on economically, geographically and socially marginalized communities. Since Nepal’s poorest households spend more than 75% of their income on food, high prices will continue to affect poverty alleviation efforts. The strong impact demonstrated by this evaluation suggests that the PAF is an effective program to address food insecurity. School Enrollment: School participation among 6-15 year old children also appears to have been significantly affected by PAF interventions. The impact estimate is an increase of 14 percentage points in school enrollment for all children in the given age-group but girls appear to have benefited even more, with 21 percentage points net increase. The effects are as strong among children from disadvantaged caste/ethnic groups. These are all notable and statistically significant impacts. While child education is not a direct outcome associated with PAF intervention, one can think of at least two ways by which a treated household would change its behavior in relation to this outcome. First, actual or perceived positive change in income/consumption for a household is more likely to reduce potential constraints of sending a child to school. Second, community mobilization and social-networking, as part of the PAF community organization (CO), may produce spillover effects such as motivating households towards child schooling. Child Underweight: Close to 60% of children from sampled villages are underweight (annex table 1)9. We find that there is no statistically significant impact on child underweight rates. However, one can take this as a step in the right direction as shown by the negative sign of the coefficient, a pointer to possible effects in the coming years. Child malnutrition is a serious problem in the country, particularly among children from the poorest communities. 8 Separate estimate for households belonging to Dalit, Janajatis and other caste/ethnic groups (other than Brahmin, Chhetris and Newars) shows a higher decline in food insecurity (24 percentage points). 9 Data from Nepal Demographic and Health Survey (DHS) indicates similar incidence of underweight in Midwestern development region. 10 Other Indicators: We also explore the possible impact on other indicators associated with social capital (trust, respect, relationships between different ethnic groups, community disputes, etc.) and female empowerment/decision-making. The analysis does not find any significant impact over the 2 year period of the evaluation. Direct effects and Spillover effects: The two main reasons for using villages as units of our analysis are: random phase-in is much more feasible across villages as opposed to individual households, and village-level randomization is a better option in presence of possible spillover effects. In the case of PAF intervention, the treatment villages have both direct beneficiaries and non-beneficiaries. Through social and economic interactions, non-treated households in the treatment villages may potentially be affected by PAF grants to treated households in the same villages. For example, if there is sharing of such grants among relatives or if increased demand for goods/services among treated households affects either earnings or prices faced by non-treated households, spillover (indirect) effects are plausible. Using a method suggested in Banerjee et al. (2007), we find that spillover effects are small and the net program impact is mostly driven by direct effects on the treated households. When we consider individual households as units of analysis, the magnitude of the net impact of PAF, for instance, on per capita consumption is about 38 percent change, close to double that of the estimate using villages as units of analysis10. This suggests that the impact estimates that we present in this paper using village level analysis are in fact lower bound estimates. Moreover, the lack of indirect effects and the dominance of direct effects also imply that the household level targeting in the PAF program is justified. 5. Conclusion The primary objective of this study is to evaluate the impact of Nepal Poverty Alleviation Fund on rural household welfare. PAF is a CDD-type social fund program and currently covers 40 poorest districts in Nepal, with plans to expand across the country. Between 2004 and 2011, PAF has supported some 15,000 Community Organizations and benefited more than 2.5 million people directly. Establishing causal effect of such a complex operation is difficult, particularly in a country setting where designing and implementing a robust impact assessment are not the norm. The Government took the initiative to build an impact evaluation component into the program and collaborated with an external research institution (Tribhuvan University) and a technical team from the World Bank to implement it. 10 This is consistent with the findings from earlier version of this paper that considered individual households as units of analysis, where treated households were compared with non-treated households. 11 The identification of the PAF impact is based on a design that involves a randomized phase- in approach, in which certain localities were randomly assigned for earlier intervention than others. Not surprisingly, the actual PAF intervention does not perfectly comply with the random assignment. However, there is a strong correlation of planned and actual intervention status. Using two rounds of survey data (pre and post-intervention) and difference-in-difference combined with instrumental variable estimation method, we find significant PAF effect on key welfare indicators. The treatment-on-the-treated impact is 19 percent growth on real per capita consumption is, 19 percentage points decline on incidence of food insecurity (defined as food sufficiency for six months or less) and 15 percentage points increase in school enrollment rate among 6-15 year olds. Impacts (positive or negative) are yet to be detected on indicators associated with child malnutrition, social capital and female empowerment. The policy implications of these results should be of interest to the government and to development partners in determining what may be effective instruments to deliver services to marginalized communities in what remains a fragile and difficult political environment. The study contributes to evidence on social funds evaluation in three ways. We use an experimental design to evaluate the impact of a large-scale intervention. Moreover, we estimate the causal impact even when there is no perfect compliance between the actual intervention and the randomized assignment. Imperfect compliance is a reasonable real world setting and allowance for such can be deemed to maintain external validity of the impact estimates. Lastly, in addition to using per capita household consumption, we also report impact on food security, an important indicator of vulnerability in conflict settings. Like most evaluations, our study has its limitations. First, the treatment-on-the-treated instrumental variable estimate does not necessarily represent the average treatment effect for the entire population, but only of the sub-population that was actually treated because of random assignment. This is a well-documented qualification of the IV method. Second, our analysis has focused on the outcome indicators only but there are important process related questions that this impact evaluation does not yet answer. For example, while we do not find any statistically significant impact on social capital and female empowerment, project documents including socio- economic surveys of assisted households at baseline and currently suggest that women have greater control over their earnings and make more diversified investments than men. Seventy- four percent of all PAF members are women. Future evaluation should carefully explore these relationships. 12 Finally, the household-level panel data provide an opportunity for further analysis over time which may help us understand changes in poverty dynamics and for further counterfactual analysis of the outcomes reported in this paper. As such, a continuation of the survey methodology and of the impact evaluation analysis would be an important tool for policy makers and for the implementation of the poverty alleviation fund at the national level. 13 References Abadie, Alberto (2005), “Semiparametric Difference-in-Differences Estimators�, Review of Economic Studies 72:1–19. Abou-Ali, Hala, Hesham El-Azony, Heba El-Laithy, and Shahidur Khandker (2010), “Evaluating the Impact of Egyptian Social Fund for Development Programs�, Journal of Development Effectiveness, 2(4): 521-555. Angrist, oshua David, and rn-Steffen Pischke (2009). Mostly Harmless Econometrics: An Empiricist's Companion. Princeton: Princeton University Press. Baum, C.F., Schaffer, M.E., Stillman, S. (2010). ivreg2: Stata module for extended instrumental variables/2SLS, GMM and AC/HAC, LIML and k-class regression. Beath, Andrew, Fotini Christia, Ruben Enikolopov, and Shahim Kabuli (2010), Randomized impact evaluation of phase-II of Afghanistan's National Solidarity Program. Working Paper Series, The World Bank. Banerjee, Abhijit, Shawn Cole, Esther Duflo, and Leigh Linden (2007), “Remedying Education: Evidence from Two Randomized Experiments in India," Quarterly Journal of Economics. 122(3): 1235-1264. Duflo, Esther, Rachel Glennerster, and Michael Kremer (2008), "Using Randomization in Development Economics Research: A Toolkit." Chapter 61 in Handbook of Development Economics. Vol. 4. Edited by T. Paul Schultz and John Strauss. North Holland. alan, yotsna and Martin Ravallion (1998), “Are there Dynamic Gains from a Poor-Area Development Program?� Journal of Public Economics, 67(1): 65-86. Olken, Benjamin A., Junko Onishi, and Susan Wong (2010), Indonesia’s PNPM Generasi Program: Interim Impact Evaluation Report Olken, Benjamin A., unko Onishi, and Susan Wong (2011), Indonesia’s PNPM Generasi Program: Final Impact Evaluation Report Paul J. Gertler, Sebastian Martinez, Patrick Premand, Laura B. Rawlings, Christel M. J. Vermeersch (2011), Impact Evaluation in Practice, The World Bank. http://www.worldbank.org/ieinpractice Ravallion, Martin (2008), Evaluating Anti-Poverty Programs. Handbook of Agricultural Economics. Vol. 4. Edited by Robert E. Evenson and T. Paul Schultz. North-Holland. Rosenbaum, Paul, and Donald Rubin (1983), “The Central Role of the Propensity Score in Observational Studies for Causal Effects�. Biometrika 70: 41-55. Tribhuvan University, Kathmandu (2007), Poverty Alleviation Fund Monitoring and Evaluation Baseline Survey Report. Tribhuvan University, Kathmandu (2010), Report on the Follow-Up Survey of Poverty Alleviation Fund Program. The World Bank (2006), Community-Driven Development in the Context of Conflict-Affected Countries: Challenges and Opportunities. 14 Annex Table 1: Baseline Means of Key Variables and Differences Between Assigned Treatment and Control p-Value for Assigned Treatment Assigned Control difference Per Capita consumption, 2007 NRS 14863 14866 0.996 Per Capita consumption, log 9.512 9.512 0.988 Food sufficiency, # months 8.055 7.88 0.480 Food poor, food sufficient for less than 6 months 0.364 0.385 0.493 Enrollment rate, 6-15 year olds 0.726 0.735 0.731 Underweight rate, 0-4 year olds 0.578 0.582 0.872 Household Size 5.877 5.843 0.792 Female Headed HH 0.108 0.117 0.577 Dalit or other disadvantaged HH 0.381 0.400 0.708 Land value, log 4.665 4.545 0.232 Receives remittance 0.141 0.149 0.695 Poorest third of VDCs by PAF ranking 0.500 0.520 0.775 mean elevation ('000m) above sea level, vdc 1.371 1.489 0.485 Distance, km, to DHQ 19.463 21.728 0.086 %Operated Agri area<0.1Ha,ward 0.179 0.193 0.561 %15+ pop with no education, ward 0.740 0.755 0.529 Notes: 1. Sample size: 3,000 HHs from 200 PSUs; 2. P-Value is generated from regressions of each variable on assigned treatment group (0,1) and standard error corrected for clustering at the PSU level 15 Annex Table 2: First Stage Regression in the IV regression Dependent Variable: Actual PAF Intervention (1,0) ----------------------------------------------------------- Co-efficient (standard error) ----------------------------------------------------------- Intent-to-treat(IV) 0.382*** (0.059) Poorest third of VDCs by PAF rank (1,0) 0.141** (0.060) mean elevation ('000m) above sea level, vdc 0.274*** (0.036) mean slope (as %), vdc -0.006* (0.003) Distance, km, to DHQ 0.003 (0.003) %Operated Agri area<0.1Ha,ward 0.400** (0.186) %15+ pop with no education, ward 0.136 (0.195) %dalit & other disadvantaged H, ward 0.044 (0.125) %HH Head of Hindu Religion, ward 0.071 (0.143) %HH Head Nepali mother tongue, ward -0.044 (0.139) Average age of females at 1st marriage,vdc -0.100*** (0.036) %own radio, ward -0.249** (0.116) Constant 1.704** (0.694) ----------------------------------------------------------- Notes: 1. * p<0.10, ** p<0.05, *** p<0.01; 2.Standard errors are corrected for clustering at PSU level; 3. There are 200 PSUs in the sample 16 Annex Table 3: Impact Estimation for Per Capita Consumption (log) ----------------------------------------------------------------------------------------------------------- (1) (2) (3) (4) DIFF-IN-DIFF DIFF-IN-DIFF Second Round Second Round Intent-to-treat IV Intent-to-treat IV ----------------------------------------------------------------------------------------------------------- Intent-to-treat 0.071** 0.075** (0.033) (0.030) PAF IG Village 0.187** 0.196** (0.091) (0.082) 1=Poorest third of VDCs by PAF rank -0.075** -0.102*** -0.028 -0.056* (0.033) (0.036) (0.028) (0.030) mean elevation ('000m) above sea level, vdc 0.028 -0.023 0.160*** 0.107*** (0.028) (0.037) (0.025) (0.033) mean slope (as %), vdc 0.006*** 0.007*** -0.003* -0.002 (0.002) (0.002) (0.002) (0.002) Distance, km, to DHQ 0.005*** 0.005*** 0.003** 0.003* (0.002) (0.002) (0.002) (0.002) %Operated Agri area<0.1Ha,ward -0.193* -0.268** 0.145 0.067 (0.099) (0.111) (0.110) (0.118) %15+ pop with no education, ward -0.158 -0.183* -0.288*** -0.315*** (0.106) (0.110) (0.106) (0.114) %dalit&other disad, ward 0.108 0.100 -0.145** -0.154*** (0.067) (0.068) (0.059) (0.055) %HH Head of Hindu Religion, ward 0.257*** 0.244*** 0.125* 0.112 (0.079) (0.082) (0.071) (0.073) %HH Head Nepali mother tongue, ward -0.326*** -0.317*** -0.261*** -0.252*** (0.081) (0.087) (0.065) (0.068) Average age of females at 1st marriage,vdc 0.028 0.047** 0.023 0.043** (0.020) (0.023) (0.017) (0.021) %own radio, ward -0.130* -0.083 -0.089 -0.040 (0.070) (0.073) (0.070) (0.071) Constant -0.675* -0.994** 9.212*** 8.879*** (0.363) (0.422) (0.319) (0.391) ----------------------------------------------------------------------------------------------------------- Observations 2774 2774 2999 2999 ----------------------------------------------------------------------------------------------------------- Notes: 1. * p<0.10, ** p<0.05, *** p<0.01; 2. Standard errors shown in parentheses are corrected for clustering at PSU level; 3. There are 200 PSUs in the sample 17 Annex Table 4: Impact Estimation for Food Insecurity, Food sufficiency less than 6 months ----------------------------------------------------------------------------------------------------------- (1) (2) (3) (4) DIFF-IN-DIFF DIFF-IN-DIFF Second Round Second Round Intent-to-treat IV Intent-to-treat IV -- --------------------------------------------------------------------------------------------------------- Intent-to-treat -0.071* -0.084*** (0.039) (0.027) PAF IG Village -0.187* -0.219*** (0.101) (0.078) 1=Poorest third of VDCs by PAF rank -0.087** -0.061 0.008 0.039 (0.040) (0.041) (0.029) (0.032) mean elevation ('000m) above sea level, vdc 0.006 0.057 -0.011 0.049 (0.026) (0.038) (0.021) (0.033) mean slope (as %), vdc -0.001 -0.002 -0.002 -0.004** (0.002) (0.002) (0.002) (0.002) Distance, km, to DHQ 0.001 0.001 -0.002 -0.002 (0.002) (0.002) (0.002) (0.001) %Operated Agri area<0.1Ha,ward 0.003 0.078 0.048 0.136 (0.122) (0.121) (0.098) (0.104) %15+ pop with no education, ward 0.111 0.136 0.281*** 0.312*** (0.116) (0.118) (0.092) (0.095) %dalit&other disad, ward -0.025 -0.016 0.152** 0.162** (0.088) (0.083) (0.068) (0.066) %HH Head of Hindu Religion, ward -0.126 -0.113 -0.013 0.003 (0.097) (0.094) (0.071) (0.072) %HH Head Nepali mother tongue, ward 0.119 0.111 0.107* 0.097 (0.092) (0.091) (0.064) (0.063) Average age of females at 1st marriage,vdc -0.039 -0.058** 0.017 -0.005 (0.025) (0.026) (0.020) (0.021) %own radio, ward 0.037 -0.009 0.053 -0.001 (0.083) (0.079) (0.060) (0.067) Constant 0.700 1.019** -0.093 0.278 (0.478) (0.492) (0.387) (0.419) ----------------------------------------------------------------------------------------------------------- Observations 2774 2774 2999 2999 ----------------------------------------------------------------------------------------------------------- Notes: 1. * p<0.10, ** p<0.05, *** p<0.01; 2.Standard errors shown in parentheses are corrected for clustering at PSU level; 3. There are 200 PSUs in the sample 18 Annex Table 5: Impact Estimation for School Enrollment, 6-15 year olds --------------------------------------------------------------------------------------------------------------------------- (1) (2) (3) (4) (5) Intent-to-treat IV IV IV (Girls) IV (Girls) --------------------------------------------------------------------------------------------------------------------------- Intent-to-treat*Round2 0.048** (0.019) PAF Village*Round2 0.139** 0.145** 0.213** 0.208** (0.060) (0.060) (0.085) (0.086) Intent-to-treat -0.030 (0.025) PAF IG Village -0.087 -0.111** -0.038 -0.081 (0.070) (0.055) (0.092) (0.075) Round2 0.050*** -0.001 -0.005 0.001 -0.000 (0.014) (0.034) (0.034) (0.051) (0.052) Age 0.002 -0.008*** (0.002) (0.002) Female -0.119*** (0.009) 1=Poorest third of VDCs by PAF rank 0.009 0.004 (0.016) (0.023) mean elevation ('000m) above sea level, vdc -0.008 -0.022 (0.016) (0.023) mean slope (as %), vdc 0.002*** 0.002* (0.001) (0.001) Distance, km, to DHQ 0.002* 0.001 (0.001) (0.001) %Operated Agri area<0.1Ha,ward 0.103* 0.090 (0.054) (0.081) %15+ pop with no education, ward -0.199*** -0.310*** (0.052) (0.072) %dalit&other disad, ward -0.026 -0.022 (0.047) (0.059) %HH Head of Hindu Religion, ward 0.125** 0.088 (0.048) (0.060) %HH Head Nepali mother tongue, ward 0.040 0.067 (0.044) (0.060) Average age of females at 1st marriage,vdc 0.006 0.019 (0.009) (0.014) %own radio, ward 0.096*** 0.120*** (0.029) (0.043) Constant 0.752*** 0.785*** 0.584*** 0.675*** 0.426 (0.016) (0.039) (0.178) (0.053) (0.259) --------------------------------------------------------------------------------------------------------------------------- Observations 10990 10990 10990 5360 5360 --------------------------------------------------------------------------------------------------------------------------- Notes: 1. * p<0.10, ** p<0.05, *** p<0.01; 2.Standard errors shown in parentheses are corrected for clustering at PSU level; 3. There are 200 PSUs in the sample 19 Annex Table 6: Impact Estimation for Child Underweight, 0-4 year olds --------------------------------------------------------------------------------------------------------------------------- (1) (2) (3) (4) (5) Intent-to-treat IV IV IV (Girls) IV (Girls) --------------------------------------------------------------------------------------------------------------------------- Intent-to-treat*Round2 -0.018 (0.034) PAF Village*Round2 -0.056 -0.053 -0.050 -0.041 (0.105) (0.104) (0.125) (0.124) Intent-to-treat 0.004 (0.026) PAF IG Village 0.012 0.038 0.036 0.041 (0.077) (0.072) (0.090) (0.087) Round2 -0.029 -0.005 -0.006 -0.026 -0.027 (0.023) (0.061) (0.061) (0.076) (0.076) Age(months) -0.000 0.000 (0.000) (0.001) Female -0.026* (0.014) 1=Poorest third of VDCs by PAF rank 0.032* 0.039* (0.018) (0.024) mean elevation ('000m) above sea level, vdc 0.008 -0.001 (0.019) (0.021) mean slope (as %), vdc 0.002 0.001 (0.001) (0.001) Distance, km, to DHQ -0.001 -0.002 (0.001) (0.001) %Operated Agri area<0.1Ha,ward 0.038 0.006 (0.065) (0.079) %15+ pop with no education, ward 0.162** 0.187** (0.064) (0.092) %dalit&other disad, ward 0.011 0.013 (0.042) (0.053) %HH Head of Hindu Religion, ward 0.045 0.041 (0.050) (0.066) %HH Head Nepali mother tongue, ward 0.007 0.047 (0.044) (0.052) Average age of females at 1st marriage,vdc 0.008 0.001 (0.013) (0.016) %own radio, ward -0.043 -0.042 (0.039) (0.049) Constant 0.587*** 0.583*** 0.236 0.565*** 0.325 (0.018) (0.044) (0.238) (0.056) (0.289) --------------------------------------------------------------------------------------------------------------------------- Observations 4549 4549 4549 2251 2251 --------------------------------------------------------------------------------------------------------------------------- Notes: 1. * p<0.10, ** p<0.05, *** p<0.01; 2.Standard errors shown in parentheses are corrected for clustering at PSU level; 3. There are 200 PSUs in the sample 20