81338 Impact Evaluation of NSP: Using Evaluation Findings to Strengthen the Program1 July 21, 2013 1. Introduction The NSP is the largest development program in Afghanistan, now in phase III, and with disbursement to rural communities of more than US$ 1.1 billion. Phase I of NSP was implemented from 2003 to 2007, phase II from 2007 to 2011, and phase III began in 2010 and will close in 2015. The program is structured around two major village-level interventions: (i) creating a gender-balanced Community Development Council (CDC) through a secret ballot, universal suffrage election; and (ii) disbursement of block grants to fund village-level sub- projects that are selected, designed, and managed by the CDC in consultation with village community – the community driven development (CDD) approach. Since inception in 2003, NSP has mobilized more than 34,000 communities and formed CDCs throughout all 34 provinces. The goal is to eventually cover all communities in Afghanistan. The program is implemented by the Ministry of Rural Rehabilitation and Development (MRRD) through NGO facilitating partners (FPs). The role of FP’s include mobilizing village communities, organizing CDC elections, building capacity of CDCs and facilitating communities to select, design, and manage village level sub-projects funded through a block-grant. 2. The Impact Evaluation of NSP In 2006, a decision was made to subject NSP to an independent and rigorous impact evaluation. The decision reflected a commitment of MRRD, donors, and the World Bank to obtain evidence on impact of NSP. MRRD leadership deserves a lot of credit for facilitating the evaluation. Other ministries and development partners are strongly encouraged to follow this example and facilitate evidence-based service delivery to communities that are in dire need of public services. Impact evaluations are an important tool for development. Evaluations provide evidence on what works and what doesn’t work, with a goal to identify opportunities for strengthening programs, whether in terms of policy or operational processes. In particular, the findings of impact evaluation for NSP provide important insights to serve as inputs to ongoing dialogue on future of the program and to inform issues regarding operations policy and processes. Section 4 of this note will discuss lessons from the findings that could be applied to strengthen NSP. The evaluation was designed to provide evidence on the impact of NSP – as opposed to comparing impacts between NSP and alternative service delivery methods. This is important for interpreting the findings. NSP follows a CDD approach. A possible alternative to CDD is top- down method where the decision-making power rests with government with limited involvement of communities in decision-making. To the extent that this impact evaluation is not a comparative evaluation between CDD and a top-down method, and since there are no impact evaluations of top-down methods for service delivery in Afghanistan, the evaluation findings 1 This note was prepared by Elliot Mghenyi, Task Team Leader for Impact Evaluation of NSP. The note borrows vastly from the Impact Evaluation Report and a Policy Note on Transition and Sustainability Issues for NSP to discuss how evaluation findings could be used to strengthen the program 1 cannot be used to say whether NSP is more or less effective than a top-down method where the decision-making power rests with government. It also means the findings do not provide insights for deciding resource allocations between NSP and any alternative method for delivering similar services. Therefore, the findings cannot be used to discuss or decide budget allocations for competing development programs. Below is a brief description of the impact evaluation of NSP. The impact evaluation of NSP is a multi-year randomized control trial designed to estimate the effects of the program across a broad range of indicators, including local governance, democratic processes, access to infrastructure and services, economic outcomes, and state- building. The evaluation relies on data from a sample of 500 villages, where half of them (250) received phase II of NSP and the other 250 are control villages that did not receive NSP during the evaluation period. The evaluation villages were drawn from the following 10 districts; Balkh, Khost Wa Firing, Sang Takht, Daulina, Adraskan, Chisht-e Sharif, Gulran, Fersi, Hisarak, and Sherzad. These 10 districts were purposefully selected to reduce the risk of evaluation villages becoming unreachable for due to insecurity (which would impede data collection) and also to ensure the districts have enough number of villages to allow for randomized selection of villages. The evaluation villages were randomly selected through a process called “matched-pair cluster randomization” where clusters comprised of villages in close proximity. In summary, the process involved matching villages in each district with the most similar village within the same district. Next, the best 25 matches (pairs) in each district were selected forming 250 pairs in all 10 districts. Finally, for each pair one village was randomly assigned to receive NSP (treatment) and the other would not receive NSP (control) – while ensuring all villages in a cluster were assigned the same treatment status. This sampling process was deliberately chosen to design a rigorous evaluation that would provide credible estimates of the impact of NSP. The impact evaluation is the first large-sample and rigorous quantitative study for NSP. Data used to estimate the impacts of NSP was collected in 3 rounds – baseline in 2007, midline in 2009, and endline in 2011. The baseline survey was conducted before villages were mobilized for NSP II. The midline survey was conducted about 2 years into implementation of NSP II. At the time of midline survey, CDC elections had already been conducted in all treatment villages, all sub-projects had started implementing with majority of them still under implementation. Only about 18 percent of sub-projects had been completed. Therefore, the midline data could only be used to estimate preliminary effects of NSP. The endline survey was conducted after 4 years of project implementation, when nearly all (about 99%) of the sub- projects were completed and with most sub-projects completed within 1.5 years before the survey. Together the 3 rounds of surveys were deliberately timed to provide data for assessing the preliminary effects of the program during sub-project implementation and to estimate the impacts of the program after project completion. In total more than 25,000 households were interviewed during the 3 rounds. Household interviews were conducted using structured survey questionnaires administered to both male and female villagers. In addition to household surveys, more than 2600 focus group questionnaires were administered to male and female village leaders in a group setting. 2 3. Summary of Evaluation Findings The evaluation findings are attributable to two major village level interventions in NSP II: (i) a gender-balanced Community Development Council (CDC) created through a secret ballot, universal suffrage election; and (ii) a single block grant, valued at $200 per household and up to maximum of $60,000 per community. The grant is used to fund village-level sub-projects that are selected, designed, and managed by the CDC in consultation with its respective village community and with facilitation by an FP. The contracted FPs were a mixture of local and international NGOs. Although the FPs would use a common operational manual to facilitate communities, each district in the evaluation sample was covered by a different FP, and this could have introduced some variation in the process of mobilizing CDC elections as well as in facilitating sub-project selection and implementation. In addition, the CDD process of managing block grants allowed communities to prioritize and self-select the sub-projects they wanted, which introduced variation in sub-projects implemented across villages. The distribution of selected sub-projects across evaluation villages is as follows; transport (26% of sub-projects and 30% of block grants), water supply and sanitation (24% of sub-projects and 22% of block grants), irrigation (20% of sub-projects and 21% of block grants), power (11% of sub-projects and 16% of block grants), literacy and vocational training (8% of sub-projects and 1% of block grants), and others (10% of sub-projects and 9% of block grants ). The following is a summary of evaluation findings. (a) Impact on access to utilities, services and infrastructure NSP improves access to clean drinking water and use of electricity. The NSP drinking water sub-projects increase usage of protected water sources by 36 percent. The NSP also induces a 5 percent reduction in the time that households spend collecting water. However, there is no lasting impact on perceived water quality or on the incidence of water shortages. The NSP electricity sub-projects substantially increases usage of electricity by 26 percent. NSP is also associated with increased access to education, health care, and counseling services for women, although NSP does not usually fund such services. The impacts appear to arise indirectly from other changes induced by NSP. While there is no impact on boys’ school attendance, NSP increases girls’ school attendance and their quality of learning. The only health outcomes associated with NSP are increased child doctor and prenatal visits and increased probability that a medical professional attends to an illness or injury. NSP irrigation sub-projects appear to be less successful. In particular, irrigation sub-projects appear to have no noticeable impact on the ability of land-holding villagers to access sufficient irrigation. Similarly, NSP transportation sub-projects appear to be less successful. Although there is weak evidence that transportation sub-projects increase village accessibility 2 years into project implementation, the impact does not persist after sub-project completion, which suggests the findings on transportation sub-projects are inconclusive. There are several hypotheses on irrigation and transportation sub-projects being less successful compared to water supply and electricity sub-projects. Water supply and hydropower sub-projects are likely to either benefit most households in a community or provide fairly equal services to those who benefit. In contrast, irrigation sub-projects would only benefit households with cultivable land, and even among them, more benefits are likely to accrue to landowners at the “head” of the canal compared to those in the “middle” and “tail” ends. Transportation 3 (tertiary roads) sub-projects suffer due to a different problem in that although they are selected through CDD processes, the sub-projects were not big enough to connect one village to another, or to connect a village to a district or market center. The sub-projects were certainly at village level. However, these possible explanations remain just hypotheses since there is no data to test for causal relationships. It is a subject for further analysis. The evaluation findings also show weak evidence that NSP satisfies ex-ante development priorities of male villagers. The weak evidence suggests that, once complete, the NSP sub- projects fulfill the development needs of male villagers, as measured by relative changes in the types of projects identified as most needed by the village. In particular, the NSP seems to reduce the need for drinking water sub-projects, which were ex-ante (at baseline) prioritized by a higher proportion of male villagers than any other sub-projects. (b) Impact on economic welfare The NSP affects the economic perceptions and optimism of villagers, particularly women. Female villagers exhibit improved perceptions of the current economic situation and are more optimistic, both during sub-project implementation and after completion of sub-projects. The impact on female economic perceptions demonstrates the broader improvements brought to women’s lives by female participation in CDCs and by NSP-funded sub-project activities. While the economic perceptions and optimism of male villagers improve during sub-project implementation, there is only weak evidence of an impact on optimism of male villagers – and no evidence of impact on perceptions – after sub-projects are completed. The impact of NSP on levels of income, income regularity, consumption levels, consumption allocations, assets, or food insecurity is inconclusive. However, there are some observed economic effects during sub-project implementation – some weak evidence that NSP induces small increases in the diversity of household income sources and in caloric intake. These transitory effects appear to reflect a temporary economic stimulus driven primarily by the infusion of block grant resources, most likely through hiring local labor and use of local materials. However, the effects are not conclusively observed after sub-project completion. There is no evidence that NSP impacts agricultural production or marketing outcomes. NSP has no discernible impacts on agricultural yields, productivity, or the proportion of harvests sold, although during sub-project implementation there is a bump in agricultural sales revenue. The NSP does not affect whether households sell animals, or animal products, or the revenue derived from such sales. While NSP increases handicraft sales and sales revenue during sub-project implementation, these impacts disappear following sub-project completion. However, there is evidence that NSP reduces out-migration from villages both during and after sub-project implementation. The results also suggest weak evidence of impact on the amount borrowed by households after sub-projects are completed. (c) Impact on local governance NSP impacts the structure of local governance by increasing by two-and-a-half times the proportion of formal local assemblies that contain at least one woman member. The creation of CDCs causes customary leaders to affiliate with representative assemblies during project implementation, although this effect is not observed after project completion. However, there is no evidence that NSP changes the composition of local leadership or introduces new leaders into the core group of village decision-makers. 4 The creation of CDCs by NSP induces an increase in the provision of local governance services, the activity level of customary authorities, and the role served by representative assemblies in providing local governance services. These impacts are observed during project implementation and not beyond. However, the NSP does increase the number of meetings held annually by representative assemblies. There is also strong evidence that NSP induces a durable increase in the provision of local governance services specific to women. The NSP increases participation of villagers in local governance during project implementation – as measured by meeting attendance and a desire to change decisions of leaders. NSP also increases demand for the involvement of representative assemblies in local governance. However, while the desire to change decisions of leaders persist after project completion, NSP has no durable impact on the probability of villagers attending assembly meetings, or on the extent to which they believe assemblies should be involved in local governance. The NSP induces an increase in the extent to which village leaders are perceived as being responsive to the needs of women, but this effect is observed only during project implementation and not after completion. (d) Impact on political attitudes and state-building There is strong evidence that NSP increased voting in the 2010 parliamentary elections, with the proportion of male and female villagers who claimed to have cast a ballot increasing by 4 percent and 10 percent respectively. NSP also appears to raise appreciation of democratic elections, at least as manifested by a 24 percent increase in the proportion of male villagers who prefer that the village headman is elected. However, NSP has no effect on female views about democratic elections or participatory decision-making procedures. There is also no effect on the (already-high) proportion of male villagers who believe the President or provincial governor should be elected; or on those who believe it is appropriate to publicly discuss governance; or on those who support the participatory resolution of major village issues. There is only weak evidence that NSP increases the legitimacy of the central government. In particular, NSP has no impact on whether villagers believe that the government should exercise jurisdiction over local crimes, set the school curriculum, issue ID cards, or collect income tax. Similarly, the NSP has no impact on whether villagers prefer a centralized state or a weak federation, or identify primarily as Afghan or as a member of a specific ethnic group. During sub-project implementation, treatment villages experience an increase in linkages with government officials and representatives of the Afghan National Security Forces, although these effects dissipate once sub-projects are completed. There is strong evidence that NSP improves perceptions of government, but the effects dissipate after project completion. Similarly, during project implementation, NSP induces a strongly significant increase in the reported benevolence of a wide-range of government entities, but the impact mostly fades following project completion, with only weak positive impacts persisting for the President and central government officials. This pattern is also true for NGO officials. However, the positive impact on perceptions of ISAF soldiers appears durable. NSP does not appear to affect the likelihood of villages suffering violent attacks, at least as reported by the villagers themselves, both during and after project implementation. There is also no evidence that NSP affects the ability of insurgent groups to expropriate harvests. However, 5 NSP does improve perceptions of the local security situation among both male and female villagers during project implementation, but only those for male villagers persist after project completion. The NSP has a positive impact on perceptions of government. However, the impact is mostly confined to the period of project implementation, and villagers revert to original attitudes on government once project funds are expended. This suggests that NSP is viewed as a program of government, and further that its legitimacy is tied to regular provision of public goods and engaging communities in participatory processes that bring people together and use local talent to solve community problems. (e) Impact on social norms and cohesion In line with the observation in social development literature that decisions on use of public resources could sometimes aggravate intra-communal divisions, there is weak evidence that during project implementation, NSP increases the incidence of disputes and feuds, while reducing resolution rates. Once projects are completed, this general effect disappears, and the results indicate weak evidence that NSP slightly reduces intra-village disputes. There is also some evidence that NSP increases interpersonal trust among male (not female) villagers during project implementation, although the impact is not visible beyond project completion. The small magnitude of the observed changes leads to the conclusion that there is no overall evidence of an impact on social cohesion. During project implementation, NSP improves basic literacy and computation skills of male and female villagers, but these impacts do not last. There is also some weak evidence that NSP makes villagers happier. In particular, there is weak evidence of a reduction in the proportion of female villagers who report they are unhappy, a result which could be caused by increased availability of counseling services for women, increased female participation in local governance, and/or increased access to basic utilities. The NSP increases men’s acceptance of female participation in political activity and local governance. In particular, the program increases male acceptance of female participation in elections, national candidacy by women, and women holding civil service or NGO positions by 3, 4, and 6 percent respectively. The NSP also causes a 22 percent increase in acceptance of female membership in village councils and a 15 percent increase in acceptance of female participation in selecting village headmen. The impact of NSP on women’s views on female participation in political activity and local governance is not significant. The NSP also appears to have limited impacts on cultural constraints to the education of women. The NSP has durable positive impacts on the participation of women in local governance. In particular, the results indicate 21 percent increase in participation of women in dispute mediation and 14 percent increase in the involvement of women in aid allocation. Although NSP does not appear to impact intra-village mobility of women, female socialization, or female participation in economic activity or household decision-making, the program does produce a durable increase in the ability of women to travel beyond their village. Women in NSP villages are 13 percent more likely to have visited the nearest village in the past year and 11 percent more likely to have visited the district center in the past month. 6 4. Lessons from the Evaluation Findings to Strengthen NSP in future There is ongoing dialogue within government on many issues regarding the future direction for NSP. The discussions are motivated by the fact that NSP has been in operation for 10 years and the experiences gathered during the many years of implementation should be used to transform the program for the better. The government’s desire for improved and strengthened NSP is to be encouraged. In this context, the Bank has prepared a policy note to identify key transition and sustainability issues for NSP. The policy note is aimed at guiding “informed policy debate and facilitate decision making process by the Government of Islamic Republic of Afghanistan (GoIRA)”. The following key transition and sustainability issues have been identified; (i) institutionalization of CDCs as village level governance structures, (ii) size and frequency of block grants, (iii) reduction of NSP facilitation costs, (iv) sustainability of NSP sub- projects, (v) reduction of program management costs, and (vi) linkages between CDC and sub- national governance structures. The issues represent key policy and design aspects of the program that need strengthening to improve delivery of services to the people of Afghanistan. The impact evaluation findings provide important insights to inform these issues and strengthen the program. In particular, the evaluation findings provide insights to facilitate decision-making on institutionalization of CDCs as village level governance structures, size and frequency of block grants, sustainability of NSP sub-projects, and linkages between CDCs and structures for sub-national governance. (i) Institutionalization of CDCs as village level governance structures The NSP formed CDCs to perform governance and development functions. However, legitimacy for village governance is often contested, even within government. The main issue is that the constitution provides for Village Council’s (VC) to be constituted through elections planned and supervised by the Independent Electoral Commission (IEC) – not CDCs created through NSP. There is concern that CDCs cannot be institutionalized to VCs because their elections were not supervised by the IEC. Furthermore, most line ministries neither use CDCs nor recognize the MRRD’s by-law under which CDCs are established. Overall, the evaluation findings suggest that existence of CDCs leads to increased provision of governance services, although this impact is observed more during sub-project implementation compared to after sub-project completion. For example, NSP increases participation of villagers in local governance during sub-project implementation, but not after sub-project completion. Similarly, during sub-project implementation, NSP increases the extent to which village leaders are perceived as being responsive to the needs of women (but not after sub-project completion). Furthermore, when sub-projects are completed, the perceptions of male villagers regarding impact of NSP on local governance quality tend to decline. One lesson coming out of these findings is that effectiveness of CDCs in delivering governance services is tied to CDCs being actively involved in implementing sub-projects. The evaluation also finds that perceptions on government are more positive when CDCs are actively implementing sub-projects. This suggests that NSP is rightly liked to the government, and further that CDC’s legitimacy for governance services is derived through engaging communities in participatory processes that bring people together and use local talent to solve community programs – in addition to delivering sub-projects. Similar expectations would likely apply to VCs. It would mean VCs are unlikely to be viewed more legitimate – relative to CDCs – unless 7 they can effectively involve people in participatory processes to solve community problems and deliver hard infrastructure through sub-projects. Therefore, whether the eventual GoIRA decision is to formalize CDCs as VCs – or to start afresh and independently elect VCs – it is extremely important to ensure the institution acquires and maintains adequate capacity to deliver services using participatory processes. There have been suggestions to institutionalize CDCs into VCs that would continue to perform both governance and development functions, but with governance functions overseen by the Independent Directorate of Local Governance (IDLG) while development functions continue to be the responsibility of MRRD. The evaluation findings suggest that legitimacy for governance is through active performance of development functions. Therefore, the optimal approach is not to split oversight of these functions because that could jeopardize the role of VCs in local governance. (ii) Size and frequency of block grants. The average size of a block grant from NSP is US$33,500 per community, which translates to approximately US$17-33 per capita. This seems high compared to other CDD programs in South East Asia and East Africa where block grants provide approximately US$1-3 per capita every year. However, unlike some of these programs NSP does not provide block grants every year and communities have received only a single block (although NSP III is now providing second round block grants to communities that had completed their sub-projects and fully utilized a block grant received between 2003 and 2010). Going forward there are two options being considered regarding size and frequency of block grants. The first is to provide a reduced amount of block grants to all CDCs, but more frequently – annually or bi-annually. The second is to provide the current level of block grants, but through a district level competitive process that could leave out some CDCs. The evaluation findings are consistent with the option to provide more frequent block grants to all CDCs – because CDCs effectiveness as governance institutions is tied to actively implementing sub-projects in a participatory approach. Furthermore, there is arguably very little merit for district-level competitive allocation of funds, because it could deny funds to less competitive CDCs that quite possibly represent most needy communities. If CDCs are institutionalized to VCs, competitive funding would become quite problematic to implement because it would stifle uncompetitive but constitutional local governance entities. The correct way to help weak institutions is to strengthen them through capacity building efforts, for example a customized facilitation process for weak CDCs. Smaller but regular block grants build continuity in participatory processes that use local talent to solve community problems, continuity in provision of governance services, and continuity in injecting funds for sub-project activities – including operations and maintenance (O&M). (iii) Sustainability of NSP sub-projects. The evaluation findings indicate that of the various sub-projects funded through block grants, water supply and hydropower sub-projects effectively improve access to these utilities. On the other hand, irrigation and transportation sub-projects appear to be less effective. It is not clear why. As discussed in the summary of findings, a plausible interpretation is that water supply and hydropower sub-projects are likely to either benefit most households in a community 8 or provide fairly equal services to those who benefit. In contrast, irrigation sub-projects would only benefit households with cultivable land, and even among this group, more benefits are likely to accrue to landowners at the “head” of the canal relative to those in the “middle” and “tail” ends. Although selected through CDD processes, transportation (tertiary roads) sub- projects are implemented at village level and perhaps not big enough to connect one village to another or to connect a village to a district or market center. A relevant policy question is whether the more effective water and electricity sub- projects are more sustainable compared to the less effective irrigation and roads sub-projects. A good measure of sustainability is quality of infrastructure measured several years after sub- project completion. A recent study2 used this indicator to explore sustainability of NSP sub- projects. The study found that the quality of completed water supply sub-projects – which are effective according to evaluation findings, is not better than completed irrigation sub-projects – which are ineffective according to evaluation findings. In particular, about 36% of water supply sub-projects are “functional and in good condition” compared to 72% of karez irrigation, 71% of canal irrigation, and 38% of irrigation protection wall. The proportion of “not functional” water supply sub-projects is a whopping 45%, much higher than the irrigation sub-projects (0% for Karez, 14% for canal, 0% for protection wall). Hydropower and tertiary road sub-projects are very similar on the sustainability scale. In particular, none of these sub-projects are classified as “functional and in good condition” – which is worrying. However, in both sub-project types more than 90% are “functional and in poor condition” and less than 10% “not functional”. Clearly, there appears to be no link between effectiveness of sub-projects and sustainability, at least as measured by quality of infrastructure several years after sub-project completion. Even subprojects that are effective in providing services are at risk of deteriorating. This finding reinforces the need to ensure O&M budgets for maintaining sub-projects are available and institutional arrangements such as O&M committees and plans are always in place and active. (iii) Economic impacts and linkages between CDCs and sub-national governance structures. The distribution of sub-projects in the evaluation sample is as follows; Irrigation (25%), water supply and sanitation (21%), transportation (19%), literacy and vocational training (17%), hydropower (10%), community centers (4%), tailoring machines or flour mills (3%). Irrigation, transportation, and hydropower projects are economic infrastructure – and they collectively account for about 55% of sub-projects and 69% of sub-project expenditures. However, the evaluation findings suggest no conclusive impact on economic outcomes beyond improved perception of wellbeing. In particular, there are no conclusive impacts on income levels, income regularity, consumption levels and allocations, assets, or food insecurity. This suggests economic sub-projects selected by the communities (through CDD) are hardly effective in improving economic outcomes, and therefore should not be viewed as substitutes for similar or other economic public infrastructure delivered through non-CDD approaches. 2 The study is titled “The Sustainability of CDCs” by Altai Consulting 9 Often times there have been suggestions for other line ministries to work through CDCs. The apparent lack of economic impacts from sub-projects selected and implemented by CDCs is reason to consider the proposition cautiously. In this context, the following issues need better understanding. First is whether NSP implemented sub-projects have enough scale for significant economic impacts to obtain. For example, while NSP gives on average US$33,000 per village for all types of infrastructure, two Bank-funded irrigation projects in Afghanistan spend a lot more. The Emergency Irrigation Rehabilitation Project3 (EIRP) spends about US$ 300,000 per irrigation scheme and On-Farm Water Management project4 (OFWM) spends about US$130, 000 per scheme, which are respectively about 9 and 4 times the size of the entire NSP block grant. Scale matters for development outcomes. Therefore, it is plausible to expect that relatively smaller NSP irrigation sub-projects would have relatively small impacts, at least compared to larger irrigation projects. Secondly, is whether NSP CDCs have enough capacity to implement scaled up sub- projects of the size implemented by other ministries/programs. The irrigation sub-projects done by EIRP and OFWM projects are on average about 9 and 4 times the size of an average block grant. It is not obvious that CDCs would have the capacity to effectively implement projects of that scale. Thirdly, is whether the CDD process prioritizes other considerations ahead of economic soundness of sub-projects - for example the critical needs of the community, however defined. Fourthly, is whether the village is an appropriate level for considering economic infrastructure investments – as opposed to, for example district or cluster levels. Sub-national governance structures – e.g. the proposed District Development Committees – could have a major role in selecting connective inter-village infrastructure. Since roads linking several villages or a village to a market center are more likely to give economic impacts compared to within village tertiary roads (which is what CDCs are doing) – it would seem that sub-national governance structures should have a role in planning and executing connective inter-village infrastructure. The NSP is piloting cluster CDCs. With proper incentives the clusters could potentially be involved in connective infrastructure, for example providing cluster CDCs with “earmarked cluster grants” amounting to some proportion of any connective infrastructure agreed in a cluster development plan and targeted facilitation to implement the activity. 5. Key Issues for further analysis and literature on CDD evaluations The impact evaluation of NSP was designed to estimate the average impacts of the program – the Average Treatment Effect (ATE) – across villages assigned to receive the program. The design hardly involved giving sub-samples of treatment villages variations of the program to examine the relative effectiveness of alternative mechanisms/processes to deliver the same program and facilitate learning about factors that drive impacts. In the context of NSP, it is plausible that the ATE estimates masks important differences in impacts across the sample. In 3 Ministry of Water and Energy 4 Ministry of Agriculture, Irrigation and Livestock 10 other words, the project might have succeeded relatively more in some villages compared to others, but the evaluation design did not (ex-ante) anticipate such differences. The evaluation cannot test hypothesis on what might drive the differences. Factors that could drive variation in NSP impacts include land inequality, local power relations, ethnic fragmentation, the presence of warlords, the relationship of village structures with sub-national governance institutions etc. Some of these factors are not observable, and this makes it harder to estimate the separate effects of these variables on program impacts. However, it certainly remains an important issue for future research. The findings show that drinking water and electricity sub-projects are more successful compared to irrigation and transportation sub-projects. There are several inconclusive hypotheses on what might drive these findings (see section 4 above). An important policy question is how these sub-projects built using the CDD bottom-up approach compare against similar sub-projects built using the traditional top-down approach where the decision-making power rests with the government. Which approach is more cost-effective? Where are the differences in terms of impacts? Does community involvement in the bottom-up approach explain any differences in impacts? Alternatively, are other aspects such as using local labor, local knowledge, and local oversight more important? Are the projects constructed what the community really needs from an engineer/urban planner standpoint? There is evidence in the report that the projects constructed were identified as necessary by males. Nevertheless, would a top-down government led approach have resulted in the same projects? The evaluation is not designed to address these questions. Analyzing these issues would require a fairly demanding impact evaluation in the future. Since the local context matters, results of similar evaluations in different countries may not be very helpful. It is an important issue for further research in the context of Afghanistan. Finally, the impact of NSP on economic welfare is inconclusive and therefore remains an important area for further research. Summary of other literature on CDD evaluations The literature on evaluations of CDD projects is growing rapidly and without doubt the impact evaluation of NSP is an important addition. Below is a summary of results from evaluations that estimated impacts on indicators similar to some of those examined for NSP. The summary is not presented for the purpose of comparing performance of NSP with those projects, but rather to give a sense of what impacts are being observed in other countries. Comparing impacts across projects may be misleading, even when the projects follow similar CDD approaches, because there are important differences in characteristics of targeted beneficiaries, broader policy environment, duration of implementation, size and frequency of grants, and of course varying institutional and country contexts. For example, while NSP II provided a single block grant over 4 years of implementation (2007-2011), the World bank-funded Kecamantan Development Program (KDP) in Indonesia provided grants in 3 cycles of disbursement within 5 years period (2002-2007). A summary of findings from other CDD evaluations is presented in the table below. 11 Summary of findings from other CDD evaluations Project evaluated and Evaluation design Summary of results country implemented Social and institutional Economic outcomes outcomes NGO funded Community Experimental Increases use of Weakly increases Driven Reconstruction democratic processes: employment and asset (CDR) program in 2 increases trust, reduces holdings, districts of Liberia social tension, and increases acceptance of the marginalized. But no impact on local decision making World Bank-funded Experimental No impact on trust, or Increases market activity, GoBifo program in 2 collective action, or asset ownership districts of Sierra Leone participation, or women empowerment outside the project area DfID funded CDR program Experimental Impact on bottom-up No impact on income, in Democratic Republic of accountability, increased agricultural productivity, Congo trust among ex- assets, and housing combatants, and quality increased gender inclusion, but no impacts on transparency World bank-funded Quasi-experimental Improves access to increases consumption, Kecamantan Development (propensity score health services reduces poverty, reduces Program (KDP) in matching) unemployment Indonesia World bank-funded BRA- Quasi experimental Increases participation of Increases asset ownership, KDP5 program in Aceh, (propensity score women groups. No agricultural activity, Indonesia matching) impact on associational economic perceptions; but activities, trust in no impact on employment institutions, acceptance of returning groups, and social tensions World Bank funded Quasi experimental Increased participation in Increases consumption, KALAHI-CIDSS6 program local governance, public employment, in Philippines affairs, inter-personal diversification, access to trust; but no impact on markets for Agricultural collective action produce 5 Community-Based Reintegration assistance for Conflict Victims (BRA-KDP) 6 Kapit-Bising Laban Sa Kahirapan Comprehensive and Integrated Delivery of Social Services (KALAHI-CIDSS) program 12