Report No. ACS21095 The Impact of Social Accountability on the Quality of Community Projects in Uganda: Impact Evaluation Results from NUSAF2 Transparency, Accountability and Anti-Corruption Sub-Component February 14, 2017 Document of the World Bank Social Protection, Labor and Jobs Global Practice; Africa Region (GSP01) 1 Standard Disclaimer: The findings, interpretations, and conclusions expressed in this paper do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. 2 Table of Contents Acknowledgements....................................................................................................................................... 7 1. Introduction .......................................................................................................................................... 9 2. The TAAC Social Accountability and Community Monitoring Intervention........................................ 15 2.1 SACM Design and Content ................................................................................................................ 15 2.2 Additional Incentives ........................................................................................................................ 19 2.3 Theory of Change .............................................................................................................................. 20 2.3 Quality of SACM Implementation ..................................................................................................... 23 3. Impact Evaluation Design and Data ........................................................................................................ 27 3.1 Impact Evaluation Design.................................................................................................................. 27 3.2 Baseline Data and Baseline Balance ................................................................................................. 29 3.3 Follow-up Data, Main Outcomes and Intermediary Outcomes ........................................................ 30 4. Empirical Estimation Strategy ................................................................................................................. 33 5. Impact Evaluation Results ....................................................................................................................... 35 5.1 Impacts on Main Community Project Outcomes .............................................................................. 35 5.2 Impacts on Intermediary Outcomes and Mechanisms ..................................................................... 38 6. Conclusion ............................................................................................................................................... 43 References .................................................................................................................................................. 47 Tables .......................................................................................................................................................... 51 Annex: Components of the main outcomes indices ................................................................................... 61 3 List of Figures Figure 1: Accountability Relationships in Service Delivery ......................................................................... 21 Figure 2: Impacts on Main Community Project Outcomes, All Project Types ............................................ 35 Figure 3: Impacts on Main Community Project Outcomes, Livestock Projects Only .................................. 36 Figure 4: Impacts on Components of Livestock Project Quality Score ....................................................... 37 Figure 5: Impacts on Intermediary Outcomes: Procurement and Contracting Process ............................. 38 Figure 6: Impacts on Intermediary Outcomes: Community Monitoring .................................................... 40 Figure 7: Impacts on Intermediary Outcomes: Complaints by Individuals about NUSAF2 Sub-projects ... 40 Figure 8: Impacts on Intermediary Outcomes: Complaints by Communities about Other Government Services ....................................................................................................................................................... 41 Figure 9: Impacts on Intermediary Outcomes: Payments and Satisfaction with Officials.......................... 42 Figure 10: Impacts on Intermediary Outcomes: Trust in Local Leaders and Public Officials ...................... 42 List of Tables Table 1: Sample Projects by Type and Status ............................................................................................. 28 Table 2: Baseline Balance............................................................................................................................ 51 Table 3: Impacts on Main Outcomes (All Project Types) ............................................................................ 52 Table 4: Impacts on Main Outcomes (Livestock Projects Only).................................................................. 53 Table 5: Impacts on Main Outcomes (Livestock Projects Only, Beneficiary-level indicators) .................... 54 Table 6: Component of Main Project Score (Livestock Projects only, Beneficiary-level Indicators) .......... 55 Table 7: Main Intermediary Outcomes, Procurement & Contracting ........................................................ 56 Table 8: Main Intermediary Outcomes, Community Monitoring and Complaints about NUSAF2 sub- projects ....................................................................................................................................................... 57 Table 9: Spillovers in Community Monitoring, Complaints about Non-NUSAF2 Government Services .... 58 Table 10: Main Intermediary Outcomes, Payment and Satisfaction with Officials .................................... 59 Table 11: Main Intermediary Outcomes, Trust in Local Leaders and Public officials ................................. 60 4 Abbreviations ASSIP Accountability Sector Strategic Investment Plan CAO District Chief Administrative Officer CDD Community-Driven Development (Program) CMG Community Monitoring Group CPMC Community Project Management Committee CPC Community Procurement Committee CSO Civil Society Organizations CT Community Trainer DIME Development Impact Evaluation (at World Bank) IG Inspectorate of Government IPA Innovations for Poverty Action ITT Intention to treat NDO NUSAF2 Desk Officer NDP National Development Plan NTF Nordic Trust Fund NUSAF1 First Northern Uganda Social Action Fund NUSAF2 Second Northern Uganda Social Action Fund LC1 Local Council I (village) chairman LC3 Local Council III (sub-county) chairman OPM Office of the Prime Minister PRDP Peace, Recovery and Development Plan RDC Resident District Commissioner SAC Social accountability committees SACM Social accountability and community monitoring intervention TAAC Transparency, Accountability and Anti-Corruption Component UDN Uganda Debt Network 5 6 Acknowledgements The Impact Evaluation of the Transparency, Accountability and Anti-corruption (TAAC) sub- component of the NUSAF2 project would not have been possible without the collaborative effort between the Government of Uganda, the World Bank, and partner researchers. The scientific aspects of the design, implementation and analysis of the impact evaluation were led by co-principal investigators Nathan Fiala (University of Connecticut) and Patrick Premand (World Bank), who wrote this report. A World Bank team was instrumental in the design and implementation of the TAAC sub- component, SACM intervention and related impact evaluation. Suleiman Namara was the task team leader during the design phase of NUSAF2, the conceptualization of the TAAC sub- component, and initiated the impact evaluation activity. Endeshaw Tadesse was the task team leader for the supervision of NUSAF2 implementation and managed the impact evaluation activity. Filder Aryemo and Jillian Larsen provided implementation support to the TAAC sub- component, oversaw the SACM curriculum development and roll-out, and contributed to the set- up and implementation of the impact evaluation. Filder Aryemo also contributed to this report. The NUSAF2 project was managed and implemented by the Government of Uganda. The Office of the Prime Minister was responsible for the overall coordination, resources and implementation of the project, while the Inspectorate of Government (IG) was responsible for implementing the TAAC sub-component. The IG team who managed the TAAC sub-component, oversaw the SACM intervention, and were engaged in the design and implementation of the impact evaluation included Mr. James Penywii (Director Project Monitoring) and Ms. Munira Ali (Head, Public and International Relations). The IG worked in partnership with a consortium of CSOs for the delivery of the SACM intervention, and in particular with a UDN team working under the stewardship of Uganda Debt Network led by Mr. Gilbert Musinguzi (National Coordinator), Mr. Julius Kapwepwe (Program Director) and Mr. Patrick Tumwebaze (Executive Director). Collaboration with the NUSAF2 Technical Support Team (TST), in particular Dr. Robert LimLim (Director) and Sam Sakwa (Field Operations Coordinator) is gratefully acknowledged. 7 Special thanks also go to the agencies that undertook data collection and their field staff, including Robert Sentamu at Wilksen Agencies for baseline data collection, as well as Areum Han, Kalie Pierce, Simon Robertson and Mariajose Silva Vargas at Innovations for Poverty Action (IPA) for the implementation study and follow-up sub-project assessment data collection. The research team is grateful for comments provided at various points of this study by the Inspectorate of Government (IG) team, the World Bank team, Pia Raffler, as well as Colin Andrews, Bénédicte de la Brière, Robert Chase, Deon Filmer, Vincenzo Di Maro, Ezequiel Molina, Obert Pimhidzai, Christina Malmberg Calvo, Pia Raffler, Ritva Reinikka, Lynne Sherburne-Benz and Dena Ringold at the World Bank. Financial support from the DIME i2i Trust Fund and Nordic Trust Fund at the World Bank, as well as a vanguard charitable trust through IPA, is greatly acknowledged. 8 1. Introduction Promoting citizens’ voice and participation has the potential to improve the quality of government services and development projects, as well as reduce leakage or corruption. A fast- growing body of international literature shows how corruption can undermine development by generating costs for society1. Finding the most successful and cost-effective approaches to reduce corruption remains a topic of active debate. The 2004 World Development Report presents a simple framework that articulates the relationships between citizens or program beneficiaries, service providers and policymakers. Governance and anti-corruption interventions can take the (vertical) long-route from clients influencing policymakers (bottom-up) and policymakers influencing service providers (top-down), or the (horizontal) short-route with citizens and program beneficiaries directly influencing service providers. Most traditional governance and anti-corruption interventions take the long route towards accountability, for instance by conducting (top-down) audits of programs and officials. However, these can often be costly to implement and can themselves become subject to elite interference. Besides, auditing may prove relatively ineffective in weak institutional environments (Serra, 2008; Barr and Serra, 2010). Recent research has shed light on the promise of using the (horizontal) short route for accountability. Providing populations with information and voice can be seen as an objective in its own right, including in the context of promoting human rights. Engaging local populations and encouraging transparency on the performance of local leaders and service providers can improve public governance by increasing the demand for accountability and ultimately reducing 1 Those costs can take different forms and range from an increase in bureaucratic hurdles to extract greater payments (Bertrand et al., 2007), to the creation of an unappealing economic environment for foreign investments (Woo, 2010) or a reduction of human capital stemming from bad-quality delivery of health or education services (Reinikka and Svensson, 2004, Bjorkman and Svensson, 2009). Not only does corruption represent an efficiency loss for society, but it also has a negative equity impact, affecting more severely those with less voice but with greater need for public services (Olken, 2006, Hunt, 2007). 9 corruption (Björkman and Svensson, 2009, Deininger and Mpuga, 2004, Banerjee et al., 2010, Banerjee et al., 2011, Djankov et al., 2010, Ferraz and Finan, 2008, Serra et al., 2011). Social accountability can be less effective in the presence of elite capture, or in cases where the population is not able to fully affect service providers’ incentives (Reinikka and Svensson, 2004; Björkman and Svensson, 2010, Barr and Zeitlin, 2011). Empirical evidence testing the relative effectiveness of alternative ways to design social accountability interventions in the context of large-scale national programs remains limited (Devarajan et al, 2011). This is particularly important as a range of design features can affect the likelihood of project success (Khwaja, 2009). In Uganda, many observers have highlighted that challenges in terms of transparency, accountability and corruption are affecting the quality of service delivery. The recent systematic country diagnostic for Uganda highlights some of these challenges (World Bank, 2015). According to the World Governance Indicators, the effectiveness and regulatory quality of the Government of Uganda are on a declining trend. The voice and accountability environment, which improved between 2003 and 2008, has also deteriorated2. Transparency International’s Global Corruption Barometer Survey (2013) found that 61 percent of Ugandans had paid a bribe to one of the eight institutions (judiciary, education, tax services and customs, permits/registries, medical services, police, land services, or utilities) in the last 12 months3. The average value of small bribes paid was approximately 20 percent of citizen’s annual income (IG 2014 a). Corruption was also highlighted as the most significant obstacle to doing business in Uganda by a recent global competitiveness report. How best to address the governance challenges in Uganda remains subject to active debates. The government’s commitment to increasing transparency and accountability in public service delivery is enshrined in Vision 2040, the National Development Plan (NDP), and the 2 Uganda’s capacity to control corruption appears to have reached an all-time low and is only above that of Kenya’s, East Africa’s worst performer. 3 The quality of the decentralization framework has worsened since 2005 mainly due to the proliferation of districts without adequate manpower and budget. This has negatively affected service delivery. Uganda’s performance is good on transparency but weak on budget credibility, controls, and compliance, essentially highlighting implementation issues. 10 Accountability Sector Strategic Investment Plan (ASSIP). In the last two decades, the government has actively pursued decentralization, anticorruption, and a range of reforms (World Bank, 2015). However, while legislation is in place, gaps in implementation of these regulations remain, including the ability to apply effective sanctions. At the same time, the effectiveness of encouraging citizen engagement and social accountability approaches remains unclear. Still, improving the demand for accountability was highlighted as one of the priority actions to improve the issue of public service delivery in the recent Systematic Country Diagnostic for Uganda (World Bank, 2015). The second phase of the Northern Uganda Social Action Fund (NUSAF2) was a large-scale Community-Driven Development (CDD) Program implemented by the Office of the Prime Minister in coordination with district and sub-county authorities. Between 2011 and 2016, over 10,000 sub-projects of various types were implemented throughout the Eastern, Northern, West Nile and Karamoja regions. These sub-projects included (i) public works (mostly for road-building and tree-planting), (ii) livelihood investment (including livestock and enterprise support) and (iii) infrastructure rehabilitation (such as staff house, boreholes or fencing). As part of the program, communities, with the assistance of community facilitators, were invited to formulate projects and submit proposals to the government. Sub-project approval included desk reviews at the sub- county and district as well as visits to the community before submission to the office of the prime minister for final approval. Once projects were approved, funds were managed directly by committees elected by the communities. In particular, Community Project Management Committees and Community Procurement committees were in charge of delivering the selected projects. Social Accountability Committees were formed with the responsibility to monitor project progress and provide oversight on behalf of the communities. Sub-county and district authorities undertook monitoring activities and provided oversight in coordination with project staff. The NUSAF2 project has been seen as instrumental in supporting the Government of Uganda to strengthen mechanisms to improve transparency and social accountability. One of the innovations of the NUSAF2 project is that it includes a Transparency, Accountability and Anti- 11 Corruption (TAAC) sub-component. The objective of the TAAC sub-component is to strengthen transparency, accountability and anti-corruption in Northern Uganda, using the project as an entry point to develop improved systems to be used by communities, local authorities and others with the mandate to promote improved governance in development. The TAAC sub-component is managed by the Inspectorate of Government (IG)4, and seeks to mainstream an anti-corruption prevention intervention in a Community Driven Development (CDD) project. Among other activities, some of the key innovations developed by the NUSAF2 TAAC sub-component included the set-up of a free anticorruption hotline, as well as the design and implementation of a social accountability and community monitoring (SACM) intervention. The NUSAF2 TAAC sub-component was designed to build on lessons from NUSAF1, where implementation of community projects faced several challenges. In particular, NUSAF2 pulls together the lessons of NUSAF1 on the promotion of public and social accountability in the design of a stronger service delivery mechanism mainstreamed into existing government systems. NUSAF1 introduced bottom-up accountability in a fragile environment. NUSAF2 further strengthened bottom-up accountability, among others through the TAAC sub-component. NUSAF2 also aimed to contribute to broader objectives of the Ugandan Peace, Recovery and Development Plan (PRDP), the overarching framework for post-conflict reconstruction in Uganda. The Social Accountability and Community Monitoring (SACM) intervention included thorough training on social accountability and community monitoring of NUSAF2 sub-projects communities. Follow-up visits by trainers were also conducted to provide on-going support to the communities to monitor implementation of NUSAF2 sub-projects. Community assemblies were organized to discuss the principles of social accountability and community monitoring. As part of this mobilization phase, new representatives were elected by the whole community to strengthen the NUSAF2 accountability structure, the social accountability committees (SACs). Members of the expanded committees, referred to as Community Monitoring Groups (CMGs), made a public pledge to participate in a training program, undertake monitoring of the project 4 The Inspectoral of Government also has field offices throughout the country. 12 on behalf of the community, and report back to the community. The training provided background on social accountability and NUSAF2, how to monitor projects and provided tools to monitor NUSAF2 sub-projects. A comprehensive training curriculum was developed (IG, 2015b). The training also provided hands-on skills in writing reports, providing feedback to the community, generating a community action plan and applying monitoring skills to government services other than NUSAF2 in the community. Given that SACM was large-scale and innovative in nature, a prospective randomized impact evaluation of the social accountability and community monitoring intervention was put in place. The impact evaluation was designed and implemented in collaboration between the World Bank, the IG and partner researchers. The objective of the impact evaluation was to assess the effectiveness of the social accountability and community monitoring (SACM) intervention implemented as part of the NUSAF2 TAAC sub-component. The impact evaluation provides evidence on the potential to integrate and mainstream social accountability interventions into large-scale operations. The study is particularly noteworthy given its scale: it focuses on a large-scale government-led intervention implemented in collaboration with local NGOs throughout the Eastern, Northern, West Nile and Karamoja regions of Uganda. The impact evaluation sample includes 940 communities, of which 634 communities were randomized to receive the social accountability and community monitoring (SACM) training, while 306 communities form a control group. As such, the results provide information to the IG and Government of Uganda on the value of promoting social accountability interventions in national programs. Given that it is one of the largest impact evaluation of a social accountability intervention around the world, the results of the impact evaluation will also be of interest for policymakers and development practitioners beyond Uganda. This report contains the main results from the social accountability and community monitoring (SACM) intervention impact evaluation. The results are mostly based on a large-scale follow-up sub-project assessment survey covering 634 NUSAF2 sub-projects randomized to receive the SACM intervention, and 306 comparable NUSAF2 sub-projects randomized to serve as control. Some results are also drawn from a household-level follow-up survey in treatment and control 13 communities. The impact evaluation results show that the social accountability and community monitoring (SACM) intervention improved community project quality by a small but significant magnitude amounting to 0.12 standard deviations. This overall improvement in community sub- projects is driven by a mix of increases in the quality and quantity of outputs. Livestock sub- projects are the most common type of sub-projects in the sample. In these projects, the social accountability and community monitoring intervention led communities to have healthier animals, as well as reduced by 25% (from 17.8% to 13.1%) the livestock that could not be found in the communities at the time of the follow-up sub-project assessment. The main results are shown to be similar for the standard social accountability and community monitoring intervention, and a treatment arm including additional incentives. Overall, the impact evaluation shows that community monitoring of large-scale decentralized development interventions can improve the quality of community projects. The study also analyzes the main channels through which the social accountability and community monitoring intervention led to improvements in community sub-projects. Results show that stronger monitoring of NUSAF2 sub-projects by communities is the main mechanism through which the SACM intervention was effective. Community members were more engaged in ensuring they were receiving quality project outputs, and more likely to report issues to local, sub-county and district officials, as well as to the Inspectorate of Government. Some spill-overs from the community monitoring are also observed, as complaints from community members about government services others than NUSAF2 also increase. In contrast, no impacts from the SACM intervention are found on the process used by communities to procure goods and materials. No impacts are found on payment made to district officials, satisfaction or trust in public officials, although beneficiaries report lower trust in community sub-project leaders. The report is structured as follows. Section 2 describes the content of the SACM intervention, and the underlying theory of change of the impact evaluation. It also summarizes the result of an implementation study undertaken in a sub-sample of communities that participated in the SACM training. Section 3 presents the impact evaluation design and data, including the scope of the baseline survey and follow-up surveys, as well as the main outcome measures considered for the 14 impact evaluation. Section 4 outlines the empirical estimation strategy. Section 5 presents the impact evaluation findings. Section 6 concludes. 2. The TAAC Social Accountability and Community Monitoring Intervention 2.1 SACM Design and Content The Transparency, Social Accountability and Anti-Corruption (TAAC) component of NUSAF2 included various activities managed by the inspectorate of government (IG). The component implemented anti-corruption prevention activities such as public awareness through radio talk shows, sensitization workshops and training of NUSAF2 social accountability committees. Several information, education and communication material were produced and distributed. A free corruption reporting hotline (“6009”) was set-up to ease reporting of complaints to the IG. A detailed curriculum for a social accountability and community monitoring (SACM) intervention was developed, piloted and implemented. Development of the SACM training curriculum included a review of key literature as well as consultations with NGOs/CSOs that are involved in social accountability interventions across Uganda. A team of local and international consultants from the World Bank worked closely with the IG to draft, pilot and validate the SACM training curriculum, which was published in 2014. The SACM intervention was implemented by a consortium of CSOs contracted by the IG. This collaboration created a unique partnership between a government institution and civil society organizations. The lead CSO was the Uganda Debt Network, which worked with 9 grassroots CSOs across the project area5. UDN and its partners selected 42 community trainers who underwent 5 The lead NGO was UDN and partners included 1) Church of Uganda-Teso Diocese Planning & Development Office (COU-TEDDO); 2) Community Empowerment for Rural Development (CEFORD) – West Nile; 3) Gulu District NGO FORUM (GDNF); 4) Lira NGO Forum (LNF); 5) Moroto-Nakapiripirit Religious Leaders’ Initiative for Peace (MONARLIP); 6) Pader NGO Forum (PNF); 7 Recreation for Development and Peace (RDP) – Uganda; 8) Tororo Anti- Corruption Coalition – TAC and 9) Uganda Joint Christian Council (UJCC). 15 two weeks of rigorous training of trainers organized by the IG. One full-time community trainer was based in each NUSAF2 district covered by the SACM intervention. Community trainers conducted their work with technical support and supervision from UDN, IG and the World Bank teams. The intervention was implemented between May 2014 and February 2016, with training in the evaluation sample concluding by October 2015. Achievement of the targets set for community trainers was monitored through a locally designed remote web-based technology hosted by the Grameen foundation in Uganda. Additional monitoring included field spot checks as well as periodic progress review meetings with the consortium partner heads, regional managers and community trainers. The SACM intervention curriculum focused on promoting bottom-up social accountability. Community mobilization assemblies took place to enroll additional members of the wider community into a community monitoring group. The assemblies were organized by community trainers with the help of village local leaders (LC1). The purpose was to introduce basic principles of social accountability and community monitoring, validate the role of the sub project social accountability committees (SAC) and invite additional members of the community to join existing SAC members to form a Community Monitoring Group (CMG). Specifically, four additional willing members of the community were elected by the community during the enrollment meetings to join and strengthen social accountability committees. The four newly elected members of the community and the five originally elected SAC members formed an expanded monitoring group called the community monitoring group (CMG). Members of the committees made a public pledge to participate in a training program, undertake monitoring of the project on behalf of the community, and report back to their communities. Once the community monitoring group was set-up, its members participated in 3 full days of training, complemented with 3 follow-up support visits. The three-day training was based on the SACM curriculum developed by the IG and the World Bank. The curriculum addressed three main areas: how to identify problems in community sub-project implementation, how community members can resolve implementation problems, and how to report or petition relevant local authorities about potential issues, including use of the toll free corruption reporting hotline. The SACM curriculum was developed to be delivered to semi-literate and illiterate populations, with intensive piloting and heavy use 16 of visual-based learning. The intervention was focused on improving community monitoring and encouraging the communities to follow-up on findings. It was made clear that it was not the role of community trainers to conduct monitoring and take actions themselves. The first curriculum module covers community mobilization and an introduction to social accountability. This module includes 2 to 3 hours of interaction with mobilized members of the community where a NUSAF2 sub project is implemented. In the meeting, the community trainer leads a discussion on key concepts of accountability and community engagement, the roles and responsibilities of the Social Accountability Committee (SAC). An election is then organized to select four willing members of the community to strengthen the existing SAC and form a Community Monitoring Group (CMG). Part of the meeting also identifies existing government programs operating in the community, provides an overview of NUSAF2 and why it is important for the wider community members to monitor these projects even if they are not direct beneficiaries. Discussions on key concepts of accountability cover: a) common types of corruption at the central, local government and community levels such as bribery, embezzlement, nepotism, absenteeism and solicitation of favors; as well as b) social accountability and the constitutional right of every Ugandan to participate in conducting accountability and combating corruption. This session is concluded with brainstorming on key actions the community can take as individuals or group to conduct social accountability, combat corruption and thus improve project outcomes. The module ends with the election and introduction of the community monitoring group (CMG), which includes original SAC members and four newly elected members. Preceding the election, a presentation reviews the roles of the monitoring group and characteristics of people who would be suitable for this role. Both the SAC chairman and the coordinator of the newly formed CMG are given an opportunity to give short speeches on how they will execute their duties to meet the expectations of the community. The CMG members are then invited for a three-day training at a selected venue and date. The second module discusses social accountability more specifically in the context of NUSAF2. It takes place on the first day of the three-day comprehensive training. It reviews in details all the basic concepts discussed at the enrollment meeting as well as provides a deeper understanding of the different stages of implementation of the NUSAF2 sub-project, including the procurement 17 rules and procedures to select and monitor contractors to provide goods and services to the community. The community trainer leads the community in identifying key implementation areas that are more prone to mismanagement and explores ways in which the community can engage in monitoring to obtain better project outcomes. The third module tackles community monitoring skills. This module aims at providing basic skills in community monitoring of the NUSAF2 projects. The CMGs review steps in monitoring, such as identifying sources of information, gathering and managing monitoring data. The module includes practical exercises to help CMGs to ask critical questions to monitor the procurement, timeliness, technical support, financial management and quality of inputs for the NUSAF2 sub- project in their own community. The fourth module covers post-monitoring activities. The module provides a basic understanding on how to review, store and manage monitoring data and findings. It covers the production of simple reports using monitoring data for submission to relevant authorities. Practical exercises include conducting a mock monitoring session and writing a simple report. The module ends with a session on providing feedback on monitoring findings to other community members as well as exploring possible actions to follow-up. The fifth module discusses how to generate a community action plan. This is a practical step- by-step session on how to develop an action plan relevant to the NUSAF2 community sub-project. CMGs are taken through a participatory discussion that results into key action plans that will be implemented and reviewed with the community trainer during the first follow-up support visit. The session works on planning monitoring activities and allocation of tasks among the CMGs. The sixth module covers follow-up support visits by the community trainer. This module provides step by step guidance on how the CMGs can review the action plan generated in module 5 and how the community training can provide technical support or refresher training depending on identified gaps. The module ends with guidance on how to revise and create new action plans at the end of every follow-up support visit. The final module reviews how to apply lessons learnt from the training to other government services. The aim of this module is to help CMGs apply the skills they gained from monitoring 18 NUSAF2 to other government programs in their communities. The module uses an example of teacher absenteeism from the education sector to help CMGs learn and apply their monitoring skills to other sectors. The module ends with a practical session on creating a monitoring checklist for teacher absenteeism. 2.2 Additional Incentives The incentives for community members to engage in project monitoring are not clear a priori, in particular for non-beneficiaries. The NUSAF2 Operational Manual outlines that social accountability committees are expected to complete “participatory monitoring tools” to ensure active involvement of all stakeholders in monitoring of project activities and outputs, secure commitment, and build the capacity of communities to be able to analyze, reflect, and take corrective action. It is expected that regular reporting will improve accountability and project outcomes. However, communities may lack the incentives to engage in these activities, particularly if there is fear of retaliation. The Whistleblowers Protection Act 2010 provides that a whistleblower shall be rewarded for his or her disclosure five percent of the net liquidated sum of money recovered based on that disclosure. However, this incentive for reporting faces long delays, and few people know about it. In addition to receiving the standard SACM model, some of communities were offered additional incentives for regular participation in training and active project monitoring. The incentives were designed to include individual rewards as well as group rewards. For individual rewards, individuals who completed the preliminary training and participated in follow-up visits and monitoring received an official “Community Monitor” pin from the IG. This pin helped to identify the community monitors who have been trained and empowered to monitor NUSAF2 projects and other government projects within their communities. They also received a contact card to enable them to reach out to the IG regional office and the Director of Project Monitoring for any support. As such, the incentives increase the salience of the link between CMGs and the Inspectorate of Government. As part of the group reward, communities that completed the 19 entire training, support visits, and conducted the community monitoring resulting in the final report were intended to be recognized by the IG for their hard work on local radio stations. The individual reward was designed to be based on attendance and participation in the training and follow-up visits. The group reward was designed to be based on timely production from monthly reports on their community monitoring findings. In practice, group rewards were not implemented, and the additional incentives mostly consisted of the symbolic IG pin and IG contact card, which together strengthen the visibility of the link between community monitoring groups and the Inspectorate of Government. 2.3 Theory of Change The 2004 World Development Report presents a governance framework for the relationships between citizens or program beneficiaries, service providers and public officials. Governance and anti-corruption interventions can take the (vertical) long-route from communities influencing policymakers (bottom-up) and policymakers influencing service providers (top- down), or the (horizontal) short-route with communities and program beneficiaries directly influencing service providers (see Figure 1, World Bank, 2003). This simplified framework has proved influential to analyze the pathways through which social accountability interventions can lead to better development outcomes (Ringold et al., 2012), including for a finer analysis of the theory of change from these interventions (Molina, 2014). 20 Figure 1: Accountability Relationships in Service Delivery State (Political and Technical Officials) Community (Beneficiaries Providers and (CMC, CPC, Contractors SAC/CMG) and non-beneficiaries Source: Simplified framework from World Bank (2003) In light of this framework, and given the scope of the SACM training described in the previous section, we consider three main potential pathways through which the SACM intervention can improve the quality of community projects. We start by considering two pathways that mostly relate to the short accountability route, i.e. how the community can directly influence service providers and contractors. First, as part of the NUSAF2 community-driven intervention, communities select providers and contractors through a procurement process. A more transparent and smoother procurement process can contribute to the selection of better or more accountable contractors or service providers, which may ultimately improve the quality of services and goods received by communities. We therefore analyze whether the SACM intervention leads to changes in the process by which communities select contractors, and ultimately their choice of contractor. Second, we assess whether more intensive community monitoring leads to the delivery of better project outputs. This second pathway also relates to the short accountability route, but here the influence of the community occurs mostly after the contractor has been chosen, i.e. when the project outputs are being delivered. Related to this, 21 more intensive community monitoring can also lead to additional complaints by communities to public officials about the performance of service providers and contractors. Finally, we consider a third pathway that relates mostly to the long accountability route. Specifically, we assess whether there are changes in the interactions between communities and their local officials. We consider both local political officials, as well as local technical officers that are involved in NUSAF2 project implementation. Specifically, we consider whether there are changes in the payments made by communities to these officials, the satisfaction with the services provided by the various officials, as well as the trust in local leaders and public officials. Changes in interactions between community members and officials could indicate that the SACM intervention affects project outcomes through the long accountability route. In this context, the impact evaluation tests to what extent the SACM intervention led to improvements in project quality, and assess which of these various channels is most likely to explain the observed impacts on project quality. In practice, the various mechanisms can interplay and jointly explain improvements in project quality. While the mechanisms cannot be fully disentangled, the impact evaluation can shed light on the strength of the effects on these various mechanisms, and as such indicate which mechanisms, or which combination of mechanisms, is more likely to explain overall impacts on project quality. 22 2.3 Quality of SACM Implementation An implementation study was undertaken in a sub-sample of communities that participated in the SACM training. The objective of the implementation study was to document the quality of implementation of the training, and as such to complement and facilitate the interpretation of the main impact evaluation results presented in section 5 below. Given the large scale of the SACM intervention and its disperse geographical scope, it is important to understand the extent to which implementation was consistent with the intended design. This enables us to understand whether the impact evaluation results are affected by potential implementation issues. The implementation study was undertaken in a random sample of 97 communities trained as part of the SACM intervention, or approximately 15% of all sub-projects and communities selected to be included in the impact evaluation study. As such, the implementation study sample is large enough to provide good measures of quality of implementation. The first round of surveys took place in March 2015, the second round of surveys in early May 2015, and the third round of surveys at the end of June 2015. The implementation study was based on a short instrument which was administered to a minimum of five members of the CMG, in either local language or English depending on the CMG's preference. Communities were mobilized via phone or in-person and most interviews took place on the same day as mobilization and in person. The instrument covered issues related to mobilization and group formation, timing of the training, understanding of the training curriculum, disbursement of funds to the sub-projects, self- confidence, civic understanding and social accountability. The implementation study shows that over 90% of the sample correctly nominated and elected their CMG members in line with the intended protocol and received the full three-day training. As part of the SACM intervention, every community was expected to be mobilized within one day and receive three days of training. In the random sample selected for the implementation study, only 72% were mobilized as planned in one day, but over 90% of the sample received the full 3 23 days of training6. The majority of NUSAF2 communities followed the correct procedure for picking their CMG members. For over 90% of the sample, the community directly nominated and elected members to join the CMG7. Over 85% of the sample agreed that the CMG is accountable to the community and must report its activities and findings to them. A relatively small number of sub-projects in the implementation study sample (4%) started using their NUSAF2 funds at least a month before they received training. The usefulness of the training in part depends on the timeliness of its implementation, so that the CMGs can indeed monitor key sub-project activities on time. SACM training may not be as effective in communities where substantial progress has been made, for instance purchase of materials taking place several weeks before they received training. Overall, the implementation study shows that most training took place in timely fashion. In the implementation study sample, 15% of communities received their training after they started using their funds. However, 2/3 of those communities received training within a couple weeks of starting to use their funds. Only 4% of communities in the sample received the training more than one month after starting to use the funds. Over 75% of the CMGs found it "easy" to understand the training. The SACM training curriculum was developed with the objective to be delivered to semi-literate and illiterate populations using visual supports. The implementation study highlighted a good understanding of the curriculum by the communities. Most trainers used the local language and some English to teach the curriculum. Over 75% of the CMGs declared that they found it "easy" to understand the training, while 20% found it "a little difficult". Roughly 90% of CMGs were able to articulate concrete action plans in response to examples of potential corruption or service delivery issues. During the implementation study interview, a few "vignettes" containing hypothetical scenarios were described to CMGs8, who were then 6 A small number of communities - roughly 7% of the sample - received only one or two days of training. A couple of communities, however, had not received any training at all at the time of implementation study. Both of these communities were from the Karamoja region. 7 A small number of communities did not follow correct procedure. In five communities, no additional CMG members were selected by the community. In four communities, the trainer or facilitator selected the additional CMG members themselves (rather than allowing the community to nominate and elect them). 8 The vignettes covered the following scenarios: 1) Imagine that you are monitoring a livestock project and you find out that in distributing the animals, the CPMC executives took more animals for themselves than they gave to 24 asked how they would respond to such a situation. A particularly evocative example is a case when the CMG is asked how they would respond if the procurement committee were to deliver low quality animals to the community. Over 95% of the sample agreed that they would reject the animals and take some form of action. For all four scenarios presented to the communities, at least 93% of the sample answered with a plan of action consistent with the training curriculum9. In terms of the types of actions taken if they experienced problems with service delivery, approximately 40% of the suggested actions involved contacting the IG, and most of the other suggested actions involved reporting to other authorities such as the CAO, RDC, Sub-county Chief, NUSAF officer, LC1, or police. Many communities also suggested conducting their own investigation. Most CMGs were able to come up with 3 different plans of action. Only a few communities gave answers that indicated lack of empowerment, saying that there isn’t anything they could do to address the issue. The implementation study also contained questions related to communities’ understanding of potential cases of corruption. The results showed that the majority of the trained communities understood potential corrupt practices. When incorrect answers were given, communities and CMGs lean toward being extra critical of project management10. other group members. What would you do? 2) Imagine that the community facilitator - who assisted your group to initiate this project - insists on completely taking over managing the finances of your group project fund. After all, he or she assisted your group in writing the project proposal and in most cases is more educated than the members of your group. What would you do? 3) Imagine that the contractor of your sub-project got the first advance of 8 million shillings to start work and then disappeared. What would you do? 4) Imagine that you are monitoring a livestock project and the supplier brings you very poor quality animals. You suggest that the group reject these animals and instead have the supplier come back with animals of higher quality. The CPMC and the CPC, however, insist that the group accept the animals that are of poorer quality. What would you do? 9 Examples of responses included: "We report to the community development officer at the sub-county," "Report to the sub-county and if it fails to act, we shall report to the IG," "Investigate further, after finding the truth, then we shall ask for fresh re-distribution," "We meet as a group to decide, thereafter report to LC 1 and we will ensure that animals are distributed afresh if the group decides”, and "We call them, question them on how distribution was done and demand redistribution if they refuse we report to IG and other district leaders." 10 The potential cases presented to communities were as follows: i) Imagine someone in the CPMC is given money from the sub-project to buy materials. However, their child is in a candidate class and needs to go back to school but they have no money to pay school fees. They decide to use the sub-project money to pay school fees. Do you consider their act a case of corruption? ii) Imagine that the supplier spent 50,000 shillings and came back with less goods than expected but he brought his receipts and his receipts show that the goods were simply more expensive than had been budgeted for. Do you consider this a case of corruption? 96% of communities answered correctly to the first case. A little over 60% gave the correct answer to the second case, which offered a much more complicated scenarios. Of those who failed to give the correct answer to the second question, however, over 90% 25 The implementation study highlighted some areas through which the training may have affected community actions. About 88% of the CMGs stated that they were confident in their ability to improve service delivery of the NUSAF2 project. 70% of the CMGs in the sample said that they had applied their training to projects others than NUSAF2. They gave examples with regards to monitoring of NUSAF2 project, but also with regards to questioning the quality of other public services, for instance in health centers. Ultimately, the implementation study reveals a satisfactory level of implementation of the social accountability and community monitoring (SACM) intervention. The training was mostly implemented as designed, despite the large-scale nature of the program. Some delays took place but few communities received the training too late for it not to be useful during project implementation11. An important caveat is to be noted with regards to implementation in the Karamoja region, which suffered additional delays. Finally, and as mentioned above, the incentives were only partly implemented, and as such actual incentives are relatively weak. Some areas for improvements were highlighted through the implementation study as well. Interviews with group members reveal some confusion over who was in which committee and what was the purpose of each committee. Almost 14% of the CMGs in the sample did not know that the CPC was in charge of procurement for the NUSAF2 sub-projects. Finally, the majority of CMGs interviewed asked for some kind of compensation for their service and stated their desire to be considered as beneficiaries in the next round of the NUSAF project. answered "Yes," indicating that the CMGs lean toward being extra critical of project management rather than under critical. 11 Inefficiencies in the flow of funds between the IG, UDN, the CSO partners and community field trainers contributed to some of the delays. 26 3. Impact Evaluation Design and Data 3.1 Impact Evaluation Design The goal of the SACM training was to improve the quality of implementation of NUSAF2 community sub-projects and overall service delivery. In this context, the main impact evaluation question of interest is whether the social accountability and community monitoring intervention was effective in improving the quality of NUSAF2 community projects. Beyond the effectiveness of the overall SACM intervention, a secondary question addresses what is the most effective modality for the social accountability intervention, and in particular whether additional incentives are needed to encourage community monitoring. A third question relates to the mechanisms which are more likely to explain the observed impacts on project quality, in particular the broad areas highlighted in section 2: (i) the procurement and contracting process, (ii) community monitoring and reporting of complaints, and (iii) community interactions with district officials, as well as satisfaction and trust in public officials and local leaders. In order to answer these questions, a large-sample, randomized control trial was put in place. In a random selection of communities benefitting from NUSAF2, a social accountability and community monitoring (SACM) intervention was introduced. The intervention included two treatment modalities: (1) the standard model in which communities were trained to monitor the implementation of their local NUSAF2 development projects and address any issues with project implementers and government officials; and (2) the standard model with additional incentives for communities to report corruption and produce quality project outputs. By comparing these two intervention models against a randomized control group of NUSAF2 projects that did not receive the SACM intervention, the causal impact of the social accountability and community monitoring training can be obtained. In addition, each intervention can also be compared against each other, allowing for an identification of the most effective intervention model, and the value- added of the extra incentives. The social accountability and community training intervention was implemented in a large number of NUSASF2 sub-projects, covering various types of community projects, with a 27 prevalence of livestock sub-projects. As mentioned above, as part of NUSAF2, communities were invited to formulate sub-projects and submit proposals the local government office for appraisal. The submitted projects can fall under three main categories: (i) public works (mostly for road- building and tree-planting), (ii) livelihood investment (including livestock and enterprise support) and (iii) infrastructure rehabilitation (such as staff house, boreholes or fencing). As such, the social accountability and community monitoring intervention was implemented in different sub- project types. Table 1 provides the number of each sub-project type assigned to the control group, the standard SACM modality and the SACM training with added incentives. Randomization was done at the project level without stratification on project type or location. Livelihood investment projects (including livestock and enterprise support) amount to approximately 2/3 of the sample. The rest of the sample is mostly composed of infrastructure rehabilitation projects. Due to the range of project types and the difficulty of comparing outcomes for these different projects, approximately 20 projects among the least common are dropped from the final analysis. (including valley damn and valley tank projects). Given livestock sub-projects are predominant in the sample, the analysis provides results for all sub-project types, and then separately for livestock. Table 1: Sample Projects by Type and Status Treatment Status Project Type Control SACM SACM + Total Incentives Livelihood Investment Livestock 212 206 217 635 Enterprise 23 29 29 81 Infrastructure Rehabilitation Borehole 5 3 6 14 Fencing 9 11 7 27 Staff House 11 22 14 47 Dormitory 2 2 5 9 Classroom 2 2 3 7 Health Center 3 4 4 11 Valley Tank 2 1 0 3 Valley Dam 1 0 0 1 Public Works Roads 9 12 10 31 Tree Planting 27 29 18 74 Total 306 321 313 940 28 3.2 Baseline Data and Baseline Balance A baseline survey was conducted between January 2012 and March 201312. The baseline survey was implemented among local officials throughout the 45 districts and 485 sub-counties in areas where NUSAF2 operated at that time. Sub-county officials included in the survey were sub-county chiefs, LC3 and NUSAF2 desk officers. The survey sought to capture local leaders’ attitudes towards corruption and governance. The survey also collected socio-economic data on local leaders (including education, work experience, earnings and assets), their knowledge of NUSAF2 operating rules, as well as leaders’ cognitive and behavioral skills13. The baseline survey among local officials was meant to complement available administrative data on community sub-project characteristics. The baseline data shows that the randomization procedure produced treatment and comparison groups that were similar before the roll-out of the SACM intervention, with satisfactory balance in observed characteristics. Balance tests are conducted on three sets of data. The first two, project budget and type, is from the administrative data obtained from the Office of the Prime Minister of Uganda. The third type of data is from the follow-up project assessment survey. Communities were asked when they received their NUSAF2 grants, which was coded as a continuous date. This variable, while not fully objective, gives a proxy of when the project was able to start14. In order to document balance between the treatment and control group, we test whether there is a statistically significant difference between treatment assignments on these indicators. Table 2 in annex provides the results. All of the coefficients are small and none are statistically significant. On this limited set of variables we conclude that there 12 Note that it was originally expected that the SACM intervention would start in 2013. However, delays in the procurement process to select and contract the implementing NGO implied that the intervention only started in May 2014. The intervention was implemented until October 2015. This explains the time lag between the baseline survey and follow-up sub-project assessment survey. 13 It was originally anticipated that randomization could be implemented at the sub-country level, and as such that baseline balance could be tested at the sub-country level. However, due to the fact that NUSAF2 project implementation was more advanced in some sub-counties, the remaining number of sub-counties was not sufficient and randomization had to be performed at the sub-project level. As such, balance tests on the leaders’ survey cannot be performed as originally intended. The information from the baseline survey will mostly be used in extensions of the analysis to consider heterogeneity in program impacts by local characteristics. 14 83 of the communities (10% of the sample) reported that they did not know when the money came and so this variable is coded as a missing value. 29 was generally good balance across the various treatment and control groups. This means that the sub-projects assigned to the various control and treatment groups were similar before the roll-out of the SACM training. Therefore, program impacts can be estimated by taking first- difference in outcomes observed in the follow-up survey. We return to the empirical estimation strategy in section 4 below. 3.3 Follow-up Data, Main Outcomes and Intermediary Outcomes The main source of follow-up data for the impact evaluation consists of a sub-project assessment collected after the completion of SACM implementation, between December 2015 and February 2016. The sub-project assessment included an observation of community projects by a team of enumerators. For projects with a single output (e.g. a staff house or a borehole), enumerators directly observed characteristics of the output. For livelihood support projects where outputs were distributed to beneficiaries, a sample of beneficiaries was drawn and beneficiary-level outputs were observed15. For example, for livestock projects, a sample of beneficiaries was selected and enumerators visited the sampled beneficiaries to observe the animals provided by the project. The sub-project assessment data allows for the measurement of a set of core outcomes for the impact evaluation, but also of intermediary outcomes (or main underlying mechanisms) that can lead to changes in final outcomes. For each domain, the sub- project assessment allows capturing a range of variables, which can later be aggregated into indices. The next sub-sections provide additional information on the main outcomes and intermediary outcomes to be tested and the indicators that were collected to measure them. The annex provides tables with the full list of variables composing the various indices. 15 In livestock and enterprise projects, the enumeration team collected the names of all participants. From this list, five people were selected at random to be interviewed. These five people were sought and replaced only in the instance that they had died (in which case attempts were made to track the animal they had originally been given), or were living with their animals very far away. Replacements were restricted to these special cases. The total number of individuals replaced was less than 3% of the sample. Data collection for all other projects was at the community level. 30 Complementary data is obtained from a follow-up household surveys, which covered NUSAF2 beneficiaries in SACM treatment and control communities and was collected between May and July 2016. The follow-up household survey provides complementary information at the household level, including on intermediate indicators related to individual complaints to local leaders and officials, and trust in local leaders and officials16. The primary outcome is a measure of a project overall score, which is composed of indices that measure the quality of the project and the quantity of outputs delivered17. The project overall score is the main outcome for the analysis. It is built as an interaction of a quality measure and quantity measure. This allows accounting for the situation where a community received more of output from a project, but at lesser quality, and vice versa. The quality and quantity indices are also analyzed separately. As the quality and quantity indicators are created across different product types, the indices constructed are normalized within each project type to ensure comparability. Project quality is measured within each project type through direct observation of a range of attributes of the project output. For livestock, the project quality score is an additive index of whether the animal received was of the appropriate age, whether it was a local or improved breed of animal, whether the animal was productive when visited by the survey team 18, and whether the animal displayed any signs of illness. For staff houses, quality is whether the walls, roof, windows, doors, ceilings and floors meet quality standards. For enterprise projects this is whether individuals have access to materials, transportation, credit, labor and markets. Roads are assessed based on the material used in the construction, and tree planting based on whether the seeds or seedlings were certified. The annex provides the full set of indicators that compose the quality index. 16 Additional information from the follow-up survey will be used in extensions of the analysis that will also consider impacts on household-level assets. 17 All indices are additive from the individual components. 18 Productivity is defined by animal type. For instance, if the animal is a dairy cow it is determined to be productive if it is producing milk. 31 The quantity measure captures the outputs delivered as part of the community project. It is determined by the number of animals received, length and height of the building constructed, number of people engaged in the enterprise, length of the road constructed and the number of trees planted. These measures are obtained from direct observations of the outputs by enumerators during the sub-project assessment. In cases where the output could not be observed, the quantity measure takes a value of 0. This happens for livestock sub-project when the livestock has died or is missing at the time of the sub-project assessment. The annex provides the full list of quantity indicators. To complement the observed measures of sub-project quality and quantity of outputs, an index of project implementation is also constructed. This score is composed of subjective questions asked to the community about whether they felt the project was useful, whether they felt the project was completed as expected, and whether the materials met expectations and were not deemed to be very expensive. The final indicator considered is whether the project appeared to be a “potential problem project”. 23 of the projects could not be found at the time of the follow-up sub-project assessment. These projects are categorized as “potential problem projects”. In practice, the implementing agency and the IG verified that funds were disbursed to the communities and the sub-projects implemented, ruling out major issues with these projects. Still, the data collection agency could not undertake the sub-project assessment in these projects and they could not be included in the sample. As part of the analysis, we test whether the share of potential problem projects varies between treatment and control communities. In addition to the primary outcomes, the sub-project assessment also measures three types of intermediary outcomes that capture the main underlying mechanisms that can explain changes in final outcomes. As mentioned above, the three main domains of intermediary outcomes are (i) the procurement and contracting process, (ii) community monitoring and reporting of complaints, and (iii) community interactions with district officials, as well as satisfaction and trust in public officials and local leaders. These three domains relate to some of the key areas covered by the social accountability and community monitoring curriculum. Indicators on the 32 procurement and contracting process include an index of challenges faced by communities in the procurement process, an index of satisfaction with suppliers of goods and materials, and whether the community hired a contractor. For communities that did hire a contractor, indicators also include an index of challenges faced by communities in the contracting process, and an index of satisfaction with the contractor. The second main domain for intermediary outcomes include indicators on community monitoring, in particular an index for the intensity of project community monitoring, and an index for intensity of monitoring by the Social Accountability committee (SAC). Indicators are also built from the follow-up household survey on the number of complaints made by individuals to officials, including LC1 chairperson, sub-county officials, district officials and the Inspectorate of Government. These complaints are reported separately for the NUSAF2 sub-projects and for government services unrelated to NUSAF2, in order to proxy for potential spill-overs on service delivery at the local level. Finally, the third main domain for intermediary outcomes captures interactions between communities and local officials. This includes indicators for whether a payment was made to district official or staff, an index of satisfaction with the sub- county NDO and district veterinarian officer, as well as measures of trust in local leaders and public officials. 4. Empirical Estimation Strategy The empirical estimation strategy relies on the randomization of the SACM training at the sub- project level, which allows identifying the impact of SACM training based on first-difference in outcomes at follow-up. Specifically, data from the follow-up sub-project assessment and follow- up household surveys are used to test the differential effects of each program modality on the outcomes and intermediary outcomes presented in Section 3. The impact of the social accountability and community monitoring training is estimated through the following intention to treat (ITT) regression model: Yi = α + βT + γXi + δR + εi (1) 33 where i refers to a community, Y is the outcome (or intermediary outcome) of interest, T is a set of dummy variables for which treatment a community was assigned to, X is a matrix of project controls (administrative data on project type, start date and budget)19, R is a set of region, district and sub-county dummies, and ε is the error term. The estimation is performed separately for a pooled treatment (combining the standard SACM model and the SACM model with incentives), and separately for each type of treatment modality. In addition to the main outcomes and intermediary outcomes listed in Section 3, the same estimation method is applied to components of the individual indices. While heterogeneity by sub-project type cannot be formally estimated due to small sample sizes, program impacts are also estimated separately for livestock sub-projects only. This gives a sense of the robustness of the results for the main project type in the sample, but also allows looking into the specific components of the overall project score for this type of sub-project. In addition, since the sub-project assessment was performed for a sub-sample of beneficiaries for the livestock projects, the main indicators can be analyzed at the beneficiary level for these projects, providing some additional statistical power. In order to ensure the transparency of the study, the analysis to be conducted was specified in a technical pre-analysis plan that was registered prior to the analysis being undertaken. The pre-analysis plan also outlines two extensions of the analysis beyond the scope of the current report20. First, additional research will assess to what extent variations in the density of social accountability training across sub-counties lead to spill-over or displacement effects. Second, additional research will assess heterogeneity in the training effectiveness by characteristics measured from the baseline leader survey. 19 These community controls are used in robustness tests only. 20 See Fiala and Premand (2016). 34 5. Impact Evaluation Results 5.1 Impacts on Main Community Project Outcomes Results show that the social accountability and community monitoring training intervention improved the overall quality of community projects. Figure 2 and Table 3 present the results for the five main outcomes: total sub-project score, score separated by quality and quantity, whether a project is a potential problem project, and the project implementation score. We find a statistically significant effect on the overall project score of 0.12 standard deviations. This is a moderate effect size and suggests that there was an important change in the communities due to the SACM intervention21. This overall improvement in community sub-projects is driven by a mix of increases in the quality and quantity of outputs, though improvements in the quality and quantity scores taken separately are marginally not significant. There is no significant difference in the share of potential problem across the treatment and control groups. There is no change in the subjective indicator of project implementation either. Figure 2: Impacts on Main Community Project Outcomes, All Project Types 0.14 0.12 * 0.10 0.08 0.06 0.04 0.02 0.00 -0.02 Sub-project Overall Sub-project quality Sub-project Potential Problem Project Score (std) score (std) quantity score (std) Project implementation score 21 A cleaner interpretation of the magnitude of the effects can be provided for livestock project. A one standard deviation increase in the quantity score is equivalent to a community receiving five extra cows. For the quality indicator, a one standard deviation increase would represent all of the sub-project's animals changing from either unproductive to productive, bought at the incorrect age to the correct age, having an illness to not having an illness or being local instead of hybrid. 35 No statistical difference was found in the impact of the standard SACM model, and on the SACM model with added incentives. There is no evidence that the incentives had a significant value-added on final outcomes beyond the standard SACM training. This is not really surprising given the incentives were relatively weak (and low-cost), and mostly consisted in individual incentives that strengthened the salience of the link between community monitoring groups and the Inspectorate of Government. Therefore, in light of these results, we pool the two treatment modalities for the rest of the analysis since they do not appear to lead to different impacts. Given the livestock projects are the most prevalent and constitute 2/3 of the sample, they can also be analyzed separately. Table 4 present the same results as in Table 3, but for the livestock projects only. The point estimates are broadly consistent with results for the full set of projects. They point to improvements in overall project score, mostly coming from improvements in the quality of livestock sub-project. However, these indicators at the project level are marginally not significant, in part due to the fact that the sample of livestock projects is slightly smaller, providing less statistical power. In order to gain additional statistical power, Table 5 and Figure 3 display similar results for indicators measured at the beneficiary level. Results again point to improvements in overall livestock project score that are marginally not significant, mostly coming from improvements in the quality of livestock sub-project, including a significant improvement in animal health. Figure 3: Impacts on Main Community Project Outcomes, Livestock Projects Only (Indicators at the beneficiary level) 0.20 0.15 0.10 * 0.05 0.00 Sub-project Overall Sub-project quality Sub-project Sub-project quality Score (std) score (std) quantity score (std) score (std) - health -0.05 36 The SACM intervention led communities to have healthier animals, and decreased the share of animals that could not be found during the follow-up subproject assessment. Figure 4 and Table 6 present impacts on the individual components of the livestock sub-project score. This helps to determine which component of the indices is driving the results documented above. We do not find statistically significant effects on animal age, whether the animal is an improved breed, or whether the animal is productive. We do find that the social accountability and community monitoring intervention led communities to have healthier animals, with a 15% decrease in the number of illness among animals delivered in livestock projects (from 0.187 to 0.16). In addition, a lower number of animals were missing and could not be found during the follow-up sub-project assessment. Specifically, results show a statistically significant effect on the fraction of animals that were dead, stolen or sold in communities. The SACM training decreased the likelihood of missing animals by approximately 25% (-0.047/0.178), a relatively large effect. This means that either the SACM intervention increased the number of animals actually received by communities, or increased the quality of the animals received and as such their probability of survival until the follow-up sub-project assessment. Figure 4: Impacts on Components of Livestock Project Quality Score 1.00 0.90 * 0.80 0.70 0.60 0.50 0.40 ** 0.30 0.20 0.10 0.00 Share Animals of Share Animals of Share of Productive Index of animal Share of Missing Correct Age Improved Breed Animals Health Livestock Mean in Treatment Mean in control 37 5.2 Impacts on Intermediary Outcomes and Mechanisms This sub-section explores the underlying mechanisms that can contribute to explain the overall impacts documented above. As suggested in the section on the theory of change, three main mechanisms are considered, namely (i) the procurement and contracting process, (ii) community monitoring and reporting of complaints, and (iii) community interactions with district officials, as well as satisfaction and trust in public officials and local leaders. First, the SACM training did not lead to changes in the ways communities undertook procurement and contracting processes, i.e. in their selection of providers of goods and services. Figure 5 and Table 7 present the main results on intermediary outcomes related to the procurement of goods, services and materials, as well as the contracting of suppliers to deliver goods and services to the community. No impacts are observed on these indicators. Treatment communities do not report significantly less challenges in the procurement process and significantly higher satisfaction with the suppliers of goods and materials. Sub-projects in communities that benefitted from SACM are as likely as control sub-projects to rely on contractors to deliver community projects. They do not face significantly less challenges in dealing with contractors, and do not report higher satisfaction with contractors. Figure 5: Impacts on Intermediary Outcomes: Procurement and Contracting Process 6.00 5.00 4.00 3.00 2.00 1.00 0.00 Challenges in Satisfaction with Hired a Contractor Index of challenges Satisfaction with Procurement supplier Index to Implement in Contracting contractor Index Process Index Project Process Mean in Treatment Mean in control 38 A finer analysis of the individual variables that enter in the various aggregate indices reveals no changes in any of the procurement variables. Almost none of the contracting variables are significant either. The only exception is a significantly higher reliance on single-sourcing in treatment communities, as well as a significantly lower share of communities complaining that their opinion was not taken into account by the contractor during project implementation. The latter effect reflects the stronger community monitoring that took place in these sub-projects, as we now discuss in relation to the second mechanism. Second, there is evidence of higher intensity of sub-project monitoring by communities assigned to the SACM intervention. As such, stronger community monitoring contributes to explain the improvements in community projects noted above. Figure 6 and Table 8 present the results on community monitoring indices. The SACM training led to an increase in overall sub- project monitoring by communities. For instance, communities that benefitted from SACM were more likely to produce accountability reports or monitoring reports. The SACM intervention led to a particularly strong increase in the intensity of sub-project monitoring by members of the community monitoring groups. The intervention was effective in making the social accountability committees more active: social accountability committees in communities assigned to SACM undertook approximately 40% more monitoring activities compared to committees in communities that were not assigned to the SACM training. Social accountability committees were more likely to produce written monitoring reports, monitor project implementation, and monitor the selection of livestock and material. Although these changes are large in relative terms, the level of monitoring by the social accountability committees in the control group is rather low. As such, the absolute increase in community monitoring is moderate, which is in line with the small overall improvements in the quality of community projects. 39 Figure 6: Impacts on Intermediary Outcomes: Community Monitoring 3.50 3.00 * 2.50 2.00 1.50 1.00 *** 0.50 0.00 Index for Intensity of Project Community Monitoring Index for Intensity of Social Accountability Committee Project Monitoring Mean in treatment Mean in control Consistent with increased community monitoring, individuals in communities that received the SACM training are also more likely to report having complained to public officials about issues with their sub-project. Figure 7 and Table 8 present impacts on the number of complaints about the NUSAF2 community sub-project made to various public officials. The overall number of complaints increased by 13% in treatment communities. The number of complaints made to LC1 chairperson, sub-county officials, district officials and the Inspectorate of Government all increased. The impact is particularly strong for complaints made to the Inspectorate of Government, although this type of complaint is less frequent to start with. Figure 7: Impacts on Intermediary Outcomes: Complaints by Individuals about NUSAF2 Sub- projects 2.50 *** 2.00 1.50 1.00 ** 0.50 * ** *** 0.00 Total # Complaints to # Complaints to LC1 # Complaints to # Complaints to # Complaints to Officials Chairperson Subcounty Officials District Officials Inspectorate of Government Mean in treatment Mean in control 40 Some spill-overs from community monitoring are also observed, as complaints from community members about government services others than NUSAF2 also increase. Figure 8 and Table 9 present impacts on the number of complaints made by community members about government services others than the NUSAF2 community sub-project. Results reveal a strong overall increase of 20% of the number of complaints made about other government services. The number of complaints made to various officials increase, including LC1 chairperson, sub-county officials, district officials and the Inspectorate of Government. These results suggest that the SACM intervention increased community monitoring of other government services as well, which could lead to impacts on service delivery beyond those documented above on the overall quality of NUSAF2 community projects. Figure 8: Impacts on Intermediary Outcomes: Complaints by Communities about Other Government Services 2.50 *** 2.00 1.50 1.00 ** * 0.50 ** *** 0.00 Total # Complaints to # Complaints to LC1 # Complaints to # Complaints to # Complaints to Officials Chairperson Subcounty Officials District Officials Inspectorate of Government Mean in treatment Mean in control Finally, no impacts are observed on indicators capturing community interactions with district officials, as well as satisfaction and trust in public officials and local leaders. Figure 9 and Table 10 present the results on community interactions with officials, both technical officers (such as district veterinarians, engineers or NUSAF2 desk officer) and local leaders. Some indicators capture whether payments were made to local officials. When payments are reported, the data does not allow distinguishing whether the payments were legitimate (such as when communities 41 pay district veterinarians for medicine) or illegitimate (such as bribes). Other indicators capture whether communities were satisfied with the quality of service they delivered, as well as on trust in local leaders. Figure 10 and Table 11 present impacts from the SACM intervention on trust in local leaders and public officials. No impacts are observed on any of the indicators on payments made to officials and satisfaction with these officials, suggesting that the training did not affect communities’ interactions with officials, although it did improve the quality and quantity of goods and services delivered to the communities. Trust in public officials in the local and central government is unaffected by the intervention. However, SACM communities report lower trust in community sub-project leaders. Figure 9: Impacts on Intermediary Outcomes: Payments and Satisfaction with Officials 1.00 2.45 0.80 2.40 0.60 0.40 2.35 0.20 2.30 0.00 Payment was made to Payment was made to 2.25 district official district officer (vet, Satisfaction with NDO Satisfaction with engineer Index Disctrict Vet Index Mean in treatment Mean in control Mean in treatment Mean in control Figure 10: Impacts on Intermediary Outcomes: Trust in Local Leaders and Public Officials 4.00 3.50 *** 3.00 2.50 2.00 1.50 1.00 0.50 0.00 Sub-project LC3 Subcounty LC5 District Central Leaders Chairperson Officials Chairperson Officials Government Mean in treatment Mean in control 42 6. Conclusion This report presented the main results from the social accountability and community monitoring (SACM) intervention impact evaluation implemented as part of the TAAC sub- component of NUSAF2. The impact evaluation results show that the social accountability and community monitoring (SACM) intervention improved community project quality by a small but significant magnitude amounting to 0.12 standard deviations. This overall improvement in community sub-projects is driven by a mix of increases in the quality and quantity of outputs. For livestock sub-projects – the most common type of sub-projects in the sample, the social accountability and community monitoring intervention led communities to have healthier animals, as well as reduced by 25% (from 17.8% to 13.1%) the livestock that could not be found in the communities at the time of the follow-up sub-project assessment. The main results are similar for the standard social accountability and community monitoring intervention and a treatment arm including additional incentives. The study analyzes the main channels through which the social accountability and community monitoring intervention led to improvements in community sub-projects. It shows that stronger community monitoring is the main channel through which the intervention was effective. Community members were more engaged in ensuring they were receiving quality project outputs, and more likely to report issues to local, sub-county and district officials, as well as to the Inspectorate of Government. Some spill-overs from the community monitoring were also documented, as complaints from community members about government services others than NUSAF2 also increased. On the other hand, no impacts were observed on the process used by communities to procure goods, services and materials, or on interactions between communities and public officials, although beneficiaries report lower trust in community sub- project leaders. The table below summarizes the main findings for each impact evaluation question. 43 Summary of Main Results by Impact Evaluation Question Main Study Question Main Findings 1. Is the social accountability and community Yes. The SACM intervention has statistically monitoring (SACM) intervention effective in significant impacts on an overall index of improving the quality of community projects? community project quality. Results suggest that communities get a mix of additional and better-quality outputs from their NUSAF2 projects. The magnitude of the impacts are not very large, however. 2. Are additional incentives effective beyond the No, the additional incentives do not have standard SACM model? impacts beyond the standard SACM model. 3. What are the mechanisms that are more likely to explain the observed impacts on project quality: i) changes in the way communities procure No impacts are observed on intermediary goods and services? outcomes that capture how communities procure goods and services. ii) more intensive community monitoring and This is the most likely mechanism. Impacts additional complaints to public officials? are observed on the intensity of community monitoring, as well as on the number of complaints made by community members to public officials. iii) changes in interactions with local leaders and No impacts are observed on intermediary public officials, as well as satisfaction and trust outcomes that capture interactions between in these leaders? communities and local officials (aside from a decrease in trust for community sub-project leaders). Results from the impact evaluation of the TAAC SACM intervention highlights the promise of a novel approach to social accountability. The program is of a significantly larger size and geographic spread than has previously been experimentally tested, which is likely to make the results more generalizable. The trainings are also of greater intensity than has previously been 44 attempted and implemented in a way that is more likely to be scaled-up by governments around the world. As such, the results have the potential to get researchers and policy makers to consider how bottom-up accountability can lead to real changes in national community-based development programs. An important caveat is that the cost to benefit ratio of the intervention is not fully established, and would be hard to ascertain without being able to fully take into account potential spill- overs. A key question relates to the value-for-money of the SACM intervention itself. Due to challenges in contracting, the scope of the training, and the fact that the implementing NGO/CSO was a consortium of various partners, the program was relatively expensive to implement. The cost of the training per community is estimated at approximately $900-$1000. This can be compared to the overall cost of the community projects: a majority of livestock projects were given 12,000,000 USH (approximately $3,636), which is also the average amount received per community. As such, the SACM training would need to increase the quantity or quality of outputs delivered to communities by close to 1/4 of the total value of community projects for direct SACM impacts on NUSAF2 projects alone to justify the cost of the program. The magnitude of the impacts documented in this report are smaller. Still, the increase in the quantity of outputs delivered to communities is not the only benefit from the program. Indeed, the training also had positive spill-over effects by strengthening monitoring of other government programs and services, which would be very challenging to quantify. The program may also have led to valuable individual-level changes on beneficiaries’ empowerment or attitudes. In any case, finding ways to reduce the cost of the training would be needed to ensure its cost-effectiveness. It is likely that the same training could be delivered at a significantly lower cost, in particular as the delivery of the training for the first time at a large scale induced some fixed costs and inefficiencies. This report also points to a set of questions that would deserve additional research. If it was possible to predict where the program is most likely to work best, and for which types of sub- projects, it could be targeted and the overall impacts could be larger, further contributing to better cost-effectiveness. Future research will explore this question, as well as heterogeneity in program impacts by local characteristics or local program density. Additional analysis beyond the scope of this report would also be required to assess impacts on individual beneficiaries’ assets. 45 Lessons from implementation of the NUSAF2 TAAC sub-component and SACM intervention have already had broader policy influence. They informed the design of NUSAF3 TAAC sub- component. For instance, in line with the approach in the SACM training, the Community Monitoring Groups (CMGs) have replaced the Social Accountability Committees (SACs) and have been given an expanded mandate. At the same time, training of these communities will build upon the SACM curriculum. In addition, the experience of the TAAC sub-component has already been emulated in social protection programs in Liberia and Sierra Leone. The NUSAF2 TAAC sub- component is also mentioned as an example in the World Bank new strategic framework for mainstreaming citizen engagement (World Bank, 2014). 46 References BAIRD, S., B. OZLER, C. McIntosh, (2011). The regressive demands of demand-driven development, The World Bank. BANERJEE, A., GREEN, D., GREEN, J. & PANDE, R. 2010. Can voters be primed to choose better legislators? Experimental evidence from rural India. BANERJEE, A., S. KUMAR, R. PANDE & SU, F. 2011. Do Informed Voters Make Better Choices? Experimental Evidence from Urban India. BARR, A. and D. SERRA (2009). "The effects of externalities and framing on bribery in a petty corruption experiment." Experimental Economics 12(4): 488-503. BARR, A. and A. ZEITLIN. 2011. Conflict of interest as a barrier to local accountability. CSAE working paper, Oxford University. BERTRAND, M., DJANKOV, S., HANNA, R. & MULLAINATHAN, S. 2007. Obtaining a driver's license in India: An experimental approach to studying corruption. Quarterly Journal of Economics, 122, 1639-1676. BHARGAVA, V. E. 2006. Global Issues for Global Citizens: An Introduction to Key Development Challenges, Washington, D.C.: World Bank. BJORKMAN, M. & SVENSSON, J. 2009. Power to the People: Evidence from a Randomized Field Experiment on Community-Based Monitoring in Uganda. Quarterly Journal of Economics, 124, 735-769. BJORKMAN, M. & SVENSSON, J., 2010. "When is community-based monitoring effective? Evidence from a randomized experiment in primary health in Uganda." Journal of the European Economic Association 8(2-3): 571-581. 47 DEININGER, K. and P. MPUGA, 2004. Does greater accountability improve the quality of delivery of public services? Evidence from Uganda. World Bank Policy Research Working Paper. 3277. DEVARAJAN, S., S. KHEMANI, and M. WALTON. (2011). Civil society, public action and accountability in Africa, The World Bank. DJANKOV, S., LA PORTA, R., LOPEZ-DE-SILANES, F. & SHLEIFER, A. 2010. Disclosure by Politicians. American Economic Journal: Applied Economics, 2, 179-209. FERRAZ, C. & FINAN, F. 2008. Exposing Corrupt Politicians: The Effects of Brazil's Publicly Released Audits on Electoral Outcomes. The Quarterly Journal of Economics, 123, 703-745. FISZBEIN, A, D. RINGOLD and H. ROGERS (2009). Making Services Work; Indicators, Assessments, and Benchmarking of the Quality and Governance of Public Service Delivery in the Human Development Sectors. Mimeo, the World Bank, Washington DC. HUNT, J. 2007. How Corruption Hits People When They Are Down. Journal of Development Economics, 84, 574-589. IG, 2014a. Inspectorate of Government Report to Parliament (July – December, 2014). Inspectorate of Government, Kampala. IG, 2014b. Social Accountability and Community Monitoring Training Curriculum. Inspectorate of Government, Kampala. KOZIOL, M. and C. TOLMIE (2010). Using Public Expenditure Tracking Surveys to Monitor Projects and Small-Scale Programs. Washington DC, World Bank. LITSCHIG, S. and Y. ZAMBONI (2011). Audit risk and rent extraction: Evidence from a randomized evaluation in Brazil, Department of Economics and Business, Universitat Pompeu Fabra. MOLINA, E., 2014. “Can Bottom-Up Institutional Reform Improve Service Delivery?” IDB Working Paper Series No. IDB-WP-513. 48 OLKEN, B. & PANDE, R. 2012. Corruption in Developing Countries. Annual Review of Economics, 4. OLKEN, B. A. 2006. Corruption and the costs of redistribution: Micro evidence from Indonesia. Journal of Public Economics, 90, 853-870. FIALA, N. and P. Premand. 2016. "The Impact of Social Accountability on Service Delivery: Experimental Evidence from a Large-scale Community-driven Development Program in Uganda." AEA RCT Registry. REINIKKA, R. & SVENSSON, J. 2004. Local Capture: Evidence from a Central Government Transfer Program in Uganda. The Quarterly Journal of Economics, 119, 679-705. REINIKKA, R. & SVENSSON, J. 2006. "Using Micro-Surveys to Measure and Explain Corruption." World Development 34(2): 359-370. REINIKKA, R. & SVENSSON, J. 2011. "The power of information in public services: Evidence from education in Uganda." Journal of Public Economics 95(7-8): 956-966. RINGOLD, D., A. HOLLA, M. KOZIOL, S. SRINAVASAN, 2012. “Citizens and Service Delivery, Assessing the Use of Social Accountability Approaches in the Human Development Sectors.” World Bank, Washington DC. ROSE-ACKERMAN, S. 2004. Governance and Corruption. In: LOMBORG, B. (ed.) Global crises, global solutions. Cambridge; New York and Melbourne: Cambridge University Press. RUBIO, G. M. 2011. Measuring governance and service delivery in safety net programs, World Bank Social Protection Working Paper Series, Washing ton DC. SERRA, D., 2009. "Combining Top-down and Bottom-up Accountability: Evidence from a Bribery Experiment", Centre for the Study of African Economies, University of Oxford. 49 SERRA, D., A. BARR, T. PACKARD, 2011. Education outcomes, school governance and parents' demand for accountability: evidence from Albania, World Bank Policy Research Working Paper Series, Washington DC. VAN STOLK, C. and E. TESLIUC, 2010. Toolkit on Tackling Error, Fraud and Corruption in Social Protection Programs. World Bank Social Protection Working Paper Series, Washing ton DC. WOO, J.-Y. 2010. The Impact of Corruption on a Country's FDI Attractiveness: A Panel Data Analysis, 1984-2004. Journal of International and Area Studies, 17, 71-91. WORLD BANK. 2003. World Development Report 2004: Making Services Work for Poor People. New York: Oxford University Press; Washington, DC: World Bank WORLD BANK, 2014, Strategic Framework for Mainstreaming Citizen Engagement in World Bank Group Operations, Engaging with Citizens to Improve Results. WORLD BANK, 2015, Uganda Systematic Country Diagnostic, Boosting Inclusive Growth and Accelerating Poverty Reduction. World Bank, Washington DC. 50 Tables Table 2: Baseline Balance (1) (2) (3) (4) (5) (6) (7) (8) Fencing Project start date Sub-project Livestock Enterprise project Roads Tree project Staff House (When grants were VARIABLES budget project (0/1) project (0/1) (0/1) project (0/1) (0/1) project (0/1) received) Treatment==SAC 0.336 -0.010 0.003 -0.001 0.012 -0.014 0.009 0.060 [0.526] [0.029] [0.018] [0.009] [0.015] [0.023] [0.012] [0.361] Treatment==SAC Plus -0.071 0.016 0.006 -0.010 0.011 -0.028 0.005 -0.018 [0.528] [0.030] [0.018] [0.009] [0.015] [0.023] [0.012] [0.359] Observations 895 895 895 895 895 895 895 812 R-squared 0.976 0.625 0.638 0.727 0.421 0.385 0.737 0.357 Mean in Control 20.549 0.729 0.079 0.031 0.031 0.093 0.038 38.380 p-value for equality of treatments 0.433 0.385 0.880 0.332 0.952 0.524 0.763 0.827 Standard errors in brackets Controls include sub-county and project type dummies *** p<0.01, ** p<0.05, * p<0.10 51 Table 3: Impacts on Main Outcomes (All Project Types) (1) (2) (3) (4) (5) Sub-project quality Sub-project Overall score Sub-project quantity Potential Problem Project VARIABLES Score (standardized) (standardized) score (standardized) Project implementation score Treated 0.119* 0.094 0.059 -0.003 0.034 [0.071] [0.075] [0.049] [0.012] [0.075] Observations 864 864 855 895 710 R-squared 0.354 0.374 0.419 0.28 0.411 Mean in Control -0.026 0.01 -0.041 0.027 2.377 Standard errors in brackets Controls include sub-county and project type dummies *** p<0.01, ** p<0.05, * p<0.10 52 Table 4: Impacts on Main Outcomes (Livestock Projects Only) (1) (2) (3) (4) (5) Sub-project Sub-project quality Sub-project Project Overall Score score quantity score Potential Problem implementation VARIABLES (standardized) (standardized) (standardized) Project score Treated 0.101 0.105 0.017 -0.007 0.001 [0.090] [0.087] [0.055] [0.011] [0.074] Observations 606 614 606 635 589 R-squared 0.375 0.400 0.441 0.336 0.425 Mean in 0.027 0.023 0.004 0.028 2.308 Control Standard errors in brackets Controls include sub-county and project type dummies *** p<0.01, ** p<0.05, * p<0.10 53 Table 5: Impacts on Main Outcomes (Livestock Projects Only, Beneficiary-level indicators) (1) (2) (3) (4) Sub-project Sub-project quality Sub-project Sub-project health Overall Score score quantity score score VARIABLES (standardized) (standardized) (standardized) (standardized) Treated 0.114 0.120 -0.017 0.109* [0.079] [0.081] [0.040] [0.058] Observations 2,788 2,798 2,783 2,811 R-squared 0.391 0.389 0.495 0.230 Mean in Control -0.016 0.015 -0.03 -0.001 Standard errors in brackets Controls include sub-county and project type dummies *** p<0.01, ** p<0.05, * p<0.10 54 Table 6: Component of Main Project Score (Livestock Projects only, Beneficiary-level Indicators) (1) (2) (3) (4) (5) Fraction Of Animal Is An Animal Health By Animals Not Animal Bought At The Improved/Crossed/Hybrid Animal Is Productive Mean Number Of Observed VARIABLES Correct Age (0/1) Breed (0/1) (0/1) Illnesses (Dead/Stolen/Sold) Treated 0.024 0.018 -0.013 0.027* -0.047** [0.029] [0.011] [0.027] [0.015] [0.019] Observations 2,814 2,812 2,645 2,645 2,814 R-squared 0.271 0.761 0.234 0.187 0.178 Mean in control 0.345 0.167 0.397 0.859 0.336 Standard errors in brackets Controls include sub-county and project type dummies *** p<0.01, ** p<0.05, * p<0.10 55 Table 7: Main Intermediary Outcomes, Procurement & Contracting (1) (2) (3) (4) (5) Challenges in Hired a Index of Satisfaction Procurement Satisfaction Contractor to challenges in with Process with supplier Implement Contracting contractor VARIABLES Index Index Project Process Index Treated -0.056 0.136 -0.001 -0.123 -0.004 [0.083] [0.134] [0.031] [0.291] [0.231] Observations 742 686 833 307 501 R-squared 0.273 0.362 0.619 0.554 0.432 Mean in 2.046 5.544 0.41 3.623 5.273 control Standard errors in brackets Controls include sub-county and project type dummies *** p<0.01, ** p<0.05, * p<0.10 56 Table 8: Main Intermediary Outcomes, Community Monitoring and Complaints about NUSAF2 sub-projects (1) (2) (3) (4) (5) (6) (7) Index for Intensity Index for Intensity of Social of Project Accountability Total # # Complaints # Complaints # Complaints # Complaints to Community Committee Project Complaints to to LC1 to Sub-county to District Inspectorate of VARIABLES Monitoring Monitoring Officials Chairperson Officials Officials Government Treated 0.155* 0.245*** 0.300*** 0.075** 0.055* 0.059** 0.117*** [0.086] [0.067] [0.102] [0.031] [0.030] [0.027] [0.027] Observations 855 907 6,002 6,581 6,445 6,328 6,286 R-squared 0.486 0.434 0.230 0.171 0.177 0.181 0.205 Mean in 2.767 0.628 1.929 0.74 0.611 0.422 0.237 control Standard errors in brackets Controls include sub-county and project type dummies *** p<0.01, ** p<0.05, * p<0.10 57 Table 9: Spillovers in Community Monitoring, Complaints about Non-NUSAF2 Government Services (1) (2) (3) (4) (5) # Complaints to Total # Complaints # Complaints to LC1 # Complaints to sub- # Complaints to Inspectorate of VARIABLES to Officials Chairperson county Officials District Officials Government Treated 0.174** 0.095*** 0.088*** 0.038** 0.054*** [0.074] [0.032] [0.026] [0.018] [0.018] Mean in 0.872 0.461 0.317 0.158 0.154 control Standard errors in brackets Controls include sub-county and project type dummies *** p<0.01, ** p<0.05, * p<0.10 58 Table 10: Main Intermediary Outcomes, Payment and Satisfaction with Officials (1) (2) (3) (4) Payment Payment was made was made to district Satisfaction Satisfaction to district officer (vet, with NDO with District VARIABLES official engineer Index Vet Index Treated -0.003 -0.013 -0.040 0.093 [0.027] [0.028] [0.122] [0.141] Observations 871 871 881 572 R-squared 0.347 0.343 0.332 0.429 Mean in 0.141 0.166 2.377 2.308 control Standard errors in brackets Controls include sub-county and project type dummies *** p<0.01, ** p<0.05, * p<0.10 59 Table 11: Main Intermediary Outcomes, Trust in Local Leaders and Public officials (1) (2) (3) (4) (5) (6) Sub-project LC3 Sub-county LC5 District Central VARIABLES Leaders Chairperson Bureaucrats Chairperson Bureaucrats Government Treated -0.099*** -0.042 -0.026 -0.037 -0.037 0.026 [0.028] [0.034] [0.029] [0.038] [0.033] [0.021] 7292 7278 7260 7233 7271 7273 Observations R-squared 0.115 0.129 0.104 0.096 0.097 0.102 Mean in 3.651 3.199 3.309 3.027 3.322 3.62 control Standard errors in brackets Controls include sub-county and project type dummies *** p<0.01, ** p<0.05, * p<0.10 60 Annex: Components of the main outcomes indices The table below summarizes which variables enter the main outcomes indices, separately for the quantity indices (first column) and quality indices (second column). The overall indices aggregates both the quality and quantity indices, and as such includes the variables in the first and second columns. The variables entering the indices are presented separately by sub-project types. When analyzing results across sub-project types, sub-project indices are normalized before being aggregated together. Sub-Project Type Quantity Indicator(s) Quality Indicator(s) Livestock Total number of animals the subproject received Age when animals were received (as recalled by beneficiaries). Age of observed animals. Caluculated by examining the animals teeth. The breed of the animal. Either local or crossed/imporved/hybrid. Indicates that an oxen is ploughing Indicates that a female cow has given birth Indicates that a breeding bull has bred Cow is pregnant Goat is pregnant Female animal is lactating female animal is plough (for Karamoja) Weight of cow measured using a tape around the chest Weight of goats measured using a tape around the chest. Indicates wounded skin Indicates scarred skin Indicates swollen skin Indicates abnormal discharges from the ears. Indicates abnormal discharges from the eyes. Indicates abnormal discharges from the mouth. Indicates abnormal discharges from the nostrils. Indicates abnormal discharges from the rectum. Indicates abnormal discharges from the vulva. Indicates abnormal discharges from the udder. Indicates swollen lymph nodes Indicates limping Indicates lameness Indicates fractures Indicates blindness Indicates bloating Indicates swollen joints Indicates diarrhorea Indicates slow dullness Indicates slow breathing Indicates rapid breathing Indicates reduced appetite Indicates coughing Indicates parasite infestation. We consider an animal ill if it is severly infested. Indicates the temperament of the animals. We consider an animal ill if it has an aggressive or dull temperament. Indicates inflamed udder. Indicates issues with the teats. We consider an animal to be ill if the teats are not functional, small, missing, swollen or with soares Indicates the state of the testis. We consider an animal to be ill if the testis are not functional, small or there is only one (uncastrated animals only). Staff House Length of staff house Indicates walls are satisfactory. Width of staff house Indicates roof is satisfactory. Height of front wall of staff house 61 Indicates windows are satisfactory. Height of rear wall of staff house Indicates doors are satisfactory. Indicates ceiling is satisfactory. Indicates floor is satisfactory. Indicates painting satisfactory. Indicates inflamed udder. Indicates issues with the teats. We consider an animal to be ill if the teats are not functional, small, missing, swollen or with soares Indicates the state of the testis. We consider an animal to be ill if the testis are not functional, small or there is only one (uncastrated animals only). Staff House Length of staff house Indicates walls are satisfactory. Width of staff house Indicates roof is satisfactory. Height of front wall of staff house Indicates windows are satisfactory. Height of rear wall of staff house Indicates doors are satisfactory. Indicates ceiling is satisfactory. Indicates floor is satisfactory. Indicates painting satisfactory. Enterprise Number of people currently engaged in the enterprise Current access to materials Current access to transportation Current access to credit Current access to skilled labour Current access to markets How successful they feel their enterprise is. Fencing Length of the fence. Roads geopoints for either end of the road and any Material the road is made from. corners. With this we can calculate the length of the road Width of the road. (Measured in 3 places) Tree Planting Size of the land used. Indicates whether the seeds/seedlings bought were certified. Number of seedlings planted. Number of seeds planted. 62