93422 Process Evaluation of the Program Nasional Pemberdayaan Masyarakat (PNPM)- Urban November, 2011 OPEN PUBLICATION Acknowledgements The authors would like to thank the management and field staff of SurveyMeter, notably Bondan Sikoki, Wayan Suriastini and Dani Alfah , as well as Cynthia Clapp-Wincek, Blane Lewis, Victoria Beard, Juliana Wilson and Bonnie Walter. John Strauss facilitated work with the IFLS data, while Arie Kapteyn, Krishna Kumar, Yashodhara Rana and Lisa Miyashiro provided important feedback and input into the study. From the World Bank we thank George Soraya, Evi Hermisari and Yuliy Widyawati, as well as Judy Baker, Fatima Shah and Vivi Alatas. All errors remaining are our own. 2 Glossary Abbreviation Bahasa English AD/ART Anggaran Dasar/Anggaran Budget /Household Budget Rumah Tangga ANDAL Analisis Dampak Lingkungan Environmental Impact Assessment APBD Anggaran Pendapatan dan District Income and Spending Belanja Daerah Budget APBN Anggaran Pendapatan dan National Income and Spending Belanja Negara Budget Bangda Pembangunan Daerah Regional Development BAPPD Berita Acara Penarikan dan Penggunaan Dana Bappekot/kab Badan Perencanaan City/District Development Pembangunan Kota/Kabupaten Planning Body Bappeprop Badan Perencanaan Provincial Development Planning Pembangunan Propinsi Body Bappenas Badan Perencanaan National Development Planning Pembangunan Nasional Body BI Bank Indonesia Bank of Indonesia BKKBN Badan Koordinasi Keluarga The National Coordinating Body Berencana Nasional for Family Planning BKM Badan Keswadayaan Community Voluntary Masyarakat Contribution Committee (now LKM) BLM Bantuan Langsung Masyarakat Community Block Grants BOP Biaya Operasional Operational Funds BPD Badan Perwakilan Desa Village Representative Body BPKP Badan Pemeriksa Keuangan dan State Audit Agency Pembangunan CBD Community Based Development CDD Community Driven Development CSS Pemetaan Swadaya Community Self Survey DED Detailed Engineering Design DIPA Daftar Isian Pelaksanaan Detailed Budget Document that Anggaran authorizes the release of funds – at Central and District levels DJP Direktorat Jenderal Director General of the Treasury, Perbendaharaan, Departemen Ministry of Finance Keuangan DKT Diskusi Kelompok Terarah Focus Group Discussion Dokumen SPK-D Dokument Strategi Strategic Document on Poverty Penanggulangan Kemiskinan di Alleviation Daerah Departemen Ministry of Finance Abbreviation Bahasa English Keuangan DPRD Dewan Perwakilan Rakyat Regional Representative Body Daerah DPT Diskusi Partisipatif Terpadu Unified Participatory Discussion Fasilitator Tenaga Pengembangan P2KP Community Development Masyarakat P2KP Staff FKA-BKM Forum Komunikasi Antar BKM Inter-BKM Communication Forum Tingkat Kota/Kabupaten FMR Financial Management Report KA Konsultan Advisory Advisory Consultation KBK Komunitas Belajar Kelurahan Kelurahan Study Group KBP Komunitas Belajar Perkotaan City Study Group KDP Program Pengembangan Kecamatan Development Projec Kecamatan KE Konsultan Evaluasi Evaluation Consultant KMP Konsultan Manajemen Pusat Central Management Consultant KMW Konsultan Manajemen Wilayah Regional Management Consultant Korkot Koordinator Kota, KMW City Coordinator KPK-D Komite Penanggulangan Regional Poverty Alleviation Kemiskinan di Daerah Committee (at Provincial and City/District level) KSM Kelompok Swadaya Masyarakat Community Voluntary Contribution Group LKM Lembaga Keswadayaan The new name for BKM Masyarakat, merupakan nama generik yang dahulu dinamakan BKM LKMD Lembaga Ketahanan Organization Masyarakat Desa LSM Lembaga Swadaya Masyarakat NGO Musrenbang Musyawarah Perencanaan Annual local government Pembangunan development planning meeting and discussion process ND Pembangunan Lingkungan Neighborhood Development Permukiman Kelurahan Kelurahan Settlement P2KP Proyek Penanggulangan UPP Kemiskinan di Perkotaan PAKET/PAPG Penanggulangan Kemiskinan Poverty Alleviation Partnership Terpadu, sama dengan PAPG Grant (PAPG) PBL Penataan Bangunan dan Environment and Development Lingkungan Regulation PDAM Perusahaan Daerah Air Minum Water Company PDMDKE Pemberdayaan Daerah dalam Empowering the Region in 4 Abbreviation Bahasa English Mengatasi Dampak Krisis Mitigating the Effects of Economic Ekonomi Crisis PJM Perencanaan Jangka Menengah Community Development Plan PJOK Penanggung Jawab Operasional Project Manager at Kecamatan Kegiatan Level Pej.PK Pejabat Pembuat Komitment The Office of Commitments PKK Pembinaan Kesejahteraan Family Welfare Keluarga PMU Program Management Unit Program Management Unit PODES Potensi Desa Village Potential Survey POM Project Operational Manual Project Operational Manual PPM Penanganan Pengaduan Community Complaints Masyarakat Mechanism Pronangkis Program Penanggulangan Poverty Alleviation Program Kemiskinan PRA Participatory Rural Appraisal Participatory Rural Appraisal PS Pemetaan Swadaya Voluntary Contribution Map PU Pekerjaan Umum Public Works Relawan tanpa Volunteer pamrih Renta Rencana Tahunan Annual Planning Document RK Refleksi Kemiskinan Poverty Reflection RKL Rencana Pengelolaan Environmental Protection Plan Lingkungan RKM Rembug Kesiapan Masyarakat Community Preparedness Meeting RPJM Rencana Pembangunan Jangka Mid-Term Development Plan Menengah RT/RW Rukun Tetangga/Rukun Warga Rukun Tetangga/Rukun Warga RWT Rembug Warga Tahunan Annual Residents Meeting SA Special Account (Rekening Special Account (Rekening Khusus) Khusus) SATKER-P2KP Satuan Kerja Program Working Unit on Poverty Penanggulangan Kemiskinan di Alleviation Programs in Urban Perkotaan areas SIM Sistem Informasi Manajemen Management Information System SKPD Satuan Kerja Perangkat Daerah Coordinating body on developmemt planning at the District/City level SKS Satuan Kerja Sementara Temporary Working Unit SNVT Satuan Kerja Non Vertikal di Non-Vertical Working Unit tingkat Propinsi Provincial Level SP2D Surat Perintah Pencairan Dana SPK Strategi Penanggulangan Poverty Alleviation Strategy 5 Abbreviation Bahasa English Kemisikinan SPKD Strategi Penanggulangan Regional (District or City) Poverty Kemiskinan Daerah Alleviation Strategy TA Technical Assistance Tim Interdept di Tim Pengarah dan Kelompok National level interdepartmental Tingkat Nasional Kerja Antar Departemen team Terkait TKPKD Tim Koordinasi Regional Poverty Reduction Penanggulangan Kemiskinan Coordinating Team Daerah TKPP Tim Koordinasi Pelaksanaan P2KP Implementation P2KP (Kota/Kabupaten) Coordinating Team TNA Training Needs Assessment Tridaya Pemberdayaan Lingkungan, Infrastructure, Social and Pemberdayaan Sosial dan Economic Empowerment Pemberdayaan Ekonomi TOR Terms of Reference Terms of Reference TOT Training of Trainers Training of Trainers UKL Unit Kelola Lingkungan Infrastructure Management Unit UMR Upah Minimum Regional Regional Minimum Wage UP Unit Pengelola yang dibentuk Management Unit created by BKM BKM UPK Unit Pengelola Keuangan Financial Management Unit UPL Unit Pengelola Lingkungan Infrastructure Management Unit UPS Unit Pengelola Sosial Social Management Unit UPP Urban Poverty Project (P2KP) Urban Poverty Project (P2KP) … 6 1 Introduction This report is a process evaluation of the Program Nasional Pemberdayaana Masyarakat- Urban (PNPM-Urban). As the Government of Indonesia’s leading (and now virtually universal) community-driven development (CDD) urban poverty program, the Program Nasional Pemberdayaan Masyarakat-Urban (PNPM-Urban) has evolved substantially since its initial incarnation as the Urban Poverty Program in 1999. With an estimated allocation of USD 1.7 billion per year, provided from national and local budgets, community contributions, and external support from the World Bank and other donors, PNPM-Urban's overall objective is to ensure that the urban poor in participating kelurahan benefit from improved socio- economic and local governance conditions through: (a) the formation and institutionalization of elected representative organizations (Badan Keswadayaan Masyarakat or BKM, now known as the LKM) at the kelurahan (or urban ward) level that are accountable to communities; (b) provision of block grants to communities directly and transparently to finance poverty alleviation activities related to infrastructure, social and economic objectives; and (c) enhancing the capacity of central and local governments to partner with community organizations in service provision.1 The World Bank has provided financial and technical support to the program since 1999, with the majority of funds supporting small-scale infrastructure construction and maintenance. This process evaluation focuses largely on documenting the implementation of activities under the current phase of PNPM-Urban as well as under the pilot Neighborhood Development (ND) scheme undertaken as part of PNPM-Urban, and examines to what extent project objectives are being attained, as well as best practices and lessons learned for the future. The report proceeds as follows. In the first section, we review the history of CDD as well as the genesis and current structure of the PNPM-Urban program. Next, we describe the methodology used and the conduct of the evaluation. We then present the evaluation findings. The report concludes with a summary and implications focusing on process improvements and overall program design. Instruments, protocols and detailed descriptions of the methods used are found in the Annexes, which also include survey results and an illustrative analysis of a subsample of data from the Management Information System. 1 We use the Bahasa Indonesia terms for administrative geographies throughout, including kelurahan, kecamatan, and kota. Some of the sites in our sample used “BKM” while others, where PNPM -Urban had started more recently, used “LKM.” To promote readability and simplicity, we will use “BKM” throughout Additionally, we use the term kelurahan and desa interchangeable. Although some sites in our sample are desa, we will use kelurahan throughout for simplicity. 2 Background 2.1 What is The Community-Driven Development Approach? What distinguishes a CDD program from other models of community empowerment? After decolonization, many developing countries adopted a centralized approach to policymaking. However, in the 1950s, realizing that endemic rural poverty in areas with limited access would be difficult to resolve under such an approach, some countries adopted rural development programs with a community approach (such as the Community Development Program in India or the Comilla rural development program in Bangladesh). Such programs generally included local planning and institutional development but for technical and political reasons were then supplanted by more sectoral approaches in which line ministries or agencies focused on specific technologies or services. Beginning in the 1970s, the World Bank and other donors initiated poverty-reduction programs that focused on local participation and development, starting with Area Development Projects (ADPs) and Integrated Rural Development Projects (IRDPs). In the 1980s, such programs evolved from a community consultation model (in which agencies operate as direct-service providers but consult communities about their needs and preferences) into a community participation model (in which communities take part in planning, co- financing, and operating program investments). The 1990s saw the growth of a third approach, the community empowerment model, on a large scale in a number of countries. This model goes further by shifting the locus of responsibility for managing project implementation and funding entirely to communities. Finally, the community-driven development (CDD) movement in development policy has seen the implementation of a model with even deeper engagement by the World Bank, among other development organizations, across a range of countries and sectors, including health, nutrition, infrastructure, and microenterprise support.2 What distinguishes the CDD approach is the role of the community in decision-making: communities identify unmet needs and potential solutions, set priorities, formulate and implement plans, and, in the long-term, develop a sense of ownership and accountability (Mansuri and Rao, 2004; Binswanger et al. 2010). CDD requires that communities engage politically and economically with the development process, but also that local and central government adapt to this through appropriate decentralization, adjusted management, control bodies, and frameworks. CDD programs thus typically call for building management and technical capacity on the part of all three actors while improving vertical and horizontal accountability (upward to governments and donors, downwards to users, and across the community itself). 2.3 CDD and Urban Poverty in Indonesia Participatory development programs in Indonesia can be traced back to the 1970s. At this time, the Kampung Improvement Project (KIP) aimed to upgrade physical infrastructure in various housing areas within major Indonesian cities, but it was essentially managed by the 2 Binswanger et al. (2010) provides a detailed account of the evolution of CDD for the interested reader. government, with community relationships conducted through the local Camat. Devas’ (1981) case study of the KIP found that the improvement programs tended to benefit the affluent relative to the poor. Moreover, based on data from Jakarta, the program appeared to stimulate house improvement/repair, but did not have any significant effect on such factors as residents’ income, health, or employment. Furthermore, infrastructure project maintenance was an issue. Notably, one subproject within the KIP seeking to adopt a much more comprehensive, socio- economic focus and an explicitly participatory approach encountered administrative and organizational delays. Nonetheless, the program was highly popular and had a high degree of political support. This was followed by GoI’s Inpres Desa Tertinggal (Program for Left- Behind Villages), a poverty-alleviation project consisting of block grants given to poor villages across the country, while the World Bank-funded, GoI-managed Village Infrastructure Project (VIP) provided villages with a one-time grant to build any of five types of public infrastructure. The history of CDD programs supported by the World Bank in Indonesia begins with the first phase of the Urban Poverty Program (UPP) in 1999 and its rural counterpart, the Kecamatan Development Program (KDP) in 1998. As many have observed, there are two important contextual facts to note about the genesis of the UPP and KDP. Firstly, the internal architectures of these programs were built around a pre-existing body of experience with community development project design. At the same time, the UPP and KDP were created with a sense of deep urgency against a backdrop of significant social and political upheaval, with the confluence of rising poverty following the East Asian financial crisis and a tide of political decentralization after the end of the Suharto regime. UPP was designed to promote the development of community-based organizations (BKM) at the kelurahan level to receive block grants and manage an open menu of activities in three domains (referred to collectively as tridaya): infrastructure, social protection, and economic activities. Only a small number of activities were excluded (such as the sale of tobacco), and for qualifying kelurahan, the menu also permitted communities to the setup a revolving loan fund (RLF) for microcredit loans, subject to some qualifications. Notably, all technical functions were outsourced to individual private consultants and firms. The program was introduced to 2621 kelurahan over the period of 1999-2004. UPP1 was conducted in only six provinces, comprising the densely populated urban corridor of northern Java, the kota (district/city) of Bandung, the special area of Yogyakarta, and the kota of Malang.3 In 2002, UPP2 was funded (with a second cycle in 2007, UPP2-AF), further extending the program first to 2059 and then 2,911 new kelurahan and increasing coverage to 13 provinces in Sulawesi, Kalimantan, NTB and the southern part of Java. A third stage of the program, UPP3 was begun in 2005 for 1,726 new kelurahan in an additional 15 provinces, covering large areas of Sumatra, East Kalimantan, NTT, the Moluccas and Irian Jaya. 4 During this time of expansion, the GoI was also able to use the UPP as a support platform for community-based housing reconstruction in Aceh following the tsunami of 2004, attracting funds from other donors. In addition, the multi-donor Java Reconstruction Fund has collaborated with UPP to support housing reconstruction for Yogyakarta and Java. The GoI 3 We use kota to refer to both “city” and “district” interchangeably. Some of the kelurahan in our sample reside in kabupaten, but for simplicity we refer to these using the term “kota.” 4 Selection criteria for participating sub-districts were the same as in UPP2 9 has also leveraged the UPP to deliver other similar programs such as the ADB’s Neighborhood Upgrading and Shelter Sector Project (NUSSP) for slum upgrading and post- disaster reconstruction. The slate of social protection programs have continued to evolve and expand, and new ones introduced, notably Health Insurance for the Poor (Askeskin, now Jamkesmas); unconditional cash transfers (BLT or Bantuan Langsung Tunai, provided in 2005 and again in 2008-9, both times to offset impacts of fuel price increases); conditional cash transfers (PKH or Hopeful Families Program, started in 2007) for very poor households, and several other programs directed at the extreme poor of particularly vulnerable populations such as the elderly and disabled.5 Overall expenditures on social assistance have increased significantly since 2005 and now account for about 0.5 percent of GDP. However, this is still well below the average for developing countries (1.5%) and for other countries in East Asia and the Pacific (1%). (World Bank 2011). In 2006, after ten years of experience with the KDP and UPP, as well as the prior history of the KIP and VIP before that, President Yudhoyono announced a national policy and operational umbrella for all community empowerment programs, Program Nasional Pemberdayaan Masyarakat (PNPM), which would become a flagship poverty alleviation program scaled up to all areas of the country. At present PNPM includes five programs: PNPM-Urban (formerly the UPP), PNPM-Rural (formerly the KDP); Support for Poor and Disadvantaged Areas (SPADA); PNPM Rural Infrastructure Support Program (MS-PNPM) or PPIP (Program Pembangunan Infrastruktur Perdesaan), and PNPM-Infrastructure for Socio- Economic Development (RISE). The initial PNPM-Urban began in 2008, extending the UPP model to a further 2090 kelurahan. A second wave of financing, PNPM-AF, allowed the program to scale-up and reach all urban kelurahan in the country. Since 2010 a new governmental body, TNP2K has been charged with developing and coordinating the country’s poverty reduction strategy, including increasing the focus on community-driven development (CDD) via the PNPM Mandiri (National Program for Community Empowerment). TNP2K uses a three part framework used to categorize programs and policies: Cluster 1 includes social protection programs such as RASKIN and Jamkesmas; Cluster 2 focuses on empowerment and includes primarily PNPM; and Cluster 3, which is much smaller than the previous two clusters in terms of resources, includes programs for increasing incomes in the longer term via credit for micro and small scale enterprises. This strategy includes the third wave of PNPM-Urban, PNPM-Urban III, initiated in 2010. 2.4 PNPM-Urban (UPP3) Program Description In this section, we review key features of PNPM-Urban in its current form. As shown below, there are three main components to PNPM-Urban: developing local government and community capacity for decision-making, providing block grants (Bantuan Langsung Masyarakat, or BLM) directly to communities, and executing projects with technical 5 See Sumarto and Bazzi (2011) for a comprehensive account of the evolution of social protection policy in Indonesia since 1998. 10 assistance from consultants and facilitators, supported by contributions from the local government and the community. Figure : Main Program Components Community and local Kelurahan Block Grants Implementation Support government capacity building • Community awareness and • Infrastructure: Specific • Technical support for socialization increased by priority investments e.g project implementation facilitators through social roads and competitive grants provided by professional intermediation activities and to community groups for consultants training specific infrastructure to • Local government staff services projects • Facilitators aid in accounting training • Social : Assistance to and monitoring practices particularly vulnerable social groups • Economic : For qualifying kelurahan, could also be used to provide microcredit via revolving loan funds In each kelurahan, project implementation takes place over a cycle of three calendar years. In Year 1, with the help of facilitators, the community undergoes a participatory socialization process beginning with awareness raising, a community self-survey and poverty mapping exercise, that culminates in the election of the all-volunteer BKM the formation of community self-help groups (Kelompok Swadaya Masyarakat, or KSM) and the formulation of an development plan (PJM Pronangkis, or Medium-Term Community Development Plan, CDP) to guide activities during the three year cycle. In the current PNPM-Urban, the CDP is required to be linked to city and local government as well as to explicitly develop goals that are linked to the UN Human Development Index and the Millennium Development Goals. For the remainder of the project cycle, Years 2 – 3, the BKM controls and administers the grant with the assistance of the facilitator. Every BKM forms a small team for project administration and a subproject implementation team. A financial management unit (Unit Pengelola Keuangan, or UPK) is established for accounting and record keeping. The BKM receives funding in three tranches every year for activities designed to meet the goals of the community development plans. The grants range from USD 15,000 to USD 30,000 (based on population size) per kelurahan and may be disbursed to the KSMs for individual project proposals that are consistent with the CDP, used for provision of social assistance (including grants and training programs), or for tertiary infrastructure investments pre-identified by the community during the CDP. The block grant also includes funds to finance overhead costs. Facilitators assist the BKMs and KSMs to prepare financial reports on grant and sub-grant implementation, and where RLFs (microcredit program) are in place, facilitators assist the UPK with revolving fund administration. At present, the program has begun to phase out of the RLF, placing instead a greater emphasis on infrastructure investments. 11 Figure : Administrative Structure of PNPM-Urba 12 Annual year-end residents’ meetings (Rembug Warga Tahunan, or RWT) and other appraisals are held, in which project reports are made public to hold the KSM and BKM accountable for past activities and to confirm the Renta (Annual Plan) for the following year. Community volunteers play an important role from initial planning to actual administration to the implementation and maintenance of projects. At the end of the three-year cycle, the BKM steps down and a new cycle of election and planning begins. After receiving grants three times under the PNPM-Urban, however, a kelurahan is expected to graduate to other “channeling” programs and is eligible for coordination funding to do so. PNPM-Urban promotes collaboration with local government through the Renta, which is presented to the kelurahan Musrenbang at the start of each year to be integrated or adopted into the local government’s planning. At each administrative level, the Ministry of Public Works works with the local government to form a coordination working group (Tim Kordinasi Pengelolaan Program) to provide information, technical support and to solve administrative problems. In addition, PNPM-Urban facilitators work with the BKMs to form associations (BKM Forums at the kecamatan and kota level) to work collectively and to influence government planning, while also building local government capacity to work with BKMs and the BKM Forum. The overall project is housed within the Ministry of Public Works, which is the executing agency. The program Steering Committee is composed of representatives from the Ministries of Public Works, Finance, Home affairs, People’s Welfare, Cooperation and Small Medium Enterprises, and Industry and Trade. It is chaired by the Coordinating Ministry of People’s Welfare, which also chairs the National Poverty Reduction Committee. Within the Ministry of Public Works, day-to-day administration is conducted by a center Project Management Unit (PMU) that reports to the Director General Cipta Karya (Director General of Human Settlements, or DGCK) and works through smaller administrative units or satkers at the central, provincial and local levels. The PMU hires two kinds of agents, consultants and facilitators, to assist in project implementation. The consultant team is organized as a hierarchy with a National Management Consultant (NMC) and a team of regional-level oversight consultants (OC), while the much larger team of facilitators is deployed at the local level. The provincial satkers manage the facilitators and korot/askot (see Figure 1). Finally, two important pilot initiatives added to the program since UPP2 are the Poverty Alleviation Partnership Grant (PAPG) and the Neighborhood Upgrading Schemes (NUS) or Neighborhood Development (ND). Under PAPG, for the 100 or so local governments selected, in years three to five of the project, a Regional Poverty Alleviation Strategy was created and additional matching funds for activities jointly proposed/cost-shared by the local government and the community is awarded to local governments. The program aims to make formal a consultative relationship between the local government and community and to finance activities that lie beyond the typical scope of PNPM-Urban, i.e., larger than the typical BLM allows, requiring local government contributions, or covering more than one kelurahan. A second (ongoing) initiative, called Neighborhood Development (ND) is a strategy that originated as the Neighborhood Upgrading Scheme, a program of slum upgrading. The ND program initially started with 18 pilot sites, with sites chosen based on good prior performance 13 on the part of the BKMs; it has since been expanded to 255 more sites. ND aims to expand the PNPM-Urban infrastructure activity, both in terms of level of resources and project scope, and to increase the involvement of local government. As with PAPG, ND is designed to produce one project that is larger than those under PNPM-Urban or produce a series of integrated projects. To promote collaboration between local government and community-lead initiatives to develop a participatory and sustainable program for settlement planning, the program expects that the investments will be made in coordination with (and with substantial buy-in from) local government, other government, and private sources. The core difference between ND and the rest of PNPM-Urban, however, is the element of spatial planning. The ND mission is focused on integrating the planning and management of community settlements into official development planning. While the grant amount is significantly larger (1 billion Rupiah) and is mainly for infrastructure that can contribute to economic or social development, a considerable fraction (30 percent) of the funding is allocated to planning (including hiring outside experts) and “marketing activities.” The purpose of the planning and marketing support is to increase the community involvement in the project and obtain cooperation and funds. All ND activities and grant funds must be focused on one priority area, which is selected based on several parameters, including the poverty rates and other measures of need. Unlike the PNPM-Urban, a first round of funds is immediately available. The selected sites receive Rp 200 million for socialization and hiring an urban planner who helps build a spatial plan for the community. The block grants are to be used in one or a series of integrated projects within the single priority area to meet the described needs for upgrading. After the initial planning process, the BKMs receive a second tranche of funds (Rp 400 million for construction and Rp 100 million for additional planning and marketing activities. The third tranche of funds consists of Rp 300 million to complete construction. The planning phase is led by the Planning Team or TIPP, which consists of the BKM, the hired urban planner, community leaders, the lurah and representatives from local government agencies such as Public Works and BAPPEDA. Implementation is carried out by an Implementation Team, which is formed by the members of the BKM in charge of infrastructure, the lurah, and community volunteers. 2.5 Objectives of PNPM-Urban We discuss the objectives of PNPM-Urban for the time frame ending in 2011, as grouped into a set of key indicators for the program’s outputs and outcomes, based on its components as shown below. Based on initial program appraisal documents, the three key outputs of the program are community empowerment, local government support and service provision, and local capacity building. Based on these program outputs, the overall outcomes to be achieved are better living conditions within an environment of improved local governance. With respect to community empowerment, PNPM-Urban seeks to include a significant fraction of poor and female households taking part in the planning and decision-making processes leading up to project implementation (40% each). In addition, although administration is delegated to the BKMs, the project seeks to mobilize broad-based participation in the election process, aiming to have 30% of the overall adult population participating in votes. The project also seeks to have a 30% rate of female participation in the KSMs. As an indicator of local government buy-in, the program aims to have the majority of 14 local governments providing cost-sharing, with target amounts based on their fiscal capacity. The program aims for the majority of kelurahan to have completed all three types of activities, and for most of the infrastructure works to be certified as good quality (using local government standards). Finally, the program also aims to develop local administrative capacity to the point where MIS data reporting is almost universal and a high fraction of BKM audits are complete. Performance indicators for the program outcomes are increased household expenditures / improved access to economic and social services in 80% of kelurahan, a 20% cost-savings on infrastructure and satisfaction from most beneficiaries. Figure : Key objectives for PNPM-Urban III, World Bank Improved socioeconomic living conditions and local governance  Increased household expenditures / improved access to economic and social services in 80% of kelurahan by 2011  Infrastructure built is 20% less expensive in 80% of kelurahan  80% satisfaction levels from beneficiaries regarding services and local governance Local government and services to Technical assistance Community Empowerment the poor and implementation support  40% poor household  80% of local governments  90% providing participation rate and provide cost sharing: 20% for timely and 40% female participation those with low fiscal capacity accurate data to rate in planning and and 50% for those with high MIS decision making fiscal capacity  70% of BKM  30% of adult population  80% of wards to have complete annual voting in BKM/ LKM infrastructure, economic and financial audits elections social activities completed  90% of kelurahan form a  70% of infrastructure works BKM/LKM certified as good quality  90% of kelurahan form and  Additional metrics for ratify CDP kelurahan with revolving loan  30% female participation funds rate in KSMs Source: PNPM III Project Action Document 3 Methodology 15 The evaluation team included researchers from RAND, a nonprofit research organization based in the United States; SurveyMETER, a research organization based in Jogjakarta, Indonesia and consultants Bonnie Walter, Juliana Wilson, Blane Lewis and Cynthia Clapp- Wincek. The process evaluation, including fieldwork and report-writing was carried out between March - October 2011. The main field work commenced from May 28 through June 25, 2011. 3.1. Study Objectives The evaluation aimed to answer a set of research questions that addressed the extent to which PNPM-Urban successfully delivered its key program outputs and its intended/unintended effects on individuals and institutions. This evaluation was designed to be responsive to specific questions that have been raised regarding the program, focusing primarily on overall governance issues and the infrastructure component of the program, although we discuss where possible the social and economic components as well. The broad evaluation questions posed to the evaluation team were as follows: 1. Assessing the role of CDD: The CDD approach has been attributed to a number of objectives including building social capital and improving living conditions. a. To what extent is the CDD model realized by PNPM-Urban, in terms of planning, voting and contribution to program activities? 2. Links with local government a. How does PNPM-Urban link with local government and with sectoral programs (details on types of programs, interactions, etc.)? b. Is there much collaboration / duplication? c. In communities where collaboration works well, what are the contributing factors? Similarly, where it doesn‘t work well, why is that? 3. Capacity building and effectiveness of the facilitators a. How effective are the current facilitators and do they adequately meet the needs of the communities? b. What characteristics contribute to strong facilitators versus weak? 4. Goernance and Control Mechanisms a. How effective are the current governance structure/ control mechanisms that 5. Quality of infrastructure projects: a. Overall how is the quality of the infrastructure investments? b. What factors contribute to high- and low-quality works? c. Are current standards adequate? d. How cost effective are infrastructure investments? e. are in place? Are they adequate or could they be improved? If so, how? 6. PAPG/ND and the adequacy of block grants: a. Currently communities receive relatively small grants. Under the Neighborhood Development (ND) scheme currently being piloted, the amount of those grants is substantially higher. How do experiences and outcomes vary under PAPG/ND? What are the differential impacts? 16 b. For communities that may not have capacity for implementing larger grants, what are some alternatives? In the text, we address Questions 1-6 as separate research themes, and in the conclusion return to the broader question of the specific role/s and objective/s of CDD in the urban context, and how effective it is in achieving these roles/objectives. 3.2. Methods For this process evaluation, the principal methods used were:  Literature Review With the assistance of the World Bank team, RAND conducted a literature review of numerous completed and ongoing related studies on urban poverty, CDD and PNPM-Urban. We reviewed evaluations of previous incarnations of both PNPM-Urban and PNPM-Rural. This process was also informed by consultation with experts on PNPM-Urban, CDD and local governance in Indonesia.  Formative Research In addition to background literature review, RAND staff conducted a pre-field visit, conducting interviews with PNPM-Urban and other government officials, including from Bappenas and the Ministries of Social Welfare, Home Affairs, and Public Works, and several local (kota) level officials. Staff also met extensively with World Bank Staff and the SurveyMETER team. Based on these meetings, and two pilot field tests, protocols were developed and revised.  Collection and Analysis of Qualitative Data and Rapid Surveys RAND conducted primary data collection in collaboration with SurveyMETER in 16 kelurahan across Indonesia (including three participating in the ND Program described in Section 2). Data gathering included key informant interviews, focus groups and in-depth interviews with poor households as well as PNPM-Urban beneficiaries, rapid surveys of households and PNPM-Urban community group leaders and infrastructure inspections by engineers.  Analysis of PNPM-Urban Administrative Data Finally, RAND conducted an assessment of the PNPM-Urban MIS data made available publicly through www.p2kp.org and internally through the World Bank Jakarta office. The complete MIS data include site-by-site indicators of community participation, budgeting and project information going back to program inception, as well as a database of complaints registered using the system; however, only a subset of the available data could be extracted, cleaned and merged in a format suitable for analysis. With considerable assistance from the World Bank Jakarta office, we constructed a smaller master dataset of validated data from the component parts, performed summary analysis and identified major issues with data outliers and anomalies, and key areas for improvement. 17 3.3. Detailed Description of Primary Data Collection 3.3.1 Site Selection Criteria for site-selection were determined, with discussion and input from the World Bank team, to be: broad geographical representation, variation in the level of poverty and experience with larger vs. smaller grants and Neighborhood Development (ND) projects. 13 of the 16 kelurahan were randomly chosen from the sample of urban communities in the Indonesian Family Life Survey.6 Regional stratification was performed based on discussions with World Bank teams in the U.S. and PNPM-Urban specialists at the World Bank in Jakarta, as well as other experts, so as to achieve an appropriate geographical representation, based both on urban population and on the allocation of PNPM-Urban funds across regions. Two “field test” sites were selected in Yogyakarta, the location of SurveyMETER. To stratify on wealth, a simple index was created out of the following variables from the IFLS community data: an indicator of whether the predominant road type in the kelurahan is cement/asphalt, the percentage of homes with electricity, an indicator of whether there is a sewage system in place, the percentage of homes that receive water from a pipe, and the share of households receiving the rice ration. These variables were standardized and summed to create the community wealth index.7 To choose the West and East Java sites, we categorized kelurahan8 into three levels of wealth based on the value of this index: less than or equal to the 33rd percentile of the national distribution of the index for all urban kelurahan in the IFLS (i.e., ‘poor’); between the 33rd and 66th percentiles (`moderate’); and above the 66th percentile (‘wealthy’). We then randomly selected one kelurahan from each category in West Java, and one for each category in East Java. For Sulawesi and North Sumatra, from which we were selecting two sites each, we carried out a similar procedure that selected one kelurahan from below and one from above the 50th percentile of the wealth index (`wealthy’). In Central Java, 6 Sampling from the IFLS sites was done for a variety of reasons. First, it provides a unique opportunity to match the original data we collect to a range of information about the same communities collected before the introduction of PNPM-Urban. This information on ‘baseline’ conditions, coming largely from the community surveys conducted as part of the IFLS in each round (including 2000 and 2007 and earlier) includes not just information on public services and infrastructure in the kelurahan, but also indicators of community collective action and operation of NGOs and other groups in the community. These prior characteristics may strongly condition the success of PNPM-Urban, and understanding if this is the case is a very useful part of this evaluation. Second, reliable kelurahan-level data on poverty is difficult to obtain. Initial plans to stratify using the data on poverty rates generated during the poverty mapping phase of PNPM-Urban implementation available via the MIS were not feasible (as will be discussed in the following section). With the IFLS data, stratifying the community sample both by region and by wealth or poverty is straightforward. Finally, since the IFLS is a representative national survey (covering provinces where 80 percent of the Indonesian population lives), the sample of communities chosen is reasonably representative of urban communities across the country. We note that although the IFLS does not cover all provinces, the small sample size as well as logistical considerations in any case prevented sampling in all provinces. We are grateful to John Strauss and Bondan Sikoki, PIs of the IFLS, for allowing us to sample from the IFLS community lists and providing access to the kelurahan IDs for this purpose. 7 For a measure of mean kelurahan well being, one could more directly simply calculate mean per capita consumption for the households in the IFLS sample for the cluster. However, within each sample kelurahan in the IFLS, households are sampled from a single, randomly chosen RT (neighborhood), not over the kelurahan as a whole. Hence this group may not be representative of all households in a particular cluster. Even if households were randomly sampled throughout the kelurahan this problem would arise, given the small number of households per cluster. That said, the correlation of our index —which is based on kelurahan-level indicators--and mean per capita household expenditure in the cluster is .3278 and highly significant (p=0.00). 18 where we had only one site after the field test sites were (randomly) selected for Yogyakarta, we simply selected one site randomly. Replacement sites were also chosen randomly within each region/wealth stratum.9 Finally, to ensure that all the sites selected contained infrastructure projects at a stage of completion such that they could be feasibly assessed, we required that the community be listed in the PNPM-Urban MIS and that it had some projects listed in the MIS that were at least two years old to ensure that engineering assessments could judge the maintenance or deterioration of PNPM-Urban infrastructure projects. The table below lists these 13 sites for the study. Table: Study Sites (Non-ND locations) Kelurahan Kabupaten/Kota Province Wealth index Triharjo Kulon Progo D.I. Yogyakarta High Karo Pematang Siantar North Sumatra High Cengkareng Timur Jakarta Barat Jakarta High Tambakrejo Surabaya East Java High Pancuran Gerobak Sibolga North Sumatra Moderate Astana Cirebon West Java Moderate Kauman Surakarta Central Java Moderate Lirboyo Kediri East Java Moderate Antang Ujung Pandang South Sulawesi Moderate Ngestiharjo Kulon Progo D.I. Yogyakarta Low Wiroborang Probolinggo East Java Low Rantepao Tana Toraja South Sulawesi Low Hulu Banteng Lor Cirebon West Java Low The final three sites were allocated to ND communities. Given that kelurahan in the ND pilot were by definition good performers under PNPM-Urban (rather than a representative sample of kelurahan), the outcomes observed in these sites are not a reliable indicator of the effects of a general scale up of ND across sites where PNPM-Urban achieved varying degrees of success. To this end, ND sites were chosen to capture variation in implementation experience. The PNPM-Urban team identified one ND community that was deemed “successful,” one that was relatively “unsuccessful,” and one that was “typical”, based on community participation (e.g., meetings held, number of participants), local government involvement, and the quality of ND planning products (e.g., the Community Settlement Plan). Given these explicit selection metrics and to ensure the confidentiality of the sites and respondents in this small sample, we refer to individual ND sites using numbers. Site 1, located in Central Java, is the site categorized as “typical.” Site 2 is in West Java and is considered the “unsuccessful” site. Finally, Site 3 is the “successful” site and is also located in Central Java. 3.3.2 Data Collection Activities 9 The selection process itself worked in the following way: within each combined region/wealth stratum, we assigned a random number to each kelurahan and then ranked the kelurahan in the stratum by that number. The site with the highest number in each was selected. The site with the second highest was chosen as a “replacement” site. 19 To cover the various actors in PNPM-Urban, data gathering for the qualitative research encompassed a range of instruments and approaches, including key informant interviews, rapid surveys of BKM and KSM leaders, rapid surveys of poor households, focus groups and in-depth interviews, infrastructure beneficiary interviews and engineering evaluations. Site visits were conducted by four teams, each consisting of a team leader (RAND researcher or consultant), two SurveyMETER staff (moderator and note-taker) to conduct focus groups, in- depth interviews, and KSM group interviews, an interpreter for key informant interviews and beneficiary interviews conducted by the team leader and an engineer. Key Informant Interviews In each site, semi-structured interviews were conducted with a range of PNPM-Urban and other government officials, as well as non-governmental organization (NGO) representatives were to gain a broad overview of the PNPM-Urban functioning, successes and problems. Targeted informants included  PNPM-Urban District Program manager  PNPM-Urban City coordinator (korkot) or Assistant for Infrastructure (askot/consultant)  PNPM-Urban facilitators  District officials from the Ministry of Public Works;  Bappeda head or deputy (Development Planning Agency)  Kota Deputy Mayor  Camat (kecamatan leader)  Lurah (kelurahan leader)  NGO representatives working in the kelurahan (or higher level if necessary)  BKM leader  Infrastructure KSM members (one or two interviews simultaneously)  Planning Committee and Implementation Committee members (for ND sites, in place of KSM)  Planners (for ND sites) All key informant interviews were conducted by the RAND team leader, with the assistance of the interpreter, except in the case of KSM members and Planning Committee and Implementation committee members in ND communities where interviews were conducted by SurveyMETER . Rapid Surveys of BKM and KSM members For BKM and KSM, in addition to the semi- structured interviews, short questionnaires were administered to the BKM head and the KSM leader. In the case of the BKM heads, who tended to be relatively well educated, the questionnaire was self-administered. The infrastructure KSM leader’s questionnaire was administered by SurveyMETER. Information collected included the composition of the groups (number of members, number of women), number and frequency of meetings, project proposals developed/reviewed/accepted, frequency of meetings with the facilitator, and so forth. Poor Households – Focus Groups, In-Depth Interviews and Rapid Survey Two sets of focus groups and key informant interviews (one focused on PNPM-Urban, and one on general issues 20 pertaining to poverty) were conducted on eligible households sampled from the lists of poor households developed as part of the PNPM-Urban implementation10; in addition, all households drawn from these lists and screened for the focus groups were administered a short survey. To keep enumeration costs to a reasonable level, two sets of households were selected from poverty lists for the three or four RTs in the kelurahan with PNPM-Urban projects and the highest numbers of poor people. Screeners visited households selected from the lists for each RT until enough eligible individuals and replacements were recruited. For the PNPM- Urban focus groups and in depth interviews, a key criterion was that participants have experience with PNPM-Urban11. All visited households were administered a brief survey questionnaire. In addition to asking about awareness of PNPM-Urban and familiarity with local PNPM-Urban infrastructure projects, the screening questionnaire asks if the respondent or a family member ever participated in a PNPM-Urban meeting or voted in an election related to PNPM-Urban, and if the respondent knows what the KSM is and the BKM is. For the poverty component, to obtain up to 16 focus group participants and 2 interviewees plus backups, an average of 27 households were visited per kelurahan. To obtain the same number of participants for the PNPM-Urban component, 41 households on average were visited. The greater number reflects the difficulty of finding willing individuals with the minimum knowledge of PNPM-Urban. Two PNPM-focused focus groups were conducted in each site, one for men and one for women. The focus groups had 6 participants on average. A standardized protocol (slightly different for the men’s and the women’s groups) was developed to explore issues including the following:  Perceptions of the community’s needs;  Understanding of and familiarity with PNPM;  Participants’ role in PNPM-Urban activities and decision-making;  Perceptions of whether the program is directed to those in need (including with regard to the placement of infrastructure projects) and open in terms of decision making.  Perceptions of the strengths and weaknesses of the three PNPM-Urban areas (infrastructure, social/assistance, economy/loans) in addressing the needs of the community;  Participants’ experiences with PNPM-Urban infrastructure projects and perceptions of their quality and usefulness;  Awareness of and views about other anti-poverty projects and programs. 10 While this list was potentially slightly outdated in some cases (as it was created at the start of the program so in some communities could date from 2007), it was deemed more useful than other possible sources, such as the roster of households receiving the rice ration, which is thought to contain a large share of non-poor households. 11 Initial piloting by SurveyMETER using draft screening forms revealed a generally low level of knowledge of PNPM- Urban (generally confirmed in the survey field work): most participants had heard of PNPM-Urban by name, but few could describe its purpose or operation or identify specific projects funded by PNPM-Urban. Respondents who exhibited the strongest name-familiarity with PNPM-Urban were a select group with direct involvement in PNPM-Urban such as KSM members (who were to be separately interviewed in any case) and others who were likely to have a strong pro-PNPM-Urban bias. A person was thus deemedeligible if he or she (a) had heard of PNPM-Urban and (b) was aware of one or more PNPM- Urban infrastructure projects in their neighborhood (RT) read to them from a list by the enumerator. Focus group and interview protocols were designed to capture participants’ general opinions about these projects (quality, usefulness, etc). A similar approach was taken when asking about other PNPM-Urban activities (loans and assistance). 21 Emphasis in the protocols is placed on PNPM-Urban infrastructure activities. However, questions are also asked about Economy (loans) and Social (assistance) activities. First the moderator asks if anyone in the group participated in these activities as a beneficiary. If yes, their experiences and perceptions are elicited. However, given that the formative work suggested that many or even most participants would not be familiar with these PNPM-Urban activities (at least by name), the protocols also cover participants’ perceptions about these activities in general terms (whether they are useful, and their value relative to infrastructure). The same approach is used for infrastructure itself: recall from above that the criterion for knowing about infrastructure did not require that candidates for the focus groups actually know that PNPM-Urban was responsible for the infrastructure projects they identified. For the second set of more general focus groups, information was gathered on:  Respondents’ perceptions of their own poverty and its causes, and barriers to moving out of poverty;  Strategies for coping with inadequate resources (both permanent and temporary shortfalls);  Differences in the causes and impacts of poverty for men and women  Perceived needs, including forms of government assistance.  Participation in and perceptions of assistance programs (including efficiency, fairness, value of benefits, convenience, corruption, etc.) Two in-depth interviews per kelurahan were conducted, one each with a female and a male resident. These interviews aimed to capture, in greater detail, experiences, views and perceptions of the program, permitted more intensive follow-up questions, elicited personal narratives of poverty and experiences with PNPM-Urban and potentially made it easier for respondents to discuss issues they would not be willing to talk about in the group setting. The in-depth interviews with program beneficiaries were conducted by SurveyMETER. Infrastructure beneficiary interviews Shorter, semi-structured interviews were administered to people living near selected PNPM-Urban infrastructure projects. The interviews were conducted by the RAND team leader with the assistance of the interpreter. Typically between one and three areas in the kelurahan were visited that each contained one or more PNPM infrastructure projects. In these areas, the interviewer would approach a random sample of approximately 5 to 10 respondents to interview, often individuals living in nearby homes. Respondents were selected in essentially a random fashion, but with some effort to ensure variation in gender and age.12 The aim of the beneficiary interviews is to shed light on whether and how these projects affect the lives of people living close by, whether these beneficiaries are aware that funding was from PNPM-Urban, and the extent to which they or their neighbors were involved in the development, implementation and maintenance of these projects. Engineering Quality Assessments The goal of the infrastructure quality assessment was to determine whether PNPM-Urban projects met basic construction and materials standards and 12 At times more than one respondent per household visited was interviewed, while in some cases respondents did not actually live near the site. 22 practices. Scoring sheets with multiple criteria, including materials used, project specifications (e.g., road surface width), maintenance, and performance. To assess the quality of infrastructure projects completed under PNPM-Urban in the visited communities, a team of local engineers evaluated selected projects based on standardized protocols for specific project types (roads, drainage, etc.) Lists of completed projects were obtained with the assistance of the World Bank office in Jakarta. These lists were supplemented by lists that the engineers obtained locally once at the site (in many cases, the two lists of completed projects--MIS and local--were not consistent). In each kelurahan, the engineer reviewed up to 20 infrastructure project across six categories: roads, bridges, drainage, public toilets, housing projects, and water projects.13 In kelurahan where fewer than 20 projects had been completed, the engineer would assess all completed projects. For sites where more than 20 projects had been completed, projects were selected so that the distribution by category was roughly proportionally to the actual frequencies of each type. The sampling was also conducted so as to ensure a mix of projects from different PNPM-Urban cycles in each site. The scoring protocols were developed and previously used by the Government of Indonesia, were provided to us by the World Bank and modified for use in the field (see appendix). The output of the infrastructure assessment is a score for each sampled project and overall summary scores for six types of infrastructure. The engineers’ projects assessments were conducted using a combination of observation and, where necessary, direct communication with knowledgeable parties (e.g., BKM members, facilitators). Materials samples were not collected due to time and feasibility constraints. MIS Data Following a detailed conversation with the PNPM-Urban MIS team in Jakarta, we requested for purposes of the analysis a single, master data file with data on participation records, projects completed, and BLM disbursements from 2007 to 2010. The master MIS data file is an unbalanced panel of 12,999 kelurahan, from 2007-2010, including key variables for self-reported community characteristics (including population and poverty), participation statistics for PNPM socialization activities undertaken in that year’s cycle, total funds disbursed (by funder), proposed funding (by funder), number of projects actually implemented (by category), funding of projects realized (by category and funder), the number of beneficiaries (KSM and public, by type - poor and nonpoor), information on inputs e.g. timber use, physical quantities of infrastructure produced (by categories) and certification, number of complaints registered, financial performance. In addition, a separate datafile contained identifiers for the 265 PAPG kelurahans14. After data cleaning, the final sample suitable for analysis contained a subset of approximately 8000 kelurahans from the 2009 funding cycle only. Given the strong likelihood of selection biases in the final MIS sample, we do not present the results in the main report, but attach it as an Annex B as illustrative of the type of analysis that could be carried out with the MIS. 13 Projects that fell outside of these categories were not assessed. However, in many cases project categories were interpreted broadly to reduce the number of projects that could not be scored. 14 Note that 26, or almost 10%, of the PAPG kelurahans were not found in the “exhaustive” master MIS database. 23 3.3.3 Data Analysis Once all interviews were conducted and transcripts prepared, the data from semi-structured interviews with key informants (n=127), PNPM-Urban focus group (n=32) and in-depth interviews (n=32) as well as semi-structured interviews with infrastructure beneficiaries (n=75) were coded and analysed using Atlas.ti. For data analysis, we used a multi-stage technique described by Lincoln and Guba (1985) and elaborated by Ryan and Bernard (2003)15 to extract information of the major themes and subthemes addressed by the evaluation questions. For the MIS and survey data, information was received in Excel, and coded and analyzed using the statistical software Stata. In the next section of the report, we present the evaluation findings for each research question, based on the data collected. In each section, we highlight what appears to be working well in the program, areas for improvement and promising best practices for future directions. 15 For all four sources of data, team members first developed a temporary codelist by outlining key themes and sub-themes. The themes align with the subject areas outlined in the TORs for the poverty analysis and the PNPM-Urban evaluation. We had earlier also used these themes to construct our interview and focus group protocols. Using this temporary codelist, the coding team used the text management software Atlas.ti to mark major themes in a few of the transcripts. As additional themes emerged, the coding team along with two members of the research team revised the codelist and finalized it. Once the codelist for each of the four sources of data was finalized, two coders independently used Atlas.ti to mark themes according to the finalized codelists. To ensure that we have accurately and consistently captured all the text, we had one of the team members supervise the two coders. When coders were unsure about matching texts with themes, they left memos next to the coded texts. At the end of the coding process, the coder and the supervisor discussed all the memos. In the case of disagreement, the supervisor made the final decision. Once all the texts had been identified based on themes and sub-themes, these texts were extracted and we had data in three forms: narrative description, short qualitative answers and close-ended questions. In the case of narrative descriptions and short qualitative answers, we created free-lists and calculated the range and central tendency for key themes. We also noted outlier cases. Since most of the close-ended questions consisted of quantitative data, we calculated range of responses on key variables. In order to make comparisons across sites, in all four sources of data, we noted whether the site was ND and from which kelurahan. In addition, in the case of key informant interviews, we also recorded whether the respondent was a local government official, Public Works/ PNPM-Urban official, PNPM-Urban community based informant or an NGO official. These categories helped us make comparisons across individuals having different levels of engagement with PNPM-Urban. 24 4 Assessing the CDD approach Binswanger et al (2010) describe a successful CDD program as undertaking “the co - production of outputs by a joint venture of communities, local governments and the central government, with support from the private sector and civil society.” To assess the CDD model in PNPM-Urban, we divide our analysis into four main themes: the success of overall socialization in building basic awareness and understanding of PNPM-Urban’s processes, funding and projects; community participation in planning, decision-making activities and implementation; inclusiveness towards women and the extent to which PNPM-Urban meets the needs of the poor. As part of the UPP expansion, a prospective quantitative impact evaluation built into the program during UPP2 between 2004 and 2007. Although UPP2 was not randomized, the quantitative mid-term evaluation of UPP by Pradhan et al. (2010) was able to use a rigorous pre-post design by comparing treatment and control communities matched based on district location and poverty scores. In brief, the Pradhan et al. evaluation demonstrated no statistically significant impact on household welfare in UPP treatment communities (as measured by consumption, asset building, and access to credit). The evaluation suggested a small increase in household-reported access to sanitation, particularly for the poor. In terms of participation, individuals that were already more socially connected were more likely to be participating. The evaluation also found a decrease in the perceived power of the local lurah in poorer treatment areas as well as the leadership ranking of the RW and local activists in richer treatment areas. There were large improvements in participation in community organizations and welfare for households in both treatment and control communities, hence these could not be attributed to UPP. Overall, the authors report that most people living in treatment communities were aware of UPP and a high percentage (40 percent) reported being satisfied with the program, but respondents seldom cited UPP as the program that met the communities’ most pressing needs. Our research below revisits and expands upon some of these issues in the context of present conditions in PNPM-Urban. A. Overall Socialization: Community Awareness and Understanding of PNPM-Urban PNPM-Urban is a program that is simple in concept, but can become complicated in practice, given the many entities involved and various and modifications in the program over time. As Table 4.1 suggest, a basic element of capacity-building during the socialization process is educating community members and local government officials about the concept of CDD and the specific features of PNPM-Urban. A second basic element is ensuring that knowledge about the program and information about its activities is widely disseminated to the public, including how and when to participate in voting and planning and the existence/status of projects undertaken. What is working well, and why? Community awareness of the opportunity to express their needs and direct resources (whether or not they choose to exercise it) as well as the outcomes of project implementation is a first- order and necessary condition for establishing any kind of downward accountability. In our study communities, at the level of officials and community leaders, there is generally good understanding of PNPM-Urban’s goals and processes. Within communities, there is a high level of recognition for PNPM-Urban funded projects. Overall, all types of key informants (even those who were not themselves involved with PNPM-Urban) were able to demonstrate understanding of the PNPM-Urban program design and able to describe the fundamental principles behind the participatory process. In our study areas, as described in the methodology above, the screening survey for focus groups and interviews sampled poor households in RTs where there were large shares of the poor. Across all sites, 47 percent of our sample reported having heard of the PNPM-Urban program. When asked about the specific PNPM-funded infrastructure projects in their neighborhoods, almost all respondents were aware of at least some of the projects. Most focus group respondents16 were able to identify the broad purpose of PNPM-Urban as poverty reduction and improvement in well-being. For instance: “it is a program from the government to help the community in capital or in infrastructure;” “to make society prosperous by providing them with the needed infrastructure;” “a kind of government project to face the poverty problem.” Most frequently, respondents associated PNPM-Urban with infrastructure projects, notably roads, ditch and housing improvements; although respondents also mentioned credit and social assistance; and/or with various training courses for economic activities (usually for particularly women). Among the sample of beneficiaries purposively recruited because of their proximity to infrastructure projects, respondents were overwhelmingly aware of the projects in their areas and when they were built. Beneficiaries were also often aware of how projects were built and in many case participated in one form or another on the construction process (we discuss this more below).17 What areas are there for improvement? Both key informants and households themselves reported that community members are less aware of the specific organizations and processes associated with PNPM-Urban. Key informants overall reported that community level understanding of the program was relatively low, although some recognized participation was tied to knowledge and understanding - one Askor observed that participation increased as individuals began to realize project benefits, and a BKM leader reported that the lack of information and awareness of the program and its benefits was a major constraint to building interest and trust and that more communication with their community was needed. A major problem noted is that the program’s target group, the poor, do not understand the PNPM-Urban process. Turning to household survey 16 We note again that recruitment for the focus groups (from the broader pool of low income respondents identified in the screening survey) necessarily set the knowledge threshold low: participants had to have familiarity with PNPM-Urban projects in their neighborhood, but did not have to know that they were funded by PNPM-Urban, or be aware of specific program features of PNPM-Urban. 17In some locations respondents were aware of the PNPM projects but would refer to the program by its previous name, P2KP. To ensure we capture respondents’ knowledge of program activities accurately we used both names during beneficiary interviews. 26 respondents, when specific program entities (e.g., BKM) were mentioned, far fewer expressed any familiarity: less than 20 percent reported knowing of the KSM and the BKM. Furthermore, when asked about the specific PNPM-funded infrastructure projects in their neighborhoods, while all household survey respondents were aware of at least some of the projects, only about half of respondents (46 percent) could identify at least one them as funded by PNPM-Urban18. Among focus group respondents19, across sites, participants showed a lack of familiarity with details of PNPM-Urban. In one site, respondents agreed that “we only know the name, P2KP and PNPM.” In one site, when asked about the participatory process, the respondent remarked: “I don’t know about that. I only know that there have been trash wagons, that’s it.” For the most part, respondents were also unaware of the roles of entities such as BKM and KSM. Some confused PNPM-Urban projects and activities with other social programs. One respondent remarked that “the community never knows what the next step is. So …what is BKM, who are the persons in there, what are the activities, we do not know.” This corresponds to the fact that most program implementers interviewed were not undertaking sufficient socialization ex-ante. Importantly, while many were able to cite some outcomes of the program, few focus group participants spontaneously noted that the program was meant for “the grassroots” or had a “community empowerment” aspect. Among those respondents who did identify “community empowerment” as a part of PNPM-Urban, few appeared to define this in terms of a participatory decision making process. One respondent instead framed it as the community working collaboratively (but not deciding) together: “The mason might help in establishing the foundation. The laborers might help in gathering the sands. The others might help in fixing the materials. All work together.” Only a few respondents were able to articulate the process of funding, the decision-making process and community mobilization during implementation, and even so the details are not always accurate: for instance, one respondent described PNPM- Urban as “a kind of grant from central government to certain communities where the community is given a chance to manage all activities funded by the central government.” Even when benefits are realized, often the source of funding is not attributed to PNPM-Urban. A significant proportion of beneficiaries in all sites had limited knowledge of PNPM’s role in the projects they were benefiting from. Most were unable to give any response when asked where the funds for the projects came from. Only a small number were able to correctly identify PNPM-Urban as the source of funds; many others incorrectly thought that the funds were from the local government or other institutions (with responses including, for example, the Lurah, the village government and even Mercy Corps. Unlike the household survey respondents, two-thirds of the beneficiary interview respondents were not aware of PNPM- Urban by name (or by any of its older names). 18 We note that this is consistent with Pradhan et al(2010), most households living in UPP areas reported knowing about UPP. Some of them, about 20 percent, could also say what UPP is about, that is, aid for the poor although, notably, more than that (25 percent) ever attended a meeting organized by UPP. 42 percent claimed to have benefited from the presence of UPP in the community, with infrastructure improvement mentioned most frequently. 19 Recall that recruitment for the focus groups (from the broader pool of low income respondents identified in the screening survey) necessarily set the knowledge threshold low: participants had to have familiarity with PNPM-Urban projects in their neighborhood, but did not have to know that they were funded by PNPM-Urban, or be aware of specific program features of PNPM-Urban. 27 In the majority of cases, community members report learning about PNPM-Urban through channels other than the program’s socialization efforts as listed in Table 4.1. Most focus group respondents did not refer to socialization activities but instead said that they had gained knowledge of PNPM-Urban from their friends, relatives or acquaintances, or they had seen a reference on TV. A smaller number reported seeing invitations posted in the village or otherwise being invited personally to meetings (note that the key informant interviews described these two less common modalities as being tied to community mobilization efforts); others reported seeing projects and finding out or being told they were done by PNPM-Urban. Consistent with this, key informants tended to report that the community was informed about project outcomes or decisions (and then only when it was considered relevant), and less frequently reported systematic efforts to inform the public about participation in planning and decision making. One facilitator noted that typically “many people are aware of PNPM, but not all; most learn about the program when a project is completed near where they live.” In one site, a KSM leader even rationalized keeping the poor uninformed, because in the event that their proposals were not executed, “we have a moral obligation” and such individuals would be disappointed. In one case, the facilitator noted that socialization meetings were attended only by RTs, RWs, respected community leaders, and the local women's group leader, and they were unsure if poor people were even present at the meetings. B. Community Participation in Planning, Decisionmaking and Implementation This section describes findings related to the implementation of the PNPM model relative to its objectives for community participation as noted in Section 2.5 and the processes of socialization as laid out in the table below Cycle Stage Goal Initial Socialization  Develop picture of social dynamics  Disseminate information about PNPM  Seek permission from Heads of Household/village to implement the PNPM process  Call for volunteers Community Preparedness  Create community commitment to accept or reject PNPM Meeting (RKM)  Recruit volunteers with the relevant criteria for leading the various activities in the cycle Poverty Reflection (RK)  Raise awareness that poverty is a common problem t  Identify the roots of poverty Voluntary Contribution Mapping  Generate knowledge/awareness about the realities of problems (PS) and potentials of the community  Build motivation to resolve problems Formation of BKM (BKM)  Implement Organization and Leadership discussions  Build community understanding of the organizational criteria for the BKM and create the BKM Establishment Committee  Create draft Budget  Elect delegates that embody “noble values” (if the total number of RT are great screening can be done at the RW) to create a credible BKM Compiling of PJM Pronangkis /  Drafting of program activities for poverty alleviation (3-year RENTA and annual) 28 Organization of KSM  Creation of self-help groups that reflect community needs Source: PNPM-Urban Implementation Manual; discussions with stakeholders Community members can participate in decisionmaking by taking part in the planning process, proposing projects. voting or standing for election for the BKM or KSM, and then taking an active role in the BKM and KSM itself. and selecting projects (e.g., by attending BKM meetings), elected representatives and community leaders play an important role, too. Moreover, it would be impractical to have even a modest fraction of households in a kelurahan participate in meetings. Nevertheless, community input, in part through formal PNPM-Urban meetings, is expected, and it is against these program design-driven benchmarks that we assess the program. To the extent that community members express their views or desires informally, for example through a conversation on the street with a community leader, we may not capture this type of participation fully. The program also defines objective criteria for participation, especially for population sub-groups like women and the poor. For example, women are expected to constitute 30 percent of attendees at meetings, although this does not guarantee that women will contribute the same share of active participation Data from all types of respondents suggests that community participation in decision-making across the sites varies in practice, but in most sites community leaders (although not facilitators) play a critical role in making decisions. The PNPM-Urban program is designed to promote active community participation in project selection and implementation, while relying on representative leadership to manage key project tasks. Most respondents reported that participation in PNPM-Urban decision-making and project implementation varies by site and by activity. Direct participation by households is higher for project implementation than project selection or design. There is some evidence that households participate in decision making indirectly (e.g., through community leaders or PNPM-Urban officials), but it is not possible to determine the extent and effectiveness of indirect or representative decision- making. Respondents of all types generally described the extent of community participation in the project implementation process as significant (and the most significant type of community participation relative to other aspects of the CDD model). Participation most often takes the form of labor, either directly or indirectly supporting project construction. The majority (about three-fourths) of beneficiaries across all sites reported some form of contribution to project implementation. In most kelurahan, respondents told us that their contribution to infrastructure project construction came in the form of preparing or cleaning the site (e.g., cleaning canals, collecting debris after a project is completed, carrying materials, etc); carrying out main construction tasks; providing snacks and beverages to the workers, and giving small amounts of money. With respect to compensation, accepted practice appears to vary across sites. Focus group respondents in more than one site reported that they not only provided free labor but provided materials as well. In other sites, respondents reported that they provided volunteer labor, although there were also skilled laborers who were hired and paid a wage. Consistent with the focus groups, many of the beneficiaries explained that they provided volunteer labor. However, in a few sites, respondents explained that all construction work was done by paid skilled laborers, sometimes from outside the community. What is working well and why? 29 As we discuss in detail, PNPM-Urban is itself a system of delegated representation. Within the program structure, participatory processes appear to function according to plan, at least nominally. Once elected, in most of our sites, community members report involvement with PNPM-Urban that is consistent with program guidelines and objectives. Table A2 in the Annex shows that most BKM leaders reported that the BKM itself meets once every month. One BKM met every three months, and one reported meeting only once every year. At least one KSM infrastructure leader was also interviewed at each site. Table A3-A4 show there is significant variation in the frequency of the KSM’s own meetings – the majority of leaders reported meeting once or more per month, while another 30 percent reported meeting quarterly or fewer times a year. Public meetings were held much less frequently by the BKM: only two BKM leaders reported also holding public meetings monthly. Three reported having meetings quarterly and three every six months. One reported an annual meeting and one reported never holding any public meetings at all. However, all but one KSM reported holding public meetings and all but one reported that any citizen could attend and speak at these meetings. In practice, in many instances, the structure of PNPM-Urban works through and reinforces existing local elite leadership structures, either formally through the RT/RW leadership or informally through religious and other leaders to facilitate community participation in proposing projects. Many facilitators themselves reported that RT/ RW leaders or Lurahs work together with PNPM-Urban to help organize the community. One remarked that local leaders had the most useful knowledge about the community, and another noted that they were more credible to the population. Notably, where suggestions for projects were reported to originate from the community, these are usually sent to the BKM via the RT leader. When households did participate, it was usually done through an existing local representative, typically including BKM and KSM members, RT and RW leaders, and the head of the women’s group. One respondent explained, “The community gives their opinion to the Pak RT [RT leader] and he in turn takes it to the PNPM-Urban meeting.” According to another respondent, “community representatives attend[ed] meetings because it is impossible to have meetings where everyone attends.” Furthermore, in the majority of sites, focus group respondents expressed the view that attending meetings was valuable to them as members of the broader public. Community members were said to be able to use meetings to propose projects and discuss their needs. The content of meetings varied substantially across sites, reflecting in some cases the fact that different sites were at different points in the PNPM-Urban project cycle. In more than one site, respondents related discussions and socialization activities, including the weighing and final selection of projects by community members. One respondent noted that although the frequency of meetings and activities was high, they did not include much community-level participation. A clear sense from the focus groups was that the purpose of the meetings later in the cycle was to present decisions already made about projects, not to engage residents in debate and decision-making on them (for instance one respondent remarked “we are only given the announcement that there will be assistance [funds] from PNPM-Urban, our road will be repaired, we hope for cooperation”). However, in many other cases even if meetings essentially only informed or updated the public on the program, respondents appreciated and felt that these updates were useful. Respondents also noted that some meetings of this type also played the role of reinforcing accountability. 30 In most communities, voluntary participation through labor was reportedly easy, and individuals were eager to contribute. One key informant noted that “although money and materials are difficult, labor is easy to get.” In only one case, community members were reported to have donated materials to increase the project volume. In many sites, focus group respondents similarly reported that projects would spontaneously generate community contributions in the form of labor. This was facilitated by the concept of gotong royong (mutual contribution) or kerja bakti (socially-responsible work), already a feature of their communities. Some respondents explained that the willingness to provide labor was directly linked to the perception that PNPM-Urban projects originated within the community; whereas if projects were perceived as “aid” or coming from the government it would be appropriate to ask for monetary compensation. Others mentioned that it was their “duty” given the “self- help” nature of the scheme. Those that did not have the time would contribute cigarettes, snacks and drinks (or money to buy them) instead. What areas are there for improvement? The consistently-reported finding that participation takes place via delegation to representatives who are supported by or themselves are members of local elites is itself not necessarily a poor outcome. However, worryingly, as previous authors have stressed, this opens the door to elite control, although the extent to which elite capture is actually a problem is difficult to assess, particularly given capacity constraints in local areas. BKM representatives, RT leaders and RW leaders regularly attend KSM meetings in almost all sites; although RT/RW leaders are less represented at the BKM level. A common concern about the common system of delegated representation is that the availability and political weight of the delegates: one respondent noted that “there were no representatives from RT 1, which is isolated, so it is always left behind.” As indicated, limits on the number of members on the BKM may leave some RTs unrepresented in larger kelurahan. In most of the BKMs, not every RT in the kelurahan had a representative on the BKM itself; and only half the BKMs reported having community leaders join in their meetings. In addition, the number of public participants at meetings varies widely. At the low end of the distribution, 30 percent of the leaders reported ten or fewer typical attendees while at the upper end, 30 percent of the leaders reported 35 or more. Multiple factors were cited by key informants as influencing community members’ participation as representatives within the structure of PNPM-Urban. The most commonly- cited barrier, especially for PNPM-Urban volunteers in KSM and BKM—was the time-cost of the process, i.e., that community members did not have the time. Indeed, for active volunteers, such as the BKM head, the opportunity costs are high: PNPM-Urban implementation requires a substantial time commitment, taking time away from businesses and families. Many interviewees stated that the BKMs lost members as time went on. One BKM member stated that they lost most of their female members who eventually left as they felt were sacrificing time with their families, especially children. This opinion was echoed by an interviewee who stated that people feel they “spend too much time on PNPM-Urban and not enough time with their families.” Frequently, PNPM-Urban implementers stated that participation was also limited because of the volunteer nature of the work, which applies both to elites (related to BKM participation) and for the poor. One respondent noted that it was difficult to recruit men 31 for the BKM because the position provides no honorarium for such a significant time commitment. Another KSM leader noted that everyone was busy with their own job. In another area, the korkot gave examples of communities in industrial areas where reportedly 60 people would be invited to meetings and less than 5 would show up, attributing this to heavy work schedules. In that same location, the KSM leaders agreed that people could not leave their jobs to participate. Many of the key informants expressed the opinion that only community members who were likely to be direct beneficiaries participated. In one site, the interviewee noted that the community was easy to mobilize because they knew that they would receive funds. In another, the interviewee remarked that sometimes community members attended the meetings expecting to get benefits, but when they did not, they would stop attending meetings. . Several respondents indicated that it was difficult to sustain participation rates: the initial surge of participation is often followed by declining participation rates as the community becomes “bored” or “tired.” Others noted that they received complaints because project selection and design takes too long. Notably, one respondent suggested that meetings about infrastructure projects typically did not have high participation because the benefits are indirect. These responses suggest that mobilization of the community for infrastructure (the overwhelming focus of PNPN program) may be hard relative to the other PNPM components which potentially yield more direct welfare benefits to individuals. Finally, in some locations, cultures and traditions were cited as making it difficult to empower people, including feelings of stigma or embarrassment about asking for or accepting assistance. Norms about individual vs. communal rights and responsibilities were also mentioned; some respondents noted that participation was low where the poor were more “individualistic,” and in some areas, effective implementation was reportedly blocked by people who were unwilling to contribute towards the public good. However, the perspective often given by community members suggests that PNPM-Urban elected leadership and other elites can also be exclusionary: participation in meetings reflects pre-selection by the BKM, and/or often RT heads, RW heads, and other community leaders. Among respondents to the household screening survey, an overwhelming majority (approximately 80 percent) reported that neither themselves nor any of their family members had participated in PNPM-Urban meetings or voted in any related election. The majority of focus group respondents also did not participate in any meetings, and said they did not attend meetings not because they did not wish to, but because they were not invited - those who indicated that they had been personally invited to a meeting (by their RT leader for example) noted that they would certainly attend. Respondents in one site reported never going to meetings at all, but simply participating in surveys. Some respondents expressed frustration that the process was not, in their view, truly participatory. One respondent noted that they no longer attended meetings because their opinion was not respected, while another stated that meetings were a waste of time because “higher folk” would not accept proposals in the interest of the “common folk.” Others report that information about decision making made by community leaders at meetings is not always made available afterwards. It was common for focus group respondents to say that they only learned about infrastructure projects after they had been decided upon and were being announced. Some beneficiaries reported that they had only found out about PNPM-Urban after having been approached (and agreed) to help implement a project. In two sites, beneficiaries living around completed 32 infrastructure projects also reported that decisions about project selection and implementation were made by PNPM-Urban community representatives and local leaders as well as informal community leaders, with significantly less direct household participation, even though these respondents were the most likely to be directly affected (positively or negatively) by the projects. The structures of PNPM-Urban may also not allow those affected by the projects but excluded from decision making to intervene. In one site, a respondent remarked that “officials” did not give enough information to people in his community about one specific project and tried to move it along too fast. This respondent helped organize a community meeting to discuss potential PNPM-Urban infrastructure activities, but he stated that the nature of PNPM-Urban was not well explained by the BKM and the community was not adequately brought in and prepared. Surprisingly, facilitators were not described as having a more direct role in fostering decision making related to project inception or selection. One important exception was a site in which the facilitator was reported to make final decisions regarding project design and materials after they had been determined by the KSM, however this degree of involvement may have been inappropriate. In a few sites, key informants reported limited community participation in project implementation, involving just a small group of volunteers and direct beneficiaries. In these instances, a number of informants reported that residents working on projects expected compensation, especially if volunteering with PNPM-Urban took them away from paying work. There may be a fine line between encouragement to volunteer in implementation and the perception of social pressure from the BKM or KSM. A few key informants noted that people felt not simply willing but obligated to contribute. Some focus group respondents also reported being repeatedly requested by the BKM or other community leaders to contribute to the work. A respondent in one such site complained that “all of us are busy people. Every time we get a job, it should be paid.” It was also noted by key informants that more complicated projects require more specialized labor, for which recruiting and relying on volunteers is not efficient. C. Participation By Women Fostering participation for marginalized groups where the marginalization is perpetrated by communities themselves (groups such as women, recent migrants, or indigenous people) is a problem inherent to all CDD programs. As noted above, PNPM-Urban relies on pre-existing structures for community mobilization, especially at the sub-kelurahan level (RT/RW). As a result, community members who are marginalized from discussions prior to PNPM-Urban may also be excluded during PNPM-Urban decision-making processes, including women In 2007, a study on female participation in CDD programs was conducted with the explicit aim of developing a gender mainstreaming strategy for PNPM-Urban going forward (Joint Donor and Government Mission, 2008). The report’s findings were based on field visits and interviews. The authors noted that women’s experience and self-confidence were internal barriers to their involvement, while the formal participatory processes led to biases against women even when there were able candidates. In UPP, nationally, although women made up almost 50 percent of the votes, only 19 percent of candidates and 16 percent of representatives 33 were women (this was especially marked in Nusa Tenggara Barat (NTB), with less than 10 percdent of women in the selected BKM). Elsewhere, the authors found that the requirement for senior high school education might eliminated many women, and both men and women preferred to vote for men for decision-making positions. Furthermore, even when women’s participation was established, the scope was generally limited to their own village, with low representation at higher levels such as the kota/ kabupaten. The report’s findings suggest that even in cases where a relatively high percentage of women volunteered for PNPM-Urban activities, the underlying drivers of this were somewhat mixed. One important factor was the historical presence of women volunteers connected to the local implementation of Family Welfare Program (PKK) activities such as running the Posyandu. However, another reason was that unpaid positions were simply not attractive to men. As a result, the women’s participation in some cases interacted with issues related to local elitism: active volunteers often ended up being elite women. Furthermore, across the various PNPM-Urban projects, the report noted that the types of jobs for which women are selected tended to build on traditional roles. Even when programs had structures in place to promote diversity, such as quotas for implementation and monitoring teams, for example, women tended to hold the position of treasurer or secretary. In UPP sites, 51 percent of the financial management unit members were women, but only 11 percent of the infrastructure management units were members, while there was wide variation in women’s participation as laborers. The report noted that project rules could thus reinforce traditional roles, promoting elite women and leaving the poorest out, and recommended (in addition to the implementation and monitoring of project features such as gender quotas for participation in meetings and decisionmaking) enhancing the capacity via additional support and training and ensuring the presence of women staff and facilitators who act as role models. What works well and why? In our sites, we found there were no strong gender differences in knowledge of PNPM-Urban. Women were just as likely (or unlikely) as men to have heard of PNPM-Urban and its components. There were also no differences in reports of participation at public meetings. However, this may be because the question asked whether respondents themselves or household members attended meetings, thus also covering men in the household. Most participants in both male and female focus groups noted that women did contribute to the projects (especially if their husbands were working as volunteer laborers). A number of respondents also noted that women and men both would volunteer if the need arose, although this did not mean they are viewed as equivalent. As one male participant put it, .”.it doesn’t mean that the woman should do the work equal with the man, because man and woman have different power[s]. They will help what they can help based on their capability.” With rare exceptions, both focus group and beneficiaries reported that women’s participation in implementation of infrastructure projects consisted of providing (mostly male) workers with snacks and drinks. In only one site were women were reported to be actively engaged in building and construction. What areas are there for improvement? 34 The findings of our study suggest that many if not almost all of the issues raised in 2007 persist in PNPM-Urban today. Although the household survey results show that women are just as able to recognize the term PNPM-Urban and/or projects, actual knowledge about the specifics of the process as revealed in the focus groups appears significantly lower for women, which may be a result of not attending meetings. The majority of the key informants across the 13 sites indicated that female participation was lower than men, both in decision making and in implementation. Women are a minority in decision making bodies and at community meetings (commonly reported figures are on the order of 2 out of 12 committee members, or close to 20 percent) Informants reported that women rarely participated in planning and decision making, and when they did, it tended to be in PNPM-Urban social and economy meetings rather than infrastructure. This is borne out by our BKM and KSM survey results. Table A2 shows that only in one case did the BKM have more than 50 percent female representation. However, this is clearly not the norm. Two of the BKMs had no female members at all, while the modal number was 2. In 8 of the 13 main sites, women made up less than a quarter of the BKM membership. Table A3 shows that at level of the Infrastructure KSM, female participation is no higher. 26 percent of KSM in our sample reported no female representation on the KSM and almost 70 percent report female membership under 25 percent. Table A4 shows that in terms of public meetings almost all of the KSM leaders reported that more men than women attended public KSM meetings. Only two reported an equal or greater number of women attending. Some respondents expressed the opinion that this was due to women’s own lack of desire or ability to participate. Participation of women is made even more difficult by social norms and family roles. Typical statements made included “women are expected to stay home,” “are busy with their families” and would have to “make sacrifices especially concerning their families.” Another noted that “social jealousy” (albeit of an undefined nature) made it difficult to raise female participation. In one site, it was noted that the community itself was not concerned with women’s participation and that the 40 percent participation target was hence difficult to reach. However, other key informants acknowledged that features of the program and its implementation also lead to women’s relatively low participation. Low participation rates are at least partially attributable to inconsistent or non-female inclusive invitations to program meetings. Firstly, women are often not invited to meetings implicitly because they are typically not the household head. One respondent noted that women tended to participate only as substitutes for their husbands when the latter could not do so. A facilitator in one site noted that in that location, this was even precluded because invitations were issued to individuals rather than households, and invariably listed the man. In another site, the BKM leader noted that women are simply not invited to the infrastructure meetings. Consistent with this, in the women’s focus groups, most respondents reported never having been invited to, or attended, meetings themselves – rather their husbands would go. As with the women in the focus groups, the vast majority of female infrastructure beneficiaries stated that they were not invited to PNPM-Urban meetings. Most of those interviewed had not attended meetings, although a few reported that their husbands or sons had. Female beneficiaries confirmed observations by key informants that invitations were typically sent addressed to the male heads of household, and others commented on the fact that the overwhelming majority of attendants at meetings for infrastructure projects are men. 35 Even when women are explicitly invited, meeting logistics are not designed inclusively: for instance, in one site the BKM leader reported that meetings were generally held in the evening, and that no informal discussion could take place outside such meetings, precluding women’s attendance because of their obligations to care for the family at that time, as well as other opportunities for them to provide input otherwise. With respect to the poor, a key problem for the poor is the question of opportunity costs associated with uncompensated contributions of time and labor. In one location, the Sekta reported that when he was head of BAPPEDA, he had asked that the poor be invited to meetings, but they did not attend saying they had to earn a living and did not want to waste their time at a meeting that “wouldn’t deliver anything.” The respondent further observed “the poor have immediate needs and these usually outweigh longer term issues. So paying for/attending training, for example, will be sacrificed to meet today’s needs.” D. Does PNPM-Urban meet the needs of the poor? What works well and why? There were divergent views across sites and among respondents about (a) the needs of the poor and (b) whether PNPM-Urban met those needs. In general, respondents felt that PNPM- Urban was meeting the needs of the poor. Key informants were asked what they felt where the main needs of the poor in their communities (or for city level officials, their districts). The responses were quite varied. Across the sites, there was an almost equal three way split among those who said that the poor need more jobs or loans, more social assistance programs, or more infrastructure. The specific types of social programs respondents mentioned as key needs were diverse and included: increased access to healthcare, affordable food, increased access to education, availability of entrepreneurship programs. When infrastructure was mentioned, the types of infrastructure development projects participants listed ranged from private housing improvements to roads, garbage disposal, ditches and clean water sources; generally the types of investments realized under PNPM. For instance, various participants said: “[We need] sanitation, improvements in garbage disposal. Because of the absence of trash bins, we throw garbage into the trenches,” “In my neighborhood, the street cannot be passed through by riding motorcycle. I think the street must be repaired. We also need clean water,”“[There are] too few trash wagons, causing illness” and “[We need] a good drain for the areas subject to floods. It has been missed by the government so far.” Interestingly, a few focus group and in-depth interview participants argued that one of the main needs of the poor is more information (also referred to as ‘socialization’) about existing assistance and welfare programs. One stated that the community’s poor need “more socialization since people feel that they don’t have enough information about programs.” Even respondents who expressed a desire for more economy or social projects recognize that those project may be more difficult to implement than infrastructure projects and may not benefit as many people as infrastructure projects. 36 What areas are there for improvement? Many key informants as well as community members reported that the current mix of infrastructure, economy, and social projects was not ideal. However, there was relatively little consensus as to the appropriate mix, although many respondents stated that more emphasis should be placed on economy and social projects. The wide range of priorities for the poor that informants mentioned is found within kelurahan (or kota) as well as between them. That is, there was little consensus among informants from particular kelurahan on what are the main needs of the poor in their communities. For instance, in Pematang Siantar responses were as follows: “Cheap food is the most beneficial because it addresses people’s daily needs;” “The most urgent need is loans for people to start businesses;” “Infrastructure is ok, but more than that people need social programs and knowledge;” “The most significant needs are economy projects and jobs”. In Probolinggo, responses included: “Daily needs are urgent – people need food;” “The poor in [in this community] need access to health care;” “The community needs access to health care and micro-enterprise support;” “The most requested projects are paving and rehabilitation of private homes.” Informants who felt infrastructure development was the key need of the poor in the community included lurah and sekta, as well as korkot, satker, BKM coordinators and KSM members. Within this group, views on what specific type of infrastructure was most needed varied. The most widely held view across fieldwork sites was that housing improvements are the key infrastructure development needed in the community. This was followed, in order of the frequency of mentions, by: drainage, roads, toilets, and clean water. Turning to the poor themselves, the diverging views within kelurahan on what constitutes the main needs of the poor in the community are also found in the responses from focus groups and in-depth interviews. A significant proportion of respondents mentioned jobs, loans and capital, training, and access to healthcare and education. Cheaper prices, especially for food, were also mentioned by many. A typical comment on this issue included: “The prices are expensive. If possible, lower down the prices.” There were very few noticeable differences in the views of men and women regarding the main needs of the community. Among those who argued that low interest loans and capital are the greatest needs of the poor, a common complaint was that lenders have extremely high interest rates. A focus group participant made a comment that exemplifies many others: “Borrowing money from others is difficult and the interest can be up to 20 percent. Sometimes paying the amount of the interest only is hard to do. Sometimes we have to give away our belonging as the collateral.” Two explanations were offered for why low interest loans are important to the poor. First, participants explained that loans would enable the poor to start their own business, given that jobs are scarce in their communities. One in-depth interview participant, for example, said “If we need money there isn’t any guarantee that we can get a loan, so I think we need loans program. The ladies here want such kind of program in this area. So they can use the money to run business and to create jobs for jobless young men who don’t go to school.” The second reason offered is that, as many of the poor already have their own businesses, loans would enable them to expand and grow. For instance, a focus group participant explained: “I need a loan. I don’t work in my own place; I rent a poskamling [location]. [My] income will rise if I have my own place, because renting a place makes me spend money.” Among those who said various kinds of assistance were the main needs of the poor, three were mentioned most often: 37 better access to health services; support with school expenses (such as uniforms and books); and skills training (in particular for people to start their own business). A small number argued for comprehensive programs that include an assistance element and a loan element. For example: “To get rid of poverty, there should be trainings for the female members about entrepreneurship for instance how to make marketable food. Upon the completion of the training, they should be lent some amount of money to start the business. If they already have the skills but do not have the money, they won’t be able to start the business.” In a similar vein, another participant said:” [What we need is] assistance for the community in form of capital and training so that people can be independent.” Infrastructure development was mentioned less frequently, although many individual participants listed a number of these as the key needs of poor people in their community. In response to this question, for example, a focus group participant said: “I think education, health, sanitation, and widening of the alleys and the streets because in this village, the alleys are narrow and not paved.” It is interesting that infrastructure was mentioned less often than jobs, capital, and access to healthcare and education. We could not, from the focus groups discussions, assess whether this is because community members feel that infrastructure improvements are needed but secondary to others, or because the quality and scale of existing infrastructure is seen as adequate. Although this may appear at odds with community members’ statements that they are happy with existing (infrastructure-focused) projects, we believe that community members express satisfaction relative to the alternative of no PNPM- funding—it is difficult to elicit responses in which people consider alternative past project mixes. Best Practices and Recommendations for Consideration Where observed, meaningful efforts to raise overall public consciousness are highly labor- intensive and require collaboration between PNPM-Urban implementers and community leaders at the RT/RW level. Facilitators can play a role in outreach to community members through holding informational exhibitions and publications. In many cases, the BKM’s level of effort was cited (albeit sometimes by BKM members themselves) as responsible for the success of socialization. A typical mechanism is to take advantage of existing community-based schedules to deliver information about projects and meetings during regular RT/RW meetings or other meetings held by local leaders. A facilitator in one site acknowledged that socialization at the RT/RW level is left to community leaders, as they themselves do not have enough time to do this activity, but another reason is credibility: “when we give the explanation, they don’t really pay attention to what we say, but if the leaders do that, the community will listen”. In two sites, PNPM-Urban implementers noted that volunteers went house-to-house to do socialization. Finally, in terms of linking outcomes to PNPM-Urban, it is worth noting that only in a handful of the communities we visited were infrastructure projects labeled as PNPM; in most, there was no sign to indicate the provenance of the project. However, in one location, facilitators reported significant efforts taken to publicize projects including weekly exhibitions in the city square. Areas which reported successfully encouraging community members to take an active role in PNPM-Urban, whether as volunteers or simply by participating in meetings followed two 38 approaches. In one instance, the community’s activities reflected a fuller realization of the potential of the formal program structure. In Sibolga, where various respondents described the community as driving decision making, community members were reported to attend frequent open meetings every two weeks in a public meeting space, where individuals with ideas could suggest them to the RT/RW head, who would consolidate the ideas, select some and propose them to the BKM20. In other areas with high participation, key informants reported building on pre-existing community characteristics (notably education levels, previous experience with community mobilization, and local culture) to shape the nature and extent of grassroots participation in PNPM-Urban decision making. In some instances, participation was naturally higher due to local conditions : for instance, where education is relatively high, respondents reported mobilization is relatively easy because there is a greater understanding of the program itself21. One respondent noted that education also enables the community to assimilate to the PNPM- Urban process efficiently as “people know how to organize meetings and do them effectively.” Variation in regional or social norms about discussion and mobilization is also important. The facilitator interviewed for Astana, for instance reported that it was easier to achieve community participation in his current sites because the majority of the people there were Javanese and described as “very open.” However, in the case when this is presumably not the case, sensitivity to these norms can be crucial to maintaining participation. One senior facilitator noted that PNPM-Urban formal processes could then be supported by informal methods: “formal meetings (Rembug) are bad for this - it embarrasses people, they’re afraid. If you go and talk to people informally then you understand what the conditions. Informal meetings are much more effective for learning problems.” Finally, in the few instances, where project beneficiaries interviewed did report that they had given direct input into the decision making about projects, localized outreach appears to be an important strategy. One remarked that “people” came to talk about the project before it was built, and that pipes and pathways were discussed. Another respondent explained that a member of the community had been invited to see the project site and then asked to help mobilize people to help with the project. While as in other cases, it appears that the project had already been selected when this agreement was sought, this respondent explained that there were meetings and people were asked for their agreement to conduct the project, As the experience of communities with respect to implementation appears fairly uniform, isolating best practices in this area is less relevant. However, we note that un terms of promoting the ability of the poor to participate, and potentially exploring the use of PNPM- Urban as a job-creation mechanism, in three sites, focus group respondents reported that all community-contributed labor was paid for by PNPM-Urban, although at a below market wage. Some beneficiaries also noted that workers from the community were paid, albeit in the form of a stipend or wage below market rates. However, we note that it cannot be conclusively established from these sources whether the laborers in these instances were actually poor and how this payment was viewed, as none of the respondents in our data collection reported actually receiving such payments. 20 In the interviews, projects generally mean “infrastructure projects” but respondents are free to interpre t this broadly. 21 This may be because educated residents already know more about PNPM-Urban, or because it is easier for them to understand as a result of outreach. 39 In sites where female participation was said to be “good” relative to total community participation (which may itself have been low), men and women were reported to participate equally in designing and execute projects. However, it should be noted that much of women’s reported participation in decision making almost always largely reflects the invited presence of women leaders who are already active in the PKK (family welfare groups) that focus on “women’s concerns.” In one instance, the respondent noted, “women are strong in Surabaya, they speak their mind.” In another, the lurah’s wife actively promoted women’s involvement. This pattern is noteworthy because PNPM-Urban was designed to foster greater women’s role in decision-making, not merely reflect whatever levels of female participation already were present due to other factors. Indeed, one facilitator remarked that some other facilitators were too willing to accept a “local culture” argument.” Alternatively, as noted by one respondent, facilitators could potentially explore more creative approaches to recruit other women, such as outreach through informal groups such as arisan and Koran study groups. In terms of implementation of infrastructure projects, another noted that women could be more likely to volunteer given that they tended to be at home during the day. On the broader picture on whether PNPM is meeting the needs of the poor, it seems evident that it does perform a valuable and recognized function. However, while determining a coherent set of first-order priorities is less clear given the nature of the data, more consideration could be given to the balance of funds between infrastructure, economy and social. Given that previous studies have found the provision of credit through PNPM to be less value-added, this might imply greater consideration towards “social” activities, although clearly there remains a strong constituency who value the provision of infrastructure. 40 5 Links With Local Government In 2009, at the inception of the latest project cycle, World Bank review of the program found that few BKMs were coordinating other sectoral poverty programs. The findings and “lessons learned” included the need for a broader communications strategy and training program targeting local governments (World Bank, 2009). In this section, while we do not cover the topic of sectoral poverty programs, we present findings that address the following evaluation questions: How does PNPM-Urban actually link with local government? Is there much collaboration? Is there duplication? In communities where this collaboration works well, what are the contributing factors? Similarly, where it doesn‘t work well, why is that? In principle PNPM-Urban promotes collaboration with local government through the Renta, which is presented to the kelurahan Musrenbang at the start of each year to be integrated or adopted into the local government’s planning. Local lurahs also have some responsibility to review and sign-off on community project proposals. At each administrative level, the Ministry of Public Works acts in accordance with the local government to form a coordination working group (Tim Kordinasi Pengelolaan Program) to provide information, technical support and to solve administrative problems. In addition, PNPM-Urban facilitators work with the BKMs to form associations (BKM Forums at the kecamatan and kota level) to work collectively and to influence government planning, while also building local government capacity to work with BKMs and the BKM Forum. In practice, our findings suggest that while there is coordination and little duplication, but at the same time there is also a sense of competition and a low level of true collaboration. Due to the nature of the questions, this section draws mostly on the interviews with key informants. Pilot testing of the focus group protocols made it clear that residents, who knew little about PNPM-Urban process in general, lacked knowledge of PNPM-government relationships. What is working well, and why? Respondents described PNPM-Urban and the musrenbang as useful complements: the musrenbang is a valuable mechanism to coordinate the allocation of local government funds to projects that are not being considered by PNPM-Urban. Participants in the musrenbang meeting can thus learn about projects that will be funded or carried out by the different programs. For instance, a camat stated that: “When they are meeting for musrenbang and find that there are some programs that cannot be implemented, PNPM-Urban will carry out the programs.” Likewise, a lurah of one of the sites remarked that: “programs identified through the musrenbang process are allocated to: government, PNPM-Urban, and other programs.” Others noted ways in which PNPM-Urban has a comparative advantage over for local government. respondents noted two primary advantages of PNPM-Urban relative to the existing musrenbang process which make it a useful substitute in certain contexts. First, they noted that the PNPM-Urban process is better than the musrenbang process since it is much faster (typically, the musrenbang meets annually). Second, respondents perceived PNPM- Urban as more in tune with people’s needs than the musrenbang process which they described as politicized. For example, an NGO representative in Pematang Siantar remarked that the musrenbang process was ineffective in addressing the community’s needs because the only people who attended the meetings were RT heads. A PU official in Sulawesi observed that “the musrenbang is captured by elites. The process of selecting priorities for funding is unclear. They want show-off projects that don’t satisfy the needs of the people.” Respondents also noted that “community proposals get pushed aside for government proposals in the musrenbang” and that unlike the process of PNPM-Urban, “there is no pre-musrenbang process that explains plans and policies to people in order to enable them to join the discussion” and “only general issues get raised in the musrenbang, not the details of what people need.” Actual involvement by local government officials in PNPM-Urban activities varied across sites. At the village level, lurahs that were interviewed demonstrated robust understanding of PNPM-Urban discussions and activities. In most sites, there is a good relationship between the lurah and PNPM-Urban, with little or no problem with approval of projects. A few other types of respondents also emphasized the active role played by the lurah in some of the kelurahan. In Pancuran Gerobak, one respondent remarked that there was a close working relationship with the lurah and that it was impossible to implement any project without the lurah knowing about it. Similarly, Public Works is charged with the responsibility of developing standards for material costs, verifying sites and monitoring that the construction of infrastructure projects is done according to guidelines, but the DPU’s level of engagement in PNPM-Urban activities differed across sites. In some cases, their involvement was significant and went beyond simply monitoring projects. For example, in one site the DPU provided input and feedback on the technical aspects of proposals. In another site, DPU officials heavily interacted with the KSM about the construction process to ensure that the final quality of the project was up to standards In most cases, as noted, the PNPM-Urban process is faster and more effective than the musrenbang process. This has implications for their relationship, as it means that the local government plays the more reactive role. In other words, via the musrenbang, the government adapts and decides not to do the projects that the PNPM-Urban is doing, rather than the other way around. Often, participants in musrenbang will decide to let PNPM-Urban do the urgent projects even if they had had them in the musrenbang plans from previous years. As noted by the askot in a Sumatra site: “PNPM-Urban handles projects that cannot be (immediately) done through the musrenbang but that are urgent and then PNPM-Urban coordinates with musrenbang to avoid overlap.” What areas are there for improvement? Even when relations were seen as positive, the overwhelming majority of respondents, across sites and in different positions, expressed the view that, as far as the scope of activity is concerned, PNPM-Urban and local government bodies maintain mutually-exclusive domains. While on the one hand this implied little or no duplication of activities and priorities, there was also little actual collaboration. In other words, we found little evidence of coordination of investment. Some respondents felt that there existed competition between PNPM-Urban and 42 the local government to please local communities that should be avoided. This view was especially strong in Jakarta Barat. Where relationships between PNPM-Urban and local government officials were not positive, the underlying factor most commonly reported was that the lurah was unwilling to participate in PNPM-Urban. In one case, a BKM member there noted that: “There has not been good coordination between PNPM-Urban and the current lurah; [the lurah] will sign off on projects but doesn’t care to be involved and doesn’t share information with PNPM.” He noted the lurah had been invited to meetings but did not attend them. In other cases, complaints included that their lurah was ignorant about PNPM-Urban activities, failed to attend community meetings despite getting invitations and was unwilling to cooperate. However, there are also instance of estrangement that could be attributed to lack of outreach on the part of PNPM-Urban : one lurah remarked that he hardly met with the PNPM-Urban facilitators. (In contrast, and as a marked exception, the lurah in one site in Java was perhaps even too involved: he helped raise awareness about the program, attended PNPM-Urban meetings but also monitored the work of KSMs, which is beyond his role). In a site in Sumatra, the DPU had relatively little interaction with PNPM-Urban and only provided nominal oversight. In an East Java site, the head of the BKM stated that he had never met anyone from DPU. Even the PU Department Heads remarked that, in general, DPU had very little knowledge of PNPM-Urban and quite a few misperceptions about the program. PNPM-Urban also calls for facilitators to train for local government officials. However, across all sites, there was little mention of such training. In some cases, lack of communication has created problems when PNPM-Urban starts a project that the government had already planned. The Bappeda in a large city remarked that: “PNPM-Urban is always in a rush to build projects which leads to overlapping problems .” By and large, however, as we have noted, problems with duplication and overlap do not seem significant. Some respondents felt that more coordination is still needed to help avoid some remaining inefficiencies and duplication such as approving parallel funds for the same program. Notably, however, even among those who felt that more coordination would be helpful, most wanted PNPM’s community participation structure to remain intact. The satker above emphasized that the program, if transferred, “should be given to the people to be in line with the principles of community driven development/community empowerment.” The concern on the part of many of the informants interviewed, however, is that some local governments would not be able or willing to adhere to this principle. Best Practices and Recommendations for Consideration Where links to local government work best, respondents most frequently cited the active use of the musrenbang at the kelurahan level, and secondly, the commitment of the Department of Public Works and the head of village (the lurah or kepala desa) to effectively communicate government activities with BKM members. 43 Coordination between PNPM-Urban and other government projects is said to work best when the same individuals are present at both the meetings for PNPM-Urban and those for musrenbang. For example, even when the lurah is not actively involved in PNPM-Urban decision-making, he is still usually present in the most important PNPM-Urban and musrenbang meetings. BKM members tend to be individuals very active in their communities (some are leaders of an RT, an RW, etc.) and so they are likely to participate in musrenbang meetings as well and personally disseminate information to the attendees about PNPM-Urban plans, hence leading to the avoidance of duplication. Several respondents gave specific examples of how the active engagement of the Department of Public Works (DPU) and the lurah with PNPM-Urban serve to avoid redundancy in infrastructure activities. One of the BKM members of Ngestiharjo stated that the lurah helps avoid duplication by passing information about the musrenbang to the RW heads who then pass it to the community. In some sites, DPU staff attended PNPM-Urban socialization meetings to convey information on planned government projects to avoid redundancy. Public Works officials in several locations stated that they felt PNPM-Urban was complementary to their work. In one site, the Public Works informant said that PNPM-Urban was good at small projects while DPU is better for large infrastructure work. He noted that small community projects like those done by PNPM-Urban would never be done by DPU. Any recommendation taken to draw local governments and PNPM-Urban together should be nuanced with care. Many respondents expressed mixed feelings concerning whether there should be more explicit coordination between PNPM-Urban and government at the Kota level. On the one hand, some expressed concern about further integrating PNPM-Urban with local government because they feel PNPM-Urban should remain an independent structure. On the other hand, some respondents said there was scope for more coordination, although, regardless, very few (even among Kota officials) expressed the view that PNPM-Urban should be highly integrated into local government. An extreme view of this is that PNPM-Urban should be subsumed by local government - for instance, the satker of a city in East Java considered it natural for PNPM-Urban to be eventually managed by the kota government The main concern voiced by many of our interviewees is that the involvement of local government could reduce community ownership. As a camat noted: “If the government gets involved in the program again it is possible that there will be another mess. When the government takes control of the program, I fear that … community participation and cooperation will fade away.” Similarly, other respondents expressed fear of party politics, waste and misuse of funds and in general a slowdown in execution of projects. Even among Kota officials there were many cases of opposition to the idea of enhancing the involvement of their own offices with PNPM-Urban. These respondents felt that if local government were to become more involved in PNPM-Urban, its participatory mechanisms could be jeopardized. For this reason, the Head of the PU of one Kota in Sumatra stated that he would opt to keep the PNPM-Urban and the local government structures separate and even suggested that some current PU projects should get done through the PNPM-Urban mechanism since execution is better in PNPM-Urban. 44 6 Facilitators This section presents findings on capacity building and effectiveness of PNPM-Urban facilitators. According to the project design, facilitators should be involved in every stage of the PNPM-Urban process. We explore the following questions: What roles do kelurahan facilitators actually play? How effective are they in meeting the needs of the communities? What are the characteristics of strong vs. weak facilitators? What role might kelurahan-level facilitators play going forward? The findings are from the interviews with key informants, in which facilitators were discussed in depth. The facilitators themselves were one of the key informant categories. We present their perceptions separately below. In addition, findings from the questionnaires to BKM and KSM leaders are also presented.22 A. What roles do facilitators actually play? According to the key informants, facilitators are responsible for five main types of activities:  Assisting the KSM – includes providing guidance with proposal writing, budgeting and cost control, and technical guidelines for projects; infrastructure maintenance; prioritization of projects; etc.  Assisting the BKM – includes supporting the BKM with, among other things: understanding the financial management of funds; prioritization of projects; setting up KSMs (in ND sites, they help establish core planning teams, called PIPP); and so forth.  Socialization and community outreach – through such activities as putting up signs in strategic public areas like the market to announce PNPM-Urban activities, hosting ‘exhibitions’ in city squares, promoting cooperation within the community in the areas of PNPM-Urban in which they are active (infrastructure, economy and social assistance), etc;  Supporting community mapping (either with BKM or in ND communities, with the core planning teams);  Project implementation inspections – including monitoring construction works (often alongside PDU representative), checking on materials used, ensuring compliance with guidelines, and so forth. What is working well, and why? The questionnaires used with BKM and KSM members to draw quantitative data on their activities included questions on kelurahan-level facilitators. Table A.1 in the Annex shows that among the 13 BKM in non-ND sites who completed the questionnaire, 11 meet with the facilitator every month and 2 every three months. This suggests that in terms of frequency of interaction, BKM mostly have regular access to facilitators. The main types of assistance and 22In the focus groups, awareness of facilitators’ roles and activities proved to be almost non-existent among participants. 45 support provided by facilitator to the BKM and the frequencies with which these were mentioned are as follows: All but one BKM were helped in financial reporting; most got help with project monitoring and implementation, and 8 of 13 got help in financial management. The ‘other types of assistance’ provided by the facilitator to the BKM included: assisting in composing the accountability report; assisting the korkot; composing the accountability report; conveying information from the higher level of the PNPM-Urban structure; invite KSM and the community to discuss projects; proposal mentoring; socialization about PNPM-Urban; and providing technical assistance on PNPM-Urban infrastructure. Each of these was mentioned by at least one BKM coordinator. In the KSM questionnaire (Table A.3), respondents were asked about the frequency of meetings between facilitators and KSM. Out of the 19 members of active KSM who responded to this question, 47 percent (nine) said they met with facilitators at least once every two weeks, and 31 percent (six) meet once a month. The rest (four) meet once every three months or less. This too suggests a fairly frequent interaction with facilitators. In addition to the roles above, facilitators themselves noted that they must all be ready to cross -support— i.e., the infrastructure facilitator helping with socialization, the community empowerment facilitator helping oversee infrastructure projects, etc. Most key informants demonstrated a good awareness of the roles of the facilitator. Most interviewees, including BKM and KSM members as well as local government officials, agreed on the importance of facilitators for PNPM-Urban, as they play a key role in some of the most central elements of the program, as described above. Highlighting their centrality to the PNPM-Urban process, many interviewees explained that when facilitators cannot do a good job supporting the KSM and BKM effectively, the latter two cannot in turn be effective, so it is important for these groups’ performance to have the backing and assistance of an able facilitator with adequate capacity and skills. What areas are there for improvement? In spite of the acknowledged importance of the role to PNPM-Urban in each kelurahan, many key interview respondents outside the BKM/KSM did not personally know their local facilitators; they more often knew their BKM. Interviewees’ perceptions of the effectiveness of facilitators in their various roles were mixed: some felt that the performance of facilitators they had encountered was satisfactory. However, of those who were able to comment, the majority of informants stated that facilitator performance was often below expectations. Of the latter group, most acknowledged that the problems were often due to program rules (more on this below) and other circumstances, and not because of the facilitators themselves. It was notable that many of the sekda, lurah and local NGO representatives interviewed did not know much about the facilitator in their kelurahan and were therefore unable to comment on their performance. B. What makes for good facilitator performance? What is working well, and why? Among interviewees satisfied with facilitators’ performance, positive comments included: “facilitators visited construction sites often,” “provided technical guidance to the KSM,” “share information with the people, which the people appreciate” or “did a good job preparing 46 the community” to become more empowered. A small number of respondents mentioned instances in which facilitators identified errors or problems in the development of an infrastructure project, which the community were therefore able to address prior to project completion. A few respondents, especially BKM leaders, commented on the attributes that make for a good facilitator, which included:  Institutionally savvy;  Good inter-personal skills;  Friendly;  Awareness of the needs of the community;  Understanding of local culture;  Good communication skills – including the ability to tell the community what can and cannot be done within PNPM-Urban;  A proactive attitude. One interviewee stated that facilitators must be able to say “No” to communities. He felt that they should be more active partners, assisting the community to make the best choices for local development instead of blindly following proposals. A few facilitators, korkot, and even BKM coordinators stated that facilitators must support BKM, KSM and communities more generally without creating dependence. As one facilitator noted: “One day we won’t be here anymore.” What areas are there for improvement? Of the group that felt that facilitators’ performance is often below expectations, this referred primarily to the quality and quantity of support they can provide to KSM and BKM, as well as the amount of time they can devote to socialization and community outreach. This view was held mostly by BKM coordinators and KSM members, although askot, korkot and facilitators themselves also often remarked on this issue. According to these informants, there are two main challenges to facilitators’ support of BKM and KSM:  Facilitators do not have enough time to conduct their activities to a satisfactory standard (most widely cited challenge);  Short length of (often unplanned) rotations is a problem. Regarding the first issue, a common remark was that working with community groups (BKM and KSM) to support them in their administrative tasks can become a significant burden on facilitators, given the heavy administrative requirements of the program. This is especially true of the time they spend helping the KSM in producing technical proposals and project reports. Several key informants felt that this administrative burden did not leave enough time for the facilitator to spend time in the community, building capacity or conducting PNPM-Urban outreach activities. A few respondents also said that facilitators work with a greater number of communities and groups than they can effectively handle. This reduces the extent to which they can familiarize themselves with the situation and needs of particular communities, and the number of times they can visit each site and support the KSM and BKM in their roles. One respondent explained that the workload of the facilitator in part depends on the number of RTs in a kelurahan; as a result, he argued, the range of responsibilities allocated to a facilitator 47 need to take the number of RTs per kelurahan into account. KSM members in particular remarked that they would like more assistance than they currently receive from the facilitators on tasks like project budgeting, proposal writing and training on technical standards. These respondents recognized that the insufficiency of these kinds of support often stem from the intense demands the program places on facilitators’ time. With regard to the second problem, while many felt that the length of rotations was not particularly problematic, others argued that short rotations are a hindrance to effective facilitator contributions. One BKM member, for instance, said that “if facilitators rotate often, they cannot build a strong connection with the community.” Another BKM member stated that if a facilitator is involved with a project starting one year which will be implemented the following year, the same facilitator should be in place to see the project though, as different facilitators sometimes have different requirements for how to carry out certain tasks (like fill out forms). A third, related complaint was that rotation of facilitators impedes completion of projects: positions often stand empty, communities do not have the same level of support for PNPM-Urban activities, and the result is delays that carry into the next fiscal year. More broadly, several interviewees complained that facilitators often are not sufficiently familiar with the specific circumstances and needs of the communities in which they work, possibly in part because of the short periods spent in their role in each site. One interviewee, for instance, commented that rotation of facilitators needs to allow for adaptation time. In addition, a number of interviewees (in particular BKM members), complained that facilitators are sometimes not familiar with the specific problems of the communities in which they work, a problem some but not all interviewees explicitly linked to the issue of rotation. Others argued that some facilitators are not fully aware of the rules of the program and expected activities in their areas (infrastructure, social or economy). An interviewee from Sibolga said that as he attended technical school, he could identify instances where the infrastructure facilitator provided incorrect informant. Another, from Pematang Siantar told us of a facilitator who worked in economy but knew very little about the sector, which required a more senior facilitator to come to village to assist. Facilitators themselves were keenly aware of the limitations they face in performing their roles, especially regarding time constraints. For instance, speaking of the challenges posed by heavy administrative burdens, one facilitator remarked that “spending time in the community is sacrificed for administration.” This is very important given the centrality of facilitators’ role in empowering the community, not just in assisting the KSM and BKM. Another facilitator said that facilitators often have to hold several meetings simultaneously due to lack of time to arrange each meeting individually. These comments were echoed by a third facilitator who said they have too much work – and are spread too thin across various communities – to be effective in their roles. This facilitator explained, for example, that many of the socialization activities are left to community leaders, when in fact the facilitators themselves should be organizing them. Short rotations may have harmful effects here as well. One facilitator commented that the current length of contracts (often nine months or one year) is too short to enable them to “empower the community.” 48 According to several respondents, a third challenge faced by facilitators is a lack of adequate training for their various roles and responsibilities. Interviewees argued that facilitators do not receive training of sufficient quality on many aspects of their work, including “teamwork,” “prioritization of objectives,” “administrative work,” “dealing with people” and so forth. In particular, facilitators across several districts remarked on the quality of training they receive, saying for example: “Our two week training is good, but it doesn’t necessarily prepare us for how to interact with people. Dealing with people effectively is one of the most challenging aspects of our jobs – for example, how to talk to a lot of different types of people, how to settle an argument….” An NGO representative who was aware of the work of facilitators remarked that in his kelurahan, facilitators often do not speak the local language, and the poor are not always fluent in Indonesian, a concern for facilitators’ ability to do their job effectively. Finally, a small number of respondents were of the opinion that part of the problem is that facilitators are not uniformly qualified to conduct their work effectively and efficiently; for instance, an interviewee stated that a facilitator working in the economy element of PNPM- Urban “did not know anything about that sector.” While the content of training for facilitators was discussed, and the need for improved training highlighted, the issue of training materials was not brought up. Best Practices and Recommendations for Consideration The question of how to improve on facilitators’ performance was discussed by many interviewees. Most facilitators are responsible for multiple communities and have substantial administrative burdens, which compromises their ability to dedicate sufficient time to each Kelurahan, and to certain tasks, especially socialization. This may be a contributing factor to the relatively low levels of knowledge and direct participation of the community in some PNPM-Urban sites. Because most felt that a significant challenge to good facilitator performance is presented by certain program rules (such as short rotation and high demand for administrative support to BKM), the majority felt that improvements could be achieved by addressing these specifically. Reducing the administrative burden on the facilitators could also be beneficial; facilitators spend substantial time helping BKM and KSM members complete required paperwork and reports. Reducing this burden could free more time to help with fundamental challenges (e.g., participation). It should also be emphasized, as we noted earlier, that this time burden for administrative requirements is borne as well by BKM and KSM members, who after all are citizens who usually have full time jobs. However, there are several tradeoffs to be be considered that make eliminating these tasks not straightforward. Reporting and planning tasks are be important to ensuring PNPM-Urban is well-executed and developing local technical capacity. The administrative procedures involved in BKM and KSM activities are important for the different levels of auditing and thus cannot be eliminated. Nevertheless, any measures that reduce the administrative work would be beneficial for both KSM/BKM and facilitators, and would leave more time for facilitators to carry out other tasks. 49 Given these multiple demands, and in particular the fact that mobilization and community participation needs are not being well addressed, the preferred recommendation, resources permitting, is to devote more resources to put more facilitators in the field to be better able to accomplish their multiple tasks. Otherwise, it may be necessary to revisit the roles facilitators are expected to play in communities to determine whether they should optimally focus on technical training and support to BKM and KSM, or on socialization and empowerment activities within the community at large, or to alternatively consider developing methods of training facilitators themselves to assess these tradeoffs and prioritize their workload. The problems highlighted by interviewees regarding rotation are also complex. The tendency to rotate facilitators across sites in some cases limits their ability to develop close, trusting relationship with the community in which they work. However, this practice may limit negative side effects (e.g., cronyism). At a minimum, however, the rotation of facilitators could be changed so that it better coincides with the project cycle in the community they are working in. As discussed in the previous section, many respondents recognized that good training is important to prepare facilitators for their tasks and the challenges they may face on the job. However, although respondents mentioned facilitators’ training as an area for further improvement, there was little consensus on the type of training that would most benefit facilitators in their various roles. Improved training across a number of key dimensions (interpersonal links/socialization; administrative work; etc) may be desirable. A more formal needs assessment, of course, should precede any major expansion of the training. A possible system of ‘rewards and punishment’ was also discussed by a small number of interviewees. An NGO representative familiar with PNPM and the work of facilitators, for example, suggested that “facilitators should not be paid if a project doesn’t work”. However, the subjectivity inherent in the types of projects typically undertaken may lead to issues with the implementation of such a scheme in practice, and in a particularly bad example, may motivate facilitators to seek out projects that are easy to conduct rather than truly reflective of community needs. Significantly more serious consideration would need to be given to such a suggestion than respondents have shown to date. Finally, other initiatives with potential for improving facilitators’ performance, such as mentoring or improvements in the quality of training materials could be explored in further research as additional measures to promote high levels of performance and commitment by facilitators. Piloting a system of incentives, for instance, may be informative. It should be noted that these were not discussed by interviewees. This most likely reflects the prevailing view that most challenges to facilitators’ performance results from program rules than from the facilitators’ own activities or attitudes. 50 7 Control Mechanisms and Governance In Indonesia, the challenge posed by widely distributing funds is steep. As a korkot stated: “there are many interests involved which can lead to diversion of funds.” Governance within PNPM-Urban is thus closely tied to the concept of upward and downward accountability. For day-to-day management, since projects are generally conducted in the community, community members themselves have some ability to directly observe project progress. KSM and BKM project/financial reports are required to be made public annually to hold these bodies to account, and where RLFs are in place, regular BKM meetings are to be held to review the UPK accounts GoI has established two monitoring and evaluation working groups that cover PNPM-Urban: one for Management Information Systems (MIS) and one for reviewing and finalizing performance indicators for PNPM-Urban. In the first instance, the National PNPM Steering Committee has adopted an integrated, shared MIS and website. The PNPM-Urban monitoring system has also developed an advanced web-based MIS that is open to the public, providing updates and information on project progress. Several other mechanisms to ensure good governance are built into PNPM-Urban as part of its Better Governance Action Plan, including (i) randomized audits; (ii) severe penalties for misuse of funds; (iii) rewards for well performing BKMs and KSMs; (iv) an accountability matrix for all operations; and (v) a complaints handling service. A. Are current governance and control mechanisms adequate? Program funding is directly transferred from the treasury office (KPPN) to the BKM, recorded in the government accounting system, and included in government accountability reports. The PMU prepares a separate set of project financial reports that are suitable for project monitoring purposes. The PMU and Local Project Unit (PIU) at the kabupaten level maintain separate accounting records for all payment requests (SPM) and remittance orders (SP2D), on a cash basis. The facilitator plays an important role, as he or she not only assists the KSM and BKM with financial reporting, but also has to provide countersignatures in order to enable funds to be withdrawn by the BKM contingent on acceptable plans and reports on how funds will be used . Cash balances held in the kelurahan by any party are kept to the minimum possible. Apart from GoI, the World Bank and other program funders have a critical role to play in monitoring and evaluation. The provisions of PNPM-Urban call for World Bank annual site visits to 2 percent of kelurahan, 15 percent of kota, and 100 percent of provinces. The NMC coordinates these monitoring efforts with those of other funders. What works well and why? Based on our respondent interviews, we found very few reports or complaints of misuse of funds within PNPM-Urban. While some of this may be attributed to reporting bias, in many cases respondents were able to cite many layers of control mechanisms and auditing in PNPM- Urban that help avoid these problems, including multiple levels of formal auditing of BKM and KSM finances by facilitators, askots, korkot, kota level staff from the Department of Public Works as well as independent auditors. PNPM has, by design, many layers of control mechanisms and auditing that aim to avoid misuse of funds and detect irregularities in the implementation of projects. The Korkot in a West Java site stated that “Facilitators go over the balance, make analysis of the construction, and then they have discussions”, but then added that the integrity of the finances is assured by independent auditors hired expressly for this purpose. The Sekda in one of our sites highlighted the importance of the Kota DPU in verifying and monitoring the “final quality of construction”. In addition, many respondents attributed a purportedly low prevalence of corruption to the participation and implicit monitoring of community members. Most remarked upon the importance of the combination of official guidelines with community informal monitoring. The Sekta of another Kota that we visited highlighted the role of the community and stated that: “the risk of misuse of funds under PNPM is lower because the projects are implemented by the people themselves”. In that same site, a BKM member concluded that: “projects built by PNPM are high quality because there are many control mechanisms, internal and external monitoring: [the] community, BKM, facilitators, askots, etc.” He also added that the monitoring of community members is also important: “internally, the people in RT and RW monitor while they are working so it is very effective.” What areas are there for improvement? While we did not encounter many occasions where respondents explicitly said that they knew about misuse of funds or corruption, some instances were in fact described by korkots and by NGOs. For example, an NGO respondent from a Sumatra site related a case in which a KSM member took the money from a project and left the village. No cases were related that directly referred to the kelurahan we visited (the respondents had only heard about cases that happened elsewhere, or chose to report them as such). There are two possible ways to interpret why few interviewees reported cases of corruption. One is that this shows effect of strong program governance in limiting the misuse of funds, at least compared to other public programs in Indonesia. The second possibility is that misuse cases exist but respondents are not willing to talk about them. Some interviewees commented on problems that reduce the efficacy of monitoring. In terms of project implementation, the most common problems were either that the projects built departed from plan or the mix of materials used was different to the one specified in the project proposal. Other respondents felt that the problem lies not in the inadequacy of monitoring, but in the soft enforcement of sanctions when problems are uncovered. A korkot stated that “there are no sanctions when there is a problem of misuse of funds [we just] try to keep from happening again.” A PU official explained an occasion where “irregularities were found in the audits but the next tranche of funding was still released.” One korkot acknowledged that although “there are no complaints about financial issues from the community,” “some people feel that facilitators are lax about quality control.” Limited quality control is perhaps not surprising given the large amount of work expected from 52 facilitators and the relative complexity of the documentation required for each project (see subsection 4.3). Best Practices and Recommendations for Consideration The solution often suggested by respondents for reducing the probability of leakage is to add additional reporting requirements. For example, one respondent stated that the recent simplification in proposal and project reports (PJM), make it harder to monitor correctly: “the previous PJM format is better because it made it difficult for people to steal funds. The current format makes it a bit easier to hide irregularities.” Worrying about fund leakage, the former satker said that the korkot, senior facilitator, and facilitator should be required to sign the funds disbursements. However, the PNPM-Urban reporting requirements are already quite substantial and place a significant burden on both facilitators and volunteers such as KSM and BKM members (we discuss this in detail in the subsequent subsections). This problem is compounded by the low skill-level of many of the individuals involved in reporting. One interviewee stated that “the problem is complex book-keeping system since many of the people involved have had only primary schooling” for six years. Another key informant stated that the “KSM struggle with accounting, financial reporting and claim not to know the standards and norms for expenditures and hence there were problems with financial accounts.” We discuss in a separate section (7) the role that the MIS could play in supporting reporting and auditing In terms of upward accountability, one BKM member noted that unanticipated audits are the most useful: “Askots are probably the most effective, because they show up un-announced.” A BKM in a different site also highlighted the importance of cross-monitoring between different bodies: “[KSM’s] financial books are checked monthly. Members of the [BKM] sit on the KSM as informal members to oversee the work.” In terms of downward accountability, referring back to the findings of Section 4, community awareness is critical. In particular, best practice seems to be the case in which communities are not only able to generally follow project development but also have specific knowledge of performance indicators and funds disbursed. A BKM member stated “The community knows how much is the contribution from government, and how much it is from them, so they can monitor” One facilitator in Sumatra echoed this, saying that: “[we] haven’t had any problems with implementation because standards come from PU and the community knows about all the projects” B. What is the role of the Management and Information System? The objective of the MIS is to track performance indicators for each cycle of the PNPM-Urban as agreed in the PAD. The data in the online PNPM-Urban MIS principally includes information on the implementation of projects in each level, investment output data, grant management data and complaints/feedback data. These data are collected and updated monthly by PNPM-Urban staff at the kelurahan level, who upload the data to the MIS in MS- Access databases. Much of this information is then extracted by the National Management Consultant (NMC) MIS team in the form of Excel files and made available on the web for 53 public use. What is working well, and why? The MIS hosts and organizes a vast amount of data that is highly idiosyncratic to each project cycle. Each project (UPP2, UPP3, PNPM Urban and AF, and PNPM Urban 3) has its performance indicators The amount and timeliness of information freely available for purposes of public dissemination via the PNPM-Urban website is commendable. The information is grouped into three separate data sets: a ProKel or Kelurahan Profile providing kelurahan and community participation information; a BLM file providing grant management information; and a complaints feedback dataset. The ProKel and BLM files are made available on a monthly basis for the previous few months for all kelurahan in each yearly funding cycle, both separately and in aggregate form. The complaints feedback dataset may be queried via the website. A data snapshot for individual kelurahans may be looked up via the web by any user Discussion with the World Bank PNPM-Urban team suggests that usage of the MIS for monitoring is reported to be widespread. The World WB, NMC (Monitoring-Evaluation Specialist/Monev, financial management experts, etc), OC (Regional and Provincial Consultants), City Coordinators, and facilitators are reported to use the MIS to look up information on specific kelurahan as well as aggregate data. The MIS is the principal source of information for performance indicators and a primary tool for the monitoring team in the Coordinating Ministry of People Welfare and Bappenas. In addition to its primary monitoring function, the MIS has a potentially large role as a coordinating mechanism for a wide range of stakeholders with interest in PNPM-Urban. PNPM-Urban MIS regularly feeds the data into the integrated MIS of poverty program managed by Bappenas. Local governments, particularly the provincial and kota/kabupaten project managers and some local governments who already runs e-government, also reportedly use the MIS. A final group of users are members of the public with interest in PNPM-Urban , specifically the Coordinating Ministry of Social Welfare, National Planning Agency (Bappenas) or university students for their thesis or Dissertations. A large number of quality controls are applied in all levels from facilitators up to NMC including through trapping and control system in the application. The basic procedures (ideally) are applied below:  Facilitator level: the senior facilitator verifies data filled in by their team in the logbook before reporting the data to the city coordinator  City Coordinator level : the city coordinator checks and verifies the data logbook received from facilitators  Data Management Assistant (Asmandat) at Kota/Kabupaten level: Asmandat inputs the data into the MIS application based on the logbook which has been checked and verified by the city coordinators and ensure that the data filled in into the application has been checked and verified by the city coordinators.  Provincial Team Leader, Monitoring Expert, and Senior Data Management Assistant (Asmandat) at Province level: regularly check the data uploaded by the city 54 coordinators in the website. They ensure all data from Kota/Kabupaten are complete, consistent and clean from the data anomaly problems  NMC team leader, monitoring expert, and MIS team: regularly check completion and data accuracy and provide feedback to the provincial team. The problems found by NMC or reported by the provincial or kota/kabupaten team are recorded by the NMC. To enforce the validation process, the NMC has designed a monthly MIS day during which all its agents are requested to only focus on MIS data review. The NMC also provides a teleconference during that day to communicate with the provincial and kota/kabupaten consultants and to help with problem solving. Finally, although there is no special audit applied, a monthly review is conducted to check for data completeness. What areas are there for improvement? While these quality protocols are stringent, one problem is that the validation procedures are conducted manually in all levels and staff reportedly may not adhere to the protocols. At present, almost 3050 employees are involved with data gathering from the NMC to its facilitators, of which almost 250 are dedicated staff. Reportedly, cost is not a limiting factor in improving the MIS; rather personnel capacity or incentives may not be fully engaged. It is to be recognized that the scale and scope of the MIS itself is extremely challenging and evolves on a continuous basis with the program. For instance, in 2010, the GoI altered the primary format of data reporting, rendering the glossaries for previous data inconsistent for the time being. Furthermore, information in the consolidated MIS originates from multiple sources, including the World Bank as well as other donors such as the IDB. Considerable effort is taken to ensure that similar standards apply to the end data, and slippage cannot always be avoided. This leads to particular features of the data that are difficult to navigate for the beginning user. Some users report that individual guidance from consultants is needed to navigate the website and find appropriate data; local governments specifically with low capacity may have difficulty with this resource. More broadly speaking, while the MIS data format is developed and hence adequate for its main objective of tracking project performance in each cycle, the existing database structure is not well-suited to users that want to download and use a broad set of information (e.g., for large-scale data analysis), nor are the current data designed for users who want to assess or compare data or statistics for one or more kelurahan, kecamatan, or kota. At present, for instance, constructing a comprehensive, multi-year dataset with comparable data items for all kelurahan is extremely difficult and thus presents a large obstacle to a comprehensive analysis of the PNPM-Urban MIS data. Unfortunately, given issues of inconsistencies in format, quality, and completeness of the data, the construction of such a dataset at present is highly labor-intensive. Annex 2 in the appendixes shows some illustrative analysis that can be undertaken with the MIS data but also highlights some of the issues with data quality. 55 Best Practices and Recommendations for Consideration To address some of the data quality issues in the MIS, strong efforts are currently being undertaken by the PMU and consultants to validate the data. A tool to strengthen data analysis is being developed by the NMC, which will automatically detect problems of data completion or anomalies, including input from earlier discussions with RAND, which should be extensively tested. Other data quality control measures including periodic error checking, data audit, and random check are currently under consideration. These efforts Given the demand for MIS content, and the evolution of different project cycles, a re- evaluation of the MIS database structure is timely. The NMC should explore conducting a survey of existing and potential MIS data users to determine (1) what p2kp.org data resources are currently being used and (2) what types of data key user groups would most like to see made available. This could also be supplemented by a collection of usage statistics. The MIS is a valuable resource, and developing a clearer understanding of users’ needs will help the NMC efficiently allocate resources to improving data availability and access. A benefit-cost analysis could be conducted To the extent possible, the benefits provided by the MIS data should be compared against the costs of providing that data. Namely, data or variables that are not used but are costly to collect should be evaluated for exclusion. On the other hand, key types of data could be considered for addition, including individual project data, that is, a list, by kelurahan, that enumerates all projects, indicating project type, funding, description, location, and completion date. This information is currently only provided on an aggregated basis by project type. Therefore it is not possible using the MIS to identify, for example, the location or budget for a specific infrastructure project. This data has to be obtained from individual BKMs. At the same time, in addition to content evaluation, some user interface testing could be conducted, and certain features improved For instance, staff are currently working on ensuring future glossaries and data items that are consistently harmonized across the various funding cycles, a step that will allow various sets of files for different years to be easily compared. A data catalog providing a concise list of available MIS data (e.g., BLM, complaints, PM) should be published prominently on the MIS web site. The list should include all data types (and associated glossaries), even if some data sources are no available online. Protocols for data use e.g. a data request process for obtaining sensitive data should be set up with clear requirements to obtaining access to data not published online. Resources allowing, other more advanced features such as a web-based dataset selection tool could also be considered. 56 8 Infrastructure Quality Assessments Previous work by Mulya (2010) focuses specifically on PNPM-Urban infrastructure quality and effectiveness. Mulya conducted an assessment of 86 P2KP infrastructure projects in 12 kelurahan in six cities across three Indonesian islands. The study was designsx to assess infrastructure technical quality, project usefulness to the community, financial characteristics, interactions with local government, and areas for improvement. The quality assessment found approximately 25 percent of projects scored in the highest (“good”) category, while nearly 40 percent of projects received a “poor” rating, the lowest category. Combining quality assessments with project funding data, the study concludes that lower quality projects tended to have higher costs, which Mulya argues implies wastefulness in project spending. However, they also conclude that P2KP projects are built at lower cost than equivalent government projects or contractual work, in some cases 30 percent less expensive. The researchers also conducted focus groups and in-depth interviews to assess social and programmatic factors that assess infrastructure quality. However, they find few strong associations between qualitative metrics and technical infrastructure quality. The report is not without its limitations, however. The wastefulness assessment does not account for differential project costs, which could account for correlations between project costs and quantities constructed. In addition, as the authors admit, the qualitative assessment produced limited data to support the analysis. Finally, Mulya (2010) assesses projects from 12 sites but only across six cities, leaving the analysis susceptible to selection effects. In addition, the analysis draws conclusions about many topics that are not clearly based on data collected as part of the study. In this section, we address questions related to the quality of PNPM-Urban projects, the adequacy of current standards and what factors contribute to the high and low quality, and cost-effectiveness. Our primary approach to assessing PNPM-Urban infrastructure projects quality was through a formal engineering assessment using engineers to evaluate projects based on standardized protocols for specific project types. The goal of the infrastructure quality assessment was to determine whether PNPM-Urban projects met basic construction and materials standards and practices.23 In addition, we also incorporate information from the qualitative data collection. The six infrastructure categories assessed were: roads (jalan), bridges (jembatan), drainage (drainase), public toilets (MCK), housing projects (RLH), and clean water projects (air bersih).24 Within each category there was considerable variation in the project types and project characteristics. For example, road projects included three surface types: concrete, paving block, and asphalt. “Water projects” is a broad category that includes drinking water, bathing, and washing projects, and water sources included wells and natural springs. “Housing” projects included upgrades to private homes as well as improvements to facilities, like schools, used by many community members. “Drainage” projects were often uncovered 23 The 243 projects were from the 13 non-ND sites. In the three ND sites, we assessed 69 additional PNPM (non-ND) projects. We have excluded the PNPM-Urban projects that were completed in the ND sites from the main analysis reported here, but we have included basic summary statistics for these projects in Annex 6. 24 Projects that fell outside of these categories were not assessed. However, in many cases project categories were interpreted broadly to reduce the number of projects that could not be scored. 57 drains that ran alongside roads or paths, but this category also included projects such as drain covers. The infrastructure assessment was conducted by four Indonesian civil engineers, with one engineer working with each RAND field team.25 RAND obtained lists of completed PNPM- Urban projects from the MIS database with the assistance of the World Bank office in Jakarta. The MIS lists were supplemented by lists that the engineers obtained directly from PNPM- Urban officials at each site. In many cases, the two lists of completed projects—MIS and local—were not consistent, and to ensure a complete list of projects from which to sample the engineers constructed a combined project list. In each kelurahan, the engineer reviewed up to 20 infrastructure project across the six types of projects. In kelurahan where fewer than 20 projects had been completed, the engineer assessed all completed projects. For sites where more than 20 projects had been completed, projects were selected so that the distribution by category was roughly proportionally to the actual frequencies of each project type. The sampling was also conducted so as to ensure a mix of projects from different PNPM-Urban cycles in each site. Finally, where a subset of all projects of the same was selected for review, selection was done randomly. The engineers conducted the projects assessments using a combination of visual inspection and, where necessary, communication with knowledgeable parties (e.g., BKM members, facilitators). Discussions with individuals who had knowledge of the projects were necessary in situations where operational characteristics of the project could not be addressed through direct observation (e.g., drainage overflow during rainy seasons). As noted in Section 3, projects were assessed against multi-criteria scoring sheets shown in Annex 5, which included categories such as materials used, project specifications (e.g., road surface width), maintenance, and performance. The scoring sheets, which were previously used for the UPP Qualitative Study of Infrastructure (Mulya 2010), were provided to us by the World Bank and modified for use in the field.26 The forms contain a variable number of scoring elements depending on the type of infrastructure. The scoring elements are based on Indonesian government standards for each type of infrastructure. For each scoring element, the engineer assesses a project on a three point scale ranging from good (two points) to sufficient (one point) to deficient (zero points). The scoring elements are combined into a composite score, which provides an overall assessment of infrastructure project quality. The composite score is the total points received out of the total points possible.27 Consequently, the output of the engineering assessment is a percentage score for each sampled project and overall summary scores for each of the six types of infrastructure. The engineers summarized the results of their assessments on the scoring sheets and provided detailed written comments to address any issues not covered by the scoring criteria. We compiled the engineering scores into a master infrastructure project database. We conducted no materials sampling or assessment. In total, 243 projects were scored. A. What is the overall quality of infrastructure and are current standards adequate? 25 The engineers consulted with the RAND team about assessment protocols prior to field work during the training, collectively assessed projects at the two field test sites, and then conducted their assessments of the remaining sites individually based on the finalized protocols. 26 For example, the scoring sheet for roads only covered two of the three applicable road types, so the engineering team added characteristics for asphalt roads to increase the number of projects eligible for scoring. 27 For some projects, certain scoring elements were not relevant and thus excluded from the total. Because of this, the total points possible varies not only by project type but also across specific projects. 58 Bridges constitute the smallest number of projects (3 in total, or 1 percent), while roads make up the largest project type (110; 45 percent). Table 4.1 provides summary statistics of the scores from the engineering quality assessments. Average scores for each infrastructure type range from a low of 59 percent (roads and bridges) to a high of 79 percent (housing projects and public toilets). Within each project type there is substantial variation in scores. For example, drainage scores range from a low of 11 percent to a high of 100 percent (meaning the project earned the top score for each relevant scoring element). Table 4.1 Summary statistics for infrastructure quality scores N Mean (%) SD (%) Min (%) Max (%) Road 110 59 12 32 100 Drainage 55 73 22 11 100 Housing 53 79 21 14 100 Public toilet 10 79 8 67 98 Water 12 62 15 33 89 Bridge 3 59 3 57 63 The scoring sheets classify projects as good, sufficient, or deficient based on the project’s composite score.28 Projects that score greater than 60 percent are classified as “good,” those with scores between 40 percent and 60 percent are “sufficient,” and projects scoring lower than 40 percent are deemed “deficient.” Table 4.2 summarizes the percentage of each project type that falls into each score category. It is noteworthy that very few projects received a score that put them in the deficient category—only 4.5 percent. Of the deficient projects, nearly three-quarters were roads and drainage. Thirty percent of projects received a sufficient score, while a sizeable majority (66.1 percent) received a good score. Thus, based on the engineer assessment the vast majority of PNPM-Urban infrastructure projects (95.5 percent) were scored either sufficient or good. Table 4.2: Frequencies for score categories by project type Deficient Sufficient Good N (%) (%) (%) Road 110 1.65 22.22 21.4 Drainage 55 1.65 2.06 18.93 Housing 55 0.82 2.88 18.11 Public toilet 10 0 0 4.12 Water 12 0.41 1.65 2.88 Bridge 3 0 0.82 0.41 Total 243 4.53 29.63 65.85 What is working well, and why? 28Scoring categories were provided with the scoring sheets and may not be standard for non-PNPM-Urban infrastructure projects. However, consultation with the engineering team indicates that alternative classification systems used in Indonesia are relatively similar. We use this convention as a simple way to “bin” projects by quality. 59 Focusing on the 11 projects that received deficient scores, there is no evidence that certain engineers are disproportionately scoring projects as deficient. Six kelurahan have projects that received deficient scores, but across those kelurahan deficient scores were distributed relatively evenly.29 As noted above, roads and drainage projects were more likely to receive deficient scores than other types of projects. However, as a percentage of total projects, water projects had the highest portion (8 percent) of deficient scores. Overall, however, the share of deficient scores is very low. Table 4.3 shows the share of projects for each type of infrastructure that received a deficient, sufficient, or good score. A high percentage of drainage, housing, and public toilet projects received good scores, while for roads and water projects the scores were more evenly mixed between good and sufficient. What areas are there for improvement? Only for bridges—where there were only three completed projects—did a majority of the projects receive a score less than good. As Table 4.3 reiterates, the overall quality of infrastructure projects, based on engineering assessments, is high. Even if we impose a higher standard on scores, nearly one-third of projects (N = 72) received a score of at least 75 percent. Table 4.3: Share of projects in each score category by type Deficient Sufficient Good (%) (%) (%) Road 3.64 49.09 47.27 Drainage 7.27 9.09 83.64 Housing 3.64 12.73 80.00 Public toilet 0.00 0.00 100.00 Water 8.33 33.33 58.33 Bridge 0.00 66.67 33.33 Our analysis and results on infrastructure quality are similar to Mulya (2010) in some ways. The project type distribution for the cities visited by Mulya (2010) closely matches the distribution from our engineering assessment.30 Specifically, the largest fraction of projects in Mulya(2010) was roads and drainage, which were also the two largest project types in in our 13-site sample. However, the quality distribution is quite different, despite both studies using similar scoring instruments. Mulya(2010) finds that the majority of projects (72 percent) are either “acceptable” or “poor,” the two categories equivalent to our “sufficient” and “deficient,” respectively. Moreover, roads projects receive significantly more low scores in Mulya(2010) compared to our engineering assessment, in which score distributions were relatively similar 29 Two kelurahan had one deficient project each, while the most number of deficient projects any kelurahan had was three. 30 For all projects conducted in their selected cities, Mulya (2010) finds 40 percent of projects are roads and 15 percent are drains. The sample distribution for the kelurahan assessed by Mulya differs somewhat, with 55 percent road projects and 29 percent drainage projects. 60 across infrastructure types. Differences in quality between projects in our study and Mulya (2010) could be due to differences in the survey design, including site selection or differences between the time period in which projects were constructed. Key informants had strong and clear views about PNPM-Urban infrastructure quality, and the vast majority of respondents felt that infrastructure quality is high. Nearly 90 percent of respondents who spoke about infrastructure quality believed that PNPM-Urban projects were either good or at least acceptable; a very small number stated that the projects were deficient or of poor quality. Notably, positive responses about infrastructure quality were provided by many different types of key informants, not just those involved in the PNPM-Urban program. Local government officials and NGO members provided similar assessments. Moreover, some local government officials stated that PNPM-Urban infrastructure quality is better than that of local government- built projects. Proportionally, however, local government officials were more likely not to express any view about quality, or quality relative to government infrastructure, than other types of key informants. C. What factors affect project quality? What is working well, and why? Although some key informants had little direct interaction with PNPM-Urban infrastructure projects, others had the opportunity to assess both PNPM-Urban projects and projects conducted either by the local government or by the community directly. Common views include the belief that PNPM-Urban infrastructure is “high quality” or “equally as good as other projects,” and many respondents shared the view that “PNPM[-Urban] infrastructure projects are good, generally much better than government projects.” The numerous respondents who felt that PNPM-Urban infrastructure projects were high quality argued that PNPM-Urban’s control mechanisms contributed to project quality and that the community produces better projects because they are more invested in the outcome. In each kelurahan, focus group and in-depth interview participants were asked about their perception of the quality of the PNPM-Urban infrastructure projects in their communities. Different kelurahan had a range of different projects, and not all project types were developed in each kelurahan. Some of the projects discussed with participants were ditches and dams, new roads or paving of existing roads, building construction, trash wagons or cans, pathways, drainage and/or sewage systems, and toilets. In spite of the differences in the types of projects available, most respondents stated that they thought the quality of the infrastructure was good—in line with the objective engineering assessments reported above. A few respondents in different kelurahan said that they have worked in construction in the past, and in their experience, the materials and craftsmanship involved in the projects in their community was high. One said: “Since I am working as a construction labor, I know about the quality. I saw that the cement use good quality cements, and the reconstruction way is also good.” It is not clear, however, whether people felt the quality of these projects was good in itself, or was good compared to the facilities they had before (either, of course, is a favorable outcome). Typical comments included: “Now the roads are wider and have been hardened, too;” 61 “Previously [a building] was made of bamboo; now it’s not;” “[The road] is pavement and it’s good;” “[The road] is good because it’s smooth; they should do that to the entire area.” Although beneficiaries living near PNPM-Urban project sites are not typically able to assess a project’s technical characteristics, they can provide subjective assessments of infrastructure quality. The RAND teams asked beneficiaries whether they felt infrastructure projects were well-made and of high quality. Views were mixed. Many beneficiaries felt that PNPM-Urban projects were better than government projects of a similar type—roads, in particular, were often deemed of higher quality. Despite some remaining concerns about project quality (see below), the vast majority of respondents who spoke about the usefulness of projects felt that PNPM-Urban projects were helpful for their community. Many respondents also provided responses that indicated they thought the projects were effective at solving a problem facing the community. A typical comment from beneficiaries in communities receiving road projects was: “the road is higher than before, and thus it doesn’t get flooded.” Moreover, beneficiaries in multiple communities where road projects had been completed said there was less flooding and fewer problems with muddy roads as a result of the projects completed near their homes. What areas are there for improvement? Critiques about infrastructure quality came from within the PNPM-Urban staff (facilitator), local government (PU head), and from respondents outside the government (NGO staff). Negative comments focused primarily on instances where the key informants felt that PNPM- Urban projects were constructed using poor construction techniques or inferior materials. For example, one individual stated that “technical quality needs to be raised;” another said that “community members who act as volunteers don’t know how to mix materials.” In spite of generally positive views, almost a third of focus group respondents commented on weaknesses in some of the projects. Most often, as with key informants above, these related to low quality materials having been used, although other criticisms were about project execution. A few respondents explained that projects in their areas had experienced damage since they were completed; many of these critiques referred to roads and pathways: “…in my RT the road cracked a little when a car passed;” “The quality [of the road] is not very good because it has damaged again. I think the concrete is not very good;” “I was disappointed [with] the road; a few months [and already] there are a lot of holes in the road. At first [it was] good, but after that the cement and the stone separated;” “Since paving [was done] my house has been subject to flooding because it is lower than the pavement.” Finally, a small number of respondents reported that some projects had to be reconditioned; the projects had been of low quality in the first instance and improved through repairs and upgrades: “The execution of the first project [toilet] was not good. The pipes and the methods were not good. But it was better with the second;” “The residents in this area improved the pavement. The first time, the pavement was no good. But then it was remade. Now it’s good; it’s been a year but it remains good.” There were some cases where project beneficiaries criticized project quality or construction methods, however. In this instance, critiques often focused on how projects were built—their design—rather than on materials or labor quality. For example, in one site beneficiaries complained that a new drainage system was not properly aligned with the main sewer. In another site, respondents criticized a roof repair project because it did not use a plastic barrier between the roof tiles and the wooden frame. 62 D. How are projects maintained? What is working well, and why? According to FG and in-depth interview respondents, different projects appear to be subject to maintenance in different ways. In the case of ditches, gutters or drains, and even roads and pathways, it is often the people living in the immediate vicinity of a project who are in charge of maintenance. This is particularly in evidence when there is damage to the infrastructure. In those cases, it is common for neighbors close to the damage to address the problem. Respondents stated, for instance: “[The ditch] is maintained by the person living near it;” “Sometimes there are neighbors who throw garbage on the street and I object to this because I have cleaned the street and the ditch;” “The ones living by the side of the pathway will cement it if there is minor damage.” In other cases, respondents explained that “the community” overall is in charge of maintenance; keeping infrastructure clean and fixing problems is not merely the responsibility of those living close by.31 In a small number of kelurahan, respondents explained that community members make voluntary contributions to a maintenance fund. Finally, many respondents explained that the day-to-day maintenance of certain infrastructure (such as the trash tub and the Posyandu building) is carried out by paid workers (janitors or trash collectors, for instance). In a small number of kelurahan, participants said that community members contribute money for infrastructure repairs. In one instance, a focus group participant said: “[If there is damage on a road] the people living close to the spot will collect money to repair it .” In another kelurahan, a participant told us that “there is a contribution each month for the repair [of the street lamp]. It’s Rp.1,000 each month.” In the same focus group, a participant from a different RT explained that in his area, the contribution was Rp.1500 for maintenance of the poyandu. A third participant from yet another RT remarked: “in my area a contribution is every evening which is usually put in a small container attached to the fence so that the patrolling men can collect the contribution from house to house; [the contribution] is for repairing lamps… or repairing a collapsed ditch. The repair cost is taken from the money.” In a few instances, focus group and in-depth interview participants remarked on problems with “community” maintenance of infrastructure projects. For example, speaking about a canal, one person commented: “Sometimes people [who] live on the upper side refuse to clean up [the canal]. As a result, people [who]live in lower areas suffers the disadvantages, enduring the smell.” Fewer beneficiaries had comments, positive or negative, about maintenance for PNPM-Urban infrastructure projects. In approximately half of the sites, beneficiaries reported that some maintenance of projects was undertaken. In most cases, maintenance was conducted by community members living near the sites. In some sites maintenance efforts were reported to be regular and organized, while in other sites respondents reported uncoordinated or sporadic project maintenance. In one site beneficiaries said that the kelurahan government was responsible for maintenance, and in two other kelurahan residents reported that they pay one or more community members to maintain the projects. Most reported that maintenance was “light,” consisting primarily of cleaning or minimal repairs. 31 The term “community” here is ambiguous and may mean individuals living in an RT, RW, or kelurahan. 63 D. Cost-effectiveness The study was not able to assess in detail the cost-effectiveness of projects because (a) budget data is often insufficient or questionable, (b) counterfactuals are not available, and (c) we lack suitable measures of effectiveness (in terms of impacts). While efforts were made to collect data, these were not found robust enough for analysis. MIS data on projects was downloaded but individual cost data was not available. Data collection in the field was complicated by the fact that in many cases, we found the budget for a single project was used to fund multiple projects, especially where residents were perceived to have diverse needs. In cases where a single budget was split, it was not possible to determine an accurate allocation across projects. Without comparable data for non-PNPM-Urban projects, it is difficult to assess relative effectiveness for PNPM-Urban infrastructure projects, and comparing projects across cities or regions may not be reasonable due to variation in input prices. We also note that comparisons of projects should also take into account the potential use of different quality materials, and the implicit value of volunteer labor. Nevertheless, we did ask interview respondents to provide their subjective assessment of whether PNPM-Urban projects were cost-effective. Some respondents commented that PNPM-Urban projects were often more efficient than government projects due to community contributions (lower total costs), the lack of complex tender procedures, and better oversight through community participation. Community participation in implementation was said by some focus group respondents to improve accountability as well as the quality of projects. With respect to accountability, one respondent remarked that community contributions eliminated the need for contractors who misuse money, a problem that many respondents felt was prevalent with government (i.e., non-PNPM) projects. In addition, participation in implementation appears to increase feelings of ownership and hence a desire to maintain quality in construction. One focus group respondent remarked “I was upset when the construction was damaged easily. I used to shout it loud to remind the community that the work is for the sake of the community and we have to build it well .” Other respondents noted a greater willingness to also contribute to the future maintenance of infrastructure provided by PNPM-Urban, since “government gives the facilities. If we feel reluctant to clean it, that's not a good attitude.” Best practices and recommendations for consideration On average, PNPM-Urban infrastructure projects are rated high quality by professional engineers. Respondents of all types—beneficiaries living near infrastructure projects, FG and in-depth interview participants, and key informants—corroborated the engineering assessment through their observations on or experiences with infrastructure project quality. Concerns over quality were split between poor materials and workmanship and poor project design. Although few projects were rated deficient based on engineering assessments, there is some room for improved project quality, especially for road and water projects. Although more road projects were rated middle-tier than most other types of projects, community members frequently viewed these projects as both helpful and well-constructed. Linkages and communication opportunities should be strengthened between PNPM-Urban implementers (e.g., facilitators, KSM) and the Department of Public Works (PU). This could 64 provide PNPM-Urban with access to expertise, especially on an ad hoc basis. Such an effort should be piloted to determine if it is effective and does not place undue burdens on PU staff. Basic guidance (e.g., short guides) could be provided to communities for the most common project types (e.g., roads, drainage) to address construction deficiencies that are due to lack of engineering or design knowledge. An area in which PNPM-Urban may not be performing well is maintenance, although we acknowledge that maintenance is difficult to assess. The engineering assessments could not separately assess maintenance issues, but key informants expressed some concerns about maintenance and beneficiary interviews suggest that maintenance is inconsistently conducted at least in some sites. FG and interview participants expressed fewer concerns with maintenance, although they emphasized that much of the maintenance work falls on those living in the immediate vicinity of a project. This raises important questions about where the burden of maintaining infrastructure projects lies and about community members’ ability to effectively repair and maintain PNPM-Urban projects given their resources and skills. An important finding from the focus groups, although it was explicitly discussed in only a few of cases, is that maintenance arrangements seem to vary from RT to RT. This highlights the importance of socialization and participation at this lower level of local government. Maintenance for PNPM-Urban projects should be assessed and determine whether maintenance needs are being adequately addressed within the program. If there are gaps, guidance could also be developed for how to approach project maintenance, not just project construction. For example, task KSMs with the responsibility for overseeing maintenance for a fixed period of time. Finally, one shortcoming of this evaluation is the inability to make reasonable cost comparisons for projects due to a lack of usable budget information. To enable this important analysis, budgeting guidance should be revised to remove the incentive to split budgets across multiple projects, which makes project tracking and outcome tracking difficult. 65 9 Adequacy of block grants and the ND experience In this section, we examine evidence related to the adequacy of block grants and one potential model for expansion of grants, the ND. A. Are current block grants adequate? The evidence suggests that demand for larger grants may or may not exist, but is not necessarily linked to demand for larger projects. In our field sites, relatively few respondents in non-ND sites reported that current block grant size was inadequate for project needs. Across all sites and respondent types, there were few accounts of communities not being able to fund a desired project because the BLM size in a particular year was too small. However, this may reflect that project requests were already scaled to expectations about available PNPM-Urban resources. A few respondents also speculated that limited funds may account for the use of low quality materials in some infrastructure projects. A second piece of suggestive evidence is that in practice, when infrastructure assessments were conducted, planned budgets for projects were frequently divided up into several smaller projects, rather than spent on the larger projects that were initially proposed. Finally, in the illustrative MIS data analysis (Annex B), we look at a sample of kelurahan enrolled in PAPG versus regular PNPM-Urban. Although the sample used for analysis may be highly selected, the results are suggestive. The analysis shows that PAPG communities receive higher grants per capita, but that these grants are used primarily to fund more, rather than larger projects. The size per project and type of project does not appear to change. Although the data is subject to many caveats, on the face of it, in this small sample, the patterns suggest that grant size may be a constraint in one sense i.e. in the total number of projects undertaken, but that that communities under PAPG did not use the expanded funding to pursue economies of scale. B. What can we learn from ND? A potential implication from the evidence above is that expansion of grants may also require expansion of community demand for and capacity to support larger-scale projects, in order to achieve appreciably different outcomes. To examine one potential model for expansion of grants, as described in Section 3, we turn our attention to the experiences under PAPG/ND. Due to the relatively recent introduction of the ND program, there are no relevant pre-existing evaluations except Ochoa (2011), which reviews the pilot phase of the project qualitatively via a desk review of program documentation, stakeholder interviews, and eight site visits. Analysis of the MIS data revealed that 16 percent of funding was spent on the planning specialist, 52 percent for the planning process (which consisted primarily of printing, food, events, and beverage for the meetings), and the remainder for capacity building and marketing activities. Ochoa found poor results in terms of leveraging funds from local government, which constituted only 0.2 percent of total ND funding. Most of the projects in the 18 pilot sites were footpaths, small roads and drainage, greening of public space, and market improvement (although the proportion of projects consisting of roads and drainage was substantially less than in regular PNPM). Community members reportedly did not see a high value associated to the mapping and spatial analysis exercise despite the program design emphasis design on spatial planning, and the large amounts of resources assigned to that phase. Ochoa (2011) concluded that there should be better efforts at targeting: either by selecting ND kelurahan based on poverty or by offering a “closed menu” of options to ensure activities selected are those that most benefit the poor. The author also noted that that the objectives of the program are complex and ambiguous, as well as poorly communicated, and defining the objective as “a harmonious and developed society as a basis for developing a healthy, harmonious, and sustainable environment” clearly left room for misinterpretation or misunderstanding. Many of the individuals interviewed, when asked what the ND objective was, answered that it was “infrastructure.” The study recommended requiring clearer links between the poverty mapping and investments However, a key limitation of this study is that, as described in the preceding sections, each ND project consists of several phases, from the forming committees, selecting urban planner, etc. At the time of the study, none of the pilot projects had received the third tranche of funding or finished the construction phases. It is important therefore to bear in mind the caveats (also noted by the author) regarding timing and effects that may be the result of initial unfamiliarity rather than poor design. To add to this work on ND as well as the previous chapters in this report, RAND team members visited three sites. Unlike the previous chapters, we organize this as a single case study covering multiple themes To recall, Site 1 and Site 3 are located in Central Java and represent “average” and “successful” sites, respectively. Site 2 is located in West Java and represents an “unsuccessful” site. These ND pilot sites were characterized by PNPM-Urban staff as such, based ex-ante on site characteristics, including community participation (e.g., meetings held, number of participants), local government involvement, and the quality of ND planning products (e.g., the Community Settlement Plan). Participation metrics were drawn from the PNPM-Urban management information system, while local government and planning product assessments were conducted by the National Management Consultant and the World Bank Jakarta staff. Awareness Across all sites, respondents who were directly involved with ND projects (i.e., members of BKMs and planning and implementing teams) generally were well aware of how the program functions. The larger community, including beneficiaries who come to public meetings, for example, do not think of ND as a program with its own objectives. Some see it simply as a larger version of PNPM-Urban, while others confuse it with projects sponsored by the local government that involve community participation. Community participation Participation has been a challenge in two of the three sites visited, surprisingly both in the successful and unsuccessful site. There are several different kinds of participation expected by ND and different issues have arisen with each. First, people are expected to volunteer for the Planning and Implementation Teams. Almost all important decision making about ND projects occurs in these teams. For the most part, the planning team (TIPP) is composed of community “notables” and leaders, including all BKM members, the 67 RT and RW heads as well as the Technical Team consisting of representatives of government agencies from the kota such as the DPU and Bappena. In addition to these leaders, people from the community who have time and are well-respected (especially retirees) are involved. By and large, it appears that there is a replication of the participation patterns seen in the PNPM-Urban sites, simply on a larger scale. Community participation in meetings varies depending on the subject of the meeting. Generally the turnout to discuss the kelurahan plan or decide the location of a priority area will draw many participants, for example several hundred in the “average site” and over a hundred in the “unsuccessful” site. This level of attendance (at initial meetings) far surpasses the norm for the 13 PNPM-Urban sites visited in the field work. Once the major decisions are made, however, attendance drops off. In this case, the “unsuccessful” site is instructive. Over 100 people came to the first meeting whose purpose was to discuss the kelurahan plan and priority plan, but when residents found out that the grant only went to one area with nothing more for the rest of the kelurahan, many never attended meetings again. The ND organizers could only find three volunteers for each implementation subcommittee, most of whom eventually quit, leaving the area leaders to bear most of the responsibility. Volunteer labor has only been available on weekends, the consequence of which was an unbudgeted increase in costs which has delayed the completion of the road. At least in that site, a certain amount of participation fatigue seems to have set in among volunteers. Among beneficiaries of the successful and unsuccessful site, name recognition of ND was low. Even though there were signs on walls and fences proclaiming the source of funding and name of the project, some beneficiaries could not identify what program built the ND projects; instead, beneficiaries often said simply that projects were done by “the government.” This was in stark contrast with what the team found in the “average” site. Here the community in the priority area was very-well integrated and knowledgeable not only of the projects constructed but also those that had not yet been built (because they had received only the first two tranches of the funds). The ND program puts an emphasis on “marketing activities,” where ND funds are leveraged to search for additional funds elsewhere. We found that, in the “average” site, a large fraction of the community was involved in actively seeking funds for projects designed in the spatial planning but that were not funded by ND. Unlike the usual situation in PNPM-Urban, where projects are relatively small and simple, planning for ND projects requires skilled labor which usually cannot be found as free volunteer labor among the community. Hence fewer community people are intimately involved. In contrast, with PNPM-Urban projects, participation in implementation is more significant, and is the phase where the community is most likely to get involved. The ND Implementation Committee is more open to community participation than the planning committee. As with PNPM-Urban, women participate less in infrastructure projects than in social or economy projects. The exclusive focus of ND on infrastructure results in lower female participation. According to a BKM member in the “successful” site, “[women] are not interested in infrastructure.” Similar to PNPM-Urban infrastructure projects, women usually contribute by preparing food and drinks for the workers. In all sites, it has proved very difficult to get or keep volunteers. There are several apparent reasons for this. ND requires that these committees do a lot of work for no compensation. 68 Also, many residents are probably unable to find the time to participate. These factors are no different from the situation of the KSM in PNPM-Urban, but a further constraint under ND is that this work requires higher levels of skills (e.g., accounting and reporting, budgeting, construction supervision) that are scarce in some communities. Further, as noted, the priority area for the ND infrastructure project is small and composed of a few RWs, so that the pool of potential volunteers is smaller than in regular PNPM-Urban. In both the “successful” and “unsuccessful” site, there was some volunteering in implementation but it was not extensive. The ‘average” site’s participation was larger due to the “accompanying” all-community- contribution projects (which consisted of a side-walk and planting flowers along one of the roads built by ND). In the first two sites, the limited volunteering mostly involved clean ups or unskilled work. After the initial post-construction clean-ups, there were few volunteers in the “successful” site and essentially none in the “unsuccessful” site. Finally, community members are asked to contribute funds for their projects. The degree to which the communities are willing to contribute to the ND projects varied from site to site. In the “unsuccessful” site, the community was essentially unwilling to contribute. In contrast, in the “average” site, the community was very much involved in the site’s project. This project included installing gardens along the road built by ND that led to the garbage disposal facility. The gardens were entirely built by the community (and the materials bought with community contributions) without support by ND. One difficulty for achieving wide participation in ND is that choosing a priority area limits in practice the possible level of participation. Since ND has to select a priority area (which in the three sites meant a geographical area composed of one or two RWs) participation from the other areas of the community is limited after the selection. In the “typical” site, the choice of priority areas and of projects within them was done through meetings between the TIPP and three representatives from each RW in the community. They decided to use the ND funds for a garbage collection and composting program and on two accompanying road and beautification projects. All these projects were made in a priority area that cut across two RWs but did not affect any others. This points to a potential constraint for both participation and local political support of ND. Under PNPM-Urban, it is possible to allocate resources to every RT or RW because projects are small (even if their economic impacts are correspondingly small). Under ND, many or most neighborhoods will be left out. This may be compensated for by the ND project(s) yielding community-wide economic benefits. However, as discussed below, this is difficult to ensure. The success of ND in the “average” site in getting the community to feel involved with the ND project might be attributed to the selection of the priority area at least partly based on their proposal of “companion projects.” The willingness of individuals from a certain area to contribute time and funds for a project should be considered as one factor of the decision to select a priority area. Overall, it appears that participation is more difficult to achieve under ND than under PNPM- Urban. This may be in part due to the lower the smaller number of residents directly impacted by the projects. Moreover, because larger infrastructure projects are complex, require good financial management, and demand skilled labor, it is more difficult to involve most members of the community in implementation. To improve involvement, the priority area should be carefully selected based on community involvement and willingness to build simpler “companion” projects. 69 Governance and monitoring The quality of ND infrastructure projects is generally deemed high by the community, PNPM-Urban staff, and government actors. ND, like PNPM-Urban, incorporates higher levels of auditing and greater community involvement in implementation than government-funded infrastructure, which may ensure higher project quality. However, as discussed in the previous Section, many respondents in the PNPM-Urban sites attributed the lower level of quality and higher cost of the regular government projects to the government tender process. Many stated that contractors are likely to be corrupt or do not have the incentives to build to high standards. Because many ND projects are large, and larger projects are required by law to go through a tender process, one could worry that ND projects could suffer to some extent from these problems as well. However, none of the respondents we interviewed reported problems with tendering for ND projects. As the ND facilitator in one of our sites mentioned, this could be because community members are involved in the tender process, or because of the several layers of auditing, or both. Indeed, several respondents in the “average” site stated that the tender process was monitored closely by the “auction committee” which is comprised of the BKM and village officers. Members of the BKM and Implementation Team are much closer to the community than are government officials at higher levels. They are under implicit scrutiny of the community, even if active community participation is limited as described above. In fact, many BKM and implementation team members in this site report that a difficult aspect of their job is dealing with constant suspicions from members of the community. The UPL (the “technical” member of the BKM) , who was also a technical member of the Implementation Team, stated that, “It is hard when people in the community don’t appreciate all the good work [we] do. People in the community don’t realize all the work that goes into the BKM. They are prejudiced a nd suspicious that [we] steal money.” Although an environment of mistrust is certainly not productive, overall it appears that active monitoring by the community is working to ensure adequate project quality and avoid misuse of funds (about which no complaints were voiced by respondents). We note that because ND projects are very recent, we could not evaluate their quality directly as thoroughly as we did with the regular PNPM-Urban projects. However, both our observations and the views of the interviewees reveal that the quality was in general high. ND Infrastructure Quality In the “average” site, ND decided to build a series of integrated projects in a priority area that cut across two RWs. The main project was a garbage compost and recycling facility, but ND also completed a concrete road that led to the facility. In addition, ND built a concrete road in the main street of the RW next to where the facility was located and bought bins for the recycling and compost and placed them in different parts of the kelurahan. The project was envisioned as providing an opportunity for the community to sell the recycled materials and provide income to the community members that place their recyclables in the recycling bins. In the “unsuccessful” site, there is a small river running through the neighborhood ND selected, and ND built a stone retaining wall so that the banks of the stream would not crumble during flood periods and water would not come up over the roadway. The project also involved repairs to the roadway running along the stream. The ND project in the “successful” site also involved improvements along a stream. In this site ND paved roads on both sides of stream and installed a foot bridge over the stream. 70 The engineers assessed the quality of the materials and construction for the ND projects at the three sites. In Site and Site 3 the main ND project could be subdivided into component projects (i.e., road components, bridge component), and the engineering team evaluated each component separately. Site 1, the “typical” site had three components, while Site 3, the “successful” site, had four components. In Site 2, the “unsuccessful” site, the only project the team evaluated was the one road project. In total, the team assessed eight project components across the three sites. The projects included roads (5), one bridge, one housing project, and one water project. The primary infrastructure assessment, as with the PNPM-Urban projects, was done through quantitative scoring based on engineering standards. Because each site has only four projects components at most, and because we visited only three ND sites, the small number of total project components (8) means quantitative analysis should be interpreted carefully. Of the eight projects, the lowest scoring was a housing project in the “average” site that received a score of 70 percent; the highest was a bridge project in the “successful” site that received a 69 percent. “Roads” is the only category for which there are multiple projects, and the average score for roads was 80 percent. We do not calculate site average scores, since it is not appropriate to interpret them for comparisons, but we note that scores in the “average” site are strictly dominated by scores in the other sites.32 Finally, the scores for all projects are in the good range, as they are all about 60 percent. The engineers also provided a secondary, descriptive assessment for the ND projects. As a group, the ND projects were deemed to be of high quality, with the engineers using terms such as “good materials” and “good workmanship.” The assessments uncovered some relatively minor problems with project design, such as trees that were planted too close to a retaining wall in one ND site. In another site, the finishing work on the concrete portion of a project was described as “rough.” But overall the quality of ND projects was reported to be high and the projects were described as functional. The key informants in the ND sites had relatively little to add about ND project quality, although most respondents who commented on project quality in these sites viewed quality as high. A BKM member in the “average” site commented that “the quality of construction is the same between ND and PNPM,” while an NGO representative at the same site stated simply that “infrastructure is good.” However, the same NGO member expressed concern about infrastructure maintenance, noting that there was confusion among about whom—local government or the community—should be responsible for maintain infrastructure projects. Two individuals from Site 3 also reported that infrastructure quality is good, but one noted that the Implementation Team faced some challenges, such as how to construct accurate budgets and how to effectively use volunteer versus specialized labor, and that some of these difficulties may have led to lower quality finishing work on infrastructure projects. Facilitators Facilitators are integral to ND, at least as much in PNPM-Urban. They start the ND process and help the BKM find the Urban Planners who are hired for the spatial planning phase. Key informants agreed that facilitators are badly needed and should stay with a project 32 In other words, all of the project scores in Site 1 are numerically lower than scores for any projects in the other two sites. 71 until it is completed—something that does not always happen. When facilitators are new they have little credibility with the community and members of the committees (BKM, TIPP, etc). As discussed in the PNPM-Urban section, a common problem with facilitators is the relatively little time they spend in a given community. Although short assignment periods might serve a purpose (namely avoiding opportunities for corruption), this means that facilitators have little experience with the communities. This does not mean that they have little experience with the ND program or as facilitators in general. For example, a facilitator for ND in the “average” site had been a facilitator since 2009 but had only been at that site for one month. Although he knew the program well and knew the technical aspects, he knew little about the community. The second most commonly voiced problem is that facilitators do not have enough time to spend on implementation and to guide construction. The novelty of ND—and the differences from PNPM-Urban—mean that facilitators spend a large portion of their time learning the ND rules. In one of the sites, the implementation team reported that they constructed a bridge but in so doing violated an ND rule, because they did not know about the rule. The fact that the ND facilitator was not aware of the rule (or alternatively, did not realize violating it was a problem) shows either a lack of understanding of ND rules by the facilitator, or a lack of involvement in project budgeting and construction. Presumably, these challenges are a result of the novelty of the ND program and may lessen as facilitators and others become more familiar with the program—and as the rules for ND become more set. Still there may also be a need for more thorough training of facilitators. As we learned in the regular PNPM-Urban sites, every time the rules in budgeting and reporting are changed, PNPM-Urban actors lose a lot time adapting to them. If this is true with relatively minor changes to rules, it is only natural that many mistakes and a large amount of time are wasted in the first years of ND, which has a separate set of rules from PNPM-Urban). In contrast, having only one implementation team (compared to the multiple KSMs in PNPM- Urban) makes ND implementation in some respects more straightforward. Moreover, the Implementation Team generally has a higher level of skill than the average KSM in PNPM- Urban, and is more autonomous. The team can do everything by itself, which makes things faster. At the same time, this has negative implications in terms of community participation, even as it reduces the strain on facilitators. As the UPL in one site stated, “The advantage of the PNPM-Urban method is that there is a learning experience for the KSM, because the KSM has to do everything (write proposal, implement things, write LPJ). However, it takes longer, because in the PNPM-Urban way there are many steps (proposal written by KSM then revised and approved by BKM, then it goes to [facilitator], askot and then the korkot). The ND is faster, because it is done by the Implementation Team. The disadvantage is that there is no ‘learning’ for KSM.” Local Government: ND interactions, coordination and support One of the aims of the ND program is to develop partnerships between communities and government. The success in this respect varies across the three sites: each exhibit different relationships with their respective local governments. The site with the least ND-government interactions was the “unsuccessful” site where none of our interviewees at the Kota level exhibited knowledge of the details of the ND project, and claimed they have nothing to do with it arguing that all supervision is at the level of the Province. However, this does not mean that the government in the “unsuccessful” 72 site is against the CDD approach. In fact, they are planning on copying it for a new program, using only Kota funding, to be launched in eight other kelurahan. The “successful” site offers the opposite example of good cooperation and collaboration between the local government and ND. Through the Technical Team, local government has several members on the TIPP and these individuals keep the local government apprised of ND decisions, offer data to the TIPP for planning, give advice as needed, and generally provide information to the committee on local government plans to enhance coordination. The Kota government also provided substantial financial buy-in to ND: it has funded and implemented parts of the priority plan including one stretch of one asphalt road; 70 percent of the cost of another road; public toilets; a storm water drain alongside new road paving, and a new water supply pipe and pumping station. So far the local government has spent 100 million rupiah in ND, which, although substantial, amounts to only 10 percent of the overall ND. Besides the considerable co-finance, the mayor has announced that he will fund the entire kelurahan plan within three years (although no funding was allocated in the 2011 budget). Site 1, the “average”, takes the middle ground. On the positive side, there is significant information sharing and coordination between ND/PNPM-Urban personnel and those in the government at the kota. As established by ND rules, the “Technical Team” consisting of representatives of government agencies from the kota such as the DPU and BAPPEDA, participated in the meetings of the TIPP. They serve as a reliable way for the government to stay informed of the plans for ND and vice versa. One example of the usefulness of this relationship involves the first project proposed for the priority area. This project consisted of establishing a physical space (stalls) for small sellers along a major road. The kota, however, was planning to expand that road, which would have meant tearing down the stalls. The representative of PU in the Technical Team alerted the TIPP, which decided on an alternative project–ultimately, the garbage disposal facility mentioned before. Despite this high level of good communication, the government at the kota level complained that it does not enough authority because the decision to disburse the funds is made at the province level. Related to this, there was evidence of confusion over the functions and authority of local government with respect to ND, again, probably because of the novelty of the program. Thus the head of PU admitted to being confused with respect to his role. PU wanted to stop the disbursement of funds when it realized the TIPP did not have the plans ready when in fact it PU does not have the authority to do this. The province disbursed the funds anyway. With respect to promoting connections with lower levels of government, namely the kelurahan, ND protocols call for formal involvement of the lurah (unlike in PNPM), as a member of the TIPP. We found that the lurah was indeed active in that function in both the “average” and “successful” site. Besides being member, the lurah has to “approve” the ND project, but this is evidently only a formality. His involvement with ND should ensure that the musrenbang process (which the lurah leads at the kelurahan level) is informed of the ND process, which helps to avoid duplication. Finally, we note two factors that may be reducing the potential for cooperation between local government and ND. First are political issues. For example, even in the “successful” site there was considerable animosity towards BKM members, who were also members of the local council despite rules against this. Second is the fact that control and supervision is by the 73 Province rather than by kota: This means that the kota has no control over ND as the resources are managed at the province level. This may limit incentives for buy-in to ND, which is one of the program’s key objectives. The one exception to this is the role played by the Technical Team who allows some degree of coordination with the ND. Kelurahan selection The selection of kelurahan for participation in the pilot phase of ND was based on good prior performance under PNPM-Urban, especially on the part of the BKM. However, according to some of our interviewees, the past-performance of the “unsuccessful” site in PNPM-Urban had not been strong. The BKM had proposed road projects but then constructed only half of them and used the rest of the BLM for non-approved projects, despite being warned by the facilitator that the remaining funds would be withheld. It is thus surprising that the “unsuccessful” had been selected for ND, and not surprising that it turned out in fact to be unsuccessful. Although Site 2 might be an exception, it raises the issue that the information available might not be enough to evaluate the performance of a site. We discuss issues with the MIS, which key informants identified as a main source to select ND sites, in a separate Section. The selection method assumes that a past history of good outcomes in PNPM-Urban, based on different measures, implies there will be good outcomes for ND projects. As we have seen this does not necessary follow, given the significant differences between the two programs. Still, it is plausible that kelurahan or districts that have done well under PNPM-Urban will do better under ND than areas that do not have as good performance under PNPM-Urban. If this selection criterion is maintained as ND expands (as opposed to pertaining only to the pilot), it raises a dilemma that communities most in need of the development benefits that ND is supposed to bring—those that are poorer—will not necessarily be the ones judged appropriate for the program. This will raise problems for ND to serve its objective of reducing poverty, unless poverty is taken as one explicit criterion for selection. Selecting priority sites One characteristic of selected ND projects is that they should benefit the poor. However, it was not always possible to target poor areas. There are two reasons why this was not achieved: the first is that in the sites visited, the poor are spread throughout the kelurahan rather than concentrated in specific areas within them. The 3D map of the kelurahan in Site 1 showed prevalence of poverty throughout the town, and the BKM head explained that poverty was therefore not a critical deciding factor for the priority area. Instead, they selected the priority area because it was the site best able to handle the garbage-disposal program. The use of ND funding for projects and management The ND grant consists of 700 million for construction and 300 million for planning and “marketing” activities. In the “unsuccessful” site, the TIPP has barely begun to find a use for the marketing budget. Some key informants felt that funding spent on planning and spatial planning process does not add much value and that these steps are not necessary for project selection. For example, the head of PU in the “average” site said “they spend a lot of money (250 million) in the planning documents but they provide no new information.” Given that some officials in some sites find that these tasks are not effective, the proportion of funds destined to them should be reduced, or at least the proportion destined to those purposes could be left for the community to decide subject to a maximum share. 74 Sustainability As with PNPM-Urban, ND is currently heavily dependent on volunteers (for example, for the BKMs and Implementation Team). However, the technical complexity and size of the projects leads to needs that are beyond many potential volunteers’ capacity and skills. Thus, the work is often concentrated among a few skilled volunteers. There is a risk of fatigue among these volunteers, and it might not be easy to replace them. Infrastructure maintenance is another important aspect of sustainability. Maintenance subcommittees have been formed in two of the ND sites visited. However, a principal maintenance issue remains unanswered, namely how to pay for major repairs. Local government has not accepted responsibility, and it is unclear whether the “beneficiaries” in the community will be willing or able to pay the cost of inevitable repairs, outside of volunteer labor. Given that ND is new, it is too early to tell how well projects will be maintained. Implementation Difficulties Members of both the TIPP and Implementation Teams cited difficulties in carrying out their assigned tasks and in meeting program requirements. This view is supported by facilitators, city coordinators, and assistant coordinators who maintain that the ND volunteers cannot function on their own and that advice and supervision from facilitators, the planners, and hired engineers are essential. That volunteers have difficulty meeting ND program requirements is not surprising, as they are tasked with complicated project and financial management. In spite of these difficulties, the volunteers carry out the projects surprisingly well, as evidenced by general satisfaction with project quality. However, particular challenges include:  Creating project budgets. This often results in under-budgeting for labor which makes for cost overruns or inability to finish projects. This is in part a problem due to the delays in disbursement of funds;  Inability to muster the volunteer labor needed or planned;  Inadequate understanding of potential obstacles: in Site 3, land owners were asked to donate land for a road, since the public right of way was inadequate, and this held up the project. In Site 2, residents in homes built illegally on public land have refused eviction and the TIPP had no power to intervene to get them out.33  Complex financial reporting by BKM and Implementation Team which requires significant assistance from the facilitator. Most volunteers lack experience with accounting and have no computer skills, which makes meeting the financial reporting requirements very challenging. There were cases of accountability reports refused and of project components for which money was not available to finish the project (e.g., Site 2’s road project). In summary, for ND the larger amount of money involved, larger project scope, and complexity of undertakings taxes the abilities of non-trained volunteers. In turn this burden likely limits the number of people who have the time and patience (as well as skills) to volunteer. As one interlocutor explained, ND requires too much from volunteers, even with 33When the korkot approached the kecamatan for assistance in evicting the squatters they refused to take action as this was not a government (kota) project. 75 facilitator assistance. Similar complaints were voiced by BKM and Implementation Team volunteers in the PNPM-Urban sites, but the factors discussed above appear to make the burden heavier for the KSM counterparts under ND. 76 10 Summary Below, we summarize the key findings of the report and best practice observations. i. Assessing the CDD approach We find that in general decision making in PNPM-Urban, participation is highly dependent on the existing capacity and incentives of local actors in spite of the formal procedures, PNPM-Urban may thus not fully realize the transformational potential its design intends.  Volunteer community organizations, composed of members of the community, work well, independent of pressure from other government structures. Although there are some issues and challenges going forward (such as BKM and KSM members having to spend too much of their time on PNPM-Urban activities), community institutions like the BKM can work independently.  However, direct participation from the grassroots in decision making is low, potentially subverting the intent of processes to elicit informed demand and ensure downward accountability: Decisions about project selection and implementation tend to be made by delegated representatives of the community within the PNPM-Urban structure as well as by informal community leaders. Few respondents report providing input into the process of determining what projects should be undertaken. In some sites, residents indicated they were asked only to approve a specific project in their neighborhood once that project had already been selected by others. Nevertheless, community members in focus groups and direct beneficiaries (those living near projects) generally expressed support for the infrastructure projects chosen through PNPM-Urban.  Participation is higher in project implementation: However, the degree of community involvement in these activities (e.g., construction) varies. In most communities, households reported making in-kind contributions to projects (e.g., labor or food/drinks for workers). The relative importance of paid and voluntary labor also varied substantially. Many respondents argued that PNPM-Urban infrastructure projects are of good quality (see section below) because the community is involved in their implementation.  Women’s participation in decision-making is even lower than that of men: Many female respondents reported that only male heads of household were invited to PNPM- Urban meetings. Widowed female respondents said they had not received invitations. A few others explained that from their household, their husbands alone attended meetings.  Female participation in implementation of infrastructure projects is mostly through the provision of snacks and drinks to workers. This may involve significant time and resource commitments. Relatively few women are involved in the construction of infrastructure projects. Women also reported significant involvement in the implementation of non-infrastructure activities.  The administrative burden of PNPM-Urban activities in the current structure is taxing for BKM and KSM, who volunteer their time in these roles. BKM and KSM members frequently complained about the time required for administrative tasks. Especially for KSM members, who tend to be less well educated, there is also a problem of lack of capacity to meet the detailed reporting requirements without significant support.  With regard to targeting, community members in focus groups and interviews were rarely critical of specific project choices; often they were happy that any projects had been funded/implemented at all in their neighborhoods. However, while not critical of choices made within infrastructure, many respondents identified things other than infrastructure (such as loans or training) as their most important needs.  Targeting the poor via infrastructure projects is a challenge because the urban poor tend to be geographically spread out, even within kelurahan. It became clear in the fieldwork that it is often hard to classify areas (e.g., an RT or a kelurahan) as “high poverty,” since there is substantial heterogeneity within these areas. Given the public good nature of most infrastructure, it is difficult to build projects that strictly benefit the poor. This is also a concern echoed by some government officials regarding other programs that attempt to target the poor.  There is demand for social and economic projects as well as for infrastructure projects. Preferences varied across and within sites, but many respondents stated that need and demand were greatest for training and loans, with less need for more infrastructure funding. Many are aware that the non-infrastructure projects have not worked well in the past (with problems such as low repayment of loans). Nevertheless, the findings raise the general issue of the value to low-income residents of infrastructure vs. other programs, and the extent to which PNPM-Urban is responding to community preferences. Introducing more cumbersome rules and procedures is unlikely to be the optimal approach to increasing participation given (a) the large variation in local context observed across our sites, (b) the perceived complexity and burden of existing rules, (c) the difficulty in practice of enforcing compliance with procedures. One constraint to greater participation may be lack of awareness of PNPM-Urban by the community. Current best practice for consciousness-raising includes spreading information through house-to-house visits. For the decisionmaking process, best practice in a small number of sites reflects the rules of PNPM-Urban itself i.e., socialization, need-elicitation through public meetings, and grassroots formation of a development plan. For both awareness-raising and decisionmaking, these best practices are highly labor-intensive and require commitment on the part of facilitators and/or 78 other PNPM-Urban staff. In most sites, we found that facilitators did not actively seek or monitor participation at the sub-kelurahan level. An alternative approach is regularizing interactions through scheduled community meetings. The most simple and least implemented best practice is to ensure that all adults are personally invited, regardless of gender or social status, to public meetings. This may require greater and more formal involvement of local community leaders, including the RT and RW heads. One alternative for raising participation is the option explored in the KDP context by Olken (2010), i.e., introducing plebiscites or referenda. Any such reforms need to be introduced in the simplest possible way to avoid further bureaucratic burden: for example, by having ballot boxes in public locations at a specified time. However, as one facilitator noted, it is not clear whether such practices will translate more or less easily from the rural context to the urban context (for instance, urban populations are more educated but also more diverse). Because a plebiscite or referenda approach would be a significant change for PNPM-Urban, a randomized feasibility study in a small pilot population would help identify the potential benefits or limitations to this approach. ii. Links with local government  Most government officials were aware of PNPM-Urban activities but involvement by local government34 in PNPM-Urban varies across sites, from little substantive involvement (e.g., only signing off on projects) to substantial local government support (e.g., participating in decision making and oversight).  There appears to be little or no duplication of activities or priorities. Local government representatives often cited concerns about project overlap, but there is little evidence that such overlap occurs. At the same time, there is little evidence that PNPM-Urban and the local government coordinate their actions or conduct complementary projects.  The annual local government development planning meeting, or musrenbang, was central to coordinating efforts between PNPM-Urban and local government: In most sites, the musrenbang takes into account PNPM-Urban projects and thus helps to avoid duplication.  Although links could be improved and some local government officials desire more explicit coordination with PNPM-Urban, support for more integration is not universal. In some sites communication between PNPM-Urban and government was poor and could be improved. However, in general, those involved in the operation of PNPM- Urban at the kelurahan level (BKM and KSM members) did not express support for closer integration with local government, and among local government officials, some argued that PNPM-Urban should remain separate from local government. 34Local government here includes various levels up to the kota or city level. Later in the analysis we will distinguish levels where appropriate. 79 Where interactions with local government officials (at both the kelurahan and kota level) were found to support the PNPM-Urban mission, we observed little project duplication and the implementation quality was high. The current PNPM-Urban model explicitly builds in scheduled interactions between the BKM and local government via the musrenbang. However, best practice in our sites involved a much more frequent and reciprocal relationship, with the lurah attending BKM and KSM internal and public meetings; local governments working collaboratively with PNPM-Urban to positively influence implementation, and local government officials visiting in-progress and completed projects. Our findings suggest that facilitators may wish to focus more attention on relationship-building as well as capacity development.35 iii. Capacity building and effectiveness of the facilitators  Facilitators are playing a significant part in all of the cycles of PNPM-Urban, from the socialization phase to helping BKMs and KSMs in the writing of reports and auditing of projects. However, facilitators play a relatively limited role in encouraging community participation in decision making (e.g., attending meetings). Interviews with BKM and KSM members make clear that the facilitators’ role is fundamental in all stages of the project cycle.  Facilitators’ contributions improve the quality of projects. Respondents noted that facilitators are often able to detect problems in the implementation of infrastructure projects before these reach higher-level audits. Some facilitators also provide specialized knowledge on project/program design.  However, facilitators appear to be overworked and overextended: Most facilitators are responsible for multiple communities, and their ability to dedicate sufficient time to each Kelurahan is limited. It is beyond their scope to contribute much to encouraging participation at the sub-Kelurahan level. This may be a contributing factor to the relatively low direct participation of the community in decision-making.  Rotation of facilitators may disrupt their ability to meet community needs: The tendency to rotate facilitators across sites in some cases limits their ability to develop close, trusting relationships with the communities in which they work. However, this practice may also limit negative side effects (e.g., cronyism). One problem with the current pattern of rotation is that the timing does not coincide with the project cycle in the community in which they are working.  The existing administrative burden on the facilitators for compliance purposes is substantial: Facilitators indicated that they spend a significant amount of time helping BKM and KSM complete required paperwork and reports. Reducing this burden could free more time to help with other fundamental challenges (e.g., participation), though this must be balanced against the need for adequate management and accounting. 35 Notably, facilitators spoke highly of the training the received and deemed it adequate, with the one exception being “relationship building.” Although it is inherently hard to teach people how to build effective relationships, such training might be well-received by facilitators and help foster improved coordination with local government. 80 Facilitators’ personal characteristics, experience, and training are cited often as factors that influence their effectiveness and, by extension, the success of PNPM-Urban. Best practices on the part of facilitators often reflect intensive, “tireless” activity to promote socialization and build community capacity rather than solely serving the role of “enforcing” rules. Increasing the number of facilitators may be a solution, resource permitting. Facilitators selection and training should also focus more heavily on the ability to mobilize the community, and facilitator contracts should include retention incentives to ensure that the best staff are retained (although performance incentives are not necessarily desirable) While rotating facilitators has been previously cited as important to ensure that many sites have access to the best facilitators and their knowledge and experience is disseminated, we found that rotation can be disruptive to the community. It can also be personally challenging for the facilitator. Facilitator rotation appears to be more effective when it is synchronized with the PNPM-Urban project cycle. iv. Governance and Control Mechanisms  Monitoring and governance mechanisms are a challenge, but appear to be working: o Auditing is being completed on schedule. .In most sites, auditing is being completed for all required entities on an annual basis. o Auditing has uncovered instances of inadequate project performance, and it appears to be working to identify problems of inappropriate project selection and poor implementation. Examples are audits that showed that projects were not completed according to plans, and in one ND site, auditing showed that a proposed project was not in line with ND priorities. o Most such problems are caught at an early stage by facilitators.  Auditing is key to limit or prevent misuse of funds. Given the high levels of corruption in different levels of government in Indonesia, people are aware of the potential for misuse of funds. Therefore, the many layers of auditing currently in place are important.  There is substantial room for improvement in the role of MIS in auditing and control. The quality of the data in the MIS, while much relied upon, is highly questionable. A large quantity of information is being collected, but it requires rationalization and validation. The MIS consists of several separate data sources that are not yet robustly linked and require considerable internal manipulation. As such, comparing data across years is not straightforward for professional analysts, let alone community members or program implementers. Few respondents of any types in our sites made reference to any use of the MIS. At the same time the primary role of the MIS for governance and control has vast potential, appears to be underemphasized and bears expansion. 81 Best practice to ensure accountability rests largely on a combination of regular physical audits and community monitoring as well as implicit mechanisms, such as social norms, to keep local leaders accountable to the needs of the broader community. Suggestions for improvement of the MIS must start from the bottom up, with adequate training and incentives for local data entry staff. Staff should be educated not simply in the technical aspects of data entry but also in the nature of the program itself, as mistakes in the data often reflect misunderstanding of program elements. At the same time, the requirements for what data must be collected and entered should be reassessed. Large amounts of information are collected and stored without clear objectives for whether and how that information will be used. Reducing data collection will help lighten the burden on PNPM-Urban staff and improve the data quality at point of entry. At the same time, adding basic GIS data could significantly improve the potential for using the MIS data for analysis by creating immediate crosswalk to other spatially-referenced data sources. Developing an integrated data system for the MIS, with built-in validation checks, may involve a high fixed-cost, but it could result in significant improvements in the quality and usability of data. A small pilot study to build and test a complete integrated system on a small scale is strongly recommended. Similar, a limited survey of existing and potential data users could help guide information collection requirements and inform how data are disseminated. v. Quality of infrastructure  Subjective assessments of PNPM-Urban project quality were almost universally favorable: Almost all respondents—notably including some government officials— said that PNPM-Urban projects were of high quality, and often of better quality than similar, government-constructed projects.  Objective assessments of PNPM-Urban project quality were also good: Professional engineering assessments undertaken in project sites were largely positive. There were no clear trends in project quality across different types of projects.  Respondents attribute the higher quality of PNPM-Urban infrastructure to the participatory approach and avoidance of government management: Respondents noted that community members’ involvement in implementation means better oversight and more dedication to ensuring good outcomes. Government contractors, on the other hand, are perceived as more likely to waste or misuse money.  With respect to maintenance, some respondents said that the community is more likely to maintain PNPM-Urban projects due to the perception of ownership. It should be noted, however, that in most cases it is too early to determine project longevity and the quality of maintenance. Most PNPM-Urban infrastructure projects have not been in use for a sufficiently long period of time to make such as assessment. Best practice for raising infrastructure quality included frequent KSM meetings with input from involved technical facilitators, and the explicit planning and coordination of volunteer labor to promote efficiency (i.e., using volunteers according to their skills). Other best practices included visiting sites beforehand to solicit feedback from beneficiaries who might 82 not attend community meetings and ensuring that the community was aware of the projects being performed by PNPM-Urban, for example by marking projects as “created by PNPM.”. Examples of poor practices generally related to overextending to projects that were too ambitious, or under-budgeting. In this instance, the technical facilitators have a key role to play in shaping informed demand for infrastructure projects. An important tradeoff relates final quality to cost and community participation: for instance, one official speculated that the use of community contributions led to lower quality materials; similarly, substituting volunteer labor for professional labor may lead to lowered technical efficiency, creating a tension with the benefits from community participation. Consequently, professional labor should be balanced against for incentives for long-term maintenance. In the best cases, volunteerism can create a sense of ownership by communities leading to better monitoring and the desire to voluntarily maintain the projects. A program change that has been raised is the move from an open menu of project options to a closed menu. Given that we did not find major problems with the quality of infrastructure, a closed menu may have little effect on project quality, while it could potentially compromise the local choice set and limit PNPM-Urban’s ability to meet diverse community needs. vi. vii. The adequacy of block grants and learning from ND  Relatively few non-ND sites report that current block grant size was inadequate for project needs: There were few accounts of communities not being able to fund a desired project because the BLM size in a particular year was too small. However, this may reflect that project requests were already scaled to expectations about available PNPM-Urban resources. A few respondents also speculated that limited funds may account for the use of low quality materials in some infrastructure projects. In addition, in many sites budgets for single project were split to fund multiple, smaller projects.  In one ND site there was a desire for more, smaller projects that are locally targeted. There were some concerns that larger projects were harder to manage (i.e., too complex) and that it is difficult to distribute larger funding blocks equitably across different areas or groups of residents. Because ND sites were treated separately, we note a few observations that are particular to this program below.  Participation. The site that had higher participation and community involvement managed to achieve this by attaching only community projects (funded entirely by contributions and built entirely by the community). The success of ND in Site 1 in getting the community to feel involved with the ND project might be attributed to the selection of the priority area at least partly based on their proposal of these “companion projects.” The willingness of individuals from a certain area to contribute time and funds for a project should be considered as one factor of the decision to select a priority area. 83  Monitoring. To maintain effectiveness and avoid corruption in the tender process, it is important to include community members in the committees overseeing the process. In Site 1 the tender process was monitored closely by the “auction committee” which is comprised of the BKM and village officers.  Relationship with local government: Ensuring that the representatives of local government that form the “Technical Team” assist to all meetings of the TIPP is an important mechanism to ensure compatibility between ND and government plans. In one site, the TIPP originally suggested establishing a physical space (stalls) for sellers along a major road. The kota, however, was planning to expand that road, which would have meant tearing down the stalls. The representative of PU in the Technical Team alerted the TIPP, which decided on an alternative project–ultimately, the garbage disposal facility mentioned before.  Targeting. ND should consider an alternative selection method for future sites to enhance the program’s impact in alleviating poverty. To achieve this, the program would need to choose kelurahan based on their poverty level. Although it is reasonable that the pilot sites were chosen based on readiness rather than need, the program in the future should consider to direct resources to needier kelurahan.  Allocation of funds to planning and marketing. In the sites visited, government and ND officials and volunteers did not find the planning and marketing activities to be very effective. Thus, the proportion of funds destined to them should be reduced, or at least the proportion destined to those purposes could be left for the community to decide subject to a maximum share 84 REFERENCES Beard, V.A. and A. Dasgupta. 2006. Collective Action and Community Driven Development in Rural and Urban Indonesia. Urban Studies, 43(9): 1451-1468 Binswanger- Mkhize, H. P., J. P. d. Regt, et al. (2010). Local and Community Driven Development: Moving to Scale in Theory and Practice, World Bank. Dasgupta, A. and Beard, V. A. (2007), Community Driven Development, Collective Action and Elite Capture in Indonesia. Development and Change, 38: 229–249. doi: 10.1111/j.1467- 7660.2007.00410.x Devas, Nick. 1981. “Indonesia’s Kampung Improvement Program: An Evaluative Case Study.” Ekistics 286: pp 19-36 Directorate General Cipta Karya Implementation Guide for Community Based Settlement Environment (PLPBK), Ministry of Public Works. Gibson, J. (2007). A guide to using prices in poverty analysis. Retrieved from http://siteresources.worldbank.org/INTPA/Resources/429966- 1092778639630/Prices_in_Poverty_Analysis_FINAL.pdf Gibson, J., & Glewwe, P. (2005). Analysis of Poverty Dynamics. In G. Kamanou (Ed.), Handbook on Poverty Statistics: Concepts, methods and policy use. New York, NY: United Nations Statistics Division. Glewwe, P., & Hoang Dang, H-A. (2005). Has Vietnam’s Economic Growth in the 1990’s Been Pro-Poor? An Analysis of Data from the Vietnam Living Standards Surveys. Department of Applied Economics, University of Minnesota. Glewwe, P. (2005). How Much of Observed Economic Mobility is Measurement Error? A Method to Reduce Measurement Error, with an Application to Vietnam. Department of Applied Economics, University of Minnesota. Glick, P., & Razakamanantsoa, M. (2006). The Distribution of Public Expenditures in Madagascar over the 1990s: Increasing Progressivity without Economic Growth? Journal of African Economies, 15(3), 399-433 Guggenheim, S., T. Wiranto, et al. (2004). Indonesia's Kecamatan Development Program: A Large-Scale Use of Community Development to Reduce Poverty. Scaling Up Poverty Reduction: A Global Learning Process and Conference, Shanghai, China Guggenheim, S. (n.d). Crises and Contradictions: Understanding the Origins of Community Development Project in Indonesia. Jakarta, World Bank. Joint Donor and Government Mission (2008), Gender in Community Driven Development Project: Implication for PNPM Strategy. Working paper on the findings of Joint Donor and Government Mission, (Jakarta: Government of Indonesia, The World Bank, AUSAID, ADB, DFID). Lincoln, Y.S., and E.G. Guba. "Processing Naturalistically Obtained Data." Naturalistic Inquiry (1985): 344-56. Mansuri, G. and V. Rao (2004) ‘Community-Based and -Driven Development: A Critical Review’, The World Bank Research Observer19(1): 1–39 McLauglin, Karrie, Adam Satu, Michael Hoppe (2007) " Indonesia: Kecamatan Development Program. Qualitative Impact Evaluation" Mulya, Shiddiq Sarana. (2010). PNPM-P2KP “Community-Driven Infrastructure Development Activities.” Thematic Studies on P2KP Implementation Performance: “Infrastructure.” Synopses Series (2). Ochoa, Catalina (2011). The Community Based Neighborhood Development Approach: Evolution, Definition, and Key Aspects Moving Forward. New York, NY: World Bank. Olken, Benjamin A. "Direct Democracy and Local Public Goods: Evidence from a Field Experiment in Indonesia." American Political Science Review 104.2 (2010): 243-67 Pradhan, Menno, Vijayendra Rao and Cristina Rosenberg(2010)” The Impact of the Community level activities of the Second Urban Poverty Project (UPP),” draft available at http://www.aiid.org/uploads/File/projects/12_UPP_impact.pdf?PHPSESSID=843e721de6b3e0 880444023a29242f1b Ryan, Gery, and Russell Bernard. "Techniques to Identify Themes." Field Methods 15.1 (2003). Sahn, D., & Younger, S. (2000). Expenditure Incidence in Africa: Microeconomic Evidence. Fiscal Studies, 21 (3), 329–347. Younger, S. (1999). The Relative Progressivity of Social Services in Ecuador. Public Finance Review, 27 (3), 310–352. World Bank Concept of the Poverty Alleviation Partnership Grant for Neighborhood 86 Development – PAPG-ND, World Bank. World Bank (2007). "Gender in Community Driven Development Projects: Implications for PNPM Strategy." Working Paper on the Findings of Joint Donor and Government Mission. World Bank. (2010). Targeting effectiveness of current social assistance programs in Indonesia Poverty Group. World Bank Jakarta Office. World Bank (2009). PNPM Urban – Additional Financing Project Paper, World Bank World Bank (2010). PNPM III - Project Action Document, World Bank Voss, J. (2008). Impact Evaluation of the Second Phase of the Kecamatan Development Project in Indonesia, World Bank. 87 ANNEXES Annex 1. Tables of Results for Rapid Surveys Annex 2. MIS Analysis Annex 3. Screening form for focus group and interview participants Annex 4. Focus Groups and Interview Protocols Annex 5. Key Informant Interview Protocols and Questionnaires Annex 6. Infrastructure Scoring Sheets Annex 7. Summary Statistics for PNPM-Urban Projects in ND Sites Table A1: Rapid Household Survey Results (N=635; male = 313, female = 320, missing =2) % % % All Male Female Had respondent heard of PNPM? Yes 47 53 47 No or Don't Know 53 47 53 Had respondent or anyone in their household participated in a PNPM meeting or voted in an election related to PNPM? Yes 18 19 18 No or Don't Know 82 81 83 Did respondent report knowing about the BKM? Yes 17 26 19 No or Don't Know 83 74 81 Did respondent report knowing about the KSM? Yes 15 17 13 No or Don't Know 85 83 87 Could respondent could identify one or more local infrastructure projects as funded by PNPM? Yes 46 54 46 No or Don't Know 54 46 54 89 Table A2: KSM Coordinator Survey Results (N=13) Number of % responses BKM Composition How many members does the BKM have? 9 7 54 11 2 15 13 4 31 What percent of BKM members are women? <25% 8 62 25-50% 4 31 >50% 1 8 Are all RTs in the kelurahan represented on the BKM? Yes 3 23 No 10 77 BKM Meetings How often does the BKM itself meet? Monthly 11 85 Every three months 1 8 Yearly 1 8 How often does the BKM hold public meetings? Monthly 2 15 Every three months 3 23 Every six months 3 23 Yearly 4 31 Never 1 8 Facilitator How often does the facilitator come to meet with the BKM? Monthly 11 85 Every six months 2 15 What does the facilitator help the BKM with? Financial reporting 12 92 Financial management 8 62 Monitoring and project implementation 10 77 Other 9 69 90 Table A3: KSM Infrastructure Leader Survey Results (N=24) Number of % responses For active KSM, how many members does the KSM have? 5 5 26 6 3 16 7 1 5 9 3 16 10 1 5 11 2 11 12 1 5 15 2 11 20 1 5 What percent of members are women? <25% 13 68 25-50% 5 26 >50% 1 5 How often does the KSM meet? several times a week 3 16 once a week 2 11 once every 2 weeks 4 21 once a month 6 32 about once every three months 2 11 about two times a year 1 5 less than once every six months 1 5 How many of the infrastructure proposals submitted to the BKM in this last round were approved? 33% 2 11 83% 1 5 100% 16 84 Does the KSM hold public meetings? Yes 18 95 No 1 5 Table A4: KSM Infrastructure Leader Survey Results for Active KSMs (N=18) Number of % responses BKM members usually attend 16 89 Lurah usually attends 13 72 RT/RW leaders usually attend 16 89 Any citizen can attend and speak 17 94 How many residents usually attend? 0 1 6 5 2 11 6 1 6 7 1 6 10 3 17 15 1 6 18 1 6 25 1 6 30 1 6 35 2 11 40 1 6 50 2 11 60 1 6 Do more men than women attend the meetings? More men 16 89 About the same or more women 2 11 92 Annex 2. Illustrative MIS Analysis with 2009 Subsample For the MIS data received August 26,2011, the table below shows the number of kelurahans having partial program data by cycle Year Number of kelurahan 2007 3,084 2008 5,855 2009 11,009 2010 10,930 For this report, we rely on a subsample of this data for a number of reasons. Firstly, given the timeframe, some data from 2010 remain incomplete or yet to be validated. Secondly, changes in the MIS structure over time have resulted in inconsistent data dictionaries between early and later funding cycles that are not fully resolved, and, reflecting the development of the MIS itself, earlier cycles of PNPM-Urban are more prone to data errors as described later in the report. Thirdly, our analytical focus is the most recent cycle of PNPM-Urban III. We therefore restrict attention to the presentation of information on kelurahan from the 2009 funding cycle, as being the most relevant and most robust for our purposes, with the appropriate caveat that any findings may not apply to other cycles. Of the 11,009 kelurahan in the 2009 funding cycle, 2133 are dropped because of zero or missing fund disbursements. Discussions with PNPM-Urban staff suggest that some of these may be actual missing disbursements due to unexpected budget shortages, while others may be missing information; in the absence of verifiable information, we drop all to minimize ambiguity. We also note that key characteristics – total population and total poor– are self- reported by kelurahans. A further 659 kelurahan have missing population data, and 111 have missing poverty data leaving a total of 8106. Of these remaining, only 100 are PAPG kelurahan. Key process outcomes36 are  indicator for having completed all socialization activities in a project cycle related to BKM elections.  the number of female voters taking part in the BKM election process/number of voters  the number of poor voters taking part in the BKM election process/number of voters A preliminary tabulation of this data suggest that 3580 or 44 % of the kelurahan in the restricted 2009 funding cycle sample have not yet completed their socialization activities. Taken at face value, this is a surprisingly high number. However, follow-on discussion with the PNPM-Urban MIS team suggests that this is likely to be a data entry error. If we restrict attention to kelurahan in the dataset for which the socialization completed indicator is nonzero, leaving us with a final sample of 4,526 (out of over 11,000 initial kelurahan). A further 1066 report 0 voters, which is also likely to be a data error; these are dropped, giving a final sample of 3460. 36 Other outcomes such as ratifying a CDP are in the dataset but universally zero-valued Leaving aside these controversial observations (although they form a large fraction of the data), although this final subsample is likely to be highly selected, participation statistics for the remaining sample are still suggestive. N Average SD Min Max Total voters/total adults 3454 41% 32% 0% 970% Total poor voters/total voters 3460 32% 24% 0% 100% Total female voters/total voters 3460 43% 10% 0% 76% In general, given that the sample with robust participation data is highly selected relative to the whole, we refrain from making strong conclusions based on what we see. We also observe that a number of kelurahan report no poor or female voters (in the case of total voters, because this is a clearly-identified data error, the 0% represents a vanishingly small percentage), while at the other extreme some report figures that are unlikely. However, we do note that on average, the statistics for these indicators are just above the targets for total participation and female participation, and below the program targets for poor participation. This suggests that absent further data error, meeting participation standards is likely a challenge. Number and types of funding and investments The MIS also contains the following information on the community’s planned allocations37 of total amount of funding available  Total BLM funding received  Planned allocations to infrastructure vs others Focusing in on infrastructure, we examine the allocation of sources of infrastructure funding (PNPM, local government and community) and the allocation of funding spent by type of project: roads, drains, small bridge, house, sanitation, boat moorings, waste disposal, clean water, electricity, schoolbuildings, irrigation, public health posts, markets and other. For this analysis, we set aside the participation data (as this data originate from a different source we assume for now that data errors are uncorrelated) and return to our total sample with any BLM funding and basic characteristics with respect to poverty and population (8106). Above Above Below median Below median median Pover3903 median size size poverty ty N 3921 3877 3895 Total funding (1000000 Rp) 120*** 150*** 140** 134** 37 The MIS also contains fields for actual activities but in the “best-possible” dataset, these fields are also universally zero- valued. 94 % funding on roads 45%*** 52%*** 48% 49% % funding on drains 23%*** 21%*** 23%*** 21%*** % funding on sanitation 7%** 4%*** 6%*** 6%**** % projects rated of acceptable quality (conditional on rated) 95%*** 94%*** 95% 94% Statistically significantly different, p<0.05; *** p<0.01 Funding allocations by size and poverty rate The first two columns of the table above shows that in our sample, kelurahans of larger size tend to receive, as indicated by the program itself, larger grants. We see that on average a significant allocation by project type is dedicated to road building (almost 50%), followed by drains (approximately 20%). Larger kelurahan also allocate their funds differently, choosing to spend more on roads and less on drains and sanitation, perhaps because the need to network a larger population outweigh their need for health and hygiene; they also do better on the quality of projects. We do the same exercise with poverty and see that poorer kelurahan receive smaller total grants. However, the poverty median split yield statistically but not highly economically significant differences in allocation and quality. This result may suggest either that the preferences of poor households do not vary greatly from nonpoor households, or that participation of the poor in decisionmaking is limited; however, from this data alone distinguishing between these explanations is not possible. Comparing PAPG to PNPM-Urban PNPM- Urban PAPG (N=8006) (N=100) Total population 7434** 5564** Total BLM funding (1000000 Rp) 140 130 Total infrastructure funding (PNPM) (1000000 Rp) 79** 70** Total infrastructure funding (community) (1000000 Rp) 36 32 Total infrastructure funding (local government) (1000000 Rp) 22*** 32*** Number of infrastructure projects 8.4*** 10.7*** Total infrastructure funding/projects (1000000 Rp) 21 19 Percentage projects rated of acceptable quality (conditional on rated) 94% 90% Statistically significantly different, p<0.05; *** p<0.01 The table above shows means for both regular PNPM-Urban and PAPG kelurahan in the final subsample. While there is no significant difference in total grant size between the two groups (perhaps due also in part to the sample selection issues related above), we note that population is significantly smaller in the PAPG sample, so on a per-capita basis PAPG grants are indeed 95 larger. We note that the contribution towards total infrastructure funding from local governments is also larger. The community contribution is no different in absolute terms, but is larger on a per-capita basis. However, we also note that the PAPG grants are not being used to create fewer, larger projects. In fact, summing up all sources of funding and dividing by the number of projects, the total project size across PNPM-Urban and PAPG remains the same. In other words, these simple means suggest that on average, PAPG communities did not use larger grants to embark on projects with larger economies of scale, but instead took on more of the same small-scale projects. Looking at the allocation of funding, again, splitting these out by PNPM-Urban and PAPG, we see no difference in the types of projects undertaken. PNPM- Urban PAPG Percentage funding spent on (N=8006) (N=100) Roads 48% 46% Drains 22% 19% Other 6% 9% Statistically significantly different, p<0.05; *** p<0.01 This suggests that patterns of allocation and projects do not vary significantly between PAPG and non-PAPG communities (albeit not controlling for other community characteristics). Finally a multivariate linear regression analysis with project size and funding allocation on an indicator for PAPG and controls for total population and total poor however also finds no statistically significant difference for PAPG. Dependent variable Project size % roads PAPG -11209587 -0.0 Total population -19.2 0.0 Poverty rate 667376 0.02 Constant 2.19e+0.7 0.44 N 7805 7798 Statistically significantly different, p<0.05; *** p<0.01 96