Report No: AUS0001049 . Indonesia TA on Village Transfers Indonesia Village Law: Technical Evaluation of Infrastructure Built with Village Funds Volume 1: Main Report . June 2019 . GOV . Document of the World Bank . © 2019 The World Bank 1818 H Street NW, Washington DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved This work is a product of the staff of The World Bank. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Attribution—Please cite the work as follows: “World Bank. 2019. Technical Evaluation of Infrastructure Built with Village Law. Volume 1: Main Report. © World Bank.” All queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Indonesia Village Law Technical Evaluation of Infrastructure Built with Village Funds Volume 1: Findings and recommendations World Bank 2019 1 Acknowledgements Task Team The TTLs for this report were Kathy Whimp and Ihsan Haerudin. Nick Menzies and Laura Ralston and Nick Menzies led the initial design and write up. Vic Bottini provided valuable inputs during design, execution and writing up. Consultants Neil Neate, professional engineer, led the technical evaluation. Neil was assisted by the Indonesia World Bank Country office staff and worked with a team of Indonesian rural development engineers who undertook the field assessments and data collection: Su’udi Noor, Saleh Siregar, Ghufron Efendi, Wawan Munawar, Chiaril Latief, and Bambang Warsito. Peer Reviewers Satoshi Ishihara, Senior Social Development Specialist, and Ashutosh Raina, Social Development Specialist, provided detailed and valuable peer review comments. Layout and Editing Katie Spillenkothen undertook detailed editing of the report and Alfi Naufida Ulinnuha provided the layout. 2 Table of Contents Table of Contents .................................................................................................................. 3 1. Introduction ...................................................................................................................... 6 Key Findings ............................................................................................................................................ 13 Recommendations .................................................................................................................................. 16 2. Technical evaluation methodology................................................................................... 19 2.1 Scope ......................................................................................................................................... 19 2.2 Sampling approach .................................................................................................................... 20 2.3 Evaluation criteria...................................................................................................................... 22 2.4 Overall quality rating ................................................................................................................. 25 2.5 Technical quality rating system ................................................................................................. 26 2.6 Field tools and approach to field visits ...................................................................................... 28 3. Evaluation Findings and Recommendations by Evaluation Criteria ................................... 31 3.1 Overall quality............................................................................................................................ 31 3.2 Process ....................................................................................................................................... 35 3.3 Quality ....................................................................................................................................... 49 3.4 Operations and Maintenance .................................................................................................... 55 3.5 Cost ............................................................................................................................................ 62 3.6 Other evaluation data and observations ................................................................................... 67 Volume 2: Annexes ANNEX 1 – RECOMMENDATIONS OF THE TECHNICAL EVALUATION ANNEX 2 – SITE SELECTION PROCEDURE (FROM 2012 PNPM TECHNICAL EVALUATION) ANNEX 3 – LIST OF VILLAGE INFRASTRUCTURE PROJECTS EVALUATED ANNEX 4 – TECHNICAL EVALUATION PLANNING AND METHODOLOGIES ANNEX 5 – DETAILED ANALYSIS OF INFRASTRUCTURE COMPONENT RATINGS ANNEX 6 – SAMPLE OF EVALUATION FIELD NOTES AND COMMENTARY ANNEX 7 – SAMPLE VILLAGE LAW EVALUATION FIELD INSTRUMENT ANNEX 8 – INFRASTRUCTURE COMPONENTS AND ASPECTS FOR TECHNICAL EVALUATION 3 Acronyms and Glossary ADD Alokasi Dana Desa, transfer to villages from district governments APBDes Village budget BKAD Badan Kersacama Antar Desa, inter-village community forums BPD Village representative BPS Central Bureau of Statistics CDD Community-driven development DD Dana desa, transfer to villages from central government gotong royong Mutual cooperation, village volunteer system IDR Indonesian Rupiah kabupaten Districts KDP Kecamatan Development Project KTD Kader teknis desa, village technical cadre kecamatan Sub-district MCK Public laundry/toilet facilities M&E Monitoring and Evaluation MOF Ministry of Finance MOHA Ministry of Home Affairs MOV Ministry of Villages MusDes Village planning meeting O&M Operations and maintenance Permendagri Regulation of the Ministry of Home Affairs PAUD Early childhood centers PDTI Engineers based at kecamaten level attached to District Department of Public Works (Dinas PU). Pendamping Facilitator PNPM Program Nasional Pemberdayaan Masyarakat – National Program for Community Empowerment PKD Pengkajian Kondisi Desa, Review of current village conditions RKPDes Annual village plan RPJMDes Medium term village planning swadaya Self-help swakelola Self-management, village implemented TPK Village activity implementation committees USD United States dollar VFS Village Financial Statistics VIP Village infrastructure project VL Village Law 4 5 1. Introduction The Village Law, enacted in 2014, mandated the transfer of funds to villages with the goals of reducing poverty and improving living standards in villages through village-led development and community empowerment. Village Law (VL) builds on Indonesia’s 17-year history of participatory and community-driven development (CDD) approaches such as under the Kecamatan Development Project (KDP) and Program Nasional Permberdayaan Masyarakat (PNPM).1 The changes consequent upon the closing down of PNPM and its replacement by Village Law transfers (Dana Desa and Alokasi Dana Desa)2 and implementation arrangements, form a critical backdrop to the Evaluation of Infrastructure Built with Village Funds. Why the quality of village infrastructure built under Village Law matters PNPM was a central government program, with its own program guidelines and implementation arrangements. Funding was allocated to the budget of the Ministry of Home Affairs, which disbursed block grants directly to bodies established at the sub-district (kecamatan) level, which then administered a competitive process to allocate block grants to villages. The management of the block grants, including financial management and technical standards, were governed by guidelines for the program, which were focused on participation in planning, implementation and evaluation, procedures to ensure good project quality financial accountability. The Ministry of Home Affairs hired facilitators, whose role was to help villages implement block-grant financed projects according to the program guidelines. The World Bank continued to support the implementation machinery for the program by financing the cost of facilitators and using its supervisory role to help the government safeguard the quality of facilitation, as a critical input to the effective spending of the block grants. Village Law involved substantial changes in the way village transfers were channeled and managed. Introduction of Village Law coincided with the start of a new administration under President Joko Widodo, which allocated the administration of villages to the Ministry of Villages and Transmigration, along with the responsibility for managing facilitators. The Ministry of Home Affairs’ role was narrowed to the regulation of village government administration and public financial management. Whereas the recipients of PNPM grants were village communities, Village Law transfers are directed to village governments. This introduced a new role for the Ministry of Finance in ensuring accountability for the Dana Desa. Dana Desa funds are channeled through the intergovernmental transfer system, via districts, which disburse transfers to the villages and collate their reports for transmission to the Ministries of Finance, Villages, and Home Affairs. 1 In the late 1990s the World Bank supported the Government of Indonesia to launch the KDP in around 3,000 villages. KDP was gradually scaled up (changing its name to PNPM) to reach about 67,000 villages by 2014. 2 The Village Law mandated three transfers to villages; one from the central government ( Dana Desa, equivalent to around 10% of non-earmarked transfers to the subnational level, and two from districts ( Alokasi Dana Desa, ADD, equivalent to 10% of non-earmarked revenues to districts, and PH-BRD, equivalent to 10% of local revenues and levies). 6 The resources allocated to the village level doubled in 2015 when Village Law was implemented, with significant annual increases following in 2016 and 2017 . Prior to VL implementation in 2014, annual aggregate revenue for villages was roughly IDR 26.7 trillion3 (USD 1.98 billion).4 That increased to IDR 52.1 trillion (USD 3.6 billion) in 2015. By 2017, the most recent year for which official estimates are available, village revenue was almost four times higher than it had been in 2014. The average amount of money managed by each village stood at around IDR 1.3 billion (USD 90,900) in 2017.5 Infrastructure makes up a significant portion of village spending. This has been true since the beginning of KDP in 1998 and remains so to date. Based on the World Bank’s Village Public Expenditure Review,6 at least 38% of village budgets (almost IDR 40 trillion, or USD 2.8 billion) are spent on public works (such as roads, bridges, buildings, water supply systems, irrigation and drainage) each year, and the total amount of infrastructure spending is between 50-60% (USD 3.5 billion). The Ministry of Villages reports that between 2015 and 2017 spending of Dana Desa produced more than 120,000 km of roads, almost 42,000 irrigation units, 64,000 water supply systems, and more than 21,000 village schools and other sector-specific infrastructure. Currently very little is known about the quality, including the design, functionality and sustainability of these investments. Quality of implementation is critical to the value achieved through infrastructure spending. Infrastructure which is poorly constructed or not fit for purpose is prone to deteriorate quickly or to be wasted. As the implementation of Village Law unfolded, questions emerged whether the new implementation support arrangements were adequate to ensure the quality of infrastructure, especially given that available resources had scaled up so quickly. Periodic assessments of infrastructure quality under PNPM had confirmed that infrastructure was of a good standard and cost effective, the last of which was in 2012. To better understand the situation on the ground under Village Law, in 2018 the World Bank initiated a Technical Evaluation of Village Infrastructure implemented with Village Law between 2015 and 2017 using a similar methodology to the 2012 PNPM infrastructure assessment. 7 Methodology for the Technical Evaluation The 2018 Technical Evaluation of Village Infrastructure evaluates the development process, quality, costs, and operations and maintenance (O&M) of 165 village infrastructure projects (VIPs) with budgets greater than USD 10,000, from 39 villages in six provinces. The five types projects assessed were: i) buildings (34); ii) bridges (15); iii) water supply (14); iv) roads (12) and v) irrigation (12). Assessors evaluated the physical structures and related files (budgets, design, 3 Data derived from the Village Financial Statistics (VFS) of Central Bureau of Statistics (BPS) 4 The exchange rate used is 1 USD = IDR 14.300, (Indonesian Central Bank’s average rate of 2018). 5 MOF estimates (see, DJPK presentation 2017 on “Paparan Dana Desa” at Swiss-bell hotel) 6 The World Bank conducted the Village Public Expenditure Review (ViPER) in 2018 analyzing the budgets of 1,868 villages to understand the impact of new transfer arrangements on village spending. 7 Neate, 2012. PNPM Mandiri Rural Infrastructure, Technical Evaluation Report. http://documents.worldbank.org/curated/en/954751468035470976/pdf/936920ENGLISH00aluation0Report02012 .pdf 7 approvals, etc.) implementation methods, and operations and maintenance (O&M) procedures. The technical evaluation covers VIPs in the same provinces as the 2012 under the PMPN program.8 While the methodology for the 2018 study was refined from the 2012 study, there were some key findings where comparison was significant. A comparison of key findings from the 2018 and 2012 studies is provided in Table 1. The findings and recommendations from the study are detailed in the body of the report and summarized in the following sections of this introduction. Headline messages are that infrastructure quality has declined, and attention to operations and maintenance spending by villages has dropped dramatically. The technical evaluation criteria covered four major areas of infrastructure project development lifecycle: 1) process; 2) quality; 3) cost; and 4) operations and maintenance (O&M).  The ‘Process’ section considers the infrastructure prioritization within villages, as well as the procurement processes, quality of supervision, environmental and social safeguards, and the status of the accounts and documentation of the village infrastructure projects (VIPs).  The ‘Quality’ section examines whether the infrastructure design is appropriate for the village needs that have been expressed in public meetings, as well as technical quality of the structures built and building in accordance with standards and code.  The ‘Operations and Maintenance’ section was based upon the evaluators’ interview of village leaders and committee members, together with field inspections of the actual infrastructure. The criteria evaluated under the first three sections are used to make an assessment of the ‘overall quality’ of each VIP, ranked on a 6-point scale from ‘highly satisfactory’ to ‘highly unsatisfactory’.  The ‘Cost’ section examines the cost effectiveness of Village Law investments. The VIP budgets were used to calculate unit rates for various types of infrastructure, which were compared with the standard costs as provided by the kecamatan (and confirmed by visiting construction material suppliers to gather local costs). The technical team was led by a professional engineer assisted by six experienced Indonesian rural development engineers. Two of these engineers participated in the 2012 Technical Evaluation of PNPM Mandiri. Findings from the Technical Evaluation Although 75% of projects are of acceptable overall quality, technical quality has declined since 2012. The methodology uses two key measures of quality. Overall quality is a measure of both the physical state of the infrastructure and the processes used to select and implement the project. Technical quality assesses whether infrastructure meets technical specifications. The 2012 Report notes, “the overall quality of the design and execution of [VIPs] was in full compliance 8 Ibid. 8 with the project’s [PNPM] technical goals.”9 However, since the 2012 Report did not rate each village infrastructure project specifically on the Overall quality measurement, no direct comparison with 2012 is possible. Figure 1: Criteria contributing to the assessment of overall quality Measured according to its technical quality, village infrastructure quality has declined since 2012. In 2012 82% of projects met specifications, whereas in 2018 only 46% did. A much greater percentage of projects were rated “slightly below specifications” in 2018 as compared with 2012, but overall the percentage of projects that did not meet specifications jumped—from 4% to 12%. The construction quality of water supply and irrigation projects was significantly inferior to that of buildings, roads, and bridges. Ninety-two percent of irrigation projects were rated as being of unsatisfactory or highly unsatisfactory overall quality. Not a single irrigation project was rated moderately satisfactory or above. Less than half of water supply projects were rated as being of moderately satisfactory or satisfactory quality. There is a clear correlation between declining infrastructure quality—particularly technical quality—and the effectiveness of village-level processes that ensure infrastructure quality. Areas of significant weakness identified in the 2018 Technical Evaluation include: (i) the adequacy of drawings, design and quantity measurement; (ii) technical support from qualified engineers, (iii) lack of user input, which undermines the utility of infrastructure but also weakens citizens’ sense of ownership which in turn underpins their willingness to fund ongoing maintenance, and (iv) almost complete absence of attention to operations and maintenance. Many infrastructure projects constructed from 2015-2018 lack appropriate file documentation. Of the 165 VIPs samples, 60% did not contain necessary planning and design documents. Some projects (15%) had no files at all. The evaluation found a correlation in the data between the 9 Ibid. Executive Summary, p. x 9 rating for quality of construction and file completeness for some types of VIPs. There were no complete files in place for village road and irrigation projects. Documents that did exist lacked basic engineering surveys and other key documentation. Participatory processes that were hard-wired into the PNPM arrangements appear to have weakened. Sixteen percent of villages did not meet the Village Law prioritization requirements, 85% of which were in the province of Maluku, suggesting a possible issue in that region. Participatory procurement processes were not always followed, and 53% of VIPs had little or no design consultation with user groups. All these processes are mandated by law (although village leaders observed that the regulations are confusing) but it is clear that these requirements are not always being followed in practice. Weaknesses in village level planning and implementation processes reflect less effective first- line support systems. Engineers are now less likely to be involved in adjustment of design drawings and sign off of engineering approval. Notably, the 2018 evaluation found that 46% of VIPs had no design drawings or approvals. In most cases evaluators found very little in the way of design calculations or quantity measurements with almost no designs having been approved by engineers or government representatives. This is quite different to the situation observed in 2012 where local engineers would typically signal their acceptance of construction packages by signing or initialing the drawings. Whereas engineering support under PNPM was provided by program facilitators, under the Village Law it is expected to be provided by engineers working for the District Department of Public Works but based at the sub-district (kecamatan). Village officials were asked to rate the technical facilitation support they had received on a three-point scale and for 85% of projects it was rated ‘poor’ and only in 3% was it ‘good’. Project files noted an average of 1.4 supervision visits per month. Low quality supervision is resulting in infrastructure which does not meet village needs and is not well constructed. Issues of construction quality are compounded if infrastructure is not well-maintained. Investment in operations and maintenance at the village level has declined sharply compared with 2012, contributing to the declining scores on construction quality . Whereas in 2012 95% of infrastructure was assessed as having sufficient or slightly below sufficient operations and maintenance investment, by 2018 85% of projects were judged to be poorly maintained (‘below specifications’, with only 3% ‘meeting specifications’ (and 12% ‘slightly below specifications’) in terms of quality of O&M, even though none of the projects examined was more than three years old. This is a potentially troubling finding with implications for effective use of public funds and sustainability. It highlights an area for government attention. The report recommends clarifying that funded public assets are owned by villages and thus it is the responsibility of villages to take care of and maintain public resources in their communities. The regulations should define sustainable maintenance methodologies for joint or multi-village infrastructures. The striking decline in adequacy of O&M highlights the need for more than just regulations to be in place. The PNPM program had emphasized the need for villages to invest in operations and maintenance, particularly in later years, and used the lever of not releasing block grants to villages which did not look after O&M adequately to enforce this requirement. 10 Table 1. Notable areas of comparison: 2018 vs 2012 Infrastructure Evaluation 2012 Evaluation finding 2018 Evaluation finding Technical area where a comparison between 2012 and 2018 was possible - % of VIPs Technical specifications were 82% 46% met ‘Sufficient or slightly below’ operations and maintenance 95% 15% (O&M) Existence of drawings, Local engineers typically signaled 46% had no design drawings; almost no technical design, and their acceptance of construction VIP designs were signed by engineers or quantity measurements packages by signing drawings. government representatives. Rated “High” for functionality 56% 26% by users Compliance with 78% 74% complied with environmental environmental and social procedures, but only 34% followed social safeguards safeguard mechanisms Land donation requirements Requirements more closely 3% followed followed Source: World Bank Evaluation of Infrastructure Built with Village Funds (2019) The particularly low-quality ratings for water supply and irrigation infrastructure suggest specific issues that warrant further investigation . Irrigation infrastructure was in particularly poor condition (no projects were rated moderately satisfactory or above), but the assessment of processes associated with its construction suggests technical aspects of the project process were quite good—on design completeness, technical quality, facilitation and the presence of file notes from engineers, irrigation projects were rated more highly than the other four types of infrastructure. The technical assessment team were told that central ministry engineers provided support (potentially from the Ministry of Public Works regional balai) for village irrigation projects. However, almost none of the infrastructure was being maintained or repaired by villages. One possible hypothesis is that villages view this infrastructure as belonging to the central government and so not their responsibility. Operations and maintenance practices for 100% of irrigation projects were rated below specification. It is also possible that poor involvement of users in initial design and implementation of projects undermined villagers’ ownership and willingness to contribute to operations and maintenance. The technical assessment rated the quality of participatory governance in relation to each VIP, and found ‘little, or ineffective participation’ for 77% of water supply projects and 80% of irrigation projects, as compared with an overall score of 38% for all projects. Ninety percent of irrigation projects involved no design consultation with users (compared with an average of 53% for all projects). The virtuous circle of user involvement in design→increased functionality→willingness to pay user fees→be er operations and maintenance is critical for small scale village infrastructure. 11 Poor design and construction were major contributors to the poor overall quality of water supply projects. Water supply projects were least likely to be constructed according to technical specifications, with more than 30% of projects rated ‘below specifications’. Water supply projects are complex from an engineering perspective, but project files showed little evidence of technical support. In 93% of cases the water supply project files contained no construction notes, no final inspection form signed by an engineer, and in 86% of cases there was no as-built record drawing. Finally, the technical evaluation looked at the cost of village infrastructure as compared with expected cost of inputs in the district. For the most part, village implemented projects were implemented at costs quite close to estimated district unit costs. Only for water systems and asphalt roads were costs significantly (around 25%) above those estimated by the field assessors. 12 Key Findings The key findings of this evaluation are summarized below: Overall Quality The overall quality of each VIP was rated using a six-point scale. All criteria evaluated during the Assessment contributed to this overall quality rating, except cost.  38% of VIPs were judged to be Satisfactory or better (7% Highly Satisfactory, 31% Satisfactory). A further 37% of VIPs were rated Moderately Satisfactory, thus 75% of the VIPs were of acceptable construction quality.  Of all VIPs in non-remote areas, 83% were of acceptable overall quality or better, compared to 60% of VIPs in remote areas.  For water supply projects, only 45% were rated ‘Satisfactory’ or ‘Moderately Satisfactory’; 55% were rated ‘Moderately Unsatisfactory’ or below.  All irrigation VIPs indicate some cause for concern (100% rated ‘Moderately Unsatisfactory’ or below; of this 42% were ‘Highly Unsatisfactory’). Process The ‘Process’ criteria considers the infrastructure prioritization within villages, as well as the procurement processes, quality of supervision, environmental and social safeguards, and the status of the accounts and documentation of the village infrastructure projects (VIPs).  It was found that 84% of village committees either somewhat or fully met the Village Law requirements set out in Permendagri (regulation) No.114/2014 of the Ministry of Home Affairs, (MOHA), known as the Village Development Guideline. Almost half of the VIPs, 46%, had no drawings, little in the way of design calculations, nor detailed quantity measurements. Village files were considered incomplete for 60% of the VIPs sampled.  The vast majority of the VIPs sampled, 90%, were completed using swakelola procedures (village-implemented, not third-party contractor implemented).  Only 18% of village committee’s procurement practices were found to be fully compliant with regulations. The bulk of the material viewed, 64%, was considered to somewhat comply with regulations, and 18% of files did not contain significant items that are required.  Technical supervision visits from kecamatan or kabupaten engineering departments was considered, by the villagers, to be poor (Good 2%, Average 13%, Poor 85%). VIPs showing evidence of proper inspections were few (22%); there was on average only about 1.3 inspection visits/month during VIP construction periods.  The accountability and governance of villages, measured through attendance of VIP prioritization meetings, as well as the participation of women in the process, were rated for their effectiveness. The ‘effective’ plus ‘highly effective’ participation of the general village 13 populace and of women in particular was found to be 27% and 20% respectively. ‘Ineffective’ participation by villagers in general and women in particular; percentages were 38% and 37% respectively.  The evaluators found that 74% of the VIPs had been implemented using appropriate environmental safeguard practices, but that only 34% of the VIPs had correctly followed the social safeguard mechanisms as stipulated in the Village Development Guidelines. Technical and Design Quality The ‘Quality’ section examines whether the infrastructure design is appropriate for the village needs that have been expressed in public meetings, as well as technical quality of the structures built and building in accordance with standards and code.  A majority of VIPs have been designed without direct user input; 53% of VIPs had little or no design consultation with user groups. The infrastructure that had the highest user design consultation were bridges, 67%. Buildings and water supply had about half of the user groups consulted, while roads and irrigation VIPs seem to have very little user input.  Considering the aggregated total of all VIPs evaluated, it was found that 46% of the technical components of the structures have been constructed in accordance with the plans and specifications (Meets Spec) as set out in the VIP proposals. This is down from the 2012 PNPM evaluation of 82%. A further 42% of the aggregated component ratings are Slightly Below Spec, with 12% of the components considered Below Spec. Operations and Maintenance The ‘Operations and Maintenance’ section was based upon the evaluators’ interview of village leaders and committee members, together with field inspections of the actual infrastructure.  Very few village committees are undertaking any but the most basic of O&M activities. Village committee members are not provided with any training in operational or maintenance tasks and are not provided with any budget for O&M.  A large proportion of the VIPs examined were considered to be poorly maintained (85%) with a quality rating for O&M ‘below specifications.’ Only 3% ‘met specifications’ with 12% ‘slightly below’.  Only 3% of village’s have functioning O&M Committees and even fewer collect user fees (2 irrigation VIPs only). There are no O&M funds held by any of the villages.  Functionality and utilization of the VIPs was considered. 26% of the VIPs were considered High (mostly buildings and bridges). The bulk of the sample was rated Average (64%), whlle VIPs rated Low or None are 8% and 2% respectively (some poorly built roads, water systems and 2 failed bridges). 14 Cost The cost effectiveness of Village Law infrastructure investments was determined by comparing the VIP budgets (broken down to unit rates) with standard costs as provided by the kecamatan (and confirmed by visiting construction material suppliers to gather local costs for typical items such as cement, steel, wood, etc.) The evaluators used this information to calculate local market rate unit costs (and discounting at an annual inflation rate of 3.3% for constructions in 2016 and 2015).  Buildings were found to have been constructed in a cost effective manner, with villagers negotiating an approximate 8% reduction from estimated market costs.  Bridges were found to have been delivered for very close to the evaluation’s estimated market cost (only 3% above), while water supply VIPs featuring lengthy pipe installations were constructed with budgets approximately 25% above the estimated local market rates.  Earthen and concrete roads were found to be have been constructed for amounts very close to existing market prices and can be considered cost effective investments for the villages. Gravel road budgets were found to be have been completed at 17% below market prices, while asphalt road budgets were calculated as being 23% above local market prices (although asphalt price fluctuations may be responsible for this disparity).  Irrigation VIPs are shown to be cost effectively implemented according to market rates (within the margin of error of this evaluation). 15 Recommendations The recommendations of this evaluation are summarized below: Improved technical support and supervision  Village committees should be working with competent design technicians or engineers who provide necessary liaison with relevant government sector personnel to ensure infrastructure quality and that village infrastructure conforms with government policies and programs.  Technical engineers supporting villages to design and implement construction projects should be directed to document the instructions they have given to village implementation committees and ensure these are placed in project files.  Existing design manuals and construction guidelines from previous rural development programs should be reviewed/revised to meet Village Law requirements, and issued to village committees, PDTI, and Kabupaten engineers. Such standard designs and specifications for village infrastructure do exist and should be made available and their use mandated.  Senior government should consider assigning additional technical resources to kabupaten/kecamatan levels, including more PDTI or kader teknis desa (KTD, village technical cadre), to ensure remote sites receive adequate technical support.  All infrastructure projects should have accurate and representative drawings and specifications. Standard drawings and details can be used but should be revised to suit the specific dimensions of the proposed infrastructure. Kabupaten engineers or a capable designate (PDTI) should inspect and sign-off all drawings of village infrastructure. Technical inspection by Kabupaten engineer/ designate PDTI should take place at all key stages of the project lifecycle (planning, construction, anniversary of completion). No funding from the Kabupaten should be approved without proper drawings in place.  Monitoring and evaluation of the construction program should be conducted at key points of the implementation cycle: planning, design approval, construction (e.g. 25% complete, 50%, 100%), and include an operational anniversary inspection (including O&M assessment). Participatory processes for implementation  Inter-Village Forums should be held (at least) three times annually with an agenda to include public discussions regarding the development, operations, and maintenance of infrastructure that is shared between communities.  Quorums for Mus Des (village planning meetings) should be required to guarantee attendance at important sessions (with a stipulated % of women). A survey of villagers’ impressions of the structure and format of these meetings may prove useful to order to make changes, encouraging attendance. 16  Village populations should be provided an opportunity to comment on VIP design criteria, including location, size, orientation and type of proposed infrastructures. Detailed rural infrastructure planning guidelines should be provided to the village committees. These resources should include descriptions of proper public input sessions that should be conducted as part of each VIPs’ planning.  Socialization and training of villages in the concept of user consultation should emphasize the relationship between user consultation, increased functionality of infrastructure and the willingness of village residents to pay for maintenance—the virtuous cycle of utility and sustainability.  Villages should be guided to allocate sufficient budget for community forums. This could be included in the annual prioritization guidance to villages from MOV.  Villages should be encouraged to establish procurement committees. Training  A simplified version of the Village Law regulations (a step-by-step guideline) should be developed for village committee use, with a training module developed to explain proper procedures and practices. Click here for relevant section.  Construction quality could be improved by identifying key construction problems and developing training materials to show proper techniques to correct them. Existing training materials for village activity implementation committees (TPK) should be inventoried, reviewed and improved/expanded to help villagers understand the various steps that should be executed during VIP implementation and the documentation required.  Training of village O&M committees should include, amongst other topics, a section on operations and maintenance activities appropriate to the infrastructure and advice on the collection of local user fees to fund such work. Villages should be made aware that Dana Desa funding can and should be used for O&M to ensure sustained functionality.  A procurement training course should be conducted where proper accounting and procurement practices are described and modeled for village committees, each year.  PDTI (district engineers) personnel should be provided annual technical training to improve their construction supervision skills. Regulatory changes  Land donation practices need to be improved through the issuance of clear instructions (by project type), including requiring donation letters and land transfer forms.  MOHA and MOV should add clarification to the regulations, emphasizing that the funded public assets are owned by villages and that future operation and maintenance duties and budgets are the responsibility of the villages. The regulations should define sustainable maintenance methodologies for joint or multi-village infrastructures. 17 Issues requiring more in-depth diagnosis and analysis  Water supply and irrigation projects (including those visited as part of this study) should be reviewed by relevant government agencies to determine if there are systemic problems that can be identified and avoided in the future.  Central government authorities responsible for support and supervision of Village Law implementation should undertake a deeper dive into performance information for villages in Maluku to identify if there may be specific performance issues in villages within that Province that need further attention. 18 2. Technical evaluation methodology 2.1 Scope The 2018 technical evaluation assessed 165 VIPs infrastructure built from 2015-2017 with village funds in six of the provinces surveyed in the 2012 study. The evaluation team visited six of the provinces surveyed in 2012 (spread across the archipelago) and included a random selection of the same villages, aiming for a mix of villages considered Remote vs. Not Remote. The provinces selected for this evaluation are Aceh, West Kalimantan, West Java, West Sulawesi, Maluku, and NTT. The VIPs were randomly chosen with an intention to spread the evaluation sample through the years of Village Law (2015, 2016, 2017); see the table below. In other words, during the VIP selection process in the villages, the evaluation team made sure to choose a diverse range of VIPs in both type and year of construction. Table 2. Construction Year for Sample’s Village Law VIPs Number of VIPs in Sample 2015 43 2016 60 2017 62 Total 165 Source: Author’s calculations, based on field notes of Village Infrastructure Technical Assessment (2018) For this evaluation’s results to be compared with the 2012 PNPM evaluation, the same classification system for VIP types was used. The VIP types identified for the study are as follows: Table 3. VIP types Project Type Examples of Sectors Represented Within This Total Sample Building Schools, early childhood education centres, MCK 34 (public laundry/toilet), community meeting hall, etc. Bridge Pedestrian, vehicle 15 Water Supply Gravity fed (GFWS), borehole, pond, reservoir, etc. 14 Road Road works, drainage 90 Irrigation Irrigation headworks and canals 12 Total 165 Source: Author’s calculations, based on field notes of Village Infrastructure Technical Assessment (2018) 19 2.2 Sampling approach The approach used to identify the sample villages and infrastructure projects for this 2018 evaluation is similar to the approached used in the 2012 study. The assessors (many of whom also participated in the 2012 study) refined their techniques for the current evaluation. The application of a similar methodology in 2018 allows for, in some cases, comparing findings and results. There are some instances where comparison was meaningful and stark. To understand the approach used in 2018 it is helpful to understand the methodology employed in 2012. Instances where the 2018 approach differed from 2012 are noted in the body and relevant section. 2.2.1 PNPM 2012 Sampling vs. Village Law Sampling The sampling of villages in the 2012 technical evaluation of PNPM was performed randomly within 12 provinces. In total, 1,765 VIPs were assed in that study. The methodology used included the following steps: A. A total of 12 provinces were selected ensuring that they would span Indonesia from west to east and north to south; B. Both rich and poor provinces were included; C. Sampling of districts (kabupaten) within provinces depended upon the total number of districts within each province; D. A sampling of three districts was taken for those provinces having ten or more districts. Two districts were selected from those with less than 10 districts. The sole exception to this is Central Java which had four districts selected; E. Four sub-districts (kecamatan) were sampled within each district. Sub-districts are rated in the BPS system as to level of difficulty of access – normal, hard, very hard and extreme. The random selection process ensured that all levels were represented; F. The selection of the villages within each of these sub-districts was left to the technical evaluation team to determine at each UPK office in the sub-district. The 2012 methodology is fully described in that report (Section 4: Site Selection Procedure for Technical Evaluation) and is included with this report as Annex 2. At the villages the evaluators were generally greeted by the Village Head and provided with a list of infrastructure projects financed under the Village Law. From the lists, evaluators chose a variety of VIPs to more closely examine, up to three in each village. Road improvement VIPs tended make up the majority of village lists, followed by buildings. For a more diverse sample, evaluators selected bridges, water supply and irrigation VIPs when they did appear on village lists. The table below provides a summary of the VIPs evaluated in the six provinces according to infrastructure type. The number of villages visited in each province is also shown below. A 20 complete list of the VIPs that were evaluated is provided in Annex 3. This study did not have access to a master database of all Village Law VIPs and cannot state that this evaluation’s relative percentages of infrastructure types is representative. Table 4. Number of Village Infrastructure Project by Type in Each Province Province # of Villages Building Bridge Water Road Irrigation Total in each Supply province Aceh 8 13 3 3 20 8 47 West 4 6 3 1 13 1 24 Kalimantan West Java 3 1 1 - 5 1 8 NTT 8 7 2 2 29 1 41 Maluku 8 5 1 6 12 - 24 West 8 2 5 2 11 1 21 Sulawesi Total 39 34 15 14 90 12 165 % of Total 21% 9% 8% 55% 7% 100% VIPs Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) 21 2.3 Evaluation criteria 2.3.1 Overall quality Criteria A: What is the overall quality rating ascribed to a VIP after assessing the process, quality, and O&M criteria? Overall quality is a measure of both the physical state of the infrastructure and the processes used to select and implement the project, considering all of the criteria assessed except cost. This measurement uses a six-point scale from highly satisfactory to highly unsatisfactory. This measurement takes into account sub-criteria in the ‘Technical and Design Quality’ section including the appropriateness of design and technical quality of sub-components of each VIP type but is distinct from the Technical quality measurement which assesses whether infrastructure meets technical specifications. 2.3.2 Process Criteria B: Did the process of infrastructure prioritization follow Village Law requirements? The guidelines for prioritizing the development of village infrastructure projects (VIPs) is regulated by Permendagri (Ministry of Home Affairs, MOHA), Regulation No.114/2014, known as Village Development Guideline. Evaluators questioned village representatives concerning these topics and steps and sought to confirm responses via documentation in the relevant village files. Criteria C: Was the technical verification adequate? Assessments of technical verification were based on examination of design completeness (including adequacy of surveys, design calculations, detailed quantity take-offs and similar technical materials) and file completeness (looking for inspection and activity reports, certifications of material quality and work results). Criteria D: Was the procurement process, including any community procurement, in accordance with law and norms? Most of the VIPs examined during this evaluation had been implemented using swakelola (self-service) purchasing and construction procedures for goods and services. The evaluators examined VIP files for evidence that these procedures were adhered to, looking for material supplier price lists, village proposals, signed negotiation papers, etc. 22 Criteria E: What was the level and quality of construction supervision. Village leaders were quizzed about the quality of technical facilitation that they received from government sector organizations and Ministry of Village (MOV) facilitators ( PDTI and Pendamping). Criteria F: How effective were the accountability and governance arrangements: inter- village community forums (Badan Kersacama Antar Desa [BKAD] and women’s participation in procurement committees? Recognizing that it would be difficult to quickly assess village implementation teams for their effectiveness in accountability and governance arrangements, the evaluators took the level of community participation as a proxy for the village leadership’s accountability and effectiveness. Meeting attendance records were consulted. Criteria G: Were environmental and social safeguards complied with, particularly with any land acquisitions? The Village Development Guideline stipulates that the Village Implementation Team must prepare a land acquisition document (if applicable) plus an environmental and social impact analysis report. The evaluators examined village files for these items and inspected the infrastructure to confirm that any environmental items highlighted in the paperwork had been properly carried out. Criteria H: What is the status of the accounts and other documentation? Evaluators met with village leaders at community offices where they were provided with a variety of Village Law VIP files and documentation. The evaluators (all of whom are engineers) sought evidence of three types of documents to represent the completeness of village VIP implementation files: construction notes by engineer/supervisor; final inspection form; and as-built record drawing. 2.3.3 Technical and Design Quality Criteria I: Was the design appropriate for the needs expressed? Evaluators confirmed with the village leader and implementation team members that the infrastructure design was done in accordance with their wishes and sized appropriately for the number of users. Criteria J: Have the works been constructed according to standard/code? The constructed works were examined by the assessors using similar standards and techniques as developed during the 2012 PNPM evaluation. 23 2.3.4 Operations and Maintenance Criteria K: What are the provisions for and experience with operations and maintenance? Evaluators asked questions about villages’ O&M experiences in a similar manner as was done for the 2012 PNPM evaluation. The 2012 methodology is included with this report as Annex 2. Criteria L: Is the infrastructure functional and are the works being utilized? Evaluators visited the VIPs to inspect and verify the infrastructure’s functionality. The evaluation team sought to confirm that the infrastructure is still operating as originally planned or intended and is fully functional. Utilization was judged by confirming the recipient population’s use of the facility or infrastructure for its intended purpose and usage in appropriate numbers. 2.3.5 Cost Criteria M: Was the budget reasonable, including against local market unit costs? The assessors provided budget and financial records for the VIPs being evaluated and made note of the size and materials of the infrastructure, as well as the final cost for the item under construction. To compare these budget figures with the local market prices, the evaluation team visited building material supply yards in the nearest kecamatan (sub- district) to obtain prices for standard materials. They used 3.3% yearly inflation factor to calculate the construction materials’ cost for the year of construction for the VIP being considered, and through that a standard unit cost for each infrastructure. All questions are repeated throughout Section 3 ‘Evaluation Findings and Recommendations by Evaluation Criteria,’ accompanied by the findings, analysis, and recommendations. Recommendations of the technical evaluation and areas for further investigation are presented throughout the text of this report associated with each item under discussion and gathered together in Annex 1. 24 2.4 Overall quality rating The overall quality score uses a six-point rating system based on a physical inspection of the infrastructure and examination of its design, construction materials and implementation methods (supervision and procurement), including operations and maintenance. It is based on all the other criteria assessed in Process, Quality and Operations and Maintenance (O&M). Cost is not considered in the overall quality score. The second page of ‘Field Tool 1’ offers the evaluator an opportunity to rate the VIP’s ‘overall quality’ as well as several more general and less-technical areas. These “Overall Project Assessment” categories are similar the 2012 evaluation, and are as follows:  Overall project quality: rated in accordance with the World Bank’s six-point rating system (tables and analysis in the sections that follow) (Discussed in Section 4.1 ‘Overall quality’)  Design completeness: assessed using a three-point system, (Good, Fair, Poor), with opportunity to record a comment  VIP file inspection and evaluation: observed (looking for engineering calculations, design details, construction methodologies, etc.)  Design consultations with users: assessed using Yes or No, with opportunity to record a comment  VIP functionality and utilization After the evaluation of each individual village infrastructure project’s technical qualities (as outlined in the Process, Quality, and O&M sections), the team leader asked assessors to step back and assign the project with an ‘overall quality ‘rating. This rating uses a six-point scale, from ‘Highly Satisfactory’ to ‘Highly Unsatisfactory’. This rating considers all the individual sub-criteria in Process, Quality, O&M including design completeness, file completeness, design consultations with users, and VIP functionality and utilization. The 2012 technical evaluation did not assign an overall quality rating so a direct comparison here is not possible. 25 2.5 Technical quality rating system Similar to the 2012 evaluation, assessors used a field tool consisting of various technical rating parameters to assess and assign ratings based on the technical evaluation criteria. The tool, ‘Field Tool 1’, was used to rate the technical quality of the VIPs. Infrastructures’ components or aspects are rated as being one of five choices: Meets Spec (Specification); Slightly Below Spec; Below Spec; Not Inspected; and Not Applicable. These ratings are the same as used in the 2012 evaluation. The component or aspect was examined in its current condition and reasonable allowances were made for normal wear-and-tear and degradation. These ratings are defined as follows:  Meets Specification (Meets Spec) – The VIP component or aspect meets the plans, specifications, or criteria as set out in the village’s infrastructure proposal. (The evaluators noted that many of the VIP files examined did not contain sufficient technical materials to advise village implementation committees and site personnel for proper construction to take place. VIP Proposals should contain engineering calculations, detailed construction plans and specifications for materials, construction methodologies for key parts of the infrastructure, etc.  Slightly Below Specification (Slightly Below Spec) – The VIP component or aspect displays certain characteristics that could be improved upon within its design, materials, construction, operation/maintenance or environmental conditions to meet the plans, specifications or criteria presented in the VIP Proposal. This rating will normally be accompanied by written commentary describing improvements that can be made to improve technical quality and sustainability.  Below Specification (Below Spec) – The VIP component or aspect was either (i) not constructed according to the approved plans or specifications in the VIP Proposal, or (ii) presents a clear and present danger to the life or safety of users. This rating will normally be accompanied by written commentary describing improvements that must be made to ensure technical quality and sustainability.  Not Inspected – It may occasionally be impossible for the team to inspect a certain aspect of a VIP. For example, many completed buildings feature ceilings with limited or no access to the attic. The team may not be able to inspect the interior of a building’s roof structure in these instances. Evaluators will question the village and Township personnel in this instance to verify VIP details as much as possible.  Not Applicable – Some components or aspects will not be applicable to VIPs. For example, the component Ceiling is included in the Building Checklist, but many building VIPs do not include such installations. The evaluation team considered normal deterioration of components over time. The use of this rating system assumes that standard O&M tasks have been carried out. O&M is rated separately for all VIP types; and if it has not been carried out properly, the O&M VIP component would be rated Slightly Below or Below Spec according to conditions. Extreme degradation due to poor O&M is not the infrastructure’s fault (where the VIP was well designed and installed). 26 27 2.6 Field tools and approach to field visits Below is a brief synopsis of the technical evaluation’s methodologies. Field visits Assessors visited villages according to a pre-arranged schedule. Village leaders were generally well prepared for the visit, with files pertaining to Village Law VIPs available for inspection. The evaluators met with the head of the village, as well as members of the village implementation team, which could include village secretary, village treasurer, village cadres, consultants, and the chief of hamlet. Evaluators met and interviewed members of the VIP implementation teams (including at least the chairperson, secretary, or treasurer). Evaluators also met with local facilitators as well as other interested individuals from village groups, including BPD (village representatives), as available for more information and context for analysis. Evaluators provided the following information and request to the Province and District contacts in preparation for initiating the evaluation:  The independent Evaluation Team wants to visit village infrastructure developed using Village Law funds, learn about the planning, design, and implementation processes of village development, including understanding the infrastructure’s utilization;  Evaluate, if possible, 5 types of infrastructure in each village: building, bridge, water supply, road/drainage, and irrigation;  The selection of subject villages within the districts should include remote communities;  The evaluation team wants to inspect the planning documents at each village office before visiting and evaluating the selected VIPs in the field. Evaluators visited villages according to a pre-arranged schedule. Village leaders were generally well prepared for the visit, with files pertaining to Village Law VIPs available for inspection. The evaluators met with the head of the village, as well as members of the village implementation teams. Meetings could include village secretary, treasurer, cadres, consultants, chief of hamlet, chairman/secretary/treasurer from the VIP implementation team, and the local facilitator, or other interested individuals from village groups. Representatives of village government would accompany evaluation team members to view the VIPs. The team took photographs to record details of the VIPs and illustrate their written findings. Field tools In addition to ‘Field Tool 1’, which, as described above, was used to rate the technical quality of the VIPs (whether or not a VIP was constructed according to specifications), and derive the ‘overall quality’ rating, the evaluation team employed a range of field tools to collect information on each VIP to analyze findings by VIP type and attempt to identify any trends and draw conclusions. 28 The technical evaluation teams used field instruments for each project type, developed for this study using the 2012 PNPM field tools as a guide. The field instruments consist of a set of eight checklists that were to be completed at each village for each sampled project. The Field Tools are: 1) Project Location and Technical Evaluation; 2) Environmental and Social Safeguards; 3) Cost Effectiveness; 4) O&M/Sustainability; 5) Key Issues; 6) Brief VIP Description / Notes; and 7) Process Assessment. The ‘Key Issues’ field tool for example, identifies the main issues (or areas rated ‘below specifications’) that surfaced from the technical quality assessment. This information is provided for all VIP types and can provide insight into some of the areas where projects are not meeting specifications, which can be helpful for addressing specific issues in specific public works areas. Assessors used the field tools to capture notes and commentary, Field tool 7 contains blank areas where evaluators documented specific observations pertaining to the VIP under evaluation. Paper copies that were used in the field were digitized during the data entry process. A sample of typical evaluator comments is provided in Annex 6 – Evaluation Field Notes and Commentary. The complete set of field tools is available in the Annexes and provides more detailed information than given in the body of this report. The field tools are described in Annex 4 and attached to this report in Annex 7 – ‘Sample Village Law Evaluation Field Instrument’. Field and office evaluation methodologies The field tools were taken to the villages in paper format and were completed by the evaluators in the villages. The forms provided areas where simple checkmarks would record Yes or No to specific questions. Other areas required budget data input (for the cost effectiveness study), dimensions of the infrastructure, etc. The written field data was turned into digital spreadsheets later by the evaluators and sent to the team leader and World Bank support staff. The digital data from these spreadsheets was extracted and assembled into tabular form. The data was grouped by infrastructure type, province, remoteness, etc. and analyzed. Photographs were gathered and filed according to village administrative number. A complete description of the technical evaluation’s structure, planning methodologies, field instruments, village VIP selection procedures, fieldwork preparation and practices, and data entry is provided in Annex 4. 29 30 3. Evaluation Findings and Recommendations by Evaluation Criteria This section of the report presents the evaluation’s findings and recommendations. These are organized according to the evaluation criteria and associated questions: 1) overall quality, 2) process, 3) technical quality, 4) operations and maintenance, and 5) cost. These criteria were deemed to be important enough to structure and capture this way for two key reasons. First, the evaluators believed these factors are reflective of the key elements of the village infrastructure project lifecycle. Second, the way they are evaluated allows for comparison to the 2012 findings in certain cases. The evaluators refined their methodology and field tools to make for a more nuanced assessment in 2018, but by adhering to the basic approach from 2012, comparison of findings in some areas is possible. The addition of the ‘overall quality’ rating was deemed to be a useful addition to the methodology to aggregate key technical findings and draw conclusions as to the impact of Village Law on infrastructure planning and implementation as well as identify trends by infrastructure type. 3.1 Overall quality The overall quality rating was based on a physical inspection of the infrastructure and examination of its design, construction materials and implementation methods (procurement and supervision) as well as quality of the design and construction according to the design, and operations and maintenance. These elements of the overall quality are dealt with sequentially in the following sections. Criteria A: What is the overall quality rating ascribed to a VIP after assessing the process, quality, and O&M criteria? Table 5. Overall quality by score and type of infrastructure Highly Satisfactory Moderately Moderately Unsatisfactory Highly Total Satisfactory Satisfactory Unsatisfactory Unsatisfactory Building 11% 31% 38% 11% 6% 3% 100% Bridge 27% 33% 20% 13% 7% - 100% Water - 21% 24% 19% 29% 7% 100% Supply Road 4% 33% 36% 10% 14% 4% 100% Irrigation - - - 8% 50% 42% 100% All VIPs 7% (12 VIP) 31% (51) 37% (61) 11% (18) 10% (17) 5% (8) 100% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) 31 Figure 2. Overall quality by score of all Village Infrastructure Projects 40% Highly 35% Satisfactory 30% Satisfactory 25% Moderately 20% Satisfactory Moderately 15% Unsatisfactory 10% Unsatisfactory 5% Highly 0% Unsatisfactory % of VIPs Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion Although 75% of projects were deemed to be of ‘moderately satisfactory’ or better overall quality, technical quality has declined since 2012. The evaluation found the overall quality of 38% of VIPs were deemed to be ‘Satisfactory’ or better (7% Highly Satisfactory, 31% Satisfactory, See Table 5). A further 37% of VIPs were rated Moderately Satisfactory, thus 75% of the VIPs were of average or better overall quality. Table 6. Key Construction Problems Noted During Evaluation Village Infrastructure Component/Aspect Percentage of VIP Rated Below Spec Building Roof – Connection to Purlin 23% Toilet 35% Septic Tank 53% Bridge Apron/Ramp/Road Access 33% O&M 40% Water Water system design 77% Supply Public taps –Fixtures 43% Public taps – Platforms 50% O&M 58% Road 49% Faulty Materials 30% Irrigation Field outlets 30% Control structures Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) It is noted that buildings, bridges and roads were on average rated more highly than water supply VIPs (80%, 80%, 73%, and 45% respectively rated ‘Moderately Satisfactory’ or better). All irrigation VIPs indicate some cause for concern (100% rated ‘Moderately Unsatisfactory’ or below; of this 42% were ‘Highly Unsatisfactory’). Evaluators comment that many village leaders indicate that senior qualified engineers are rarely involved with the VIPs. 32 Analysis and discussion of the technical findings, along with infrastructure-specific recommendations, are provided in Annex 5. A summary of the key construction-related findings from this analysis is in the following table. The particularly low-quality ratings for water supply and irrigation infrastructure suggest specific issues that warrant further investigation. It is possible to gain more insight from looking at how the projects were rated against different technical evaluation criteria. Irrigation infrastructure was in particularly poor condition (no projects were rated moderately satisfactory or above), but the assessment of processes associated with its construction suggests technical aspects of the project process were quite good. On design completeness, technical quality (meets specs), technical facilitation and the quality of documentation, irrigation projects were rated more highly than the other four types of infrastructure. The technical assessment team were told that central ministry engineers provided support (potentially from the Ministry of Public Works regional balai) for village irrigation projects, which likely accounts for why the quality of technical facilitation is rated more highly than for other projects supported by districts. However, almost none of the infrastructure was being maintained or repaired by villages. One possible hypothesis is that villages view this infrastructure as belonging to the central government and so not their responsibility. Operations and maintenance practices for 100% of irrigation projects were rated below specification. It is also possible that poor involvement of users in initial design and implementation of projects undermined villagers’ ownership and willingness to contribute to operations and maintenance. Ninety percent of irrigation projects involved no design consultation with users (compared with an average of 53% for all projects). The technical assessment rated the quality of participatory governance in relation to each VIP, and found ‘little, or ineffective participation’ for 80% of irrigation projects, as compared with an overall score of 38% for all projects. The virtuous circle of user involvement in design→increased functionality→willingness to pay user fees→be er operations and maintenance is critical for small scale village infrastructure. Conversely, poor design and construction were the major contributors to the poor overall quality of water supply projects. Water supply projects were least likely to be constructed according to technical specifications, with more than 30% of projects rated ‘below specifications’. Water supply projects are complex from an engineering perspective, but project files showed little evidence of adequate technical support and in some cases drawings were not adapted to local conditions. In 93% of cases the water supply project files contained no construction notes, no final inspection form signed by an engineer, and in 86% of cases there was no as-built record drawing. The quality and amount of facilitation is a key determinant of overall quality. Technical verifications during the implementation of the works should be done by qualified technical facilitators. Technical facilitation is further discussed under Criteria E in Section 3.2 on ‘Process’. 33 Recommendation 1: Water supply and irrigation VIPs (including those visited as part of this study) should be reviewed by relevant government agencies to determine if there are systemic problems that can be identified and avoided in the future. 34 3.2 Process The ‘Process’ criteria section includes elements of the village infrastructure project lifecycle from prioritization and planning, to technical verification (design and file completeness), implementation (procurement and supervision), governance arrangements, compliance with environmental and social safeguards, and the status of accounts (and other documentation). Criteria B: Did the process of infrastructure prioritization follow Village Law requirements? The guidelines for prioritizing the development of VIPs are set out in MOHA Permendagri (Regulation) No.114/2014, known as the Village Development Guideline. There are several stages to this process, including RPJMDes (Medium Term Village Planning – required every six years), which include Pengkajian Kondisi Desa (PKD, review of current village conditions) and various other steps and tools. The final product of this planning each year is the RKPDes (Annual Village Plan). With this document completed, all potential funding is defined, allowing the village to produce an APBDes (Annual Village Budget). All of these steps require special village meetings, each with requirements for participation, accountability, and deadlines. Evaluators questioned village representatives concerning these topics and steps and sought to confirm responses via documentation in the relevant village files (in records of meetings, budget calculations, financial records, etc.). The evaluation team examined minutes from village meetings, attendance records with signatures of participants, village planning priority lists and final resolutions. The Village Development Guideline, Regulation 114/2014 (article 62-68) mentions that Village Implementation Team (TPK) should prepare all administration documents. Table 7. Infrastructure Prioritization Fully met Somewhat met Did not meet requirements requirements requirements Building 24% 62% 14% Bridge 19% 63% 19% Water Supply 38% 15% 46% Road, Drainage 22% 64% 14% Irrigation 50% 50% - Total 25% 59% 16% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion The evaluation team deemed a village to have met the Village Law requirements if the process followed regulated stages: meetings at hamlet, community and village levels; the infrastructure 35 listed on Village Development Plan (RPJMDes); the infrastructure listed on Village Annual Activity Plan (RKPDes); properly accounted within Village Budget (APBDes); and finally, an infrastructure proposal that includes Bill of Quantity (BoQ), design documents, etc. As can be seen in table 5, the evaluation found that 84% of village committees either somewhat or fully met these requirements (25% Fully Met, 59% Somewhat Met). The villages where one or several of the documents were missing were considered to somewhat meet the Village Law. Evaluators used their discretion and knowledge of rural development to judge these files. A water supply system with no survey nor detailed elevation calculations was viewed more seriously than poorly planned road works. Those village where very few or no documents existed were noted as not meeting the requirements. Observations from the field: VIP prioritization, planning and coordination The number of villages that did not meet Village Law prioritization requirements is high, 16% (with 85% of these in Maluku). Evaluators made note of village committee members admitting that the Village Law regulations are confusing to them. Village VIP planning and prioritization is often driven by the existing infrastructure in the vicinity of a village or group of villages. The team observed that a formal inter-village forum would aide in communication between villages, but that is not currently part of the process. Many road improvement VIPs seek to connect and upgrade local lanes to district networks. The evaluation noted that some road networking projects are politically-driven, with village heads making campaign promises for road connections to remote hamlets. Bridge VIPs are often required to replace older structures on important transportation linkages before incipient failures. The team did find some upgraded bridges explicitly planned for access to fields and gardens (for transportation of agricultural equipment, materials and final product to markets). Bridges are also occasionally planned on new alignments to shorten the travel time between hamlets. Neighbouring villages were found to frequently share the use of kindergarten buildings, medical clinics and health centres. Some road network upgrades were completed to help distant users access these public buildings. Recommendation 2: A simplified version of Village Law regulations (a step-by-step guideline) should be developed for village committee use, with a training module developed to explain proper procedures and practices. Recommendation 3: Inter-Village Forums should be held (at least) three times annually with an agenda to include public discussions regarding the development, operations, and maintenance of infrastructure that is shared between communities. Public participation in these meetings could be enhanced using incentives (e.g. provision of food or drinks). 36 Criteria C: Was the technical verification adequate? Design Completeness Evaluators examined village VIP documentation seeking to confirm that relevant engineering topographical surveys, design calculations, alignment survey/studies (for water and irrigation systems), detailed quantity take-offs and similar technical materials had been properly executed. Each type of infrastructure requires different engineering methodologies and simpler structures require lesser amounts of technical input for proper implementation. The evaluators’ experience in rural development allowed them to make these judgments. Table 8. Design Completeness Good Average Poor Total Building 22% 44% 34% 100% Bridge 21% 43% 36% 100% Water Supply 0%- 31% 69% 100% Road, Drainage 0%- 25% 75% 100% Irrigation 58% 42% 0%- 100% All VIPs 17% 36% 46% 100% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Figure 3. Design Completeness 50% 46% 45% 40% 36% 35% Good 30% 25% Average 20% 17% 15% Poor 10% 5% 0% % of VIPs Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion The evaluation team found that almost half of the VIPs, 46%, had no design drawing, little in the way of design calculations, nor detailed quantity measurements. Most of the drawings noted were standard designs, often with no alterations to suit unique parameters for specific villages. There are almost no VIP designs that are signed by engineers or government representatives (which is different than as observed during the 2012 PNPM evaluation where PNPM engineering 37 facilitators would typically signal their acceptance of construction packages by signing or initialling the drawings). Observations from the field: Village committee support for design and drawings Some of the village committees are working with skilled consulting technicians who perform design and other work. Some of these individuals gained their experience working with villages during the era of PNPM. This was noted in West Java, Aceh Seletan and Pidie Jaya. Evaluators noted that technical designers are typically following the instructions of the village implementation teams (TPK), whose standards are generally less than those practices in PNPM. The village committees in the other provinces used a variety of designers from the Kabupaten or local sources (using formal consultant company, special hired staff/engineer or by their own forces). Many VIPs did not have designs specific to their villages and were sometimes using drawings gathered from other projects. A lack of field survey was evident, where unique conditions were not shown on plans. Even those villages helped by former-PNPM technicians showed a decidedly less-well-prepared design package as might have been observed during the PNPM programming. Evaluators commented that the design process and quality (field surveys, volume and man-day calculations, supervision from senior engineer, etc.) has deteriorated over the last years. Recommendation 4: Village committees should be working with competent civil design technicians or civil engineers who provide necessary liaison with relevant government sector personnel to ensure infrastructure quality and that village infrastructure conforms with government policies and programs. Recommendation 5: All infrastructure VIPs should have accurate and representative drawings and specifications. Standard drawings and details can be used but should be revised to suit the specific dimensions of the proposed infrastructure. Kabupaten engineers or a capable designate (PDTI) should inspect and sign-off all drawings of village infrastructure. Technical inspections should take place at all key stages of VIPs (planning, construction, anniversary of completion). No funding should be approved without proper drawings in place. File Completeness Evaluators studied the entire VIP file to verify that price surveys, accountability reports (RAB), certifications of material quality and work results, inspection reports of VIP activities, progress reports, etc. had been undertaken as appropriate. The results of these investigations are presented in the following table. 38 Table 9. File Completeness Yes No File Not Available Building 31% 56% 13% Bridge 53% 27% 20% Water Supply 29% 57% 14% Road, Drainage - 85% 15% Irrigation 25% 75% - All VIPs 20% 65% 15% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion In the evaluators’ opinion, over 60% of the VIP files examined did not contain an appropriate number of relevant materials to facilitate the construction of well-designed and implemented public infrastructures and 15% had no files at all. The evaluators’ judgment of a village’s file completeness can be compared to the ‘Overall Quality’ rating. Buildings, as an example, where the files were judged to be “complete” were more highly rated in their overall quality (“Yes” file completeness rating were rated 4.8 on the overall quality rating scale of 6 (with Highly Satisfactory = 6, Moderately Satisfactory = 5, etc.). “No” files were on average rated 3.9). All infrastructure types are shown in the following table. Table 10. File Completeness vs. Overall quality Average Overall Quality Rating for File Average Overall Quality Rating for File Completeness “Yes” Completeness “No” Building 4.8 3.9 Bridge 5.1 4 Water Supply 4.5 2.3 Road, Drainage - 3.4 Irrigation 1.7 1.7 Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion Three infrastructure types display similar characteristics: the average overall quality rating is higher for those VIP files that were rated more “complete”, in some cases very much more highly rated. This is a strong indication that document creation and proper filing plays a role in producing higher quality infrastructure. There were no complete files maintained by villages for road improvement or irrigation projects, and those that were examined lacked basic engineering items such as surveys and population/irrigation command area studies, etc. Village leaders tend to feel that irrigation projects are the responsibility of government agencies and pay them little attention in terms of paperwork and documentation. This should be kept in mind when communicating with villages on their responsibilities in financing public works projects and their responsibilities with respect to O&M. 39 Recommendation 6: Construction quality can be improved by identifying key construction problems and developing training materials to show proper techniques to correct them. Existing training materials for village activity implementation committees (TPK) should be inventoried, reviewed and improved/ expanded to help villagers understand the various steps that should be executed during VIP implementation and the documentation required. Criteria D: Was the procurement process, including any community procurement, in accordance with law and norms? Most of the VIPs examined during this evaluation had been implemented using swakelola (self- service) purchasing and construction procedures for goods and services. Table 11. Construction Modality Swakelola Contractor Joint Number of VIPs (%) 149 (90%) 1 (1%) 15 (9%) Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Note: The lone VIP implemented by contractor was a road. The joint modality were: 11 roads, 2 water supply systems, 2 buildings. Figure 4. Construction modality by number of VIPs 160 149 140 120 100 Swakelola 80 Contractor 60 Joint 40 15 20 1 0 Implementation Modality Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) The procurement processes stipulated in the law are related to the following contract amount thresholds: i) For purchases < IDR 50M (USD 3,400), village committees require only one material supplier/shop and can negotiate a price; ii) For purchases IDR 50M to 200M (USD 14,000), village committees require at least one material supplier/ shop which will provide a written quotation based on the BOQ. The village can conduct a price negotiation; iii) For purchases > IDR 200M, village committees require a minimum of 2 material suppliers/shops to provide written quotations. The village will conduct open price negotiation with these businesses. The cheapest one will be awarded the supply contract. 40 All villages should have been provided with local kabupaten lists of material prices as references/benchmarks for budgeting. See section 3.5 for more information on cost. The evaluators examined VIP files for evidence that these procedures were adhered to, looking for material supplier price lists, village proposals, signed negotiation papers, etc. Where all required paperwork was found and accounted, a rating of “fully complied” was awarded. Where the paperwork was incomplete, if there were no minutes from community meetings or where unit prices were higher than accepted kabupaten unit prices, this was rated “somewhat complied with laws”. Where no records were found or no evidence that market prices had been verified with Kabupaten lists, the rating given was “did not comply with laws”. Table 12. Procurement Process Fully complied Somewhat complied Did not comply Building (28) - 76% 21% Bridge (15) 20% 67% 13% Water Supply (13) 23% 31% 46% Road, Drainage 24% 60% 16% (90) Irrigation (10) - 100% - Total 18% 64% 18% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Figure 5. Procurement Methodologies and Process 18% 18% Fully Complied with Village Law Requirements Somewhat Complied Did Not Comply 64% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion Only 18% of VIPs fully complied with the procurement requirements, with the bulk of the sample (64%) somewhat complying. There were gaps in many village records where evaluators found information incomplete, as well as instances where unit prices have been accepted that are higher than those listed in the market price lists. (Market prices may be higher than kabupaten prices due to currency fluctuations. Kabupaten prices are updated only once/year.) Many Village Infra Projects (VIPs) were tendered to only one supplier, including some with budgets over IDR 41 200M (USD 14,000). Documents regarding price quotations and negotiations with suppliers were not found in a majority of the files. Observations from the field: Three key reasons for poor procurement The team found that village procurement processes are disrupted by three main factors: 1. Village implementation committees (TPKs) do not understand the procurement procedures that are to be followed; 2. Villages do not establish Procurement teams to share the workload. All procurement is carried out by staff of village (village head or administrative coordinator); 3. Villages are not interested to carry out a lengthy procurement process; they prefer a simple process. Three types of infrastructure (buildings, bridges, roads) were found to have between 13 – 21% that did not comply with the law, relatively good compared to water supply VIPs where 46% (6 of 13 VIPs) were either missing records or managing procurement without necessary guidance or not in line with guidance from the kabupaten. Evaluators found that many villages kept procurement records pertaining to yearly expenditures for various materials and items purchased by the village committee, but seldom broken down to separate infrastructure projects. All irrigation VIPs were found to somewhat comply with the laws, perhaps a reflection of the participation of government sectors with the designs (see Table 9 showing irrigation VIPs with markedly better Design Completeness than the other sectors). This study of the procurement processes within villages is compared to the data for cost effectiveness in Section 3.5 of this report. Recommendation 7: A procurement training course should be conducted where proper accounting and procurement practices are described and modeled for village implementation committees, each year. Recommendation 8: Villages should be encouraged to establish procurement committees Criteria E: What was the level and quality of construction supervision? Village leaders were asked about the quality of technical facilitation that they received from government sector organizations and MOV facilitators (supervisors such as PDTI [designates of district engineers] and Pendamping), and asked to rate this input as Good, Average or Poor. The results of this question are shown in the table below. 42 Table 13. Technical Construction Facilitation Good Average Poor Total Building (21) - 39% 61% 100% Bridge (14) 7% 57% 36% 100% Water Supply (13) 0% 15% 85% 100% Road, Drainage (88) 4% - 96% 100% Irrigation (10) 20% 60% 20% 100% VIPs Reporting (146) 2% 13% 85% 100% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) The frequency of technical facilitation visits was estimated by counting the number of technical supervision visits (from site logbook records or other source) and dividing it by the number of construction months for each VIP. Some villages reported that they had not received any technical facilitation visits. Those villages where records were found are reported in the table below. Table 14. Frequency of Technical Facilitation Visits During Construction (Visits/Month) (No. of VIP reporting) Visits/Month Building (16) 1.5 Bridge (10) 1.0 Water Supply (4) 1.9 Road, Drainage (6) 0.9 Irrigation (9) 1.4 VIPs reporting (27%) Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion Village implementation committee members reported that technical supervision provided by government ministries and agencies was poor (85%), with only 2% reporting that they received adequate supervision. It is notable that many of the VIPs did not receive any technical supervision visits (120 VIPs had no records of supervision visits, inspection notes or technical instructions in their files). For those VIPs that did have records of technical supervision visits (45 of 165), they averaged 1.3 visits/month. Observations from the field: Quality of supervision The team noted that kabupaten senior infrastructure engineers are responsible for guiding kecamatan technicians (capable designates of district engineers, PDTI), who are mandated to oversee Village Law works. It would seem that these inspection personnel may not be competent or willing, however, to issue instructions or provide guidance to village implementation teams. The evaluators found very little written commentary or instructions from these personnel, and the villagers’ impressions of these visits is overwhelmingly poor. 43 Recommendation 9: PDTI personnel should be provided annual technical training to improve their construction supervision skills. Recommendation 10: Supervisors should be required to provide written advice on the works they are overseeing. Criteria F: How effective were the accountability and governance arrangements: inter-village community forums (Badan Kerjasama Antar Desa) and women’s participation in procurement committees? Recognizing that it would be difficult to quickly assess village implementation teams for their effectiveness in accountability and governance arrangements, the evaluation team took the level of community participation as a proxy for the village leadership’s accountability and effectiveness. Meeting attendance records were consulted. The evaluators considered more than one sheet (each containing about 35 names) as confirming highly effective public participation; one sheet was rated as effective; less than one full sheet of names moderately effective; few names or no attendance records as ineffective. Evaluators depended on the memories of village leaders when attendance records were not available. Table 15. Villager Participation - Accountability and Governance (% of VIPs) Lots of participation. Some Limited Little participation. Highly Effective participation. participation. Ineffective Effective Moderately Effective Building (29) 7% 34% 28% 31% Bridge (15) - 7% 53% 40% Water Supply (13) - 8% 15% 77% Road, Drainage (90) - 30% 40% 29% Irrigation (10) - 10% 10% 80% All VIPs 1% 26% 35% 38% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Table 16. Women’s Participation (% of Attendees) Lots of participation. Some Limited Little participation. Highly Effective participation. participation. Ineffective (<10%) (>50%) Effective Moderately (about 50%) Effective (<25%) Building (29) - 28% 28% 45% Bridge (15) - 13% 40% 47% Water Supply (13) - 8% 31% 62% Road, Drainage (90) 1% 20% 51% 28% Irrigation (10) - 10% 40% 50% All VIPs 1% 19% 43% 37% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) 44 As reflected in table 16, women’s participation was verified through minutes of meetings and attendance records. Discussion The ‘effective’ plus ‘highly effective’ participation of the general village populace and of women in particular were found to be 27% and 20% respectively. The ‘ineffective’ percentages for the same groupings was 38% and 37% respectively. For just about two-thirds of all VIPs, one can say participation is less than expected, given Indonesia’s 17-year history of entrenched participatory mechanisms under the PNPM Program. Building infrastructure proposals bring more of the general population to meetings but, perhaps counter-intuitively, fewer women. One might still assume that women would be more likely to attend meetings about schools or health centers or water systems than for road VIPs. Observations from the field: Quality of participation – Water Supply and Irrigation projects Water supply and irrigation VIPs also show high levels of disinterest amongst both the general population and women. This may be due to the technical nature of some of these VIPs. However, evaluators also comment that in many cases village leaders use quick and practical methods to gather local hamlet heads or community leaders to discuss infrastructure proposals. Evaluators note that these meetings lack in public transparency, favoring expediency and rapid arrangements. The select attendees often do not represent village populations, and there are few opportunities for public debate of ideas. Previous empowerment programs were provided with budgets to conduct such village brain-storming meetings. Recommendation 11: Villages should be guided to allocate sufficient budget for community forums. This could be included in the annual prioritization guidance to villages from MOV . Recommendation 12: Quorums for village meetings should be required to guarantee attendance of important sessions (with stipulated % of women for, say, pre-MusDes, MusDes, etc.). A survey of villagers’ impressions of the structure and format of these meetings may prove useful to order to make changes, encouraging attendance. Criteria G: Were environmental and social safeguards complied with, particularly any land acquisitions? Evaluators used their extensive experience in rural development to rate the sampled VIPs according to three questions, as below in the table. This section of the evaluation is directly comparable to the 2012 PNPM evaluation. The Village Development Guideline stipulates that the Village Implementation Team must prepare a land acquisition document (if applicable) plus an environmental and social impact analysis report. The evaluation team examined village files for these items and inspected the infrastructure to confirm that any environmental items highlighted in the paperwork had been properly carried out. 45 Table 17. Environmental and Social Safeguards – VIPs using appropriate protocols (No. of VIPs reporting) Building Bridge Water Road, Irrigation Total Supply Drainage 1 Site inspection confirms 18 of 33 10 of 15 7 (14) 75 of 90 11 of 12 74% that appropriate 56% 67% 50% 83% 92% environmental standards were followed during construction 2 Voluntary land donation 2 of 24 0 of 8 0 of 14 1 of 44 0 of 8 3% conditions met 8% 2% (VIP where no new land is required are excluded) 3 Village Law social 9 of 33 6 (15) 3 of 14 35 of 90 2 of 12 34% safeguard mechanisms 27% 21% 39% 17% followed Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion The evaluation found that 74% of the VIPs had been implemented using appropriate environmental safeguard practices, compared to 78% from the 2012 PNPM study. (See Table 17, line 1, with irrigation schemes the highest at 92% and water supply VIPs the lowest, 50%). The high involvement of government irrigation agency personnel is indicative of good results. (Explanations for the low water supply ratings are not contained in the evaluation notes and commentary.) The evaluation revealed that only 34% of the VIPs had correctly followed the social safeguard mechanisms as stipulated in the Village Development Guidelines, with water supply and irrigation VIPs being lowest amongst the infrastructure types (21% and 17% respectively). The evaluators report that the majority of contraventions of the social safeguards were concerned with the lack of documentation for land acquisitions. Water and irrigation VIPs were predominant in this regard due to the linear nature of the corridors needed for such works. Village committees rarely sought to document and legalize these rights-of-way. Very few of the VIPs that required land donations were found to have met the voluntary donation guidelines – only 3% of VIPs (3 of 98). Many VIPs require narrow alignments of private lands to be donated. Examples include: road widening, water supply pipelines, drainage ditching and irrigation channels. Agreements with agricultural and village landowners should be verified and confirmed in writing as a matter of public record, but this seems to have seldom been done. Land donation requirements were more strictly followed during the PNPM village participation process. The 2012 study found that letters from the landowners confirming the land donations were generally largely in place for the VIPs sampled in that evaluation. 46 Table 18. Comparison of PNPM 2012 with current Village Law results Environmental and Social Safeguards PNPM 2012 2018 VIP implementation practices largely 78% 74% follow requirements Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Recommendation 13: Land donation practices need to be improved through the issuance of clear instructions (by VIP type), including requiring donation letters and land transfer forms. Criteria H: What is the status of the accounts and other documentation (including reports)? The evaluations team met with village leaders at community offices where they were provided with a variety of Village Law VIP files and documentation. Occasionally files were not available at the office where the meeting took place. The evaluators (all of whom are engineers) sought evidence of three types of documents to represent the completeness of village VIP implementation files: construction notes by engineer/supervisor; final inspection form; and as- built record drawing. The evaluators made cursory inspections of the accounting records but did not verify any of the financial aspects of the VIPs. Table 19. Construction Notes to File by Engineer/Inspector Yes No File Not Available Building (32) 3% 94% 3% Bridge (17) 12% 71% 18% Water Supply (15) - 93% 7% Road, Drainage (90) - 86% 14% Irrigation (12) 83% 17% 0% Total 3% 86% 11% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Table 20. Final Inspection Form Yes No File Not Available Building (32) - 84% 16% Bridge (15) 7% 80% 13% Water Supply (14) - 93% 7% Road, Drainage (28) - 82% 18% Irrigation (10) 20% 0% 80% Total 1% 84% 15% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) 47 Table 21. As-Built Record Drawing Yes No File Not Available Building (32) - 84% 16% Bridge (15) 20% 67% 13% Water Supply (14) 7% 86% 7% Road, Drainage (26) - 88% 12% Irrigation (10) 20% 80% - Total 5% 83% 13% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion It was noted that the files were generally disorganized, and evaluators were obliged to search for forms, drawings and documents pertaining to the design and implementation of VIPs. As can be seen in Table 20, most village files did not contain any construction notes or inspection commentary (86% of village files did not contain such evidence of due diligence). Further, 84% of VIP files did not contain a final inspection form or certificate of final approval by relevant government sector personnel or consultants. Only 5% of VIPs contained a marked-up drawing of as-built conditions. It was noted that the irrigation schemes that were viewed during this evaluation showed more evidence of senior government involvement than the other sectors, indicating that the government irrigation sector forces have taken a greater amount of interest in providing engineering support to the villages than other sectors (83% of irrigation schemes were inspected and provided with written advice by senior technical personnel from government agencies). Final inspections were also provided by these forces at many sites, it would seem, although most village files were incomplete in this regard. Recommendation 14: Technical engineers supporting villages to design and implement construction projects should be directed to document the instructions they have given to village implementation committees and ensure these are placed in project files. 48 3.3 Quality Criteria I: Was the design appropriate for the needs expressed? Evaluators confirmed with the village leader and implementation team members that the infrastructure design was done in accordance with their wishes and sized appropriately for the number of users. Designs were done in a variety of ways: hiring a consultant, using Kabupaten personnel, or other local resources. The skills, training and abilities of these designers was not researched by the evaluators. Table 22. Design Consultation with Users Yes No Building (32) 56% 44% Bridge (15) 67% 33% Water Supply (14) 50% 50% Road, Drainage (28) 25% 75% Irrigation (10) 10% 90% Total 47% 53% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion A majority of VIPs have been designed without direct user input, 53% of VIPs had little or no design consultation with user groups. This finding aligns with the lack of design documentation and file completeness observed in tables 8 and 9 in Section 3.2 ‘Process’ (Criteria C on the adequacy of technical verification). The infrastructure that had the highest user design consultation were bridge VIPs at 67%. Buildings and water supply had about half of the user groups consulted, while roads and irrigation VIPs seem to have very little user input. Observations from the field: Design consultations with users Evaluators learned that design consultations generally happened during a village meeting. Village leaders noted that bridges were of strong interest to the community, with water supply and public buildings (mainly schools) just slightly less so. Many design consultation meetings were held with more select people on village committees with specific interests and skills. Building designs are often created using standard plans and specifications, resulting in user groups having less opportunity for input; this may explain the low consultation rate for buildings. Water supply, drainage and irrigation schemes, as well as road works, can be highly technical in nature, sometimes prompting engineering designers to create designs with less user group inputs. Designers should be trained to understand that when end users are consulted during the 49 inception and design stage of infrastructures, it becomes more likely that those users will participate in maintenance efforts. Recommendation 15: Village populations should be provided an opportunity to comment on VIP design criteria, including location, size, orientation and type of proposed infrastructures. Detailed rural infrastructure planning guidelines should be provided to the village committees. These resources should include descriptions of proper public input sessions that should be conducted as part of each VIPs’ planning. Recommendation 16: Socialization and training of villages in the concept of user consultation should emphasize the relationship between user consultation, increased functionality of infrastructure and the willingness of village residents to pay for maintenance—the virtuous cycle of utility and sustainability. Criteria J: Have the works been constructed according to standard/code? This section discusses the technical quality of the village infrastructure projects and whether VIPs meet technical specifications. Measured according to its technical quality, village infrastructure has declined since 2012. This section of the evaluation is very similar to the technical part of the 2012 PNPM evaluation, which found the ‘overall quality of the design and execution of sub-projects was in full compliance with the project’s technical goal’.10 The technical ratings used (Meets Specification, Slightly Below Spec.; and Below Spec.) are described above, in Section 2.5, and are similar to those ratings used in 2012. Table 23. Comparison of PNPM 2012 evaluation with 2018 Village Law results Adherence to Standards and Codes PNPM 2012 2018 Technical specifications were met 82% 46% Slightly below specifications 14% 42% Below specifications 4% 12% Total 100% 100% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Considering the aggregated total of all village infrastructure projects evaluated, it was found that 46% of the technical components of the structures have been constructed in accordance with the plans and specifications (Meets Spec) as set out in the VIP proposals. This is down from the 2012 PNPM evaluation of 82%. A further 42% of the aggregated component ratings are Slightly 10 Neate, 2012. PNPM Mandiri Rural Infrastructure, Technical Evaluation Report. http://documents.worldbank.org/curated/en/954751468035470976/pdf/936920ENGLISH00aluation0Report02012 .pdf 50 Below Spec, with 12% of the components considered Below Spec. The 2012 PNPM evaluation found 14% Slightly Below and 4% Below Spec. Figure 6. Village Law, Adherence to Specifications, All VIPs Meets 12% Specification Slightly Below 46% Specification Below 42% Specification Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) There were differences found in the technical quality amongst infrastructure type. Bridges and irrigation projects’ components were found to Meet Spec more often than not (55% and 56%). Water supply VIPs were poorest in this regard, with only 34% of their components considered to Meet Spec. About 10% of components were generally considered to be Below Spec for all infrastructure types although water supply schemes, again, are lower than other types of infrastructure, at 35% of components deemed Below Spec. Figure 7. Technical Quality Ratings - Aggregate of all VIPs by province 60% 50% 40% Meets Spec 30% Slightly Below 20% Below Spec 10% 0% Aceh JaBar NTT KalBar SulBar Maluku Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) As shown in Figure 7, there are discernable differences in the technical quality of projects across different regions of the country. In Maluku less than 25% of projects met specifications, compared with almost 60% of projects in NTT. As noted above in Section 3.2, projects which did not comply with village prioritization processes were also concentrated in Maluku. Out of 24 VIPs assessed in Maluku, 22 did not meet Village Law prioritization processes. 51 Recommendation 17: Central government authorities responsible for support and supervision of Village Law implementation should undertake a deeper dive into performance information for villages in Maluku to identify if there may be specific performance issues in villages within that Province that need further attention. Figure 8. Technical Quality Ratings by VIP Type, aggregating all components/aspects Components examined for each VIP type Technical quality rating by infrastructure type Irrigation: System layout; Reservoir design; Weir; Water level controls; Ditches; Culvert and pipes; Embankments (Fill slope – 1 vert.:4 horiz. Maximum; Cut slope – 1 vert.: 2 horiz. max.); Irrigation channel (Dimensions; Field outlets); Channel control structures; Retaining Wall (Structural Irrigation 56 33 11 integrity; Erosion protection); O&M Road: Retaining Wall (Structural integrity [batter, etc.]; Weep holes; Erosion protection; Construction techniques; Dimensions); Culvert (Layout; Construction techniques); Small bridge (Layout; Construction techniques); Road 45 45 10 O&M Water supply: Water Source (Smell, color; Chemical analysis; Watershed protection); Water system design; Borehole and pump system; Reservoir Water supply 34 30 36 (Structural integrity; Easy to clean); Transmission and distribution pipe – proper installation; Public taps (Number and locations; Fixtures; Platform; Drainage; Fencing); Water pressure and quantity; Other structures; O&M Bridge: Layout; Foundation; Erosion protection; Abutments; Pier/supports; Wingwalls; Concrete; Deck beams; Deck; Submerged concrete laneway; Bridge 55 34 11 Handrail; Connections (nails, bolts); Apron / ramp / access to road; Other structure; O&M Building: Foundation; Ground beam/plinth beam; Wall; Column; Ring beam; Truss (Structural assembly and components; Connection to ring beam); Roof Buildings 47 36 17 structure (Roof sheeting/tiles/fasteners; Connections to purlin); Floor; Plastering; Ceiling; Painting; Doors and windows; Toilet; Septic tank; Ramp and handrail; Service utilities (Water, Electrical installation, Drainage, Other 0 20 40 60 80 100 structures), O&M Meets Specifications Slightly below specifications Below specifications Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) The foregoing percentages are based on aggregated data for the full sample of VIPs. The technical rating data was gathered for specific components/aspects of each infrastructure, so that identification of problem areas is possible and specific design or construction recommendations can be provided to improve the technical quality of these items. Figure 6 shows the individual components evaluated for each VIP type. Further details of this analysis of the individual component ratings is provided in Annex 5. 52 Key Technical Issues The Key Issues field tool provided a unique list of common problems for each infrastructure. This allowed the evaluators to quickly identify and record items of concern that they noticed during plan inspection or onsite evaluation of the infrastructures. Some of the noteworthy items are shown in the table below. Table 24. Key Issues, Design and Construction - % of VIPs Evaluated Type of issue Building Bridge Water Road Irrigation (12) (33) (14) Supply (90) (14) Design and Drawings 1 Lack of construction details/elevations 61% 43% 79% 53% 75% on drawing 2 Inaccurate/ missing drawings of 45% 29% 44% structural connection details 3 Inaccurate/ missing drawings of pipe 71% network Pipes 4 Pipes not buried 50% 5 Poor pipe connections 57% Concrete 6 Absence of concrete mix design on 42% 43% 14% 38% 67% drawing 7 Honeycombing in concrete 33% 21% 14% 20% 33% 8 Improper materials or poorly mixed 30% 21% 14% 32% 33% concrete 9 Broken concrete 50% Other 10 Inadequate erosion protection 21% 11 Poor drainage 36% 54% 12 No control on field outlets 67% 13 Lack of compaction 24% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion Several of the Key Issues are common to several or all infrastructure types, as shown on lines 1, 2, 6 and 7. From the lines 1 and 2, it is evident that the designs and drawings for Village Law VIPs are lacking in clarity and detail, likely contributing to flawed construction processes, resulting in less robust infrastructures. Lines 6, 7 and 8 show that poor drawings may be contributing to improperly executed concrete, resulting is weaker structures that are more prone to cracking or failure, especially during seismic events. Most water supply VIP drawings do not show the proposed pipe network, which also indicates that an engineering analysis of the system, considering supply rates and elevations, has not been performed. The evaluators noted that very few water supply VIPs have drawings that were developed and signed by an engineer. 53 There is no comparable data from the 2012 PNPM evaluation. Recommendation 18: Existing design manuals and construction guidelines from previous rural development programs should be reviewed/revised to meet with Village Law requirements, and issued to village committees, Kabupaten engineers or a capable designate (PDTI), and Kabupaten engineers. Such standard designs and specifications for village infrastructure do exist and should be made available and their use mandated. Recommendation 19: Monitoring and evaluation of the construction program should be conducted at key points of the implementation cycle: planning, design approval, construction (e.g. 25% complete, 50%, 100%), and include an operational anniversary inspection (including O&M assessment). 54 3.4 Operations and Maintenance Criteria K: What are the provisions for and experience with operations and maintenance? Article 83 of the Village Development Guideline in MOHA Regulation (Permendagri) No. 114/2014 directs that the Head of Village form an operations and maintenance (O&M) Team in the village and prepare the O&M budget to ensure proper operation and maintenance of village infrastructures. The evaluation question regarding O&M was also featured in the 2012 PNPM evaluation, where O&M issues at each VIP site were investigated and rated as being Sufficient, Slightly Below Sufficient, and Below Specification. These terminologies are roughly analogous to this evaluation’s O&M technical ratings (Meets Spec., Slightly Below Spec., Below Spec.). These findings are based on evaluators’ observations of properly maintained infrastructures. Table 25. Quality of Operation and Maintenance Practices (2018 Village Law Technical Evaluation vs. 2012 PNPM Technical Evaluation) VIP type and year Meets Specification Slightly Below Specification Below Specification Buildings 2012 57% 37% 6% Building 2018 0% 28% 72% Bridges 2012 59% 37% 4% Bridges 2018 0% 0% 100% Water Supply 2012 62% 30% 8% Water Supply 2018 25% 17% 58% Road Drainage 2012 43% 53% 4% Road, Drainage 2018 0% 4% 96% Irrigation 2012 45% 51% 4% Irrigation 2018 0% 0% 100% All VIPs 2012 49% 46% 5% All VIPs 2018 3% 12% 85% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion This technical evaluation has found that very few village committees are undertaking any but the most basic of O&M activities (see Table 27 for specific tasks undertaken). Village leaders commented that they have not been given any training in proper O&M practices and seem to unaware of funding that is dedicated toward these activities. This situation is very different than found during the 2012 PNPM evaluation, which found nearly half of VIPs undertaking O&M. As can be seen in the table above, that technical evaluation of 1,765 VIPs found that almost half of the village committees (49%) were undertaking proper O&M measures, compared to only 3% of the current Village Law sample. 55 Evaluators also examined village files for documentation relating to O&M. The current study has expanded the set of questions used in 2012 to include additional O&M operational data. Data in the following tables was recorded by the evaluation team. O&M Committee Table 26. O&M Committee and Local Funds Yes No Rupiah In place and functioning (Number of VIPs) 5 160 O&M user fee in place (Number of VIPs) 2 163 Average funds within O&M account 0 Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion O&M Committees are rarely formed for ongoing operation and maintenance of Village Law VIPs. Only five of the sampled villages had functioning O&M groups (3% - 2 irrigation SP, 3 water supply) and fewer villages have established a user fee for the services or rural access infrastructure (the 2 irrigation schemes with O&M teams have instituted user fee systems, though no fees have yet to be collected or at least there is no savings/reverse fund in place). Routine Maintenance by Village Committees Village committee members were questioned as to the routine maintenance activities that had been undertaken in the past for the sampled VIPs. Table 34 lists individual maintenance tasks for each VIP type, along with corresponding notes. 56 Table 27. Routine Maintenance Activities Conducted by Villagers VIP Type Routine O&M activity relevant to VIP type (Number) Mechanical Buildings Roof repair Painting Plumbing Concrete Drainage Washing (34) Plaster repair repair # of VIPs (%) - - - - - 7 (21%) - - Building routine maintenance notes: Village committees have only performed washing of the buildings when necessary. They frequently mentioned that there was no money provided for the maintenance of the structures in their village. Deck repair road repair protection Apron and Bridges structure Concrete Drainage Support Railings Erosion (15) repair # of VIPs (%) - - - - - - - Bridge routine maintenance notes: Village committees reported that they did not feel responsible for the maintenance of the bridges within their precinct, and that the national government road department would look after the infrastructure. Pipe flushing replacement Water Mechanical Pipe repair exercising Filter bed Reservoir Drainage cleaning Supply repair Valve (14) # of VIPs (%) - 1 (7%) - - - - - Water supply routine maintenance notes: As noted in the findings, many village committees said that they had received no training for these tasks and are provided with no budget to undertake these tasks. Erosion control Erosion control Regrading and culverts/walls Minor repair hole/surface re-gravelling of shoulders Vegetation Roads Drainage of slopes removal (90) repair Signs Pot # of VIPs (%) 6 (7%) - - 1 (1%) 2 (2%) - - - Road routine maintenance notes: Erosion of road shoulders is a constant problem for rural roads, as is the control of runoff from adjacent lands, particularly on cut slopes above roads. Adequate road maintenance requires a lot of labor on a frequent basis. User fees are sometimes difficult to collect, so that there is little money for paid maintenance activities. Fencing repair gates, outlets Embankment (aquatic and Canal repair Irrigation Mechanical Vegetation protection Sediment (12) removal erosion land) # of VIPs (%) 5 (42%) 6 (50%) - - - - Irrigation routine maintenance notes: The percentages gathered in this section of the evaluation are as expected, if slightly low. Farmers are spending maintenance time on those items that they see as most greatly affecting the water flow. They are spending very much less time on the physical infrastructure. Each irrigation VIP is unique in its requirements for ongoing and regular maintenance of parts of the infrastructure. Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) 57 Major Repairs and Rehabilitation The evaluators questioned the village head and committee members about their experiences regarding repair or rehabilitation of the village’s various Village Law VIPs. It was noted that all repairs and rehabilitations were performed using village labor, with no contractors or government sector forces involved. Table 28. Major Repairs (% of VIPs Replying Yes) Building Bridge Water Road, Irrigation Supply Drainage Major repairs or rehabilitation 3% 7% 7% 12% Incomplete performed? data Major repairs or rehabilitation 9% 7% 13% 6% Incomplete required/ still outstanding? data Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion It can be seen in Table 35 that there have been very few building and water supply repairs done in comparison with those that are necessary (only 25% and 33% of needed repairs, respectively, have been performed). Bridges are somewhat better (50% done) while roads get the bulk of attention (66% of major repairs or rehabs have been done). The low percentage of buildings receiving repair work may not be surprising as villagers often think of buildings as being the responsibility of the relevant government ministry or agency. The relative low level of water supply work is perplexing, as the need for major repair work is generally indicative of user unhappiness with the system and calls for action. All repair and rehabilitation works have been organized by village committee members and executed by village labor forces. Evaluators’ notes on many VIPs indicate that villages employ the gotong royong system, whereby villagers volunteer on a rotating basis, usually once/month. There is no comparable data from the 2012 PNPM evaluation. Village Maintenance Plans Table 29. Village Maintenance Plans (Number of VIPs) Yes No Do VIPs within a village have a maintenance plan? 47 118 Are there linkages to line Ministries? 165 Clear division of responsibilities and costs 165 Contains estimated costs for routine/capital repair 165 Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) 58 Discussion It is seen that a minority of VIPs have maintenance plans (28%) and that these documents deal with O&M issues in a very general way. Evaluators noted that the maintenance plans normally dealt with all infrastructures in the village. None of them show any specific details for the infrastructure types or where advice or special expertise is required. The evaluation noted that villages take varying approaches toward the operation and maintenance of their infrastructures. Some villages may have one maintenance plan for all infrastructure, while other villages may decide to have several committee groupings and plans, perhaps because of a larger number of infrastructures or logical divisions of responsibilities for different hamlets. O&M Training Table 30. O&M Training Yes No Rupiah O&M training received 165 Ongoing capacity development 165 Annual training budget 0 Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion It is readily seen that village committee members are not provided with any O&M training and that the committees are not provided with funding that is identified for the ongoing operation, upkeep and repair of new structures. This is different from the situation noted during the PNPM evaluation, where all O&M activities within villages had to be voluntary or swadaya contributions. Now all O&M can be fully financed, from user fees or from the village budget (APBDes). Recommendation 20: MOHA and MOV should add clarification to the regulations, emphasizing that the funded public assets are owned by villages and that future operation and maintenance duties and budgets are the responsibility of the villages. The regulations should define sustainable maintenance methodologies for joint or multi-village infrastructures. Recommendation 21: Training of village O&M committees should include, amongst other topics, a section on operations and maintenance activities appropriate to the infrastructure and advice on the collection of local user fees to fund such work. Villages should be made aware that Dana Desa funding can and should be used for O&M to ensure sustained functionality. 59 Criteria L: Is the infrastructure functional and are the works being utilized? This rating is directly comparable to the 2012 PNPM evaluation. Evaluators visited the VIPs to inspect and verify the infrastructure’s functionality. If the infrastructure is still operating as originally planned or intended, then a rating of ‘Average’ would be considered. A ‘High’ rating for this aspect of the assessment might be represented by a VIP where the recipient community or users have independently added to or improved a VIP to increase its usefulness. A ‘Low’ rating was assigned to VIPs that were poorly designed/built and not satisfying the requirements of the local population. Utilization was judged by confirming the recipient population’s use of the facility or infrastructure for its intended purpose and in appropriate numbers. A rating of high for this aspect would be where evaluators observe a public enthusiasm for the infrastructure’s impact on village activities and daily lives. Evaluators would consider their rating as being a blend of these two aspects, with one sometimes dominating the other for special circumstances. Table 31. Functionality and Utilization High Average Low None All VIPs, 2012 PNPM evaluation 56% 34% 8% 2% All VIPs, 2018 evaluation 26% 64% 7% 3% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion The results show 26% of the VIPs considered ‘High,’ a large fall of highly rated VIPs from PNPM’s 56%. The bulk of the sample was rated ‘Average’ (64%), while the ratios of VIPs rated ‘Low’ or ‘None’ are relatively unchanged. This difference may be due to the fact that the PNPM had resources to mobilize village committee members and villagers to attend the technical evaluation and participate in its research activities. Village Law’s infrastructure component has been made sufficiently bureaucratic that village populations may simply regard it as part of the fabric of governance. This social aspect of research was not pursued by the evaluators. 60 Figure 9. Functionality and Utilization 7% 3% 26% High Average Low None 64% Source: Author’s calculations, based on field notes of Village Infrastructure Technical Assessment (2018) Functionality and utilization of the VIPs was compared with the findings on villager participation (Section 3.2 and Table 15 above). The VIP functionality/ utilization data were assigned values: High = 4, Average = 3, Low = 2, None = 1. The average functionality/utilization rating for each level of villager participation was calculated, as shown in the table below. Table 32. Average Functionality/Utilization Rating for Villager Participation levels Lots of Some Limited Little participation participation participation participation Building - 3 2.8 2.75 Bridge 4 3.25 3 1.75 Water Supply - 4 3.5 3 Road, Drainage - - 2.9 2.9 Irrigation - 3 3 3 Source: Authors calculations, based on field notes of Village Infrastructure Technical Assessment (2018) Discussion It can be seen that as the level of villager participation in the VL process declines, so too does the average level of functionality/utilization of the associated VIP (for buildings, bridges and water supply infrastructure projects). The correlation for roads was less clear (with limited to no participation from average villagers), and irrigation projects were uniformly judged of average functionality, no matter the level of villager participation. It can be inferred from this that efforts in raising a community’s participation in the VL building program will produce similar rises in the functionality and utilization of the resulting village infrastructure. 61 3.5 Cost A. Criteria M: Was the budget reasonable, including against local unit costs and norms? Evaluators were provided budget and financial records for the VIPs being evaluated and made note of the size and materials of the infrastructure, as well as the final cost for the item under consideration (excluding extra costs for unrelated parts of the contract). Detailed procurement records were available for relatively few of the VIPs, only 18% (See Table 12 for a breakdown of this by infrastructure type). The amount of money received under Village Law by villages is quite large and is normally used to fund many small village projects, infrastructure and otherwise. The villagers use the VL procurement rules for small purchases and frequently seek only one supplier quotation with little documentation. Because administration of most VIPs did not include village procurement meeting (s) 11, village leaders would simply collect material price lists from suppliers. They would normally choose the least-cost option. VIP budgets and material cost calculations were not cross-examined with construction procurement records. Determining unit rates Unit rates were derived using the budget and financial information. For example, building costs were reduced to IDR/sq.m. of building area; water supply schemes to IDR/m of pipe installed; roads to IDR/sq.m. of road surface (with differing unit rates noted for earthen, gravel, concrete or asphalt roads); irrigation schemes were compared using total canal lengths, IDR/m; etc. In order to compare these budget figures with the local market prices, evaluators visited building material supply yards in the nearest kecamatan. The following steps were taken to complete the cost effectiveness field tool: 1. At the supply years/shops evaluators gathered the local prices being charged for typical construction materials (especially cement, stone, sand, steel, pipes, bricks, etc.); 2. The team created digital spreadsheets of the detailed VIP cost estimates accessed from village documents; 3. They used 3.3% yearly inflation factor to calculate the construction materials’ cost for the year of construction of the VIP being considered, and through that a standard unit cost, to be used for comparison within the three separate construction seasons (2015, 2016 and 2017). 11 As noted in Section 3.2, Table 12, 18% of villages did not comply with procurement processes and 64% only somewhat complied. 62 In the following tables, the unit costs are averaged of the noted number of VIPs over the three construction years evaluated under Village Law. It should be noted that while VIPs may be built at a lower cost compared to cost norms, it does not mean VIPs are ‘cost effective’ as the concept of cost effectiveness implies value for money. The question of value for money is critical as the quality of construction appears to have declined compared to the 2012 PNPM evaluation. This is an issue that is often not adequately captured in many technical evaluations of CDD operations. Further analysis in this area, such as assessments of potential correlations between the cost and the quality of sample VIPs, may be the subject of future work. Such analysis may reveal insight on the types where the quality is highly dependent on cost, and which may warrant additional expenses if they lead to a higher value for money. Buildings Sufficient information was gathered for 26 buildings within the villages. These buildings ranged in size from 18 to 280 sq.m. (average 77 sq.m.) and contained from 1 to 5 rooms (average 2.8 rooms). The buildings are being used as schools, early childhood education centers (PAUD), MCK (rural sanitation and laundry facility), and community warehouse and meeting halls. Table 33. Building VIP Unit Costs Actual Unit Costs Market Rate Unit Costs Actual/Market (%) (IDR/sq.m.) (IDR/sq.m.) Building (data from 2,119,000 2,296,000 92% 26 VIPs) Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion Buildings were found to have been constructed in a cost effective manner, with villagers negotiating an approximate 8% reduction from estimated market costs. Bridges There were eight bridge VIPs where the budget data was available to calculate unit costs. The average length of these bridges is 6.4 m and all are used for vehicle traffic. Table 34. Bridge VIP Unit Costs Actual Unit Costs Market Rate Unit Costs Actual/Market (%) (IDR/sq.m.) (IDR/sq.m.) Bridge (data from 8 8,254,000 8,001,000 103% VIPs) Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion Bridges were found to have been delivered for very close to the evaluator’s estimated market cost (only 3% above – likely within the margin of error of this study). 63 Water Supply Water supply VIPs are difficult to compare with one another, as systems frequently feature unique circumstances: reservoirs of different sizes and materials (concrete, steel), various public tap stands and amenities, mechanical or hand pumps, etc. These components are used by engineers to configure systems to local populations’ needs. The evaluators were, however, able to find six VIPs where lengthy water pipelines had been installed (gravity-fed systems). These pipe alignments provided a basis upon which to compare VIPs with each other and with local supplier prices. Table 35. Water Supply VIP Unit Costs Actual Unit Costs Market Rate Unit Costs Actual/Market (%) (IDR/m) (IDR/sq.m.) Plastic pipe (data 395,518 318,499 124% from 4 VIPs) Steel pipe (data 1,199,158 934,839 128% from 2 VIPs) Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion The evaluators’ calculations indicate that water supply VIPs featuring lengthy pipe installations were constructed with budgets approximately 25% above the estimated market rates. The reason for this is not clear from the data gathered, though in many ways water supply projects built using Dana Desa are problematic. Roads Road VIPs are constructed using various materials: earthen (feasible where local soils are competent and granular), gravel, concrete, or asphalt. There were 29 VIPs where sufficient financial and design/BoQ data was available for analysis. Of these, the majority were concrete (17 VIPs) and gravel (7 VIPs). Table 36. Road VIP Unit Costs Actual Unit Costs Market Rate Unit Costs Actual/Market (%) (IDR/sq.m.) (IDR/sq.m.) Earth road (data 19,597 19,898 100% from 3 VIPs) Gravel road (data 81,061 97,921 83% from 7 VIPs) Concrete road (data 219,629 218,521 101% from 17 VIPs) Asphalt road (data 220,134 179,687 123% from 2 VIPs) Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) 64 Discussion Earthen and concrete roads (comprising 20 of 29 VIPs analyzed) were found to be have been constructed for amounts very close to existing market prices and can be considered cost effective investments for the villages. Gravel road budgets were found to be have been completed at 17% below market prices, a situation that is difficult to explain without seeking additional data from those VIP villages (are there local gravel pits that village officials were able to access, for example). Asphalt road budgets were calculated as being 23% above local market prices (although certainty of this result is low since the sample contains only 2 VIPs featuring asphaltic pavements). Asphalt has typically experienced large price fluctuations in the Indonesian marketplace, which may have contributed to this finding. Irrigation Irrigation VIPs often feature unique sets of infrastructure components at the headworks of a system (concrete water diversion works within the stream or watercourse), making comparisons between the VIPs difficult. The evaluators examined village files to find examples where canals had been constructed as part of the works and used the separated costs for these as a proxy for comparison. Table 37: Irrigation VIP Unit Costs Actual Unit Costs Market Rate Unit Costs Actual/Market (%) (IDR/m) (IDR/sq.m.) Irrigation canals 421,461 408,789 103% (data from 9 VIPs) Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion Irrigation VIPs are shown to be cost effectively implemented according to market rates (within the margin of error of this evaluation). This cannot be compared to the 2012 PNPM study as that evaluation did not collect any budgetary data for irrigation VIPs. Village Procurement Practices vs. Cost Effectiveness The data collected in the procurement section (3.2, Table 12), shows that during procurement only 18% of village implementation committees fully complied with Village Law requirements., with a majority of 64% somewhat complying. This data was correlated with the cost effectiveness information, producing the table. (Water supply projects were too variable in their infrastructure to allow averages to be made. All irrigation projects were judged Somewhat Compliant.) 65 Table 38. Procurement Practices vs. Cost Effectiveness Actual Unit Cost/sq.m. Fully complied with Somewhat complied Did not comply with (No. of VIPs Averaged) laws with laws laws Building - 2,141,000 (10) 2,266,000 (2) Bridge 4,637,000 (2 outliers) 4,244,000 (5) - Road - Gravel - 90,000 (8) 93,750 (1) Road - Concrete 158,000 (5) 178,000 (16) 253,000 (5) Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion Given the low number of observations, it is not possible to determine a clear relationship between procurement practices and cost effectiveness, except in relation to concrete roads, where the data shows a clear trend toward higher basic unit costs for lesser well-managed projects. Gravel roads and buildings show similar trends, although these are supported by less field data. 66 3.6 Other evaluation data and observations Remoteness Evaluators recorded the degree of remoteness for a VIP village. The two degrees of remoteness are as follows: Table 39. Degrees of Remoteness and Sampled Number of VIP Definition No. of VIP Not Remote <20 km from kecamatan, with reasonable transportation linkages 65 Remote >20 km from kecamatan; border areas; disadvantaged areas 100 Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) The data were sorted to determine if a village’s degree of remoteness played a significant part in the technical quality rating of VIP components. A hypothesis might be that the technical quality of a VIP will go down as the degree of remoteness goes up, due to a number of possible factors: increased difficulty for technical facilitators to visit the site; reduced number of skilled laborers being available; increased difficulty in securing proper construction materials; etc. Table 40. Technical quality ratings for all components of all VIPs, by Remoteness of location Meets Spec Slightly Below Below Spec Not Remote (65 VIP) 51% 31% 17% Remote (100 VIP) 36% 35% 28% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Overall Quality and remoteness Table 41. Overall quality by Remoteness Highly Satisfactory Moderately Moderately Unsatisfac. Highly Satisfactory Satisfactory Unsatisfac. Unsatisfac. All VIP – Not 7% 34% 42% 6% 6% 4% Remote All VIP – 7% 25% 28% 18% 16% 6% Remote Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion The data gathered during this study does indicate that there is a trend toward lower overall quality of infrastructure as the degree of remoteness rises. It can be seen in Table 40 that on the narrower rating of technical quality, the Meets Spec percentage for Not Remote VIPs is higher 67 than the Remote sites (51% vs. 36%, respectively). Table 41 confirms this trend via the evaluators’ judgment of overall quality for each of the infrastructure, although with more differentiation at lower levels of overall quality. Eighty-three percent of projects constructed in non-remote areas are of moderately satisfactory quality or better, compared with 60% in remote areas. Technical Facilitation vs. Remoteness The quality of the technical facilitation (from Table 13 in Section 3.2 ‘Process’, evaluation criterion on quality of technical facilitation) can be expressed as a score (Good = 3, Average = 2, Poor = 1) and compared with the remoteness of each VIP evaluated. Table 42. Quality of Technical Facilitation vs. VIP Remoteness Not Remote Remote Building 1.3 1.4 Bridge 1.9 1.6 Water Supply 1.3 1.1 Road, Drainage 1.2 1 Irrigation 2.4 - Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Discussion The data in this table generally shows that the quality of technical facilitation (as rated by village leaders) was deemed to be of higher quality for less remote villages (with only buildings showing the reverse). Irrigation VIPs were almost all located in Not Remote places (and were provided with better technical inputs by government agents). The data in the 2012 PNPM evaluation was analyzed in a similar way for each infrastructure type. The results also showed that Remote villages had slightly poorer quality than infrastructure constructed “Near Main Road” (this 2012 rating is congruent with this evaluation’s Not Remote category). Table 43. Comparison of PNPM 2012 evaluation with current Village Law results Remoteness Effect on SP/VIP PNPM 2012 VL 2015-17 Quality (% of components Meet Spec) (% of components Meet Spec) Not Remote 76% 51% Remote 63% 36% Source: Author’s calculations based on field notes of Village Infrastructure Technical Assessment (2018) Recommendation 22: Senior government should consider assigning additional technical resources to kabupaten/kecamatan (district/sub-district) levels, including more PDTI (designated supervisors) or kader teknis desa (KTD, village technical cadre), to ensure remote sites receive adequate technical support. 68 69