ICRR 13918 Report Number : ICRR13918 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 06/06/2014 Country : Indonesia Project ID : P076174 Appraisal Actual Project Name : Initiatives For Local US$M ): Project Costs (US$M): 46.30 44,51 Governance Reform Project L/C Number : C4078; L4790 Loan/ US$M): Loan /Credit (US$M): 29.50 29.89 Sector Board : Agriculture and Rural Cofinancing (US$M): US$M ): 12.00 8.84 Development Cofinanciers : DFID Board Approval Date : 06/09/2005 Closing Date : 09/30/2009 10/13/2011 Sector (s): General transportation sector (40%); Sub-national government administration (30%); Irrigation and drainage (15%); Water supply (10%); Power (5%) Theme (s): Rural services and infrastructure (20% - P); Participation and civic engagement (20% - P); Other accountability/anti-corruption (20% - P); Public expenditure; financial management and procurement (20% - P); Decentralization (20% - P) Prepared by : Reviewed by : ICR Review Group : Coordinator : John R. Eriksson Robert Mark Lacey Christopher David IEGPS1 Nelson 2. Project Objectives and Components: a. Objectives: The Project Appraisal Document (PAD) states that the Project Objective is to : "pilot support to district (kabupaten) governments in improving transparency, accountability and public participatory practices and in undertaking reforms in financial management and procurement " (p.3). The Bank financing was in part an IBRD Loan and in part an IDA Credit . The Loan Agreement of October 11, 2005 lacks any reference to a project objective . Schedule 2 of the Development Credit Agreement of the same date contains the following project objective : "assist the Borrower in supporting local governments of Participating Kabupatens and Additional Participating Kabupatens to improve transparency and participation practices and to undertake reforms in financial management and procurement, and financing of part of Participating Kabupatens' investment development programs " (p.20). This Review is based on the statement of objectives in the PAD as it is more monitorable . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components: The project had three components :: A: Local Governance Reform (appraisal: US$1.33 million; actual: US$1.35 million) Continue support for district governments that participated in project preparation -about 15 so-called "first batch" districts (subsequently reduced to 14 as part of restructuring) - and met minimum entry requirements to undertake more advanced reforms as stipulated in the Local Governance Reform Framework . Starting in the second year of implementation, about 25 additional districts were to be selected for participation in the project and assisted over 18-24 months to meet minimum pre-investment requirements. Sub-components included: Al Reform of the District Planning and Budgeting Process, including a focus on participation and Al. strengthening pro-poor priorities at the local level. A2. Reform of District Budget Implementation and Financial Management and Reporting to improve local procurement and financial management practices . A3. Strengthening Accountability Mechanisms to help increase information disclosure and cross -district networking. B: Poverty Targeted Investment (appraisal: US$31.18 million; actual: US$33.80 million) Provide incremental financing in about 15 "first batch" districts in order to augment pro -poor development expenditures identified and prioritized through a planning process linked to development of the district Poverty Reduction Strategy Action Plan. C: Implementation Support (appraisal: US$11.90 million; actual: US$9.36 million) Fund specialized technical, facilitation and monitoring support for Components A and B at district, regional and national levels. Sub-components included: C1. General Facilitation and Specialized Technical Assistance, to provide for facilitation and technical assistance to districts. C2. Capacity Building and Institutional Training, for district staff and other local stakeholders to institute governance reforms. C3. Monitoring, Evaluation, and Studies /Surveys, to support the overall project monitoring and evaluation framework and finance studies on local governance . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost The total actual project cost was US$ 44.51 million, which, together with an IBRD front-end fee of US$0.07 million, came to a total financing requirement of US$ 44.58 million. Financing The project was externally financed by an IDA Credit of US$ 15.39 million, an IBRD loan of US$14.50 million, and a United Kingdom (DFID) grant of US$8.84 million. The DFID grant was in the form of parallel cofinancing for Component C; following restructuring, an undisbursed portion of the grant was allocated to subproject investments under Component B. Borrower Contribution The estimated borrower contribution at appraisal was US$ 4.80 million or 10.4% of total cost (including local government contributions of $ 4.62 million). The actual borrower contribution (including local government contributions of US$4.42 million), was US$5.85 million or 13.1% of actual total cost. The increase in the borrower share results from the decline in total cost from US$ 46.30 million at appraisal to US$44.51 million at closure. Dates After the mid-term review planned in May 2007 and held in April 2009, the original closing date of 09/30/2009 was extended by two years to September 30, 2011, to compensate for earlier implementation delays . 3. Relevance of Objectives & Design: a. Relevance of Objectives: High. High The project’s objective is relevant to the Bank ’s country assistance strategy for Indonesia as set out in the Country Partnership Strategy (CPS) for the Fiscal Years 2009-2012. This sought to strengthen governance through supporting the implementation of decentralization and greater local government accountability . The CPS’s second "Cross-cutting Engagement area" in its Results Matrix is "Sub-national Government Institutions and Systems, " and the longer-term objective is to "strengthen decentralized local governments and institutions to increase the impact of public spending" (p.57). Within this overall goal, the strengthening of district level governance, especially in terms of transparency, participation, accountability and financial management, is a high priority . The objective is also relevant to the Government of Indonesia ’s strategy. By October 2004, district level governments were responsible for providing basic services to their citizens . In the past, financial and human resources were allocated and appointed by the central government in Jakarta . With decentralization, all government functions -apart from national economic and monetary affairs, defense, legal and judicial systems, and international relations and trade- were devolved to the 440 kabupatens (districts) and kotas (urban centers). Strengthening the limited capacities of these entities to fulfill their new functions effectively, responsively and transparently has required a major, long-term effort. The decentralization effort was also expected to contribute to "calming separatist sentiments and regional conflicts" in Indonesia (CPS, p.4). b. Relevance of Design: Modest . . The causal chain between the project ’s activities and the achievement of its objectives is unconvincing . In order to attain the objectives, the 40 districts chosen for the pilot were each to undertake a set of 67 detailed reforms identified in the Local Governance Reform Framework of the PAD . These reforms were in the areas of transparency (ten actions), accountability (five actions), participation (eight actions), financial management (twenty actions), and procurement reform (twenty-four actions). In addition, each district was expected to fulfill three general requirements : (i) the District Head was to sign a letter to the National Steering Committee, expressing interest in participating in the programs and commitment to undertake to establish the necessary working groups and other prerequisites for carrying out the reforms; (ii) evidence must be provided to show that the participating district had no payments arrears; and (iii) no fiduciary issues were outstanding . While the substance of the expected reforms was relevant to the overall project objective, the actions to be undertaken were so detailed and complex, and the capacity of most districts so weak, that the feasibility of accomplishing all of them in 40 pilot districts (out of a total of 440 districts in the country) would have been questionable. The decision at the mid-term review to reduce the number of districts to be included in the project to 14 confirms the lack of realism of the original design . Moreover, the financial resources allocated by the project to this ambitious reform program was modest indeed – a mere US$1.33 million. The project documents do not provide an understanding of the policy and institutional context that would enable an assessment of the feasibility of the desired reforms . Some governance considerations were not taken into account in project design, including the adverse effect of continuing modifications of the legal and regulatory environment, as well as a restriction on-granting of funds to districts with limited fiscal capacities (ICR, p.6). The largest project component was B, “Poverty Targeted Investment, � amounting to 76% of actual total costs. It was intended for district level sub -projects involving the construction, rehabilitation and upgrading of local infrastructure . Both the PAD and the ICR are clear that the main, if not the sole, reason for this component was to provide an incentive for districts to undertake governance reforms . For example, the ICR (p.10) states: “The poverty-targeted investments in Component B contributed to the overall PDO outcome, in that project funding for these sub -projects was an important incentive for participating district governments to undertake the governance reforms of Component A." (This link was confirmed by the Borrower ’s ICR (ICR, p. 41), which also commented that the emphasis on sub-projects would dilute the focus of the districts on completing their reform program .) However, it was intended that sub -project investments would be supported by this operation in only 15 of the 40 districts, with investments in the other 25 left to be financed by a follow-up project (PAD, p. 8). The 25 remaining districts were, nevertheless, expected to complete their reform programs without the incentive of the sub -project investments, at least in the short -to-medium term. M&E design was weak (see Section 10a below), which is an especially important shortcoming for a pilot operation . 4. Achievement of Objectives (Efficacy): 4. Achievement of Objectives (Efficacy): There were five objectives: pilot support to district governments in (a) improving transparency, (b) improving accountability, (c) improving public participatory practices, (d) undertaking reforms in financial management, and € undertaking reforms in procurement . The project had the following outputs , which were common to the five objectives : Poverty reduction strategies and action plans were issued in 14 districts and implemented in four district budgets prepared. 262 sub-projects had been funded by closure, compared to a revised target of 96 and an original target of 72. The ICR provides two reasons for achievement being substantially beyond target : (i) participating districts were able to complete multiple investment sub -projects, especially after project restructuring (the project team confirmed that the experience gained by performing districts prior to restructuring strengthened their capacity to design and implement sub-projects, along with the greater availability of consultants after restructuring ); and (ii) the average size of a sub -project was about 60% smaller than anticipated in the PAD. Fourteen project districts received capacity building assistance in transparency, participation, procurement and financial management) by project closure, compared to a revised target of 10 districts and an original target of 40. The pre-restructuring target of 40 districts was reduced to reflect the reduction in the number of participating districts. Few details are provided on the nature of capacity building and technical assistance activities . Eight staff were placed in the central government to support project monitoring and implementation, eight staff were placed in regions to fulfill these functions, and 26 staff were assigned to support district governments . At least 4 districts increased their poverty -targeted expenditures in 2010 over 2008, compared with a revised target of at least 10 districts. The ICR cites a survey that showed an average of 11% of 2010 budgets for poverty targeted expenditures in 10 sample districts, compared to a target of 20%. It attributes the shortfall "in part due to overall reductions in poverty in these districts ." Twelve districts issued regulations on mechanisms for public consultation & information access by mid -term and 20 other districts prepared draft local regulations by closure, in accordance with targets . Public consultations on draft budgets and regulations were held in at least 12 districts. Although the aim was to increase the extent to which consultations and hearings were reflected in public documents, there is no indication as to whether this occurred . Action plans for financial management and procurement reform were prepared and implemented in 10 districts in accordance with the revised target . Outcomes (a) Pilot support to district governments in improving transparency . Modest . Transparency is not specifically defined in the project documents . Its meaning must, therefore, be inferred . It tends to be delineated in terms of what is not transparent – for example, a prevalence of opaqueness in budgets and other public documents or a lack of clarity in how expenditures were derived (ICR, p.1). The Borrower’s ICR indicates that transparency means open provision of information to the public and involving the public in the government process (ICR, p.40). The evidence on improved transparency reported by the ICR is meager, general and impressionistic . The ICR concludes that based on a review of summaries of district budgets, accountability reports and procurement plans, “transparency increased in at least 12 districts,� compared to an original target of “greater public availability of financial and procurement information .� No indication is given as to what quantitative or qualitative evidence was used to support this conclusion . Moreover, there is no indication as to how widely information was disseminated and made accessible to district populations . (b) Pilot support to district governments in improving accountability . Modest . No direct measure of accountability is provided in the project documents, and how it was to be assessed (or was assessed) is not explicitly addressed . The PAD (p.57) refers to a "Regional Financial Accountability Assessment" to be conducted by external consultants and the Ministry of . No such assessment is mentioned in the ICR. Consumer satisfaction could be viewed as an indirect, proxy measure of accountability . According to a survey conducted as part of the final project evaluation (M&E Secretariat, National Development Planning Agency, 2011), the average customer satisfaction rate increased to 82% by 2010 for the five most commonly used public services in 14 districts, compared with a baseline of 60% (measured in 2006 in 13 districts in a survey supported in part by the Japan Social Development Fund ). There was no target. The five most commonly used public services are not defined. Without more information about the methodology and scope of the two surveys, it is not possible to conclude definitively that customer satisfaction increased between 2006 and the project closure. Establishing a mechanism for receiving complaints may be viewed as one step toward improved accountability . The PAD (p. 27) called for a central complaints unit to monitor and follow up on any complaints received by potential project-affected persons, "so that remedial action can be taken quickly " (p.27). It is not clear whether this unit was pre-existing, or what was done to strengthen it . It was also the intention to set up complaints units at the district level. Only three districts established such units (ICR, p. viii). The outcome indicator was the number of complaints received and satisfactorily resolved . There were four such cases . The ICR results framework comments that this indicator and achievement do "not fully reflect the intent of project design for participating districts to set up complaint handling units for the general public .� The project team stated that most complaints are “handled informally� and that a formal complaints unit is a new concept for the Government . (c) Pilot support to district governments in improving public participatory practices . Modest . By project closure, all 14 districts had effective local regulations in place on public consultation and access to information. A survey undertaken in participating districts found that “over 83 percent of community respondents found it easy to access public documents on planning and budgeting, and that more than 90 percent of draft local regulations underwent a process of public consultation as part of the local legislative process � (ICR, p. 20). However, no details are provided on the nature of this process . The ICR also states Data Sheet, p . iii) that there is “ample evidence� that public consultations in 12 districts “were meaningful and influenced the formulation of plans and regulations .� However, the ICR provides no evidence to support this statement . (d) Pilot support to district governments in undertaking reforms in financial management . Modest . All 14 districts participating in the restructured project were reported to be practicing “sound� financial management, with five of these districts having received “improved opinions from the Regional Financial Audit Board as verified by audits and studies at project closure � (ICR Data Sheet, pp.iv-v). However, related information reported by the ICR is inconclusive . The ICR defines “sound� as receiving unqualified audit opinions from the Regional Financial Audit Board . It is not clear whether any of the 14 districts received an unqualified opinion from this Board. Moreover, it is reported that only one district received an unqualified opinion from the Supreme Audit body (ICR, p.12). (e) Pilot support to district governments in undertaking reforms in procurement . Modest . More than 10 districts practiced “accepted procurement standards � as verified by audits and studies at project closure (ICR Data Sheet, p. iv). This result was based on a survey conducted by the M&E Secretariat of the National Development Planning Agency . The Data Sheet states that in 2006, prior to the project, “the procurement system countrywide did not function well and that districts had less than satisfactory procurement systems.� However, the project documents do not define “acceptable procurement standards � or “satisfactory procurement systems.� No information is provided regarding the methodology of the survey . No indication is given of the extent to which the procurement standards met in the 10 districts correlated with the 24 procurement reforms stipulated in the Local Governance Reform Framework, and which are outlined in Attachment 1, Annex 2, of the PAD. According to the Borrower's ICR (ICR, p. 39), local governments had "stopped implementing... the procurement survey, publication of procurement sanctions, and publication of contracted unit price " (p.39) owing to lack of funds allocated to implementation of the reform agenda . 5. Efficiency: Efficiency is rated modest . The PAD does not estimate any ex ante rate of return, since the sub -projects were demand-driven. The Results Framework in the ICR Data Sheet reports an ex post average economic internal rate of return (EIRR) of 24.6%. This estimate is from a 2009 study covering 30% of sub-projects, which implies a sample of 79 sub-projects out of a total of 262. This sample size does not correspond with that reported for apparently the same study in Annex 3 of the ICR (a targeted sample of 30 sub-projects or 11.5% of the 262 sub-project total). Projects in four categories are included in the latter analysis -roads, bridges, clean water supply, and irrigation . Average EIRRs ranged from 20% to 31% in each category. The overall average EIRR weighted by the number of sub -projects sampled in each category is estimated by IEG to be 23.3%, the figure shown below. While all sub-projects amount to 76%% of actual project cost, the ICR does not provide data on the costs of the sample of sub -projects used to estimate the EIRRs; therefore, the proportion of total project cost covered by the sample is unknown . There are a number of indications of operational and administrative inefficiency . The ICR notes that "the project overall fell significantly short of the level of efficiency targeted in the PAD " (p.13). The PAD identifies the "core group" of districts as being "at least 40" but recognizes that the project will "dedicate substantial technical and training support to ensure that [districts] fulfill these requirements" (PAD, p.22). However, these targets were reduced at project restructuring to 14 districts. Although the number of districts covered fell by about 65%, the cost of Component C, dedicated to support the implementation of Components A and B, was still some 80% of the amount planned at appraisal. Moreover, implementation took place over a period of five years, rather than the initially estimated three years. Overall, “the project ended up spending almost two -thirds as much as originally intended for implementation support, for work in only one -third the planned number of districts .� The ICR attributes these results to a combination of overly ambitious project design and weak implementation prior to restructuring (p. 13). The fact that the final Governance and Decentralization Survey was not undertaken reduced the ability to assess efficiency . Delays leading to the two-year project extension were caused by (i) significant procedural demands, particularly on district staff involved in the implementation of sub -projects; (ii) constantly changing laws and regulations pertaining to decentralization; and (iii) limited capacity in key central agencies (Ministry of Home Affairs and National Project Secretariat) leading to payment, contract and reimbursement delays (ICR, pp.7-8). Implementation improved during the post-restructuring period thanks to the increased capacity of some districts and the greater availability of consultants. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate Yes 23.3% 0% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: While the relevance of the project objective is rated high, that of project design, and of the efficacy of all five objectives is modest. The formulation of outcomes intended to contribute to the sub -objectives and their associated achievements, targets and baselines were vague and difficult to measure . This made it impossible in a majority of instances to assess the extent to which progress was made, which is a particularly serious drawback for a pilot project. Efficiency is rated modest. Although economic rates of return were estimated to range from 20% to 30%, there were considerable operational and administrative inefficiencies . The project spent almost two-thirds as much as originally planned in only one-third of the number of districts projected at appraisal . Overall outcome is assessed as Moderately Unsatisfactory . a. Outcome Rating : Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating: The overall risk to development outcome is rated as significant. Political At closure, participating districts were considered to be at “real risk� of backsliding on governance reforms . (ICR, p.14), although the project team subsequently explained that the four districts that were considered to have performed well in implementing governance reforms were not considered at risk . The extent of this risk is demonstrated by the experience with non -performing districts before restructuring, which showed "how difficult implementing and sustaining governance reforms can be " (ICR, p.14). Among the 14 districts included in the project after restructuring, the ICR identifies five non -performing districts and five medium districts along with the four well-performing districts (p.30). Implementation Procurement and financial management risks at the district level were identified as high in the PAD and have remained significant during project implementation . Two specific risks continued at project closure : (i) inefficient and delayed procurement and mobilization of project staff and (ii) lack of timely delivery of capacity building activities . The ICR (p.6) downgrades the latter risk to moderate but indicates that risk mitigation measures regarding the project procurement plan prepared prior to negotiations and the recruitment of regional consultants were insufficient, thereby implying that the risk continues . Furthermore, if the Bank’s objective is to replicate implementation of local governance reforms pursuing the model represented by this project, with its high costs and implementation problems, a serious question of the feasibility of scaling up is raised, even for a modest portion of the 440 districts in Indonesia. This category of risks is also assessed as significant. Fiscal The above-referenced report by the Borrower that districts apparently stopped undertaking procurement surveys, publishing procurement sanctions and contracted unit prices owing to lack of funds implies a potentially significant fiscal risk (section 4(5). Operations and Maintenance Although the PAD estimated that districts would be able to meet longer term operations and maintenance expenditures for the sub-project infrastructure without causing deterioration in their finances, there is no indication in the ICR as to whether adequate provision has been made . From another perspective, it could be argued that the pilot project showed the infeasibility of scaling up . In this sense, the project contributed to the avoidance of further risks. a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: a. Quality at entry: Project preparation took place over a three -year period, involving several background studies and consultations with development partners and potential participating districts, in collaboration with the National Development Planning Agency. While the Bank's preparation efforts manifested thoroughness and due diligence, including consideration of potential risks, it underestimated the Government's capacity limitations, in particular at the level of the implementing agency, the Ministry of Home Affairs . This underestimation led to subsequent problems and delays during project implementation . The project design was overly ambitious and complex . Some governance considerations were not taken into account in project design (see Section 3b above). The extent of support from the Ministry of Home Affairs and National Planning Secretariat envisaged in design was very limited during the first two years of the project In view of these circumstances, the Bank was overly ambitious in specifying a wide -ranging list of governance reforms to be supported by the project along with multiple sub -project investments, each entailing a relatively rigorous set of approval and implementation requirements . The project’s Local Governance Reform Framework consisted of an 11-page list of over three dozen reform requirements (about 70 if listed in detail) for participating districts in the areas of transparency and participation, poverty reduction plans, financial management, and procurement to be implemented in phases . Six of these requirements were subsequently dropped during project restructuring because some were seen as too difficult for districts to achieve and some were no longer valid or relevant. Several key assumptions made at the design stage did not hold : Α new Government on-granting mechanism would become effective during implementation that would allow for scaling up of project reforms to additional districts beyond the original 15, but at the time of project closure, the on-granting mechanism had not become effective (ICR, p.6) The second batch of 25 districts (ICR, p.6) would undertake specified reforms without the incentive of sub-project investment funds A “snowball� effect would emerge, with innovation among reform -minded districts leading to broad dissemination of best practices and ultimately to scaling -up of the project’s reforms nationally. At the time of project closure, scaling-up had not occurred and all plans to expand project activities beyond the 14 districts were dropped. M&E design contained important weaknesses (see Section 10 below), an especially important shortcoming for a pilot operation. at-Entry Rating : Quality -at- Moderately Unsatisfactory b. Quality of supervision: Bank supervision covered the 14 districts where the project was implemented . Missions were undertaken periodically, in some cases twice a year . These included field visits to participating districts where all aspects of project implementation, including governance reforms and execution of sub -project investments, were reviewed . The supervision effort was to a large extent organized and led by the Bank ’s office in Jakarta, allowing for longer mission durations that covered the various project aspects . Implementation issues were identified and documented in aide-memoires that contained lists of items requiring time -bound follow-up actions. The magnitude and complexity of these requirements were in part recognized at project restructuring in April 2009, when the number of participating districts was reduced from 40 to 14 and six reform requirements were dropped – some because they were superseded by new laws, regulations or institutions and others because they were recognized as still beyond existing district capacities (ICR, Annex 2, Table 5). Contributing to implementation difficulties were delays in the engagement of consultants to provide planned facilitation and support, especially to districts, and failure to bring ongoing M&E findings to bear on project execution. Bank Implementation Status Reports (ISRs) flagged these issues by rating implementation progress as Moderately Unsatisfactory over a 3-year period, from 2006 to 2009. The responsibility for taking action in these areas was primarily that of the Borrower, but the Bank bears some responsibility in its supervision role . In the event, no action was taken until the mid -term review when the decision to restructure was taken . This decision and the subsequent actions --in particular, reducing the number of targeted participating districts and reducing the number of required local governance reforms -- were appropriate but tardy responses to the earlier implementation problems. That restructuring was not undertaken earlier was partly a consequence of the delay in the mid-term review, which took place at the end of April 2009, instead of at the end of May 2007 as originally scheduled. Given the importance to both parties of procurement reforms the Bank could have taken a sample post mid -term assessment of adherence to procurement standards in the 10 reportedly well-performing districts. The Bank was insufficiently proactive in improving the design and effectiveness of the M&E system, particularly in view of the pilot nature of the project . In addition, the final Governance and Decentralization Survey, which would have been particularly important for this pilot project, did not take place . Quality of Supervision Rating : Moderately Unsatisfactory Overall Bank Performance Rating : Moderately Unsatisfactory 9. Assessment of Borrower Performance: a. Government Performance: The Government's commitment to decentralization was manifested in the continued evolution and refinement of the legal and regulatory framework . It was also shown by the active engagement by the National Development Planning Agency during project preparation . But evidence of broad-based and high level support for the project ’s objectives across central government agencies, particularly during project implementation, is mixed . The project’s National Steering Committee (comprised of high level representatives from pertinent ministries and agencies ) never met throughout the years of project preparation and implementation . Although meetings and coordination took place at the working level, the guidance and support of a high level committee could have contributed to more effective project implementation . The fact that legislation and provisions of on -granting of funds was not implemented was a significant shortcoming that has undermined the possibilities for scaling up this pilot project . During project preparation and for most of the period of project implementation, the requisite national legislation and framework for on -granting to sub-national governments were not operational, nor were they at project closure (ICR, p.18) The performance of participating district governments was also mixed . Districts undertook successive phases of reform as required by the Local Governance Reform Framework, at times even without the technical assistance and facilitation that had been promised by the project due to delays in the recruitment of consultants . In addition, they completed 262 sub-project investments, which exceeded initial expectations . But there also were a number of non-performing districts that ended up being excluded from continued involvement in the project . Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance: The Ministry of Home Affairs, and the National Project Secretariat, which came under the Ministry's purview, were the implementing agencies. They generally performed unsatisfactorily up to the time of project restructuring . The difficulties began with delays in fulfilling effectiveness conditions, which were then compounded by subsequent delays in project implementation (ICR, p.16). These difficulties impacted on overall project and financial management, including, in particular, procurement of national management consultants, regional management consultants, and district -level facilitators. The project design anticipated having 110 project staff supporting project implementation, with half of these based at the district level . However, the issues that emerged with regard to putting these staff in place meant that districts did not receive the intensive support that had been planned (an original target of 55 project staff at the district level was reduced to 20 at restructuring). Causes were partly systemic (for example, when funding for sub -projects could not be provided through the on -granting mechanism but had to go through traditional central government mechanisms ), and partly administrative due to lack of capacity at the Ministry of Home Affairs . The Ministry did not appoint "work units" in the National Project Secretariat in a timely manner, including the procurement and financial management teams of the Secretariat . Another issue concerned limited capacity for consultant procurement at the Ministry, resulting in slow and unsatisfactory selection of consultants, including the national management consultants who were intended to support the Secretariat. Further issues involved financial management consultants and facilitators who experienced delays in receiving payments, contract renewals, and reimbursements for travel to the districts, with no provision for cash advances . The final Governance and Decentralization Survey, which would have been particularly important for this pilot project, did not take place . After the 2009 restructuring, the National Project Secretariat, with the support of the Ministry of Home Affairs, became more proactive in managing the project and in addressing implementation issues . Implementation improved between mid-2010 and project closure. Implementing Agency Performance Rating : Moderately Unsatisfactory Overall Borrower Performance Rating : Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The PAD envisioned that the M&E system would provide continuous learning and feedback from a combination of internal monitoring, external monitoring and independent evaluation of the pilots supported by the project . For internal monitoring, a management information system would enable data to be collected and entered at district level, stored in a master database, and be accessible at national and district levels . Monthly reports from districts and key project consultants would be submitted to the National Project Secretariat, and district -level facilitators would report on stakeholder feedback to be analyzed from one year to the next . For external monitoring and independent evaluation, a project M&E secretariat at the National Development Planning Agency was to manage annual reviews of poverty strategies and budget planning, an economic evaluation of sub -project investments, assessments of procurement and financial reforms, qualitative studies of governance reforms, and mid -term and final project evaluations. The ICR Results Framework Analysis shows seven Project Development Objective (PDO) Indicators. Most of these are appropriately at the outcome level, except for two that are at the input or output levels : number of district budgets with increases in poverty targeted expenditures, and the economic rate of return for project -funded infrastructure. However, the PDO indicators have vague targets or baselines and one has no target . The Analysis also shows ten Intermediate Outcome Indicators. Seven of these are at the intermediate outcome level and three are at the input level. None of the indicators is at the output level . In some cases, indicators are not adequately distinguished from targets, and in others, no clear targets are set (for example, "Greater public availability of information; " Annex 1, PAD). Some indicators are phrased in a way that conflates the indicator with the target (for example, PDO Indicator 3: “At least 10 district budgets show increases in poverty targeted expenditures …�). The metrics for several indicators and targets are not apparent : for example, PDO indicator 1 ("extent to which recommendations from consultations /public hearings are incorporated ..." against "a baseline of limited public consultation in policy formulation; " and intermediate indicators 7 ("receive capacity development assistance"), 9 ("project M&E system in place and providing accurate timely data "), and 10 ("complaints received/resolved satisfactorily)." b. M&E Implementation: M&E implementation fell short of what design intended . This can be attributed principally to the project management and administration difficulties described in Sections 8 and 9 above. The M&E specialist was only recruited in late 2008, the complaints handling specialist was terminated due to poor performance and then not replaced, and the management information system specialist position was never filled . These positions were subsequently eliminated as part of project restructuring by mutual agreement between the Government and the Bank following the mid-term review. The management information system was, consequently, never fully operationalized, while regular and systematic submission and analysis of monthly reports from the districts were not achieved owing to inadequate incentives for reporting or sanctions for lack of compliance . External M&E through the National Development Planning Agency, which had a separate management structure and budget allocation for these activities, was deemed to have been generally comprehensive and were credited with producing external evaluations, including mid-term and final, that were judged to be rich in information and detail (ICR, p.8). Rather than supporting the management and implementation of the project, the sophisticated and layered M&E design placed an additional burden on project execution . The final evaluation prepared by the National Development Planning Agency argued that separating the internal and external monitoring functions rendered M&E less effective . Coordination and follow-up between the National Project Secretariat and the M&E secretariat at the National Development Planning Agency was weak throughout the project . Despite the perceived good quality of the external M&E reports, their findings on implementation problems in the field did not spur specific meetings or actions to resolve problems and so had little influence on subsequent project implementation . A particular M&E shortcoming that limited the ability to assess project outcomes was the fact that the final Governance and Decentralization Survey did not take place. This resulted from the fact that following restructuring, adjustment of the project budget was not done promptly enough to enable hiring of a contractor in time . The lack of these survey findings, which would have included a survey of governance reforms in nonparticipating districts, precluded an estimation of attribution and a consequently more robust assessment of project outcomes . M&E implementation fell short of what design intended . This can be attributed principally to the project management and administration difficulties described in Sections 8 and 9 above. The M&E specialist was only recruited in late 2008, the complaints handling specialist was terminated due to poor performance and then not replaced, and the management information system specialist position was never filled . These positions were subsequently eliminated as part of project restructuring by mutual agreement between the Government and the Bank following the mid -term review. The management information system was, consequently, never fully operationalized, while regular and systematic submission and analysis of monthly reports from the districts were not achieved owing to inadequate incentives for reporting or sanctions for lack of compliance . External M&E through the National Development Planning Agency, which had a separate management structure and budget allocation for these activities, was deemed to have been generally comprehensive and were credited with producing external evaluations, including mid-term and final, that were judged to be rich in information and detail (ICR, p.8). c. M&E Utilization: The shortcomings in design and implementation (including staffing) of M&E compromised its value for utilization purposes. M&E Quality Rating : Modest 11. Other Issues a. Safeguards: Three safeguard policies were triggered during project preparation : Environmental Assessment as a Category “B� project (OP 4.01); Indigenous Peoples (OP 4.10), and Involuntary Resettlement (OP 4.12). The PAD reports that Safeguards Frameworks were prepared on all three policies, and were incorporated in the Project Operational Manual, which also offered guidance on implementation . Safeguards monitoring during project implementation focused mainly on the implementation of the sub -project investments in participating districts . The ICR reports that safeguards reviews were undertaken periodically throughout project supervision and found that sub -project design and the framework for safeguards were sound, and that there were no serious negative impacts from the investments. Minor safeguards issues identified during field visits to sub -project sites (for example, erosion-prone slopes at a roads sub-project) were documented in aide-memoires and followed up by both the Government National Project Secretariat and Bank teams . However, neither the ICR nor the project team provided a clear statement that any of the three Safeguards policies had been complied with . The ICR (p. 9) notes that the project offered the potential opportunity to develop capacity at the district level on Safeguards through the Local Governance Reform Framework but that the weaknesses in overall project implementation resulted in this opportunity not being realized . b. Fiduciary Compliance: Project financial management performance was rated moderately unsatisfactory in supervision reports prior to project restructuring, but subsequently was found to be generally satisfactory . The issues were mainly related to difficulties and delays in Government budgeting and appointment of work units, and in payments for consultants . Quarterly financial monitoring reports were often late, due to delays in receiving underlying reports from participating districts . However, by project closure, the ICR reports that the FY 2007, 2008, 2009, and 2010 audit reports had all been received on time, with the auditors providing unqualified opinions on the financial statements, although it was noted that some follow-up actions on findings from previous years were still outstanding (these were mainly related to improvements to internal controls for project management ). Procurement performance of the project had been rated moderately unsatisfactory in supervision reports for most of the period prior to project restructuring . This reflected the project management and consultancy recruitment issues described above. Four formal complaints on sub-project procurement were received and, according to the ICR, resolved. There were no reported cases of misprocurement . c. Unintended Impacts (positive or negative): None d. Other: 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately The relevance of the project objective Satisfactory Unsatisfactory remains high but the relevance of design is modest. The efficacy of achievement of each of the 5 sub-objectives is rated modest due mainly to the unclear formulation of indicators and achievements linked to the sub-objectives or because of lack of evidence. Owing to several administrative and operational inefficiencies, efficiency is rated modest. Outcome is therefore rated moderately unsatisfactory. Risk to Development Moderate Significant While the 5 well-performing districts are Outcome : reportedly at low risk, the other 9 districts present moderate to significant reform, financial and procurement risks . Bank Performance : Moderately Moderately Bank performance manifested Satisfactory Unsatisfactory significant shortcomings. Quality at Entry reflected overestimation of central and local government capacity and an ambitious and complicated design. At the same time, M&E design, essential for a pilot project, had significant shortcomings. More proactive Bank supervision might have avoided or reduced the damaging delays in engaging consultants . Supervision did not succeed in significantly improving the M&E framework or in bringing M&E findings to bear on project execution . Borrower Performance : Moderately Moderately Both government performance and Satisfactory Unsatisfactory implementing agency performance are rated moderately unsatisfactory due to multiple delays and lack of guidance and support. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR presents seven lessons learned (pp.17-18). Several of these are Indonesia -specific and of limited applicability elsewhere. The remaining lessons are of broader applicability, and are summarized as follows : 1. A positive enabling external environment is necessary for governance reforms at the local level . A necessary but not sufficient condition for successful governance reforms at local levels is a positive enabling external environment at high central government levels within which local government units can then undertake reforms . Among key requisites are (i) an adequate and stable legal and regulatory framework that specifies the roles and responsibilities of local government units and the requisite resources to discharge their functions; (ii) timely availability of quality assistance for capacity building and technical assistance to local units; and (iii) specific incentives to undertake reforms, such as sub -project investment funds provided when local governments undertake specified reforms. In this case, the first and second requisites did not begin to be satisfactorily fulfilled until after project restructuring . 2. Successful local governance reforms require strong local government leadership committed to reforms, supported by champions in the national and local bureaucracy and civil society . The head of the local government unit must not only be supportive of reforms, but set the tone and champion them . There should be advocates for change within the local bureaucracy . A dynamic local civil society, ranging from community groups, to broader civil society groups or associations, and pro -active free media organizations, should be supported by the head of local government. Civil society constitutes a source of demand for governance reforms at local as well as national levels. According to the project team, the best -performing districts in this project had these characteristics. 3. Realistic project scope and goals increase the likelihood of project success, especially in situations where implementing entities have weak capacity . While this is a sound lesson for any project, it is particularly applicable to a project like this one, where the units of local government had weak capacities to begin with . Something along the lines of a reform framework is essential, but it should not be overly ambitious . Similarly, the number of target local governments should be realistic at the outset . 4. Separating responsibility for M&E from the implementing agency can reduce the likelihood of M&E uptake and utilization . While the rationale for separating the M&E function from the implementing agency is usually to increase the independence and objectivity of M&E findings, the effect of isolating the function can also reduce the uptake of M&E results into project decision -making, an especially negative outcome for a pilot project such as this one. An alternative design would have been to include regular M&E activities within the purview of the main executing agency, reserving only a few key reports such as the mid -term and final evaluations for a separate, independent agency. The implications for scaling up a pilot project of a failure to implement the legislation and provisions of on-granting of funds for a decentralized pilot project should not be underestimated . 14. Assessment Recommended? Yes No Why? To verify the ratings, and also to assess the approaches to improvements to local governance employed by the project, including the use of community -driven development sub-projects. The assessment could focus, inter alia, on the definition and measurability of objectives and the realism of the time frame . 15. Comments on Quality of ICR: The ICR follows the guidelines and is generally consistent and results -oriented, although the evidence for several elements presented in the Results Framework Analysis suffers from a lack of clarity and specificity . The main text of the ICR is reasonably comprehensive, and the narrative for the most part is clearly written, but at times lacking conciseness. The lessons drawn are generally sound and evidence -based. However, there are some shortcomings . First, there is a dearth of quantified indicators that would permit objective measurement . The ICR sometimes confuses indicators with targets . Second, while the ICR proves a brief impressionistic indication that there were “no serious negative� Safeguards impacts of sub-projects, there is no statement of compliance with any of the three Safeguards that were triggered during project preparation . Third, there is lack of clarity regarding the auditor ’s opinions of the quality of the districts ’ financial management. Fourth, while the ICR recognizes that the economic rate of return is a measure of efficiency, it also treats it as a measure of efficacy in the Data Sheet ’s Results Framework. This is confusing and can lead to erroneous conclusions . Finally, there are three inconsistencies : (i) citing the PAD, the ICR reports the original number of sub -projects as 93 on page.12 and as 96 on page 22, but the PAD projects the number as 72 (Annex 1, p.38); (ii) the ICR suggests that at project entry the downside risks were not fully appreciated, but nonetheless concludes that "the Bank could reasonably have expected implementation to have been successful;" and (iii) the sample size of the sub-projects selected for cost -benefit analysis differs between the Data Sheet and Annex 3. a.Quality of ICR Rating : Satisfactory