ICRR 13292 Report Number : ICRR13292 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 03/29/2010 PROJ ID : P054238 Appraisal Actual Project Name : Second Gazelle US$M ): Project Costs (US$M): 39.5 65.4 Restoration Project Country : Papua New Guinea Loan/ Loan /Credit (US$M ): US$M): 25.3 25.3 Sector Board : TR US$M ): Cofinancing (US$M): 3.2 23.8 Sector (s): General transportation sector (29%) General water sanitation and flood protection sector (21%) Power (17%) Other social services (17%) General public administration sector (16%) Theme (s): Other social development (29% - P) Access to urban services and housing (29% - P) Infrastructure services for private sector development (28% - P) Natural disaster management (14% - S) L/C Number : L4525 Board Approval Date : 12/09/1999 Partners involved : Australian AID, Closing Date : 06/30/2003 12/31/2007 European Commission, Papua New Guinea (PNG)Sustainable Development Fund, PNG Harbor Fund, PNG Cocoa and Coconut Institute, Rabaul Volcanological Disaster Relief Committee. Evaluator : Panel Reviewer : Group Manager : Group : Robert Mark Lacey Kris Hallberg IEGSE ICR Reviews IEGSE 2. Project Objectives and Components: a. Objectives: The project was initially designed as the first in a programmed series of two loans (APL 1 and APL 2). According to the PAD for APL 1, the program objectives to be supported by the two loans were "to restore the economic and social well-being of the East New Britain people that prevailed prior to the 1994 volcanic eruption, which devastated much of the then provincial capital (Rabaul Town) and surrounding villages. Furthermore, the program aims to ensure sustainability of outcomes and minimize adverse impacts of future volcanic eruptions on lives, property and livelihood." These are also the objectives of the Medium Term Restoration Program (MTRP) of the East New Britain Provincial Administration (ENBPA) -- the Gazelle Peninsula being located in the Province of East New Britain -- and the Government of Papua New Guinea (PNG). Within this program, the PAD (Annex 1) states three project development objectives (PDOs) for APL 1: "(a) to upgrade existing facilities and transportation infrastructure to support a shift in the location of settlement and economic activity towards lower risk areas centered on the new provincial capital and Kokopo; (b) to consolidate relocation sites in safe zones for evacuation of displaced communities from devastated and high risk areas; and (c) restore such infrastructure and services in Rabaul as are needed primarily to ensure the effective operation of the port as a major economic facility for the Gazelle Peninsula and the Eastern Region of PNG ." The statement of project objectives in the Loan Agreement (LA) is as follows:"to (i) provide the people of the Gazelle Peninsula with government services similar to those that were destroyed by the volcanic eruption in 1994; and (ii) ensure sustainability of social and economic well -being and minimize adverse impacts of future volcanic eruptions on lives, property and livelihood in the Gazelle Peninsula ." This ICR review uses the program objectives as stated in the PAD (the Loan Agreement does not mention program objectives), and also the project objective for APL 1 from the PAD since it is more specific and linked to APL 1. b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): The MTRP was divided into two phases -- Phase I: stabilization, including completion of emergency restoration works, relocation of displaced communities, and provision of services in and around the newly designated provincial capital; and Phase II: consolidation and normalization of the long term development process . APL 1 -- the project -- was to support Phase I and APL 2 Phase II. There were five components: A. Improving the quality of life for displaced communities (US$2.4 million at appraisal, US$6.9 million at closure) including resettlement and rehabilitation, provision of relocation sites, agricultural land, infrastructure services, and economic and community support services (for example, agricultural extension and vocational training ). B. Upgrading of the town of Kokopo as the new provincial capital (US$14.9 million at appraisal, US$31.2 at closure), including urban infrastructure and utilities, solid waste disposal, provision of back -up generating equipment, construction of educational and community facilities, an administrative center and a commercial center . C. Restoration of Rabaul as a regional port ((US$3.5 million at appraisal, US$8.6 million at closure), including rehabilitation of port-related infrastructure and essential utilities, urban infrastructure and public services . D. Development of support infrastructure and social services for the Northeast Gazelle Peninsula (US$5.4 million at appraisal, US$12.1 million at closure), including establishment of a less concentrated human settlement pattern through development of provincial townships in the Northeast area of the Peninsula . E. Economic diversification and long term development planning (US$1.6 million at appraisal, US$0.2 million at closure), including development of institutional capacity in land use, water resources and waste -water management, as well as strengthening the administration of the ENBPA especially with regard to planning and development . The project was implemented by the Gazelle Restoration Authority (GRA), which was specially created to respond to the emergency and is currently being integrated into the provincial administration . GRA was also the implementing agency for the Immediate Term Restoration Program (ITRP), supported by the Bank with the First Gazelle Restoration Project. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: There were three extensions to the closing date and four formal restructurings (but PDOs or key associated outcome targets did not change ). These were rendered necessary by lower than anticipated counterpart financing from the PNG Authorities, delays and changes in the available cofinancing from partners other than the Bank, design upgrading of some sub-components, the bringing forward of some Phase II investments, changes in the sequencing of investments, higher than expected bid -prices, and inflation during implementation . Overall project costs at closure were 66% higher than the appraisal estimate . In the case of the EU-financed components, there was a particularly damaging link between the lack of counterpart funding and the availability of foreign financing . The EU financed only construction -- design was meant to be paid for by the PNG counterparts . When the designs failed to materialize, the construction was unable to proceed before the grant's time period ran out . Other co-financiers, notably the Australian aid agency AusAID, stepped into the breach, while the EU eventually came forward with new money . The Bank increased its share of financing for civil works from an average to 78% to 90%. Despite these and other measures, lack of counterpart funding remained a constraint throughout implementation, though the situation did improve in the later stages as higher commodity prices increased fiscal resource availability . 3. Relevance of Objectives & Design: The project's objectives were highly relevant. Direct and timely support was provided to the implementation of the MTRP of the ENBPA. The PDOs were also fully consistent with the priorities of the Bank's assistance strategy for PNG, key elements of which are enhancing service provision capacity and strengthening the resilience and self-protection of local communities in areas threatened by natural disasters . The relevance of project design was modest t. Extensive consultations were undertaken with local communities and with a wide range of other stakeholders to ensure that the strategy and measures proposed were both acceptable as solutions and relevant to present and future needs . Area and regional development plans were devised in such a way as to present national and local policy makers with clear alternatives and with the consequences of each alternative fully delineated; they also charted a path towards achievement of the individual goals . Key risks were appropriately identified, the first of which was the lack of adequate and timely counterpart funding . However, given the high risk that this problem would indeed materialize plus the involvement of several donor agencies, each with its own procedures, the implementation timetable was over -optimistic.The Critical Risk table in the PAD suggests the establishment of an "Executing Agency Trust Fund " to mitigate the counterpart funding risk, but there is no further discussion of this in the text of the PAD and the idea is not mentioned in the ICR . Design might also have mitigated the risk of delays arising from multi -donor involvement by incorporating the channeling of other donor financing through the Bank. No quantitative targets were developed to measure attainment of the program objectives of restoring economic and social well -being, sustaining outcomes, and minimizing adverse impacts of future volcanic eruptions on lives, properties and livelihoods . Overall relevance of objectives and design is rated as substantial . 4. Achievement of Objectives (Efficacy): Upgrading existing facilities and transportation infrastructure to support a shift in the location of settlement and economic activity towards lower risk areas centered on the new provincial capital and Kokopo . Highly achieved . The electric power grid has been extended to relocation sites and piped water supply provided to 10,000 people. Health posts have been upgraded to full health centers and a rural hospital constructed . Community resource centers have been built and access roads rehabilitated to facilitate food and cash crop marketing in relocation areas . The target of 80% occupancy by displaced communities at relocation sites has been attained indicating more settled communities and improved livelihood. Consolidation of relocation sites in safe zones for evacuation of displaced communities from devastated and high risk areas. Highly achieved . The MTRP Phase I target of 75% completion of other infrastructure, utilities and social services, focusing especially on Kokopo as the new provincial capital was fully met . This included town roads and other local transportation infrastructure, as well as the reconstruction of the Rabaul -Kokopo road. The project leveraged additional substantial foreign and domestic financing for these works . In addition, the first phase of the new Kokopo market was completed, and a new community school for 880 pupils was constructed as well as a new provincial administrative center . Overall, Kokopo has been transformed from a small village of about 1,100 people prior to the eruption into a township with a population of over 15,000. Apart from Kokopo itself, the project supported a number of initiatives to establish a more dispersed pattern of settlements in the Gazelle Peninsula including upgrading of roads, provision of water supply, improving power supply infrastructure and constructing /rehabilitating a wide range of community services . A spatial and economic strategy framework and longer -term development institutional plan have been formulated, covering, inter alia, coastal zone management and mitigation of coastal hazards in East New Britain. Detailed arrangements for the integration of the GRA into the provincial administration have been prepared, though they are still awaiting approval . Restoring such infrastructure and services in Rabaul as are needed primarily to ensure the effective operation of the port as a major economic facility for the Gazelle Peninsula and the Eastern Region of PNG . Highly achieved . The port-related protection works in Rabaul (port roads, erosion control, silt traps and dredging ) have been fully completed, and a zoning plan has been prepared and made into law . The port itself has been rehabilitated and a management contract prepared for outsourcing its operation and maintenance . The port is now functioning at pre-eruption levels. To serve the small remaining population of Rabaul (the town plan prohibits resettlement in vacated areas), the water supply system has been rehabilitated and a market constructed . Not only were the targets for APL 1 achieved, but some of those originally designated for Phase II of the MTRP, to be supported by APL 2, were also reached. Partly because of the success of APL 1, the PNG Authorities decided not to proceed with APL2, but rather to direct Bank support to what had become higher development priorities including restoration in other areas affected by natural disasters . Although the overall program goal of restoring the social and economic well-being of the people of East New Britain is not backed with any quantitative performance indicators, it is reasonable to assume, given the attainment (and in some cases surpassing ) of the project's output targets, that the project would contribute to the program objectives . Similarly, the actions to consolidate resettlement in safer areas -- including restricted activities at Rabaul, prohibition of resettlement in vulnerable areas, establishment of a Village Development Fund to finance community -based infrastructure and services, community development, and the promotion of economic activities, and preparation of a comprehensive development plan for the Province -- should contribute to the program objectives of ensuring sustainability of outcomes and minimizing the negative impact of future eruptions. 5. Efficiency (not applicable to DPLs): No economic or financial rates of return were calculated at appraisal, although the benefits at least in some cases are measurable. They include provision or improvement of infrastructure services to populations resettled in areas less vulnerable to volcanic eruption, reasonable access to increased livelihood opportunities, ensured continued operation of Rabaul port (a vital economic asset for the Gazelle Peninsula ) and an important shift towards a sounder framework for planning development in East New Britain . No attempt is made in the ICR to measure at least some of these benefits or to compare the project's cost effectiveness with other, similar post -disaster operations. The importance of the benefits must be set against the major delays suffered by the project as a result of differing donor procedures and the need to finance the gap in counterpart funding . Taking account of these delays, and of the substantial increase in project costs during the prolonged implementation period, IEG rates project efficiency as modest . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re -estimated value at evaluation : re- Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: With weaknesses in the relevance of design and efficiency, project outcome is rated moderately satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: There are two principal risks to the project's development outcome : (a) pressure to redevelop Rabaul township rather than relocate populations to less vulnerable areas (there is a vocal minority in favor of this ); and (b) under-funded maintenance of the new and already existing infrastructure and equipment . Of these, the first presents the lesser risk. The ENBPA continues to demonstrate strong commitment to PNG government policy prohibiting redevelopment in devastated areas through ordering amendments to the Rabaul town plan which envisaged such redevelopment. With Kokopo designated as the new provincial capital, and the departure from Rabaul of the post office, government administration, the main market and several key commercial enterprises, the probability of significant population return to Rabaul is diminishing over time . Problems with counterpart funding during project implementation underline the risk of under -funded maintenance, operations and supplementary investment . The Authorities are aware of the issue -- arrangements are being made to integrate maintenance needs into budgets . Moreover, some entities -- notably the national Port Authority and the Water Board -- have good maintenance records . Nonetheless, much infrastructure maintenance, especially for roads, has been traditionally inadequate in PNG . An important decline in the ENBPA revenue base following replacement of the provincial sales tax by a national value added tax, is also a cause for concern . Under-funding of maintenance is a significant risk especially when the international prices of PNG's commodity exports fall . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: Bank performance for ensuring quality at entry was moderately satisfactory . Appropriate steps were taken in identifying priorities for investment . Relocation and resettlement plans were carefully prepared . Critical risks were analyzed and corresponding mitigating measures proposed . Tools, such as the Village Development Fund, were devised which added to flexibility and the team's ability to respond to changing circumstances during implementation. The major design shortcoming was an over -optimistic assessment of the time needed to complete Phase I of the MTRP and, partly in consequence, an underestimation of project costs . M&E was also deficient as no measures of restored economic and social well -being, sustainability of outcomes, and the minimization of adverse impacts of future volcanic eruptions on lives, property and livelihood were included . This was a challenging project to supervise, and the Bank input -- in terms of the technical strength of the supervising team, staff continuity, and frequency of missions (regular, twice-yearly supervision missions were supplemented by visits from individual experts and audio -conferences from Sydney) -- was correspondingly intense. Implementation Status Reports were candid and the ratings given appropriate . The team displayed considerable flexibility in adapting to changing circumstances and to the challenges of implementing such a complex operation in a fragile state . Measures were quickly taken to lessen the adverse impact of government funding problems, and additional donor funds were mobilized as needed . The one noteworthy shortcoming -- which is an issue as much of general country policy dialogue as of project supervision -- was that the problem of lack of counterpart funding was not fully resolved . Overall, supervision is rated as satisfactory . at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: On the positive side, Government (by which is meant both the national Government of PNG and the ENBPA ) has shown strong commitment to project goals and has, for instance, taken a firm stand against the considerable political pressure for the redevelopment of Rabaul . The ENBPA has worked well with the GRA and is taking active steps to incorporate the latter into the provincial administrative structure . The main shortcoming of government performance has been inadequate counterpart funding, despite restructuring and extension of the implementation time period. This led many stakeholders to consider the project to be at serious risk -- by the original closure date of June 30 2003, only about 20% of project funds had been disbursed . Only when international commodity prices rose in the latter years of the project, did the counterpart constraint ease, and even then not fully. Government performance is rated as moderately satisfactory . The performance of the GRA as implementing agency was highly satisfactory . Having already secured the confidence of stakeholders and the population of the Gazelle Peninsula following its effective intervention in the immediate aftermath of the eruption, GRA showed itself to be a highly effective implementing agency, able to tackle the severe complications arising from the lack of government counterpart funds and multi -donor financing. It was effective in causing the necessary analytical work to be carried out (including environmental aspects ) during preparation and in wide ranging consultations to establish project priorities . Its management of the project was fully adequate -- the appropriate expertise was swiftly recruited and a project financial and accounting system installed. GRA was equally adept in adopting Bank procurement requirements and adjusting them to its requirements. On-site supervision was constantly vigilant and pro -active. a. Government Performance :Moderately Satisfactory b. Implementing Agency Performance :Highly Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: Design: Four key performance indicators (KPI) were developed in the PAD with which to monitor progress towards the completion of Phase I. These were the triggers for the then proposed APL 2 for which KPIs were also elaborated . The Phase I KPIs were (a) completion of infrastructure services at all relocation sites with at least an 80% occupancy rate; (b) 100% completion of the Rabaul port protection works and preparation of a zoning plan prohibiting redevelopment in the devastated areas of Rabaul; (c) 75% completion of other infrastructure, utilities and services; and (d) approval by ENBPA of detailed arrangements for the integration of the GRA into the provincial administration . These indicators were appropriate for monitoring the implementation of the first Phase of the MTRP . On the other hand, no performance indicators were defined to measure the achievements of the broader program objectives -- restored economic and social well -being, sustainability of outcomes, and the minimization of adverse impacts of future volcanic eruptions on lives, property and livelihood . IEG rates M&E design as modest . Implementation. The KPIs were regularly monitored and used for measuring progress towards attaining Phase 1 targets. The fact that the PAD also developed KPIs for Phase 2 was of great help both to the GRA and to Bank supervision missions, since progress was also made during implementation in reaching a number of Phase 2 targets. Although an impact evaluation was ongoing, no beneficiary surveys were conducted . Restored economic and social well-being, sustainability of outcomes, and the minimization of adverse impacts of future volcanic eruptions on lives, property and livelihood were not measured . Implementation of M&E is rated as modest . Utilization.The KPIs developed during preparation continue to be used by GRA and the PNG Authorities for monitoring continued progress in implementing Phase 2 of the MTRP following project closure . IEG rates utilization as substantial . a. M&E Quality Rating : Modest 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Environment. For purposes of O.D. 4.01, the project was classified as Category B with the main environmental focus being on the possible adverse effects of construction and resettlement activity . Several components therefore required as disbursement conditions environmental mitigation measures that were linked not only to construction but also to maintenance and operations . These included the Rabaul/Kokopo road, urban drainage in both towns, erosion control at the port of Rabaul, and all urban solid waste and sanitation components . The environmental studies and mitigation measures were carried out in accordance with World Bank safeguards policies . Arrangements were also made to monitor constantly seismic activity and air quality during project implementation, but the large quantity of ash present in the air prevented more than basic study designs for air quality . Resettlement. The major safeguard issue in the project concerned large scale relocation and resettlement away from vulnerable areas. Following the eruption, about 80,000 people were evacuated from devastated areas, some 30,000 lost their homes and 20,000 sought permanent resettlement in other parts of the Peninsula . To minimize the negative impact of settling so many residents in a new area, a plan of open consultation with affected communities -- with special attention to the host community -- was adopted by the Authorities . Although O.D. 4.03 was not triggered since resettlement was necessitated by a natural disaster (the one exception being the acquisition of a plantation and the relocation of its workers which was carried out according to Bank guidelines, including the preparation of a Resettlement Plan), the Authorities nonetheless took steps to ensure that preferences and community feedback were taken into account to the extent possible . Although not required to do so by the legal documents, the Government was planning to conduct a social assessment (including a household survey ) of project outcomes. Fiduciary: The Borrower's ICR (Annex 3 of the ICR) refers to a fraud and corruption issue that was investigated early in 2004. Although this is not mentioned or discussed in the ICR, Country Management has informed IEG of the investigation’s conclusion that there had been no wrong -doing. However, the Project Implementation Unit, housed in the GRA, was never given any feedback on this investigation which the Borrower described as a "damaging" event with "wide implications" for the relationship with the Bank (although it went on to say that these were by outweighed by the good working relationship with the Bank and "the positive impact of Bank support ."). No other fiduciary issues arose. GRA commissioned the installation of a project financial management system to meet the Bank's reporting requirements. Annual project accounts were audited in a timely manner by the Office of the Auditor General and were consistently unqualified. Procurement was carried out satisfactorily by GRA who had already gained experience with Bank procedures during the ITRP . 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately There were weaknesses in the Satisfactory relevance of design and in efficiency (see Sections 3 and 5). Risk to Development Moderate Significant The poor record regarding counterpart Outcome : funding availability and the inadequate maintenance of some public infrastructure, together with less tax autonomy for the ENBPA, combine to create a significant risk of under -funded maintenance of project-financed investments (see section 7). Bank Performance : Satisfactory Moderately The assessment of the time needed to Satisfactory complete Phase I of the MTRP was over-optimistic and, partly in consequence, project costs were underestimated. M&E was also deficient, in that no measures of restored economic and social well-being, sustainability of outcomes, and the minimization of adverse impacts of future volcanic eruptions on lives, property and livelihood were included. Borrower Performance : Satisfactory Moderately Significant counterpart funding Satisfactory problems persisted throughout implementation. Quality of ICR : Satisfactory NOTES NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: The following lessons may be learnt from the preparation and implementation of this project : When a Borrower has a record of problems in timely and adequate provision of counterpart funding, it would behoove the Bank to take appropriate mitigating measures . Where a considerable number of different donor agencies are involved, each with its own procedures and requirements, implementation would be considerably eased if it were possible to create a mechanism whereby one of the donor's procedures were to be adopted for all procurement whether that donor is the source of funds or not. GRA's practice, following the 2005 restructuring, of using a contract administration manual together with enhanced risk management measures proved highly effective in reducing further delays and cost escalations . This practice could well be adopted in other projects both in PNG and elsewhere, provided the implementing agency is endowed with the necessary capacity . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: While the ICR is clearly written, does well in conveying the developing "story" of the implementation of this project, and presents a useful annex on "Outputs by Components," it does contain some significant drawbacks : (i) the ICR should have discussed the alleged fraud and corruption issue that was investigated early in 2004 and that is referred to in the Borrower's ICR; (ii) there is an absence of the analysis of efficiency; and (iii) the M&E section is unsystematic and does not follow the sub -headings of design, implementation and utilization . Also, some discussion of the PAD's suggested trust fund for partially overcoming counterpart funding difficulties would have been useful . a.Quality of ICR Rating : Satisfactory