ICRR 13049 Report Number : ICRR13049 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 02/23/2009 PROJ ID : P087801 Appraisal Actual Project Name : Power Sector Priority US$M ): Project Costs (US$M): 1.41 2.032 Investments Country : Timor-Leste Loan/ US$M ): Loan /Credit (US$M): 1.39 1,346 Sector Board : EMT Cofinancing (US$M ): US$M): 0.02 0.686 Sector (s): Power (100%) Theme (s): Access to urban services and housing (40% - P) Infrastructure services for private sector development (40% - P) Debt management and fiscal sustainability (20% - S) L/C Number : Board Approval Date : 07/23/2004 Partners involved : Closing Date : 03/31/2006 03/31/2008 Evaluator : Panel Reviewer : Group Manager : Group : Fan Zhang Fernando Manibog Monika Huppi IEGSG 2. Project Objectives and Components: a. Objectives: The PAD states that the objective of the Power Sector Priority Investment Project (PSPIP) was to assist the Democratic Republic of Timor-Leste in the delivery of least -cost, high-quality electricity service with minimum dependence on the government ’s budget. This objective was to be achieved by improving the generation capacity and distribution efficiency of the national power company, Electricidade de Timor -Leste (EDTL) and through savings by reducing demand with the use of energy -efficient lamps/light bulbs. The Development Credit Agreement (DCA) did not have the words "with minimum dependence on the government's budget " from the objective statement. Since in practice the project did pursue the issue of reducing the power sector ’s dependence on the public budget, the objective described in the PAD is used as the basis for this ICR Review . b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): The Project consists of four components (total component costs in brackets ) 1. Replacement of engine at unit no . 5 at Comoro Power station (appraisal estimate US$1.000 million, actual US $1.686 million). To repair Unit no. 5 (MAK 4) of the Comoro power station (the only power supplier in the capital city Dili) by replacing the existing damaged engine with a new and more efficient 2.8 MW engine. The objective is to enhance base load generation capacity and reduce fuel costs at the Comoro power station . 2. Distribution rehabilitation (appraisal estimate US $0.298 million, actual US$0.286 million). This component is a small part of an ongoing system rehabilitation exercise designed to bring the damaged and dilapidated Dili distribution network to acceptable conditions . The main activities included providing necessary connections and low-voltage restructuring in preparation for the installation of new substations, repairing and re -conductoring of dilapidated transmission lines, repairing transformer stations and supply cubicles, and providing tools and instruments for maintenance work . 3. Distribution of energy efficient lamps /light bulbs (appraisal estimate US$0.090 million, actual US$0.060 million) . To distribute free of charge 60,000 lamps/light bulbs (up to 3 each to 20,000 consumers) to EDTL customers who have acquired prepayment meters . The objective is to reduce consumer energy costs, the peak load and fuel consumption of EDTL. It was also expected that the component will increase the momentum of the prepayment meters program and to raise awareness about end -use energy efficiency. 4. Project audit (appraisal estimate US$2000, actual US$0) To conduct two project audits, including the provision of technical assistance. The PAD did not specifically identify which two projects would be audited . d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost : The total project costs were US$ 2.032 million, or 44 percent higher than the appraisal estimates of US $1.41 million. The cost overrun was caused by the decision to replace the entire generating unit with a new 4.7 MW generator instead of replacing only the defective 2.8 MW engine. The larger than expected generator required the procurement of a larger step-up transformer for power distribution, amounting to a $ 159,180 cost increase for the distribution rehabilitation component . Without additional financing available, the distribution component was postponed to be carried out under a follow -up IDA-financed Energy Service Delivery Project (ESDP). The suspension of the second component resulted in additional funds being available for the energy efficient lamps component and an amendment of DCA was made in 2007 to reallocate US$13,000 from component 2 to component 3. The extra financing was intended to be used to purchase additional energy efficient lamps . However, the procurement did not go through and the extra funds were cancelled . The last component was not implemented and the funds were not disbursed. contribution: The Trust Fund for East Timor (TFET) provided a grant of US$1.390 million Financing and borrower contribution: which was administered by IDA. US$1.346 million or 97 percent of the grant was disbursed. Government counterpart financing amounted to US$0.686 million as compared with the appraisal estimates of US$0.02 million. The cost overrun was fully covered by the Government ’s co-financing. Dates : The project start-up was delayed due to protracted negotiations of the Grant Agreement . The project was closed two years later than the original scheduled closing date of March 31, 2006. Three extensions were made due to delays in procurement, cost overrun as well as political and social unrest . 3. Relevance of Objectives & Design: Rating: Substantial Relevance of the Objective. PSPIP was the World Bank’s first power sector investment operation in Timor -Leste. The project objective was highly relevant to country conditions where power infrastructure was devastatingly destroyed following prolonged civil unrest, and the restoration of basic electricity service was critical for its economic recovery. The project objective is in line with the National Development Plan (NDP) to restore infrastructure, including power networks, and one of the targets of the First Transition Support Program to reduce power sector ’s heavy dependence on the government budget. It is also consistent with the second pillar of the Bank ’s FY 06-08 CAS, which emphasizes the delivery of sustainable energy services . The project remains relevant at the time of evaluation. Despite substantial improvement in power infrastructure, Timor-Leste’s electricity service continues facing problems of low accessibility, low efficiency and unreliable supply . The project is also highly relevant to the Bank’s increased focus on supporting post-conflict countries in transition to sustainable economic growth. Relevance of the Design. Under a post-conflict situation, the design of the project was commendably simple and clearly focused on achieving “quick wins�. However, as pointed out by the PPAR of Timor-Leste Transition Support Program, when assisting post-conflict countries, it is important to strike a balance between addressing the short-term needs and planning for the long-term development. The challenge of the long-run improvement of the energy sector may not be the availability of resources but rather of the government’s capacity to manage resources and to effectively deliver services. Efficient and sustainable service delivery requires enhanced institutional and implementation capacity. The program design largely focused on achieving physical targets without paying enough attention to addressing these capacity development needs. The only component that has involved technical assistance was not carefully prepared and was dropped eventually . Although not formally part of the DCA’s statement of the project objective, the PAD and ICR state that the project aimed to reduce the dependence on Government budget for power service through increasing generation, distribution and end-use efficiency, thereby reducing the total cost of power supply and consumption. However, as recognized by the ICR, a fundamental cause of power sector dependence on Government budget was the lack of cost recovery measures. By 2005, less than half of the household that had access to electricity had installed meters. Following the 2006 political disturbances, the situation deteriorated as incidences of meter bypassing had drastically increased. Without directly targeting the metering and billing issue, the project intervention may reduce government subsidy per kilowatt of electricity supplied, but may well increase the total amount of subsidies needed. In fact as indicated in the ICR, energy demand has increased dramatically over the course of the project . Secondly, both price and income effects of increased end-use energy efficiency may push up energy demand. To curb energy consumption, the most effective mechanism is again to strengthen the metering and billing system . The project M&E focused on intermediate outputs (see Section 10). The level and quality of the electric service, and government expenditure on power supply were not reported . Because of the inadequate M&E system, it is difficult to empirically identify the connection between project interventions and intended outcomes . 4. Achievement of Objectives (Efficacy): Rating: Modest Overall objective : the delivery of least -cost, high -quality electricity service (the PAD and ICR add : “with minimum dependence on the government ’s budget .�) The weakness in the design of M&E system makes assessment of the overall outcome difficult. There were no direct indicators measuring the change in cost or quality of electricity service attributable to the project. The estimation of fuel cost savings was less reliable without considering the incremental change in energy supply and consumption (see Section 5). The Borrower’s review notes that “the technical and non-technical losses in the Dili distribution system are still as high as 50 percent of total electricity generated� thus indicating that the distribution rehabilitation sub-component was a missed opportunity to significantly increase power supply efficiency, which mitigated the achievement of the overall objective. (Note that when insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006.) Given the lack of information to make an assessment on the achievement of the overall objective, the project overall efficacy was rated modest. Subobjective 1: the delivery of least -cost electricity service : (1) After a 17-month delay in installation and cost overrun, a new and more efficient diesel generator was Output: brought online on April 1, 2007. (2) A total of 27,600 compact fluorescent lamps, or 56 percent of the purchased energy efficient bulbs, were distributed by April 2006. The distribution was suspended due to political disturbances in May 2006. The remaining 22,400 compact fluorescent lamps (CFL) were in storage at EDTL for future distribution . Outcome: The fuel consumption of the new generator averaged 0.238l/KWh compared to 0.28 l/KWh of the displaced high-speed peak load generator. The cost saving is roughly estimated to be 121,000 l/month. However, this estimation may not be reliable (see discussion in Section 5). The ICR estimates energy saving from the deployment of CFLs amounts to $2.59 million per year assuming daily lighting demand remains constant. (As discussed in Section 3, this assumption may not be plausible.) However, no baseline or ex-post data were collected regarding consumer energy consumption behavior. Therefore, there was no direct measurement of energy savings outcome attributable to the installation of CFL lights . Subobjective 2: the delivery of high -quality electricity service Output: The replacement of unit no.5 engine at Comoro power station with a new 4.7 MW generator enhanced based load generation capacity. Upon commissioning, the generator has been operating properly (excluding one month outage for a major repair) and has achieved an average load factor of 64 percent. According to the Borrower’s ICR, the generator has reached capacity factors of 81.36 percent in 2007 and up to 99.35 percent in 2008. This was an impressive performance by international standards . Outcome: It was not reported whether the displaced high-speed generator was re-commissioned for power supply in other regions and how the additional generating capacity contributed to the overall reliability of the Dili System. Although the physical target was achieved, the outcome and the extent to which the DO was achieved is unclear . Subojective 3: with minimum dependence on the government's budget Outcome: Due to lack of performance indicator on government's spending on power supply, it was not clear whether this subobjective was achieved or not . 5. Efficiency (not applicable to DPLs): Component 1: Ex ante estimation: if only fuel cost savings are considered, the internal rate of return (IRR) is 16.5 percent. Incorporating benefits from postponed installation of another diesel generator and re-commission of the displaced high-speed engine, the IRR is 49 percent. The net present value (NPV) is estimated to be between US$0.591 to US$0.626 million. Ex post estimation: The economic and financial rate of return (ERR and FRR) is between 99 to120 percent depending on the unit cost of fuel . The NPV is US $3.851 millions. Component 3: The ex ante estimation of the NPV is US$1.939 million. Because of the cost of CFL lamps was declined, the ex post estimation of NPV is US$3.681 million. Both analyses assume ERR and FRR are the same . Project efficiency was rated substantial, however there are several issues/concerns regarding ERR and FRR calculation. They are discussed below. 1. The current economic analysis does not take into account the project-induced change in fuel inputs and energy outputs, which leads to two biases: (a) benefits from private sector increase in output following the expansion of electricity supply was not considered; (b) cost savings were overestimated because fuel inputs will be higher given the incremental increase in electricity supply. A more rigorous assessment of fuel cost saving could be achieved through the monitoring of actual fuel consumption with and without the project . 2. Labor costs were ignored in both financial and economic analysis; reduction in greenhouse gas emissions due to increased energy efficiency was not considered or mentioned as one of the economic benefits . 3. If electricity metering and billing continue to be inadequate, financial analysis may overestimate the expected revenue and the fiscal sustainability of the project . 4. US$90/bbl oil was used as a conservative assumption to predict fuel cost savings over the lifetime of the project. With hindsight, this turns out to be an overly optimistic assumption as oil is currently traded at around US$50/bbl. A sensitivity analysis that reports switching value of oil price at which the project’s NPV becomes zero would better inform the value of the project . Overall, the true economic benefits may be underestimated while the FRR may be overestimated depending on how much operating and maintenance costs are able to be recovered. In practice, ill-advised fiscal impact of the projects is more of a problem than underestimated economic benefits . ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate Yes 120% 85.9% * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Based on ratings of Substantial for relevance, Modest for efficacy, and Substantial for efficiency, the overall project outcome is rated Moderately Satisfactory . a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The main risks associated with the project and the sustainable development of the power sector are (i) the emergence of civil unrest and violence; (ii) the heavy dependence of Timor -Leste on revenues from oil and gas reserves in Bayu Undan, which, with recent declines in oil prices, may result in cutting back much -needed financial resources to improve the availability, efficiency and quality of electricity service; (iii) the major challenges remaining to strengthen government capacity to design, manage and deliver power projects; (iv) the fragmentation of donor assistance; and (v) inadequate accounting, billing and collection systems, which threatens EDTL ’s ability to recover costs of supply. Overall, the risk for development outcome is rated significant . a. Risk to Development Outcome Rating : Significant 8. Assessment of Bank Performance: at -Entry Quality -at- The project objective was highly relevant to local conditions and the Bank’s country assistance strategy. The project design was simple and clearly focused on meeting short-term emergency needs. However the project design also suffered from the following weaknesses : 1. The project preparation was insensitive to the country’s history of political and economic instability. No political or institutional risk analysis was conducted, and no risk mitigation plan was developed to respond to potential shocks that would have a significant impact on project implementation. 2. Despite the country’s weak public sector capacity, technical assistance regarding project implementation and procurement was not built into the program . 3. Cost estimate of engine replacement was based on one actual quotation. No physical or financial contingencies were included in the appraisal. Project delays and cost overrun could have been mitigated by the including price contingencies in project cost estimates . 4. The appraisal suggested the 2.8MW engine option provides the most cost-effective option. The ICR noted the new 4.7MW generator exceeded considerably the appraisal estimates of cost savings. Therefore, it is unclear whether the capacity choices were based on sound economic analysis of the electricity demand and the required size of the supply system . 5. M&E system is weak. Output and outcome indicators were blurred . at-Supervision Quality -at- The project was supervised regularly. A task team member was based in the field and carried out close supervision of the project. The Bank team also showed persistence and flexibility in assisting the Government and EDTL to overcome many challenges to the project implementation and to adjust the scope in accordance with changing circumstances. The team followed up regularly on the implementation of the project, and compliance with the Bank’s fiduciary and safeguards policies . Considering the special challenges presented in a post -conflict environment and that the Bank team was able to achieve the majority of the physical targets, the overall bank performance was rated satisfactory . at -Entry :Moderately Satisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Satisfactory 9. Assessment of Borrower Performance: The ICR notes that the Government was very committed to PSPIP and made the provision of reliable electricity service a national priority. During the project cycle, the government exhibited satisfactory support and ownership of the project. Despite the fact that the Government was weak in institutional and implementation capacity, the Government was able to rely on qualified consultants for procurement and financial management . Facing unforeseen substantial cost overrun, the Government acted decisively to open a new bidding process and provided adequate financial support to cover the entire incremental cost . Staff turnover affected the performance of the implementing agency . The ICR notes that initially the General Manager of the management contractor were very engaged and provided critical inputs for project preparation . However, the subsequent two project contractors of EDTL were indifferent about the project and made no efforts towards the achievement of the project outcomes . a. Government Performance :Satisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: M&E Design: Project milestone dates are identified as key results indicators. Although they are simple and can be used to monitor project progress, these indicators do not reflect the progress in accomplishing the objectives and do not provide warnings when there is delay in project implementation. Relying on milestone dates neither provides flexibility in addressing the issues in project implementation. The increase in generation capacity and the cost savings through the use of energy-efficient lamps were used as the outcome indicators. These indicators cannot adequately reflect the project objective of delivering least-cost, high quality electricity service with minimum dependence on the government’s budget. The more appropriate indicators would be government expenditure on power supply and the reliability and accessibility of the electricity service . M&E Implementation: Throughout the project implementation, there was no effective M&E system . Cost savings were estimated based on assumptions made in the appraisal document. Beneficiary surveys could have been conducted to understand energy consumption behavior change and to estimate the actual fuel cost savings . M&E Utilization: The implementing agency has collected electricity generation and load factor data at regular intervals. But in the absence of an appropriate M&E framework, no special efforts have been made to use the data to estimate “relevant� impact. As a result, relevant information was not available to inform decision -making and evaluate alternative options. a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Environment: The generating unit component has achieved positive environmental impacts by reducing fuel consumption and pollutants emitted per kilowatt hour of electricity produced . However, because CFL light bulbs contain mercury, the component related to the distribution of CFLs poses an environmental hazard if there is no provision for recycling. 12. 12. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Moderately Moderately Satisfactory Satisfactory Risk to Development Significant Significant Outcome : Bank Performance : Satisfactory Satisfactory Borrower Performance : Moderately Moderately Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: Based on the review of the project, the following lessons were drawn : 1. Risk assessments of political and economic shocks should be conducted during project appraisal and contingent plans should be built into project design, especially in countries and regions where there have been recent experiences of large scale civil and political unrest . 2. In difficult political and macroeconomic environments like post-conflict countries, flexibility in modifying the Credit Agreements to accommodate the country’s specific conditions was a key factor in enabling the Bank to remain engaged in providing assistance . 3. When working with low-capacity borrowers, it is important to provide technical assistance to help borrowers in day-to-day project implementation and procurement . 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The quality of the ICR is rated as satisfactory overall. The ICR presents a concise and candid description of the project objectives, components and results . The ICR would have been better if it had : 1. reported if there is any increased economic activity resulting from increased electricity supply and provided more substantial evidence such as increase in reliability and accessibility to electricity regarding the impact of the project 2. provided more detailed analysis to support the rationale for installing the higher -capacity generation unit. 3. conducted more rigorous economic and financial analysis . a.Quality of ICR Rating : Satisfactory