Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) Report Number: ICRR0022134 1. Project Data Project ID Project Name P117323 Indonesia Power Transmission Development Country Practice Area(Lead) Indonesia Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-79400 31-Dec-2015 182,064,707.60 Bank Approval Date Closing Date (Actual) 08-Jul-2010 31-Oct-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 225,000,000.00 0.00 Revised Commitment 182,064,707.60 0.00 Actual 182,064,707.60 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ihsan Kaler Hurcan Dileep M. Wagle Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Loan Agreement (p.5) dated November 18, 2010, and the Project Appraisal Document (PAD, p.3) the project objective was “to assist the Borrower to meet growing electricity demand and improve the reliability of electricity supply in Java and South-Central Sumatra by strengthening the power transmission system.” The Borrower was defined as the Republic of Indonesia. The project objective was revised in June 2016. According to the World Bank (the Bank) letter dated June 30, 2016 amending the Loan Agreement, the revised project objective was “to assist the Project Implementing Page 1 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) Entity to meet growing electricity demand, improve the reliability of electricity supply, strengthen the power transmission system, and support the preparation of hydropower projects.” The project implementing entity was defined as Perusahaan Listrik Negara (PLN), the national power utility. Because of the revision of the project objectives, a split evaluation of the project outcome will be undertaken in this review. The objectives will be parsed as follows: Objectives before the restructuring in June 2016:  Objective 1: to assist the Borrower to meet growing electricity demand and in Java and South-Central Sumatra by strengthening the power transmission system; and  Objective 2: to assist the Borrower to improve the reliability of electricity supply in Java and South- Central Sumatra by strengthening the power transmission system. Objectives after the restructuring in June 2016  Revised Objective 1: to assist the Project Implementing Entity to meet growing electricity demand (Note: Since there is no material change between Objective 1 and Revised Objective 1, the assessment for the original objective will be used for the revised objective);  Revised Objective 2: to assist the Project Implementing Entity to improve the reliability of electricity supply (Note: Since there is no material change between Objective 2 and Revised Objective 2, the assessment for the original objective will be used for the revised objective);  Revised Objective 3: to assist the Project Implementing Entity to strengthen the power transmission system; and  Revised Objective 4: to assist the Project Implementing Entity to support the preparation of hydropower projects. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 29-Jun-2016 c. Will a split evaluation be undertaken? Yes d. Components The project originally consisted of two components: A. Java-Bali Transmission Substation Expansion and Construction. (Appraisal cost: US$103.5 million including physical and price contingencies, taxes, interest and front-end fee; actual cost: US$108.8 million) Page 2 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) 1. Expansion of four existing 500/150 kV and 20 to 25 existing 150/20 kV substations through the installation of new transformers and associated equipment. 2. Construction of one new 150/20 kV substation, including installation of two new 60 MVA transformers and associated equipment. B. South-Central Sumatra Transmission Substation Upgrading and Expansion. (Appraisal cost: US$150.6 million including physical and price contingencies, taxes, interest and front-end fee; actual cost: US$79.3 million) 1. Upgrading of five existing 150 kV substations through the installation of new transformers and associated equipment. 2. Expansion of 10 to 15 existing 150/20 kV substations through the installation of new transformers and associated equipment. Revised Components At the second restructuring in June 2019, a new component was added to support the preparation of technical design and tender documents for the Poko Hydroelectric Power Project (HPP) (estimated cost US$8.0 million; actual cost US$2.25 million) and 34 substations were added to the project scope under the original two components with a cost of US$52.0 million to be financed from the US$80 million loan savings—balance US$20 million was cancelled (see paragraphs Project Cost and Financing in section 2.e below). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The total project cost was originally estimated at US$254.1 million. In October 2019, the project closed with a total cost of US$190.4 million, because the actual cost of the extension, upgrading and construction of the substations was much lower than the cost estimated at appraisal. The implementation of competitive bidding method was an important factor for the cost decrease that resulted in a loan saving of US$80.0 million and an increase in the project scope at the second restructuring, but the benchmarking of the cost only to locally available historical data was another reason for cost overestimation at appraisal. Financing: At appraisal, the International Bank for Reconstruction and Development (IBRD) loan was estimated at US$225 million. At the second restructuring, US$20 million was cancelled and the loan amount was revised to US$205 million. By project closing in October 2019, US$182 million of the IBRD loan had been disbursed, amounting to 88.8 percent of the revised loan amount, and all project funds were accounted for. Borrower contribution: At appraisal, the contribution of the Government of Indonesia (GoI) was estimated at US$29.1 million. At project closing, the GoI’s actual contribution was US$8.4 million. Fewer land appropriations and PLN’s decision not to hire a third party for contract supervision were the main reasons for lower borrower’s contribution. Restructurings: There were three project restructurings.  First Restructuring (Level 2 – December 15, 2015): The closing date was extended by 12 months from December 31, 2015 to December 31, 2016. In its fifth year of implementation, the project had Page 3 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) an undisbursed amount of US$138.3 million out of US$225 million. Issues with contract management, prolonged internal processes and reviews, and slow decision making at PLN were the reasons for delays in the implementation of all contract packages, which were procured in September and October 2013. Additionally, the GoI requested the use of US$60 million of a total of US$80 million savings to finance the upgrading of additional 34 substations in the Java-Bali and Sumatra systems and the preparation of project documents for the Poko Hydroelectric Power Plant (HPP). These changes would require a Level 1 restructuring consisting of a revision of the project objective, the preparation of updated safeguard documents since the environmental rating of the project would be revised to Category A, cancellation of some project funds, and further extension of the project closing date. Therefore, the project closing date was extended by 12 months to allow project contracts to remain valid while a Level-1 restructuring of the project was prepared (Project Paper, Report No: RES21353, p.5).  Second Restructuring (Level 1 – June 15, 2016): A new component was added, and the number of substations was increased to utilize loan savings (see Revised Components above). The project objective was revised accordingly (see section 2.a Objectives above). The environmental safeguard category was changed from B to A, and five safeguards policies were triggered, i.e., Natural Habitats, Forests, Physical Cultural Resources, Indigenous People, and Safety of Dams, because the future construction of Poko HPP was expected to have significant, irreversible and large scale environmental and social impacts (Project Paper, Report No: RES22593, p.9). Two new indicators were added to the results framework to monitor the project activities related to the Poko HPP development. The end target values of the original indicators were revised because of the change in project scope. The implementation of all contracts was 24 months behind the original schedule. Issues with contract management, prolonged internal processes and reviews, and slow decision making at PLN continued to be the reasons for delays in contract implementation; therefore, the closing date was extended by 18 months from December 31, 2016 to June 30, 2018 to allow time for the completion of the substation expansion and upgrading works, and the preparation of the studies for Poko HPP.  Third Restructuring (Level 3 – June 27, 2018): The closing date was extended by 16 months from June 30, 2018 to October 31, 2019 to allow the completion of the Environmental and Social Impact Assessment (ESIA) for the proposed Poko HPP development, the findings of which were to be key inputs for the feasibility study. The ESIA could not be completed on time because of the delay in contracting a consulting firm to carry out the assessment. In this restructuring, the end target values of some indicators were revised, too, to match the actual achievements at the time of restructuring. Dates: The project was approved on July 8, 2010 and became effective on May 30, 2011. A Mid-Term Review of the project was completed on January 12, 2015. The original closing date was December 31, 2015. In three restructurings, the closing date was extended by a total of 46 months. The project closed on October 31, 2019. The reasons for closing date extensions have been outlined in the restructuring entries above. Disbursement Percentages Following disbursement percentages will be used in deriving the weights to be applied in split evaluation. Disbursement Disbursed Amount Percentage Project Objectives ($ million) (%) Page 4 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) Original 97.42 53.51 Revised in June 2015 84.64 46.49 Total 182.06 100.00 3. Relevance of Objectives Rationale The project objective was highly aligned with the Bank strategy for Indonesia at project closing. The project sought to address the increasing electricity demand by strengthening the transmission network that was expected to lead to an increase in the availability of electricity, while improving the reliability of the electricity supply. This problem fits within the area of activity under the Engagement Area 2: Sustainable Energy and Universal Access of the Country Partnership Framework FY2016-FY2020 (CPF, pp.25-27). The project objective corresponds to one of the four strategic objectives under this engagement area: “to improve operational efficiency and reliability of services through transmission and distribution, pumped storage” (CPF, p.25). The project objectives were also aligned with the Bank’s Systematic Country Diagnostic of September 2015. The report emphasizes the importance of increasing investment in the energy supply in order to meet the growing demand for energy, which requires increased investment in associated transmission and distribution capacity (ICR, p.17). The project objectives were highly relevant to the country context. The objective was appropriately pitched for the development status of the country. To meet the power needs of the economy, the GoI aims to increase the electrification rate to close to100 percent by 2020 from 85 percent in 2015 and the installed generation capacity to 115 gigawatt (GW) in 2025 from 60GW in 2015 (National General Energy Plan 2017 - Rencana Umum Energi Nasional 2017, p.34 and 85). Achievement of such targets depend on the construction of transmission and distribution lines. According to the Electric Power Supply Plan 2019-2028 (Rencana Usaha Penyedian Tenaga Listrik 2019-2018, p.I-4), two of the strategic objectives of the PLN are to improve the reliability and quality of electricity supply and increase the efficiency of the national grid by lowering transmission and distribution losses. The Bank supported Indonesia in infrastructure development through investment lending, and in improving policy environment for infrastructure project development through development policy lending. The Bank has also provided technical assistance in reforming electricity tariff and subsidy regime, establishing incentives for geothermal resource development and capacity building (ICR, p.8). Although this project was the first in reengaging with the PLN after seven years, the Bank had sufficient experience in the energy sector in Indonesia; therefore, the project objective was adequately challenging. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL Page 5 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) OBJECTIVE 1 Objective To assist the Borrower to meet growing electricity demand in Java and South-Central Sumatra by strengthening the power transmission system. Rationale Theory of Change for the Original Objectives The project had a direct causal chain: the project inputs, i.e., loans, would finance the expansion, upgrading and construction works of numerous substations to address the transmission bottlenecks. Most of the substations were operating over 80 percent of their capacity; some could not pass the regularly implemented reliability tests; load shedding was common; and the duration of outages was long. The expected intermediate outcome was an increase in the transmission capacity in Java-Bali and South and Central Sumatra grids. Because of the increase in the transmission capacity, the electricity availability would be expected to increase to meet growing demand. Additionally, the frequency and duration of outages would be expected to decrease because of fewer tripping at the substations that would result in an improvement in the reliability of electricity supply. The causal pathways from inputs/activities to outputs and to outcomes were valid and direct, although the outcomes achieved could not be fully attributed to the project’s intervention: First, the project’s theory of change was based on two critical assumptions that the generation capacity would be adequate to supply electricity to the system and that the electricity grid would have the capacity to distribute electricity to the end-users; second, although the outcome indicators were broadly relevant—the project team commented that increase in energy sales was the best indicator given that investments were scattered in a large project area where measuring other indicators, such as unserved energy, was not possible—to capture the achievement of the project objectives, they were not measurable, because “there was no way to draw strict boundaries for attribution” (ICR, p.30); and third, the project’s intervention constituted a small part of a large substation capacity development program—3,350MVA (2.8 percent) of 118,784MVA in Java and 2,390MVA (7.2 percent) of 33,200MVA in Sumatra. Outputs In the Java-Bali transmission network, following activities (ICR, p.43) were completed as planned:  Expansion of 46 substations (150/20kV)  Construction of two 500/150kV substations in Balaraja (originally in Depok) and Pedan (originally in Krian) In the South-Central Sumatra transmission network, following activities (ICR, p.43) were completed as planned:  Expansion of 23 substations (150/20kV)  Construction of five 275/150 kV substations in Lubuk Linggau, Bangko, Muara Bungo, Kiliranjao, and Lahat Outcomes Page 6 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) The incremental capacity of substations increased as follows:  150/20 kV substations in Java-Bali: by 2,940MVA against the original target of 1,350MVA and revised target of 2,790MVA.  500/150 kV substations in Java-Bali: by 2,000MVA as targeted.  150/20 kV substations in Sumatra: by 1,230MVA against the original target of 390MVA and revised target of 1,200MVA.  275/150 kV substations in Sumatra: by 2,000MVA as targeted. The above-listed intermediate outcomes, i.e., increase in substations capacities, were to result in an increase in the availability of electricity to meet increasing demand. This outcome was measured by the amount of electricity sales (in GWh) in the project areas in both Java-Bali and Sumatra:  The electricity sales in Java-Bali increased from a baseline of 50,442GWh to 180,381GWh, which was more than the original and revised targets of 77,270GWh and 170,000GWh, respectively.  The electricity sales in Sumatra increased from a baseline of 4,919GWh to 36,294GWh, which was more than the original and revised targets of 7,770GWh and 34,000GWh, respectively. The project was highly successful in achieving the output and outcome targets. The project team commented that “The additional transformers/substations capacity has contributed to the increase in the number of customers and electricity sales while maintaining/improving the reliability of the system in constant expansion and with a fast-growing demand” (Project team’s email to IEG dated November 6,2020). The project team further commented that “the attribution might be easier explained for the Sumatra system. For Sumatra, where the backbone of the electricity is based on 150 kV transmission lines, the project contributed significantly in increasing the 150/20 kV capacity connecting north and south parts of Sumatra. There is a seasonal situation when the north Sumatra system could not rely on the supply from the hydropower generation during the dry season and then needs more supply from the south” (Project team’s email to IEG dated November 6,2020) Overall, according to the achievements of the target values and the additional explanation received from the project team, the project was highly successful in achieving the project objective to meet growing electricity demand in Java and South-Central Sumatra, but due to weaknesses in establishing full attribution between project outputs and outcomes, the efficacy of the achievement of Objective 1 is rated Substantial, rather than High. Rating Substantial OBJECTIVE 1 REVISION 1 Revised Objective To assist the Borrower to meet growing electricity demand in Java and South-Central Sumatra by strengthening the power transmission system. Revised Rationale Page 7 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) The original Objective 1 was not revised at the project restructuring. Please see Objective 1 above for the discussion of the efficacy of the achievement of this objective. Revised Rating Substantial OBJECTIVE 2 Objective To assist the Borrower to improve the reliability of electricity supply in Java and South-Central Sumatra by strengthening the power transmission system. Rationale Outputs Please see outputs under Objective 1 above. Outcomes The capacity increases in the substations financed under the project were expected to lead to a decrease in the duration and frequency of outages because of fewer tripping in the substations. This outcome was measured by “transformer outage frequency” and “transformer outage duration” indicators. The baselines, targets and achievements are given in Table 1 for Java and Table 2 for Sumatra. Table 1: Java Substations Baseline Target Actual Transformer Outage Frequency (times/transformer/year)  500/150kV 0.35 0.20 0.01  150/20kV 0.19 0.10 0.02 Transformer Outage Duration (hours/transformer/year)  500/150kV 30.22 4.00 0.065  150/20kV 4.97 4.86 0.047 Table 2: Sumatra Substations Baseline Target Actual Page 8 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) Transformer Outage Frequency (times/transformer/year)  150/20kV 0.47 0.35 0.09 Transformer Outage Duration (hours/transformer/year)  150/20kV 1.16 0.64 0.12 The project was extremely successful in improving the reliability of electricity supply in Java and South- Central Sumatra. The project’s intervention contributed to the achievement of almost negligible outage frequency and duration. The project team provided additional explanation how project contributed to this high success, and how showing full attribution was not possible to establish full attribution due to the scattered nature of investments in a wide project area. Similarly to Objective 1, the project was highly successful in achieving the project objective to improve the reliability of electricity supply in Java and South-Central Sumatra, but due to weaknesses in establishing full attribution between project outputs and outcomes, the efficacy of the achievement of Objective 2 is rated Substantial, rather than High. Rating Substantial OBJECTIVE 2 REVISION 1 Revised Objective To assist the Borrower to improve the reliability of electricity supply in Java and South-Central Sumatra by strengthening the power transmission system. Revised Rationale The original Objective 2 was not revised at the project restructuring. Please see Objective 2 above for the discussion of the efficacy of the achievement of this objective. Revised Rating Substantial OBJECTIVE 3 Objective N/A Page 9 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) Rationale There was no Objective 3 defined at appraisal. Objective 3, which was added in the project restructuring, is assessed below in the "Objective 3 Revision 1" section. Rating Not Rated/Not Applicable OBJECTIVE 3 REVISION 1 Revised Objective To assist the Project Implementing Entity to strengthen the power transmission system. Revised Rationale In the original project objective, the strengthening of the power transmission system was included as a means to achieve the project outcome of meeting electricity demand and improving electricity supply reliability. The intermediate outcome of “strengthening the power transmission system” was revised into a separate objective at the second restructuring. Outputs and Outcomes Please see outputs and outcomes under Objective 1 above. The project was highly successful in increasing the substation capacity more than the revised targets. The verification study conducted by the project team one month before project closure confirmed that all of the substations financed under the project were constructed and energized, but three of them were pending operation because of outage schedules (500kV Balaraja and Pedan substations), and migration of RTU (Ngoro substation). These substations were expected to be operational before project closure. Revised Rating High OBJECTIVE 4 Objective N/A Rationale There was no Objective 4 defined at appraisal. Objective 4, which was added in the project restructuring, is assessed below in the "Objective 4 Revision 1" section. Rating Page 10 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) Not Rated/Not Applicable OBJECTIVE 4 REVISION 1 Revised Objective To support the preparation of hydropower projects. Revised Rationale Objective 4 was added at the second restructuring to support the PLN to prepare a technical feasibility study and safeguards documents for the development of 124.5MW Poko HPP. This was a technical assistance activity added to the project to utilize the loan savings from the first component. In their email to IEG on November 6, 2020, the project team provided additional information as evidence for the outcome achieved because of the project’s technical assistance support in the preparation of hydropower projects. “With regard to the Technical Assistance (TA) component to support the preparation of hydropower projects, the project had two deliverables: (1) Feasibility Study (FS) and (2) Environmental Social Impact Assessment (ESIA). Both documents concluded that the Poko Hydropower is feasible to be developed. These studies have been completed well and provided a basis for PLN in making decision to continue developing the project. “In September 2020 (a few months after the ICR was completed), PLN took the decision to move forward with the project and officially requested the Bank to finance further preparation work for Poko Hydropower, which include tender design, and bidding document. Additional discussions will be carried out with PLN during the preparation work such as ESIA update, tender strategy, contract packaging, and development strategy (public or private or public private partnership). ESIA update is required later during appraisal process since the current document is considered as a Feasibility Stage ESIA.” Based on the additional information provided by the project team the efficacy of the achievement of this objective is rated High. Revised Rating High OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project was highly successful in increasing the capacities of the substations financed under the project. The construction and operation of the substations, except three of them, were verified by a study conducted by the project team at project closure. The project was highly successful in achieving the original objectives to meet growing electricity demand and improve the reliability of electricity supply in Java and South-Central Sumatra; however, due to weaknesses in establishing full attribution between project outputs and outcome, the overall efficacy of the achievement of the original objectives is rated Substantial, rather than High. Page 11 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) Note: A split rating will be applied to arrive at an overall Outcome rating in section 6. Outcome below. Overall Efficacy Rating Substantial OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale The project was highly successful in increasing the capacities of the substations financed under the project. The construction and operation of the substations, except three of them, were verified by a study conducted by the project team at project closure. The project was highly successful in achieving the original objectives to meet growing electricity demand and improve the reliability of electricity supply in Java and South-Central Sumatra; however, due to weaknesses in establishing full attribution between project outputs and outcome, the efficacy of the achievement of the original objectives is rated Substantial, rather than High. The project efficacy of the achievement of Objective 3 and Objective 4, which are added at the project restructuring is rated High. Overall, the efficacy of the achievement of the project objectives after project restructuring is rated Substantial. Overall Efficacy Revision 1 Rating Substantial 5. Efficiency Economic Analysis The project appraisal document (PAD) did not include a detailed explanation of the economic analysis conducted at the appraisal stage. The project was prepared under an express track (Track 1), which required fewer clearances and simpler documentation (ICR, p.24). The PAD did not provide information about what the expected benefits were but noted that the economic internal rate of returns (EIRRs) for different groups of substations in Java and Sumatra ranged from 25 percent for 275/150kV substations in Sumatra to 65 percent for 150/20kV substations in Java. Similarly, the net present value (NPV) ranged from US$34 million to US$335 million at a discount rate of 10 percent (PAD, p.7). At the second restructuring in June 2016, an economic analysis was conducted for the additional 24 substations in Java and six substations in Sumatra. The analysis resulted in an EIRR of 32.6 percent and an NPV of US$731 million for Java substations and an EIRR of 44.9 percent and an NPV of US$67 million for Sumatra substations. Page 12 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) The economic analysis conducted at project closing resulted in much higher EIRRs ranging from 33 percent for 275/150 kV Substations in Sumatra to 81 percent for 150/20 kV substations in Central Java. The NPVs ranged from US$40 million to US$355 million. Financial Analysis A project financial analysis was not conducted at appraisal (PAD, p.8). The ICR (p.47) states that, at project closing, the project’s financial internal rate of return (FIRR) was calculated at 17 percent based on the following assumptions: annual electricity growth five percent; annual electricity tariff increase three percent; net revenue of 20 percent; and a project lifetime of 30 years. Administrative and Operational Efficiency There were no disbursements for the first two and a quarter years of project implementation, on account of issues arising from PLN’s contract management. Despite an improvement in contract management after the second restructuring and the start of liquidated damages implementation to overdue packages, the PLN’s regional units’ overseeing of project works had continued to be weak until project closure, and the PLN headquarters’ intervention in improving project implementation was less than effective (ISR #11, p.4 and ISR #12, p.2). Completion times of various contracts were routinely underestimated. Changes in technical specifications in the majority of the substations resulted in large scale variation orders that also slowed project progress (ISR #5, p.2). Prolonged internal processes and reviews of invoices, resulting in delays in payments to contractors, contributed to slow progress in project implementation. These challenges delayed the project’s implementation by some 46 months against the original schedule. Project costs, as estimated at appraisal turned out to be significantly lower during implementation, resulting in a savings of some US$80 million (see Section 2e). US$52.06 million of these savings were in fact utilized towards construction and development related to substation expansion, including installation and upgrading of transformers at additional locations in Java and Sumatra, US$8 million were allocated to support the preparation of hydropower projects and US$20 million of loan funds were cancelled. Despite the numerous issues caused by contract management, the project did succeed in completing the construction of original and new substations. Taking the above factors into account, the efficiency of the project in achieving the project objectives is rated Modest. Note: The following table could not be completed, because a range was given for the EIRRs at both appraisal and project closure. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Page 13 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  0  Not Applicable 0 ICR Estimate  0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project objectives were highly aligned with the Bank strategy at project closure and Indonesia’s development status, and it was adequately challenging given the Bank’s prior experience in the country and the sector. Hence, the relevance of objectives is rated High. The efficacy of the achievement of the original objectives to meet growing electricity demand and to improve the reliability of electricity supply is rated Substantial The efficacy of the achievement of the project objectives after the project restructuring is also rated Substantial. The efficiency of the project in achieving the project objectives is rated Substantial. Overall, the Outcome of the project is rated Satisfactory. (Since the third and fourth objectives were added at the second restructuring, a split rating was applied. Please see Table 3 below.) Table 3 Split-rating Calculation 　 Original objectives Objectives after revision Relevance of Objectives High Efficacy Objective 1: Substantial Substantial Objective 2: Substantial Substantial Objective 3: N/A High Objective 4: N/A High Efficacy Substantial Substantial Efficiency Modest Outcome Rating Moderately Satisfactory Moderately Satisfactory Outcome Rating Value 4 4 Amount Disbursed (US$ million) 97.42 84.64 Disbursement (%) 53.51% 46.49% Weighted Value of Outcome Rating 2.14 1.86 Moderately Satisfactory Overall Outcome Rating (2.14 + 1.86 = 4.0) a. Outcome Rating Moderately Satisfactory Page 14 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) 7. Risk to Development Outcome Financial and technical constraints of the PLN, including its organizational deficiencies, pose a risk for the operation and maintenance(O&M) of the substations financed by the project. The transmission system in the country has been expanding rapidly, which can stretch the technical and financial capabilities of the PLN to maintain the system. If PLN’s financial situation worsens, that could weaken its O&M capabilities. Historically, PLN had been allocating three to five per cent of capital expenditures for O&M, and it is expected to continue with this practice in the future. If PLN does not continue with the expansion and strengthening of the transmission and distribution systems, the substations financed under the project might not be able to meet the electricity demand in the event of a rapid demand increase. The ICR (p.32) points out that the demand for electricity has been lower than half of what was estimated at appraisal, but it is highly likely that this demand will increase in the future, given that Indonesia has a large population and a fast-growing economy. Therefore, continuous investment to increase transmission capacity is needed, in the absence of which outages may reoccur and growing electricity demand may not be met because of the overloading of the existing substations. 8. Assessment of Bank Performance a. Quality-at-Entry The strategic relevance of the project was high; bottlenecks in the transmission network were a barrier to meet increasing electricity demand. Given the country context, the project objective was adequately challenging. Technical feasibility studies of the substations were reviewed and approved at appraisal. Economic analysis was adequate to justify the financing of the project. There were some shortcomings in the monitoring and evaluation (M&E) design, but it was sufficient to measure the outcomes of the project’s intervention, which consisted of scattered investments in a large project area. The project implementation arrangements were simple: the PLN was a reputable utility with sufficient capacity to implement the investment activities supported by the project. The overall risk assessment was adequate. Quality-at-Entry Rating Satisfactory b. Quality of supervision According to the ICR (p.32), in the initial few years, supervision missions were held approximately every eight months. During the last two years of project implementation, however, formal missions were held every four months (ISR 13, p.4) The project team prepared 17 Implementation Status and Results Reports (ISRs); they were candid in reporting the project implementation issues and the actions to be taken. During the nine-year project implementation period, there were five task team leader (TTL) changes whose location rotated between Washington, D.C. and Jakarta, though one of the last co-TTLs had been with the Page 15 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) project in various capacities since appraisal. Overall, there was a degree of continuity in the project team, including financial management, procurement and safeguards specialists. As seen from Section 5, procurement issues, arising from poor contract management, constituted a major problem during implementation, contributing to the 46-month delay in project closing. The project team made efforts to mitigate the situation, closely monitoring contract management to try and prevent any payment delays because of contract expiration. Action plans were agreed by the parties to improve project implementation starting from 2015. These plans included detailed actions to be taken by the PLN, such as application of liquidated damages under the contract for any delays caused by the contractor, hiring of additional engineers to monitor constructions at project sites, monthly meetings between the PLN headquarters and regional offices to improve contract management, and regular Bank team visits to project sites to monitor implementation progress (Restructuring Paper, Report No: 32007, p.5 and 7-8). These actions resulted in improvement in contract management and project implementation, and at project closure all the substations were operational. The project team, supported and directed by the Country Management Unit, facilitated the issuance of a Presidential Decree that allowed direct loans to state- owned-enterprises, including the PLN. As a result of this decree, the processing of invoices and payments to contractors accelerated. The project team’s supervision of fiduciary and safeguard aspects of the project was adequate. The project team conducted their own study to verify the commissioning of the substations financed under the project. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project objectives were clearly defined. The causal pathways in the results chain were plausible, but the theory of change was designed on the critical assumption that there was sufficient generation and distribution capacity to supply electricity to consumers. The project level indicators, i.e., “increased electricity sales” and “reduction in the frequency and duration of outages,” encompassed the outcomes that could be attributed to the project. The indicators were specific, measurable, and timebound. b. M&E Implementation The PLN reported the progress in construction works quarterly (ICR, p.29), which made the monitoring of increase in the substation capacity possible. The system wide measurements of the indicators, defined as outcome-level indicators, started in June 2017 after some of the substations were commissioned. The indicators related to the frequency and duration of outages and technical losses were not included in the ISRs; the outages and technical losses data were reported at project closing at the system level. The indicator added to the results framework to monitor the preparations of the project documents, i.e., Page 16 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) hydropower projects prepared (Megawatt, Custom) with a target of 128MW, did not capture the outcome expected from these activities. The output indicator added for the hydropower activities measured the completion of two studies, i.e., feasibility study and environmental social impact assessment. One month before the project closing data, the project team conducted its own study to verify the construction and commissioning of the substations financed by the project through desktop analysis of supporting documents, discussions with the officials of the PLN’s regional offices, and site- visits to five substations. c. M&E Utilization Progress reports were communicated to project stakeholders. They were used to improve project implementation by overcoming contract management related issues. At project closure, M&E data were used to provide evidence for the achievement of outcomes, but they were not sufficient to establish attribution because of methodological weaknesses. The M&E findings did not lead to any subsequent interventions. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as Category B under Environmental Assessment (OP/BP 4.01) and triggered Involuntary Resettlement (OP/BP 4.12). When the project was restructured in June 2016, the Environmental Assessment (OP/BP 4.01) classification was raised to Category A and following safeguards were triggered: Natural Habitats (OP/BP 4.04), Forests (OP/BP 4.36), Physical Cultural Resources (OP/BP 4.11), Indigenous Peoples (OP/BP 4.10), and Safety of Dams (OP/BP 4.37). Environmental Assessment (OP/BP 4.01): At appraisal, the impact of the substation projects was assessed to be short term, site-specific and reversible, such as increased level of dust and noise from the use and movement of machinery, non-toxic solid waste during construction; oil spill or leakage from machinery or transformers; and worker health and safety from existing electromagnetic fields (PAD,p.20). PLN prepared an Environmental Management Plan (EMP) acceptable to the Bank and it was disclosed in the Bank’s InfoShop and locally in Indonesia in February 2010. After the inclusion of technical assistance to support the development of the Poko HPP, the project financed the preparation of an Environmental and Social Impact Analysis (ESIA) to assess the potential impacts of the proposed HPP project. There were shortcomings in the implementation of the environmental safeguard policy. Early in project implementation, safeguards performance was not reported in the project progress reports (ISR #4, p.3) or, when reported, it was not detailed (ISR #5, p.5). There were shortcomings in the effective use of the EMP in the training of the PLN staff; housekeeping and waste management at project sites needed to be improved and further trainings of the regional PLN staff were recommended to this end (ISR #7, p.6). In May 2016, the project team noted improvements in housekeeping, use of personal protective equipment by workers and installation of some safety signs on site, but some issues, such as workers smoking during welding, Page 17 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) were observed (ISR 11, p.10). The preparation of the ESIA for the Poko HPP was delayed, because of the delay in contracting a consultancy firm to carry out the assessment (ICR, p.27). The procurement of the ESIA consultant started in February 2016 and it was signed in June 2017—it took 17 months to sign the consultancy agreement (ISR 12, p.5) This resulted in an extension of the project closing date by 16 months. At project closing, the Bank’s Regional Safeguards Secretariat reviewed the ESIA and found it as an “interim” document; there were gaps related to the final works layout, nature and scale of construction activities and workforce, time frames, technical assessment of the biodiversity management plan and downstream impacts (ICR, p.56). Involuntary Resettlement (OP/BP 4.12): Four substations in Sumatra would be expected to require land acquisition, with no resettlement or impact on building structures. PLN prepared a land acquisition and resettlement action plan (LARAP), which was disclosed in the Bank’s Infoshop and in Indonesia in February 2010. During project implementation, land acquisition for only one substation, i.e., Lahat substation, and a number of transmission towers along the Lahat-Kiliranjao corridor that were financed by the PLN was required. Land acquisition was completed in compliance to the provisions of the LARAP (ISR 14, p.11). (Note: The ICR did not provide information about the implementation of the Involuntary Resettlement policy) Natural Habitats (OP/BP 4.04), Forests (OP/BP 4.36), Physical Cultural Resources (OP/BP 4.11), Indigenous Peoples (OP/BP 4.10), and Safety of Dams (OP/BP 4.37): The ICR did not provide information about these five safeguard policies, which were triggered at the second restructuring when the technical assistance support for the development of project documents for Poko HPP was added to the project scope. According to the project documents, the construction of the Poko HPP was expected to adversely affect some indigenous people communities because of either their proximity to the construction area and the dam or their location in the watershed that would be subject to erosion management measures to keep the reservoir functioning; therefore, a grievance mechanism aspect was included in the ESIA and LARAP for the Poko HPP (ISR #13, p.2 and 13). b. Fiduciary Compliance Financial Management PLN submitted interim financial reports to the Bank regularly with occasional short delays, but some reports did not include information about counterpart funding (Aide Memoire 2013, p.2) The ICR does not provide information whether the audit reports, which were submitted on time, were qualified or not. There were lengthy delays in payments to the contractors. The ICR (p.27) lists the reasons as follows: (i) delay of the annual project budget approval by the Ministry of Finance (Mo); (ii) incomplete documentation submitted by contractors; (iii) delays in submission of the supporting documents from project office in the regional area to the PLN headquarters; (iv) delays in the regional director’s clearance of payments; and (v) lengthy verification of the invoices and documents by the MoF. The processing of invoices from submission by contractors until the payments were made by the Bank was in the range of 42 to 133 days, which was lengthy. The PLN failed to comply with one of the financial covenants, i.e., debt service coverage ratio of 1.5, in the second half of project implementation. Waivers from this covenant were granted until project closing. In 2018, the project prepared the Corporate Financial Plan—an advisory and analytical work—to Page 18 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) support the PLN to improve its financial situation. The project team confirmed that all project funds were accounted for at project closure. Procurement At the start of project implementation, procurement was delayed because of the PLN’s request to include manufacturers in the bidding consortia to lower reduce investment costs (ISR, 1, p.3). (The project team commented that the eventual inclusion of the manufacturers in the consortia was a major reason for lower than estimated investment costs for the original substation investments). However, PLN failed to advertise Special Procurement Notice for three major Java contracts in the United Nations Development Business. The Bank granted a waiver for this non-compliance with the procurement guidelines on the grounds that the bidding process had already created a high interest from bidders including international firms, and that it would have been costly for the PLN to bid the contracts again (ISR #2, p.1). Upon the project team’s suggestion, the scope of works for the additional substations was divided into two: transformer supply was procured from manufacturers, and installation packages were carried out by contractors (ICR, p.34) This was expected to provide flexibility to PLN in contract management. Despite some shortcomings in the PLN’s procurement according to Bank guidelines, which required “continued support and monitoring to ensure sustainable compliance,” the procurement was, overall, effective (ISR 3, p.3). c. Unintended impacts (Positive or Negative) None. d. Other None. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Quality of Supervision is rated Moderately Satisfactory, on Bank Performance Satisfactory Moderately Satisfactory account of delays in implementation of the project. Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons Page 19 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) This review has drawn two lessons incorporating material on lessons listed on pages 33-34 of the ICR. An intervention scattered in a large project area may make establishing attribution difficult. All 77 substations were commissioned and energized at project closure. The project objective level indicators, i.e., “increase in electricity sales” and “reduction in the frequency and duration of outages,” captured the outcomes achieved at the system level; therefore, it was not possible to measure the exact extent of the development impact of the project, and establish full attribution between the project’s intervention and the outcomes achieved as a result of this intervention. The project’s intervention was on the critical path to achieve the remarkable results in meeting increased electricity demand and improving the electricity supply reliability, but because of the nature of the project’s intervention, it became difficult to demonstrate full attribution, absence of which resulted in a Moderately Satisfactory outcome rating rather than Highly Satisfactory. Direct lending to project implementing utilities may improve project implementation efficiency. In this project, PLN had to get approval from the Ministry of Finance for annual budget and disbursement (ICR, p.34). Because of long processing times of such approvals, payments to contractors were delayed between December and February every calendar year. This resulted in delays in contract implementation. The project team, supported and directed by the Country Management Unit, facilitated the issuance of a Presidential Decree that allowed direct loans to state- owned-enterprises, including the PLN. As a result of this decree, for any future projects, the processing of invoices and payments to contractors could be accelerated, and project implementation efficiency could be improved. However, direct lending may not be applicable to all future projects since the government and PLN have specific criteria of which project to use direct lending mechanism. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provides a comprehensive overview of the project. The narrative is candid in explaining the project’s shortcomings, such as those in monitoring and evaluation (M&E) system that failed to test the links in the results chain. The discussions in the M&E design and implementation are highly evaluative. The section on key factors that affected implementation and outcome provides detailed information about the issues emerged during project implementation. The narrative is internally consistent. The interrogation of the evidence is sufficient to provide evidence for the achievement of the objectives. But there are some inconsistencies with the guidelines. The ICR does not report whether the audit reports were qualified or not; the PLN’s noncompliance with one of the financial covenants, i.e., debt service coverage ratio of 1.5, is not reported; the ICR does not provide information about the implementation of Involuntary Resettlement policy. Page 20 of 21 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Indonesia Power Transmission Development (P117323) a. Quality of ICR Rating Substantial Page 21 of 21