Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) Report Number: ICRR0022494 1. Project Data Project ID Project Name P123994 Second Power Transmission Development Country Practice Area(Lead) Indonesia Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-82800 31-Dec-2018 105,364,546.29 Bank Approval Date Closing Date (Actual) 08-Jul-2013 31-Dec-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 325,000,000.00 0.00 Revised Commitment 105,364,546.29 0.00 Actual 105,364,546.29 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ihsan Kaler Hurcan Ramachandra Jammi Ramachandra Jammi IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Loan Agreement (p.6) dated August 28, 2013 and the Project Appraisal Document (PAD, p.4), the original project objective was “to meet growing electricity demand and increase access to electricity in the project area through strengthening and expanding the capacity of the power transmission networks in the project area in a sustainable manner.” The project area included a total of 27 provinces in East Indonesia (Kalimantan, South Sulawesi, Central Sulawesi, and Southeast Sulawesi), Java-Bali, and West Indonesia (Sumatra). Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) The project objective was revised in May 2019, and the objective “to increase access to electricity” was deleted. According to the World Bank (the Bank) letter dated April 15, 2019 amending the Loan Agreement, the revised project objective was “to meet growing electricity demand in the project area through strengthening and expanding the capacity of the power transmission networks in the project area in a sustainable manner.” Reasons for not undertaking a split-rating Despite the revisions of the project objective and key associated outcome targets, this review will not undertake a split rating because of the following reasons: 1. The project did not finance any activity that would lead to an increase in access to electricity. As explained in the Components section below, the project activities consisted of only the expansion, rehabilitation, and construction of substations to address the bottlenecks in the transmission network. At appraisal, the increase in access to electricity was interpreted as a higher objective that could be achieved as improved transmission substations would increase the PLN’s grid capacity to establish new connections. Since the project did not directly support the achievement of the project objective to increase access to electricity, which was interpreted as a higher objective, this review will not assess the achievement of this objective. A rating is not applicable for this objective. 2. At the first and second restructurings, the project scope was reduced along with the cancellation of project funds. This led to a downward revision on the target values of some of the key associated indicators. The reductions in target values (see the Restructurings entry below in this section) were commensurate with the cancelled loan amount and the decrease in project scope. This supports a decision not to undertake a split rating and to assess the project’s achievement based on the revised scope. In light of the reasons given above, this review will assess the achievement of the following objective without applying a split rating: Objective: to meet growing electricity demand in the project area through strengthening and expanding the capacity of the power transmission networks in the project area in a sustainable manner. Note on the Indonesia Power Transmission Development Project (IPTD1 - P117323) and the Indonesia Second Power Transmission Development Project (IPTD2 - P123994) The IPTD1 was approved in July 2010. The IPTD2, the subject of this review, was a continuation of the IPTD1 and approved in July 2013. Upon the request of the project implementing agency, i.e., PT Perusahaan Listrik Negara (PLN – Indonesia’s state-owned power company), the IPTD2 was designed as a separate project, rather than an additional financing under the IPTD 1 (ICR, footnote 6, p.7). The IPTD2 closed in December 2019, three months after the IPTD1 closed. Both projects aimed at expanding the transmission substation capacity to meet growing electricity demand in their respective project areas. Because of the similarities between the two projects, this review will borrow from the ICR Review of IPTD1, which was disclosed on the Independent Evaluation Group’s website on December 23, 2020 (https://documents1.worldbank.org/curated/en/201911608755401615/pdf/Indonesia-Indonesia-Power- Transmission-Development.pdf) Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 10-Apr-2019 c. Will a split evaluation be undertaken? No d. Components The project consisted of one component that was to support the following substation investments to improve the transmission network in the project area. (At appraisal, the project cost was estimated at US$346.43 million including physical and price contingencies, taxes, interest during construction and front-end fee. At the second restructuring, the estimated project cost was revised to US$159.0 million because of the cancellation of some of the substation projects. The actual cost at project closing was US$124.6 million.) 1. Extension and rehabilitation of selected existing 150/20 kV and 70/20 kV substations in the project area, including the addition of new transformers and equipment, and/or replacement of existing transformers with new ones and associated equipment with higher capacity. 2. Construction of new 150/20 kV substations in the project area, including the installation of transformers and associated equipment. The project was to be implemented in two phases. In the first phase, the project was to finance the expansion and rehabilitation of 37 substations (Group 1) that were appraised and ready for implementation (PAD, pp.15 and 23). In the second phase, the PLN was to prepare a list of 71 substations, including 27 new substations, and the project was to finance these sub-projects upon being appraised by the Bank against a set of eligibility criteria (PAD, pp.15 and 23). These Group 2 sub-projects were already been identified at appraisal, but changes in the list were expected because of changing priorities and availability of alternative financing sources during implementation. The PLN’s least cost expansion plan included all the sub-projects that were to be financed by the project. Revised Components The project scope was reduced because of cancellation of substation projects under Group 2 (see Restructurings in the following section). The number of substations to be upgraded, rehabilitated, or constructed decreased from 71 at appraisal to 41 after two restructurings. After the inclusion of substation projects with voltages higher than 150kV to the project scope at the first restructuring, the definitions of subcomponents were revised as follows:  Subcomponent 1: Extension and rehabilitation of transmission substations and construction of new transmission substations under Group 1.  Subcomponent 2: Extension and rehabilitation of transmission substations and construction of new transmission substations under Group 2. Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The total project cost was originally estimated at US$346.43 million. The project cost was revised to US$181.0 million at the first restructuring and US$159.0 million at the second restructuring because of reduced scope of Group 2 activities and savings from Group 1 activities. In December 2019, the project closed with a total cost of US$124.6 million. Financing: At appraisal, the International Bank for Reconstruction and Development (IBRD) loan was estimated at US$325 million. At the first restructuring, US$165.4 million was cancelled and the loan amount was revised to US$159.6 million. At the second restructuring, an additional US$22.0 million was cancelled and the loan amount was revised to US$137.6 million. Savings from Group 1 activities and reduced scope of Group 2 activities were the main reasons for reduced Bank financing. By project closing in December 2019, US$105.4 million of the IBRD loan had been disbursed, amounting to 75.5 percent of the revised loan amount. Borrower contribution: At appraisal, the contribution of the Government of Indonesia (GoI) was estimated at US$21.4 million. At project closing, the GoI’s actual contribution was US$19.3 million. Restructurings: There were two project restructurings.  First Restructuring (Level 2 – April 15, 2018): At this restructuring, US$165.4 million of the loan was cancelled because of cost savings from Group 1 projects (US$20 million) and the decrease in the scope of Group 2 substation projects (US$145.4 million). The cancellation of a large amount under Group 2 was mainly driven by lower-than-expected electricity demand growth and the postponement of a 35GW generation program (ICR, p.11). In accordance with the decrease in project scope, target values of some of the indicators were revised down as follows: (i) growth in electricity sales in Java-Bali from 135 percent to 117.5 percent; (ii) growth in electricity sales in East Indonesia from 144 percent to 120 percent; (iii) growth in electricity sales in West Indonesia from 145 per cent to 120 per cent; and (iv) commissioned capacity of transformer substations for Group 2 from 4,130,000 KVA to 3,150,000 KVA. These reductions in target values were commensurate with the cancelled loan amount and the decrease in project scope. At this restructuring, it was confirmed that the scope of transmission investments included substations with voltages higher than 150 kV. Lastly, the project closing date was extended by 12 months from December 31,2018 to December 31,2019. This extension was needed to allow time for the completion of the substation projects in Group 2 that were delayed because of the PLN’s slow progress in finalizing the list of eligible substation projects and preparing feasibility studies.  Second Restructuring (Level 1 – April 10, 2019): The project objective was revised. The objective “to increase access” was deleted, because the project did not include any activity that would be expected to lead to the achievement of this objective. An additional US$22 million loan was cancelled. Substation projects in Group 2 were to be contracted under four packages. The offer received for Package 2 was higher than the cost estimate of US$22 million for this package. Since PLN’s own procurement rules did not allow an award price higher than the estimate, PLN decided to enter into price negotiations with the only bidder. This was against the Bank’s procurement rules. Therefore, PLN decided to continue the procurement in accordance with its own procurement guidelines and finance Package 2 using its own resources. This reduction in the project scope resulted in a decrease in the target value of the indicator “commissioned capacity of transformer Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) substations for Group 2” from 3,150,000 KVA to 2,290,000 KVA. This reduction in the target value was commensurate with the cancelled loan amount and the decrease in project scope. Dates: The project was approved on July 8, 2013 and became effective on March 26, 2014. A Mid-Term Review of the project was completed on April 1, 2016. The original closing date was December 31, 2018. In the first restructuring, the closing date was extended by 12 months. The project closed on December 31, 2019. The reasons for closing date extensions have been outlined in the first restructuring entry above. 3. Relevance of Objectives Rationale The project objective was aligned with the Bank strategy for Indonesia at project closing. The project sought to address the development problem of unserved electricity by addressing the bottlenecks in the transmission network capacity. This was expected to increase the availability of electricity and meet growing electricity demand. The project objective falls under the Engagement Area 2: Sustainable Energy and Universal Access of the Country Partnership Framework FY2016-FY2020 (CPF, pp.25-27). The project objective corresponds to one of the four strategic objectives under this engagement area: “to improve operational efficiency and reliability of services through transmission and distribution, pumped storage” (CPF, p.25). The Bank strategy defines the power deficit in Indonesia as a major bottleneck “to growth, shared prosperity and human capital formation” (CPF, p.25). The project objectives were aligned with the Bank’s Systematic Country Diagnostic of June 2020, too. The report emphasizes the importance of increasing investment in the energy sector in order to meet the growing demand for energy that requires increased investment in associated transmission and distribution capacity (ICR, p.15). The project objectives were relevant to the country context, too. The objective was appropriately pitched for the development status of the country. To meet the power needs of the economy, the GoI has been aiming to increase the electrification rate to close to100 percent by 2020 (achievement by December 2019 was 98.89 percent) from 85 percent in 2015 and the installed generation capacity to 115 gigawatt (GW) in 2025 from 60GW in 2015 (National General Energy Plan 2017 - Rencana Umum Energi Nasional 2017, p.34 and 85). Achievement of such targets depends on the expansion and strengthening of the transmission networks, including substations, and the distribution networks. According to the Electric Power Supply Plan 2019-2028 (Rencana Usaha Penyedian Tenaga Listrik 2019-2028, p.I-4), two of the strategic objectives of the PLN are to improve the reliability and quality of electricity supply and increase the efficiency of the national grid by lowering transmission and distribution losses. Expansion, rehabilitation, and construction of transmission substations would be expected to contribute to the achievement of these objectives. The Bank supported Indonesia in infrastructure development through investment lending, and in improving policy environment for infrastructure project development through development policy lending. The Bank has also provided technical assistance in reforming electricity tariff and subsidy regime, establishing incentives for geothermal resource development and capacity building (ICR, p.8). This project was the continuation of the first Indonesia Power Transmission Development Project. Therefore, the Bank had sufficient experience in the sector and the country, and the project objective was adequately challenging. Rating Relevance TBL Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To meet growing electricity demand in the project area through strengthening and expanding the capacity of the power transmission networks in the project area in a sustainable manner. Rationale Theory of Change The project had a direct causal chain: the project inputs, i.e., loans, would finance the expansion, upgrading, and construction of a total of 114 substations to address the transmission bottlenecks. Most of the substations were operating over 80 percent of their capacity that was higher than the PLN technical standard of 60 percent (ICR, p.7). Load shedding was common, and the duration of outages was long. The expected intermediate outcome was an increase in the transmission capacity in the project areas. Because of the increase in the transmission capacity, the electricity availability would be expected to increase to meet growing demand. The causal pathways from inputs/activities to outputs and to outcomes were valid and direct, although the outcomes achieved could not be fully attributed to the project’s intervention because of the following reasons: (i) the project’s theory of change was based on two critical assumptions that the generation capacity would be sufficient to supply electricity to the system, and that the electricity grid would have the capacity to distribute electricity to the end-users; and (ii) although the outcome indicators were broadly relevant, i.e., increase in energy sales, to capture the achievement of the project objectives to meet growing demand, energy sales were influenced by other factors, such as economic growth, technological advances, and tariff changes, making it difficult to establish full attribution between the project outputs and expected outcomes (ICR, p.16). Additionally, the PAD does not define “a sustainable manner.” This can be interpreted as the sustainable operation of these substations because of the operation and maintenance (O&M) capacity of the PLN, but the project did not finance any activity to further develop the O&M capacity of the PLN. The theory of change does not support the achievement of this aspect of the project objective, either. Outputs 1. Under Group 1, 37 substations were expanded, rehabilitated, or constructed as planned.  19 substations in Java-Bali  7 substations in Sumatra  5 substations in Kalimantan  4 substations in Sulawesi  1 substation in Amlapura and 1 substation in Nusa Dua/Sanur Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) 2. Under Group 2, 41 substations were expanded, rehabilitated, or constructed as planned at the second restructuring.  18 substations in Java-Bali  16 substations in Sumatra  4 substations in Kalimantan  3 substations in Sulawesi Outcomes  The incremental capacity of substations in Group 1 increased by 2,040MVA against the original target of 2,010MVA.  The incremental capacity of substations in Group 2 increased by 1,680MVA falling short of the revised target of 2,290MVA. The above-listed intermediate outcomes, i.e., increase in substations capacities, were to result in an increase in the availability of electricity to meet increasing demand. This outcome was measured by the amount of electricity sales (in GWh) in the project areas:  The electricity sales in project areas for Group 1 increased from a baseline of 10,000GWh to 14,109GWh, which was lower than both the original and revised targets of 18,580GWh and 16,000GWh, respectively.  The growth in electricity sales in project areas for Group 2 in Java-Bali was 136.4 per cent against the revised target of 117.5 percent.  The growth in electricity sales in project areas for Group 2 in East Indonesia was 126.2 per cent against the revised target of 120 percent.  The growth in electricity sales in project areas for Group 2 in West Indonesia was 152.0 per cent against the revised target of 120 percent. Overall, the project was substantially successful in achieving the output and outcome targets. The increase in energy sales in Group 1 project areas was lower than the target, despite achieving the target for substation capacity increase. On the other hand, the growth in energy sales in all Group 2 project areas was higher than the revised targets, despite the achievement of capacity increase was lower than the target. Other factors had a significant impact on the energy sales increases. In Group1 project areas, the growth of electricity demand was lower than the estimates because the economy was driven mostly by agriculture and service sector that consume less power than industrial sector. On the other hand, the growth in electricity sales in Group 2 project areas was higher than the estimates mostly because of relatively faster economic growth, especially in Jakarta, Central and East Java (ICR, p.32). Overall, the project was substantially successful in achieving the project objective to meet growing electricity demand in the project areas, but due to weaknesses in establishing full attribution between project outputs and outcomes and the weaknesses in the theory of change, the efficacy of the project in achieving the project objective is rated modest. Rating Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) Modest OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Overall, the project was substantially successful in achieving the project objective to meet growing electricity demand in the project areas, but due to weaknesses in establishing full attribution between project outputs and outcomes and the weaknesses in the theory of change, the efficacy of the project in achieving the project objective is rated modest. Overall Efficacy Rating Primary Reason Modest Insufficient evidence 5. Efficiency Economic Analysis At appraisal, a “with project” and without project” economic analysis was conducted for each of the six project implementation regions, i.e., Java-Bali, Sumatra, South Sulawesi, South Kalimantan, West Kalimantan, and Central Kalimantan. The economic benefit of the project was calculated as the value of all electricity (rather than incremental increase in the amount of electricity available to the consumers because of the project’s intervention) sold at the willingness-to-pay of unconnected consumers. The cost of self-generation by private, commercial, and industrial consumers using gasoline, diesel, and furnace oil, respectively, were used to calculate the willingness-to-pay at the medium-voltage and low-voltage levels. The economic analysis resulted in a range of economic internal rate of returns (EIRR) estimated for six regions from 42 percent in West Kalimantan to 60 percent in Central Kalimantan (PAD, p.9). The weighted average of these EIRRs was 55 percent (ICR, p.45). The economic analysis conducted at project closing resulted in a lower weighted EIRR of 46 percent. The lower electricity demand than estimated at appraisal and delays in project implementation were the main reasons for a lower EIRR at project closing. Financial Analysis At appraisal, a financial analysis of ONEA was conducted and the utility was found to be profitable because of the public service obligation (PSO) subsidies it received from the government to cover the shortfall between electricity tariffs and electricity supply costs. In 2015, PSO subsidies constituted 21 percent of PLN’s revenues (ICR, p.42). The ICR (p.41) reports that, at project closing, the project’s financial internal rate of return (FIRR) was calculated at 17 percent based on the following assumptions: annual electricity growth of five percent and annual electricity tariff increase of three percent. These assumptions and calculations are identical to those in the first Indonesia Power Transmission Development Project. Administrative and Operational Efficiency Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) Project efficiency was adversely affected by the change in government priorities following the 2014 presidential election. The PLN’s focus shifted from transmission investments to the construction of 35GW power generation capacity, a target set by the new government. This delayed the preparation of the list of substations for Group 2, which was submitted to the Bank along with feasibility studies two years after loan effectiveness. However, because of decreased electricity demand increase forecasts due to economic downturn, the uncertainty about which transmission infrastructure should urgently be built, and the limited time left to complete all substation investments, PLN decided to cancel most of the investments under Group 2 (ICR, pp.11 and 23). This resulted in the cancellation of US$145.4 million of the loan (about 45 per cent of the total loan of $325 million). Completion times of various contracts were routinely underestimated resulting in numerous time extensions (ICR, p.24). The procurement process was inefficient (ICR, p.27). The time required from bid opening to processing the advance payments to contractors took more than one year, which was too long and delayed project implementation (see section 10.b Procurement below). Prolonged internal processes and reviews of invoices by various directorates within the PLN and the Ministry of Finance delayed payments to contractors in the range of 42 to 133 months slowing project implementation (ICR, p.24). Various changes to technical specifications, too, contributed to the delayed completion of the procurement process and signing of the contracts (ICR, p.24). As a result of the reorganization of the PLN in 2014, the decision-making processes for the project were delegated to regional offices, but they did not have sufficient capacity to implement the project. This created a more complex bureaucratic process for contract management (ICR, p.22). The data flow from regional offices to the project management unit was not smooth. This complicated project coordination and also contributed to the delay in the preparation of the substations list to be included in Group 2 (ICR, p.23). Taking the above factors into account, the efficiency of the project in achieving the project objectives is rated Modest. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 100.00 Appraisal  55.00  Not Applicable 100.00 ICR Estimate  46.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project objectives were highly aligned with the Bank strategy at project closure and Indonesia’s development status, and it was adequately challenging given the Bank’s prior experience in the country and the sector. Hence, the relevance of objectives is rated High. The efficacy of the achievement of the project objective Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) to meet growing electricity demand is rated Modest because of weak attribution between the project activities and the achievement of expected outcomes. The efficiency of the project in achieving the project objectives is rated Modest, because of significant administrative and operational inefficiencies and weak methodology used in calculating the project’s economic benefits. Overall, the Outcome of the project is rated Moderately Unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome The risks identified to the development outcome of the first Indonesia Power Transmission Development Project equally apply to the development outcome of this project. Financial and technical constraints of the PLN, including its organizational deficiencies, pose a risk for the operation and maintenance(O&M) of the substations financed by the project. The transmission system in the country has been expanding rapidly, which can stretch the technical and financial capabilities of the PLN to maintain the system. If PLN’s financial situation worsens, that could weaken its O&M capabilities. Historically, PLN had been allocating three to five per cent of capital expenditures for O&M, and it is expected to continue with this practice in the future. If PLN does not continue with the expansion and strengthening of the transmission and distribution systems, the substations financed under the project might not be able to meet the electricity demand in the event of a rapid demand increase. The ICR states that the demand for electricity has been lower than the estimates at appraisal, but it is highly likely that demand for electricity will increase in the future, given that Indonesia has a large population and a potential for fast economic growth. Therefore, continuous investment to increase transmission capacity is needed, in the absence of which outages may reoccur and growing electricity demand may not be met because of the overloading of the existing substations. 8. Assessment of Bank Performance a. Quality-at-Entry At project entry, the goals of addressing the transmission bottlenecks through the expansion of the transmission substation capacity to meet the growing electricity demand of consumers and increasing access to electricity were of high strategic priority. The project’s intervention was straightforward: expansion, rehabilitation and construction of substations would increase the transmission capacity and more electricity would be available to consumers and allow more people with access to electricity. However, the project design did not include any activity to expand the transmission and distribution grids or increase connections to households or businesses. The project design did not support the achievement of the objective to increase access to electricity. This was a significant shortcoming in the formulation of the project objective. Technical feasibility studies of the 37 substations in Group 1 were Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) reviewed and approved at appraisal, and these projects were ready for implementation. However, although the 71 substations to be financed under Group 2 were known at appraisal, a final list was not prepared at appraisal due to absence of detailed technical feasibility studies. The finalization of this list was delayed significantly causing project implementation delays and cancellation of a large amount of project loans. The economic analysis was adequate to justify the financing of the project, but the economic benefit was defined as the monetary value of all electricity in the system rather than the incremental increase in the electricity amount as a result of the project’s intervention. There were some shortcomings in the monitoring and evaluation (M&E) design, but it was sufficient to measure the outcomes of the project’s intervention, which consisted of scattered investments in a large project area. The project implementation arrangements were simple: the PLN was a reputable utility with sufficient capacity and experience gained in similar projects to implement the investment activities supported by the project. The overall risk assessment was adequate, but mitigation measures for the procurement risk and design risk related to Group 2 projects were not sufficiently identified that led to cancellation of around US$167 million of the loan amount. Because of the significant shortcomings in the identification and preparation of the project, the quality at entry is rated moderately unsatisfactory. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision According to the ICR (p.28), supervision missions were held approximately every eight months, but during the last two years of project implementation, formal missions were held every four months. The project team prepared 11 Implementation Status and Results Reports (ISRs); they were candid in reporting the project implementation issues and the actions to be taken. Procurement issues and contract management constituted a major problem during implementation, contributing to project implementation delays and extension of project closing date. To improve project implementation and closely monitor the project implementation, the Bank’s project team mobilized a senior power engineer during the last 18 months of project implementation. Additionally, monthly meetings were held with PLN during the last 12 months of project to monitor project progress. These increased supervision efforts resulted in an improvement in project implementation and effectiveness of coordination between regional offices, PLN headquarters and the Bank’s project team (ICR, p.28). At project closing, all substations were energized, except one in Group 2. The project team’s supervision of fiduciary and safeguard aspects of the project was adequate. Because of the implementation delays of around two years after effectiveness and the delays in identifying substations under Group 2, the project team’s focus was more on the completion of project activities and the achievement of outputs rather than the outcomes of the project’s intervention; the shortcomings in the M&E system were not addressed to fully capture the project’s outcomes. The factors outside the control of the project team also necessitated a greater focus on the achievement of project outputs rather than outcomes, such as the change in government priorities towards generation, complex project implementation because of the reorganization of PLN delegating decision-making processes to regional offices that did not have sufficient project implementation capacity, and numerous technical specification variations. Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) Overall, the quality of Bank supervision is rated Moderately Satisfactory. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project objectives to meet growing electricity demand and to increase access to electricity were clearly defined, but the definition of “a sustainable manner” was not given in the project documents. The formulation of the project objectives did not match the activities the project was to finance. The causal pathways in the results chain were plausible for the achievement of the first objective to meet growing electricity demand. However, the project’s theory of change did not support the achievement of the objective to increase access to electricity. There was no project activity to achieve this objective. Hence, there was no indicator in the results framework to capture its achievement. The outcome level indicators of “electricity sales” and “growth in electricity sales” in project areas were proxy indicators to capture the project outcomes in meeting growing demand for electricity. However, a full attribution between these indicators and the achievement project outcomes could not be established, because there were other factors that could affect energy sales, such as tariff changes, economic downturn, and improved efficiency of electrical appliances. The indicators were selected in accordance with the PLN reporting system, therefore, indicators that would better capture the achievement of project outcomes could not be included in the results framework, such as the number of new customer registrations, the increase in customers’ electricity consumption, and customers’ applications to increase the power capacity (ICR, p.47 and footnote 11 on the same page). The indicators capturing the project’s intermediate outcomes, i.e., increases in transmission capacity, were specific, measurable, and relevant. b. M&E Implementation The PLN reported the progress in construction works quarterly, but with delays (ICR, p.25). During the first year of the implementation, the reports did not include the indicators in the results framework. This was because of the reporting format used by PLN, which was later revised to include the measurements of the indicators in the results framework. However, the quality of the data did not satisfy the requirements of the Bank; “[t]he data, however, was not provided in a structure that is consistent or in the level of granularity requested by the Bank” (ICR, footnote 31, p.26). The weaknesses in the M&E system in fully capturing the project outcomes were not corrected during implementation. The target values of indicators for Group 2 projects, such as the increase in transmission substation capacity, were reduced to match the change in project scope because of the cancellation of most of the substation projects in that group. Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) c. M&E Utilization M&E findings were communicated to project stakeholders. M&E data were used to provide evidence of application of inputs and achievement of outputs, but due to the shortcomings in the M&E design, the achievement of outcomes could not be fully captured. The M&E findings led to the downsizing of project scope and cancellation of almost 45 per cent of the loans. M&E findings are not expected to influence subsequent interventions in the near term. Overall, the M&E quality is rated modest. There were significant shortcomings in the design and implementation of the M&E system making it difficult to adequately assess the achievement of the project objectives and test the links in the results chain. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as Category B under Environmental Assessment (OP/BP 4.01) and triggered the Indigenous Peoples (OP/BP 4.10) and the Involuntary Resettlement (OP/BP 4.12) safeguard policies. Environmental Assessment (OP/BP 4.01): At appraisal, the impact of the substation projects was assessed to be modest, non-sensitive and reversible, such as increased level of dust and noise from the use and movement of machinery, non-toxic solid waste during construction; oil spill or leakage from machinery or transformers; and worker health and safety from existing electromagnetic fields (PAD, pp.23- 24). No long-term adverse impact was expected due to investment activities. PLN prepared an Environmental Management Plan (EMP) for expansion and rehabilitation of substation projects under Group 1, EMPs for two new substations to be built in Group 2 and an Environmental and Social Management Framework (ESMF) for the expansion and rehabilitation of other substation projects to be included in Group 2. The EMPs and ESMF were disclosed on PLN’s website in March 2013 and in the Bank’s InfoShop in April 2013. The PLN had a dedicated unit responsible for the project’s compliance with environmental and social safeguard policies. At project appraisal, this unit had already been implementing the safeguard policies under ongoing two Bank-financed transmission projects (PAD, p.26). During project implementation, ESMF was used to screen substation projects in Group 2 and EMPs were prepared when required. The ICR (p.26) states that “[t]he project was overall in compliance with environmental and social safeguards procedures.” Indigenous Peoples (OP/BP 4.10): Although there were no indigenous people present in or collectively attached to the project areas for the substation investments in Group 1 and the candidate substation investments to be proposed under Group 2, this safeguard policy was triggered in case the PLN could propose new substation projects in areas where indigenous people were present. The PLN prepared an Indigenous People Planning Framework (IPPF) to guide, if necessary, indigenous people development plan for substation projects in Group 2. The IPPF was disclosed on the PLN’s website on March 2013 and in the Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) Bank’s Infoshop in April 2013. During project implementation, no indigenous people were affected by the project activities because the investment works consisted of the expansion and rehabilitation of existing substations on PLN’s own premises. Involuntary Resettlement (OP/BP 4.12): This safeguard policy was triggered because the construction of two new substations in Group 2 would require land acquisition. Therefore, at project reparation Land Acquisition and Resettlement Action Plans acceptable to the Bank were prepared and they were disclosed on the PLN’s website on March 2013 and in the Bank’s Infoshop in April 2013. Because of the cancellation of most of the substation projects in Group 2 and the PLN’s decision to finance the construction of those two new substations using its own resources, the investment works consisted of the expansion and rehabilitation of existing substations on PLN’s own premises; hence, there was no land acquisition or involuntary resettlement during project implementation. b. Fiduciary Compliance Financial Management PLN submitted interim financial reports to the Bank regularly. The audit reports were unqualified and submitted according to the schedule. As was the case in first Indonesia Power Transmission Development Project, there were lengthy delays in payments to the contractors in this project, too. The ICR (p.24) lists the reasons as follows: (i) delay of the annual project budget approval by the Ministry of Finance (MoF); (ii) delays in submission of the supporting documents from regional project offices to the PLN headquarters; (iii) delays in the regional director’s clearance of payments; and (iv) lengthy verification of the invoices and documents by the MoF. The processing of invoices from submission by contractors until the payments were made by the Bank was in the range of 42 to 133 days, which was overly lengthy. The PLN failed to comply with one of the financial covenants, i.e., debt service coverage ratio of 1.5. Waivers from this covenant were granted until project closing. There were no known issues of corruption or misuse of funds associated with the project. All project funds were fully accounted for at project closure. Procurement The procurement process for the substation projects in Group 1 was inefficient. The evaluation of bids took more than three months until a bid evaluation report was submitted to the Bank for review. Another five months were required after the Bank’s no-objection to issue the notification of the award and sign the contract. The delays in the issuance of the contractor’s guarantee postponed the issuance of the advance payment to the contractor by another five months from the contract signing date. The whole process took more than one year adversely affecting the project implementation schedule. Various changes to technical specifications of the substations contributed to delayed completion of the procurement process and signing of the contract. Because of the differences between the procurement guidelines of the Bank and the PLN, the Package 2 in Group 2 could not be financed by the project. According to the PLN’s procurement guidelines, the award price of a contract must be below the cost estimated for the scope of that contract. The only bid for Package 2 was higher than the cost estimate of $22 million. Therefore, PLN decided to enter not price negotiations with the bidder, which was against the Bank’ guidelines. Consequently, that package was excluded from the project scope and $22 million from the loan amount was cancelled. Since the procurement of Package 2 did not comply with the Bank’s procurement guidelines, the PLN’s spending of $22 million could not be recognized as the borrower’s contribution to the project (ICR, footnote 15, p.12). Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) The procurement process of the Group 2 substation projects was more efficient. The bid evaluation process was completed within the 60-day bid validity period (ICR, p.28). Overall, the procurement was in compliance with the Bank’s guidelines, except Package 2 in Group 2, which was later cancelled, but the procurement process was slow and mostly inefficient. c. Unintended impacts (Positive or Negative) None. d. Other None. 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome Unsatisfactory Unsatisfactory Because of the significant shortcomings in the identification and preparation of the project, the quality at entry is rated moderately unsatisfactory. The quality of supervision is rated moderately satisfactory. According to the Bank guidance (p.23), when the assessment of Moderately Moderately Bank Performance one dimension is in the Satisfactory Unsatisfactory satisfactory range, and that of the other dimension is in the unsatisfactory range, overall Bank performance depends on the outcome rating. Since the outcome rating is moderately unsatisfactory, the Bank performance is rated moderately unsatisfactory. Quality of M&E Modest Modest Quality of ICR --- Substantial Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) 12. Lessons Uncertainty about the investments to be financed under the project could adversely affect project implementation and result in lower achievement and cancellation of loans. The project was to finance substation projects under two groups. The technical feasibility studies of Group 1 substation projects had already been reviewed and approved during project preparation, and they were ready for implementation. However, the technical feasibility studies of the Group 2 projects were to be prepared after project start, and the project was to finance these projects upon the Bank team’s appraisal of each one of them against an agreed set of eligibility criteria and acceptance for financing. However, the finalization of the list of projects for Group 2 was delayed by more than two years. As a result of changing government priorities, the project implementing agency, i.e., PLN, cancelled most of the substation investments under Group 2 that led to a cancellation of a big part of the project loan—around $145 million. On the other hand, the 37 substations projects under Group 1, which were ready for implementation at project start, were completed and all substations were energized two years before the extended project closing date. While acknowledging that "[i]t was a major and fast shift in sector and specific investment priories that led to significant revision and reduction of identified and otherwise quite certain investments," the project team commented in their email dated June 28, 2021 that "emphasis on uncertainty, which by design is embedded in framework type operations, may be slightly misleading as a lesson from this operation." A weak M&E system can fail to fully capture the achievement of the project outcomes or establish full attribution between the project activities and outcomes. The M&E system was designed in accordance with the PLN reporting system. The intermediate results indicators were sufficient to capture the increase in the transmission substation capacity. However, the outcome indicators, i.e., increase or growth in electricity sales” were insufficient to capture the project outcomes in meeting growing electricity demand in project areas, because other factors affected the achievement of these indicators, such as tariff changes, economic downturn, and efficiency in increase in appliances. Indicators, such as the number of new customer registrations, the increase in customers’ electricity consumption, and customers’ applications to increase the power capacity, were not included in the results framework. The indicators did not have the specificity and granularity to capture project outcomes and were insufficient to test the links in the results chain resulting in weaker attribution. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provides a comprehensive overview of the project. The report is sufficiently evaluative. The narrative is candid in explaining the project’s shortcomings. There has been sufficient interrogation of the available evidence. The report is focused on what occurred as a consequence of the project. The narrative mostly Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Second Power Transmission Development (P123994) supports the ratings. The report is internally consistent; there is a logical linking and integration of the various parts of the report. The report follows the majority of the Bank guidance. The section on key factors that affected implementation and outcome provides detailed information about the issues emerged during project implementation. However, the discussion of project efficiency does not include an assessment of the project’s administrative and operational efficiency. The information in the M&E Utilization section is insufficient. The Lessons and Recommendations section consists mostly of findings rather than lessons. There were numerical errors in the tables; for example, the original target of the indicator of “commissioned capacity of transformer substations for Group 2” was 4,130MVA, but the ICR report it as 1,680GVA, which was the actual achievement. The ICR is substantially longer (30 pages) than recommended in the Bank guidance (15-20 pages). a. Quality of ICR Rating Substantial Page 17 of 17