Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) Report Number : ICRR0020605 1. Project Data Project ID Project Name P096207 POWER TRANSMISSION Country Practice Area(Lead) Ukraine Energy & Extractives L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-48680 30-Jun-2012 238,000,000.00 Bank Approval Date Closing Date (Actual) 02-Aug-2007 30-Jun-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 200,000,000.00 0.00 Revised Commitment 193,821,974.72 0.00 Actual 193,821,974.72 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ihsan Kaler Hurcan George T. K. Pitman Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives In 2006, the Government of Ukraine requested the World Bank’s support for the implementation of its multiphase long-term Energy Sector Reform and Development Program (Energy Program 2005-2012), particularly through investment lending. Such support was provided by the World Bank with a series of Specific Investment Loans (SILs) which were focused on priority investments with a strong ownership on the Government’s part and a clear commitment to technical and governance excellence on part of the implementing agency. The first SIL supported the Hydropower Rehabilitation Project and was approved by the Bank in June 2005. The Power Transmission Project is the second SIL to support the Energy Program. Page 1 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) The Loan Agreement (p.5) states that the Project Development Objectives were: “(a) to improve the security, reliability, efficiency and quality of power supply through the rehabilitation of transmission substations and the strengthening of the power transmission network; and (b) to improve the institutional capacity and technical capabilities of the Project Implementing Entity to ensure a secure and reliable operation of the high voltage power grid and facilitate the unimpeded operation and opening up of the electricity market.” The Project Appraisal Document (PAD, p.7) defines the same objectives with slightly different wording in the main text as follows: “The main objective of the Power Transmission Project is to improve the security, reliability and quality of power supply through the rehabilitation of transmission substations and the strengthening of the power transmission network. The project will also aim to improve institutional capacity and technical capabilities of transmission system operator, UkrEnergo (UE), so that it can assure secure and reliable operation of the high voltage power grid, and, therefore, facilitate unimpeded operation and opening up of the electricity market.” Following IEG procedures, this ICR Review is based on the project development objectives as stated in the Loan Agreement. b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The project consisted of five components. A: Rehabilitation of transmission substations. (Appraisal cost: US$157.8 million; Actual cost: US$187.8 million). This component included replacement of outdated high voltage equipment and installation of modern protective relaying and substation automation systems in (i) three 330 kV substations in the Dnieper electric power grid; (ii) one 330 kV and one 220 kV substation in the Donbas electric power grid; and (iii) five 750 kV substations. Because of significant savings, UE suggested adding additional packages to the Procurement Plan, which was approved by the World Bank in 2014. As a result, the number of 750 kV substations rehabilitated under Component A increased from five to seven. B: Strengthening of transmission network. (Appraisal cost: US$47.0 million; Actual cost: US$31.5 million). This component included expansion of the existing 330 kV Bar substation and construction of the Page 2 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) 72 km long 330 kV transmission line to connect Bar substation with the Dniester Hydropower Pumped- Storage Plant which had been under construction. C: Stabilization of the Crimea Electric Power Grid. (Appraisal cost: US$23.2 million; Actual cost: US$9.5 million). This component included upgrading of the existing 220 kV Simferopol - Sebastopol transmission line and associated substations to the 330 kV voltage level, and upgrading of substation control systems at 330 kV and 220 kV substations in the Crimea electric power grid. Because of slow progress with the effectiveness of the project during the first year after its approval, the Bank agreed to the financing of the 220 kV Simferopol-Sevastopol line upgrade through UE’s own funds. The loan proceeds, initially set for upgrade, were allocated for the rehabilitation of the Simferopol substation. D: UE Institutional development. (Appraisal cost: US$8.7 million; Actual cost: US$5.9 million). This component included establishment of a corporate wide Management Information System (MIS) in UkrEnergo (UE). This component also included provision of technical assistance to UE in (i) capacity building in procurement and project management; (ii) designing and implementing corporate-wide MIS; and (iii) annual auditing of project and company accounts. An Asset Revaluation Study was included in 2011, and implementation of the MIS activity was moved to the Second Power Transmission Project. E: Implementation of the Grid Code. (Appraisal cost: US$1.3 million; Actual cost: US$0.0 million). This component included provision of technical assistance to UE in implementing the Grid Code and related regulations and norms of the Wholesale Electricity Market. In 2012, the World Bank agreed to exclude the ‘Implementation of the Grid Code’ package from the Procurement Plan. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The total project cost was originally estimated at US$238.00 million. Actual project cost was US$234.56 million. There were several adjustments of costs among the components as shown in the Components section above. Financing: US$193.82 million (97%) of the IBRD Loan of US$200.00 million was disbursed. US$6.18 million was cancelled at project closing. Borrower contribution: The Borrower agreed to contribute US$38.0 million at appraisal. The Borrower’s actual contribution was US$40.74 million. Dates: It took more than one year after the signing of the Loan Agreement for the project to be effective and this initial delay was exacerbated by the occurrence of force majeure in 2014. The frequent government changes, loss of Ukrainian control over Crimea and the conflict in Eastern Ukraine had Page 3 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) significantly affected project implementation. One of the Donbas Power Grid contracts which was implemented in the conflict zone could not be completed because of security considerations. The modernization of 330 kV Simferopol substation in Crimea was terminated because the substation was no longer accessible. In order to mitigate the impact of the force majeure on the electric system, UE had to maintain the operation of the Ukrainian Power System and did not allow the shut-down of substations in non-conflict zone which was necessary for the installation of new equipment. Worsening economic conditions in Ukraine also resulted in the bankruptcy of a contractor and all works at the substation in Dniprovska Grid were stopped. The loan closing date was extended three times. 1. For 2 years 6 months from June 30, 2012 to December 31, 2014 because of the slow start of the project and also the seasonal nature of the rehabilitation works which could only be done during the summer seasons of 2012, 2013 and 2014. 2 . For 1 year from December 31, 2014 to December 31, 2015. The main reason for this extension was the stoppage of works in Crimea and Eastern Ukraine because of the appearance of force majeure. The total number of years for which the project was extended after the first and second extensions being more than 3 years, Bank Management’s approval was taken before the granting of the second extension. 3 . For 6 months from December 31, 2015 to June 30, 3016 for the completion of pending works under five contracts. 3. Relevance of Objectives & Design a. Relevance of Objectives In order to address the problems in the energy sector, including those in the aging high voltage transmission network, the government of Ukraine (GoU) developed a multiphase long-term Energy Sector Reform and Development Program (Energy Program) and asked for Bank’s support to finance the rehabilitation of transmission system and some capacity building measures. The project development objectives are relevant to country conditions, because they are closely aligned with the objectives of the Energy Program of the GoU which are “to stimulate sustainable economic growth and investments in Ukraine by (i) improving the security, reliability, efficiency and quality of energy supply at reasonable cost; and (ii) supporting the country’s aspirations with regard to legal and technical harmonization and increasing integration of its energy market with the EU Internal Energy Market” (PAD, p.19). The project development objectives also remain relevant to the Bank’s Country Partnership Strategy for FY 2012-16. They correspond to Pillar 1 and Pillar 2 of the strategy which aim at, in addition to other outcomes, improving governance in the energy sector and performance of power sector. The objectives were also in line with the main priorities of the Bank’s country strategies at appraisal FY2004-2007 and during 2008-2011. Page 4 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) Ukraine continues with the rehabilitation of its power transmission system in the follow-on Second Power Transmission Project which is also funded by the Bank. The follow-on project also aims at building UE’s capacity and implementing the Grid Code. Therefore, the objectives of the project are still relevant to the current priorities of the GoU. Rating High b. Relevance of Design Although broadly defined, the project objectives were clearly stated and they were linked to the intermediate and final outcomes, particularly to those in the investment components of A, B and C, which constituted 97.5 per cent of the actual project cost. Refurbishing and modernization of selected 750 kV substations on the transmission line which crosses the country from West to East was aimed at decreasing the amount of load shedding caused by insufficient substation capacities. Furthermore, rehabilitation of five critical 330 kV substations in Dnieper and Donbas regions would help eliminate the bottlenecks during peak-loads. Construction of a 72 km long 330kV transmission line and the expansion of the 330 kV Bar substation specifically targeted the connection of the then under-construction Dniester Hydropower Pumped-Storage Plant to the national grid. And finally, the upgrading of control systems in all substations in Crimea and, as per the revised scope of this component, the rehabilitation of the Simferopol substation would improve the electricity distribution in Crimea which heavily depended on electricity transmitted from the mainland. The causal chain between the funding of these activities and the achievement of outcomes, namely improving the security, reliability, efficiency and quality of power supply in Ukraine was clear. The institutional strengthening and policy reform objectives of the project were rather ambitious and although a clear causal chain could be established between the objectives and project activities as defined in components D and E, these components had to be restructured in order to accommodate the changes in GoU’s priorities and policies during the implementation. The Management Information System (MIS) contract of Component D was moved to Second Power Transmission Project and the implementation of Grid Code package was excluded from the Procurement Plan for Component E, which was the only one related to the policy reform objective of the project. The problems faced in the design of the project caused by these components are also acknowledged by the ICR in the “Lessons Learned” section. The results framework at appraisal defined the project outcomes as to “support sustainable economic development” by improving power transmission and to “support market reforms in the energy sector” (PAD, p.41) The PDO outcomes were also defined and the use of these outcomes were clearly listed as: (1) Ukraine’s energy intensity (use of energy per unit of GDP) is reduced by 10% compared to 2005. (2) An increasing number of eligible electricity consumers are free to choose their electricity supplier; (3) Ukraine is better prepared to meet UCTE requirements related to the security and reliability of its power transmission system, and increase its electricity trade with the EU; and (4) UE’s organization and operating practices are compatible with transmission system operators in the EU Internal Electricity Market. The Results Framework Page 5 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) had to be restructured couple of times during project implementation to better align it with project activities and objectives (ICR, para. 64). Overall, there was a substantial causal chain in the design of the project between the activities and the outcomes, especially on the investment side of the project, and the intended outcomes were clear and convincing. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To improve the security of power supply. Rationale Outputs* • Rehabilitation of Pivdenna and Dniprodzerzhinska 330kV substations in the Dnieper Power Grid was completed. The contract for the rehabilitation of Pershotravneva 330 kV substation in the same grid was terminated due to the bankruptcy of the contractor. • Rehabilitation of Azovska 220 kV and Vinnistkaya 750 kV substations in the Donbas Power Grid was completed. Rehabilitation of Chaikine 330 kV substations, which is located in the Eastern Ukraine conflict zone in the same grid, could not be completed. • Out of 130 high voltage circuit breakers for 750 kV substations 124 were installed, but 6 belonging to Donbasskaya 750 kV Substation could not be installed due to the close location of the substation to the conflict zone in Eastern Ukraine. • All of the equipment for protective relaying and automation for 750 kV substations were installed. • Construction of the 72.90 km long 330 kV Bar-Dniester transmission line was completed in April 2013. The transmission line is operational. • Expansion of the 330 kV Bar Substation was completed. • Upgrading of substation control systems at 330 kV and 220 kV substations in Crimea was fully completed in 2013. • Modernization of Simferopol 330 kV substation in Crimea was terminated due to loss of Ukrainian control of the territory. * These outputs apply to Objectives 1, 2, 3 and 4. Outcomes Page 6 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) • Total net injected generation into the transmission system increased to 139.8 GWh from 138.2 GWh falling short of the target value of 155.8 GWh which was added in 2012. However, the Crimean Grid and the conflict zone in Eastern Ukraine were not included in the calculation of the actual net injected generation amount. • The ICR (para.66) states that the maintenance and repair costs were reduced by US$0.8 million per year. This was listed as a key outcome indicator. In order to achieve security of power supply, there should exist extra generation capacity compared to the expected demand for electricity in the system and necessary infrastructure must be in place to bring electricity, regardless of where it is generated, to end-users when demanded. Ukraine has ample generation capacity compared to demand and is a universal access country, but due to the bottlenecks in the transmission system caused by aging high voltage transmission network, electricity is not always available when needed. And the Ukrainian generation mix, where hydropower and natural gas have a share of 5% and 7%, respectively, (International Energy Agency, Statistics, Ukraine, 2014) makes it difficult to meet the peak demand. To address these problems related to the security of power supply, the project implementing entity UE completed the rehabilitation of most of the substations included in the project and constructed a 72.9 km long transmission line connecting the Dniester Hydropower Pumped-Storage Plant to the national grid. Connection of this hydropower generation system to the natural grid contributed to the increase in the fast- responding peaking capacity in the Ukrainian power system. The bottlenecks in the transmission system were partially eliminated and the amount of energy-not-served (ENS), which is the electricity that could not be served to the consumers due to failures in the transmission system, was decreased by at least 35 GWh/year from its baseline value of 747.3 GWh/year. Furthermore, a partial increase in total net generation injected was also observed. These achievements all support improved security of power supply. However, the occurrence of force majeure negatively affected the achievement of this objective. The decrease in the operation and maintenance (O&M) costs by US$0.8 million per year will partially decrease the budgetary pressure on UE in O&M. Nevertheless, the financial viability of UE needs to be further improved to be able to conduct O&M activities adequately in order to assure the operation of the grid without any technical problems. (Please see the discussions in UE’s financial viability under Implementing Agency Performance and Fiduciary Compliance sections.) Ukraine has also embarked upon an ambitious reform program of transforming its transmission network and legislation in order to integrate it with the European Network of Transmission System Operators for Electricity (ENTSO-e). This policy orientation can be seen as a step to secure power supply by importing electricity from the EU if need arises. This will also allow Ukraine to export its surplus generation to the EU region. In this regard, the ICR also argues that as a result of the activities supported by the project, the Ukraine Unified Power System is ready to operate under the requirements of the ENTSO-e (ICR, para.67). At the meeting with the project team, it was clarified that the substations rehabilitated under this project were technically ready to operate under the ENTSO-e. On the other hand, since the whole Ukrainian system is not technically compatible with the ENTSO-e, the Ukrainian grid’s switch to ENTSO-e has not been achieved yet. Page 7 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To improve the reliability of power supply. Rationale For Outputs please see Objective 1. Outcomes • The number of outages at two substations, Pivdenna and Dniprodzerzhinska, were reduced to zero from 29 at each substation. The number of outages at Azovska Substation was reduced from 8 to 4, which is lower than the target value of 6. Although the contractor ceased all works at Pershotravneva Substation due to insolvency, the number of outages at this substation was reduced from 28 to 12 which is lower than the target value of 13. Measures cannot be taken for the Chaikine Substation which is located in the conflict zone in Eastern Ukraine. The simple definition of power supply reliability is that power is available whenever it is needed. Thus, number of interruptions in power supply, i.e. outages, is a good measure of power supply reliability. These can be easily observed at the substation level and a good comparison can be made between the values before and after the project intervention. As a result of the substation rehabilitation activities, the number of outages at two substations were eliminated and at another substation the number of outages was decreased from 8 to 4, which is lower than the target value of 6. In the Pershotravneva Substation the number of outages fell to 12 from 29, although the works at this substation were stopped due to the bankruptcy of the contractor. On the other hand, no data could be collected from the Chaikine Substation which is located in the conflict zone. The reduction in the ENS by at least 35 GWh/year also supports the achievement of this objective. Rating Substantial PHREVDELTBL PHEFFICACYTBL Page 8 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) Objective 3 Objective To improve the efficiency of power supply. Rationale For Outputs please see Objective 1. Outcomes •　Transmission losses were reduced by 14 MW achieving the target revised in December 2014. Original target was reducing transmission losses to less than 2.5%. The ICR does not give an explanation why the target was changed from a per cent value to an absolute value. Project Papers do not provide any explanation, either. The project team commented that the end target for reduction of transmission losses had always been set as an absolute value of 14 MW (PAD, p.41). However, the restructuring paper (April 2012, p.3) gives the target as "transmission losses reduced to less than 2.5%". In the following restructuring (December 2014, p.7-8), the end target was revised from 2.5 % to 14 MW, but the comment section of the indicator reads that "end target date aligned with the extension of closing date." It does not give an explanation why it was revised back to 14 MW. The baseline figure was kept as 3.5% until project closing. •　ENS was reduced by at least 35GWh/year achieving the target. •　Electricity losses, defined by the project team as both commercial losses and technical losses per year in two project areas consisting of Dniprovska Grid and South-western Grid, were reduced to 1.67% which is better than the target value of 1.70% revised in September 2015, but higher than the original target value of 1.65% which was added as a PDO indicator in September 2012. However, the baseline value of 1.96% includes electricity losses in Dnieper Grid and the Crimean Grid, as well. By the closing date, measurements could not be taken in these two grids due to force majeure. Therefore, it is not clear how comparable the baseline value is with the achieved value. Furthermore, there is no explanation in the ICR why the target value was revised to 1.70% from 1.65% less than four months before the closing date. (When the revision was made, the closing date of the project was December 31, 2015.　 The closing date　 was extended by another six months to June 30, 2016 in December 2015.) Regarding the last point, the project team provided the following explanation: "This change was introduced due to the events of 2014-2015. The loss of access to the Crimean Grid, to some parts of the Lugansk and Donetsk regions, as well as difficulties with assessing the losses in the Dnieper Energy system (some of which were located in the conflict area) made the precise measurement of losses in all 4 grids impossible. As a result, UE has managed to fully measure losses only in 2 grids (the Dniprovska energy system and South-Western energy system) and requested to reduce the final target accordingly." •　Technical electricity losses per year in the project area were reduced to 1.64% which is a better achievement than the target value of 2.00% added in 2012. The baseline value was 2.53%. On the other hand, it is not clear how the technical loss baseline figure of 2.53% could be higher than the baseline transmission losses value of 1.96%, including both commercial losses and technical values. The project team commented that "the technical losses baseline figure of 2.53% can indeed be higher than the electricity losses value of 1.96% which included both commercial losses and technical losses due to the way UkrEnergo assesses and measures these parameters in the system". 　 Efficiency is related to the electricity losses during generation, transmission and distribution of electricity. During transmission and distribution, electricity losses happen in the wires and also in the substations. Since Page 9 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) the project activities did not include any transmission line rehabilitation, but were restricted to the rehabilitation of some selected substations, the improvement in electricity losses should be easily attributable to the substation rehabilitations. (The construction of a new transmission line connecting the Dniester PSP to the Bar Substation would improve the efficiency of the system, but its impact would be lower compared to the impact of the rehabilitated substations.) 　 The project’s initial target was to reduce transmission losses to less than 2.5 percent from a baseline value of 3.4 percent as indicated in the PDO Indicator 2. This baseline value of 3.4% does not match the baseline values of transmission losses of the Indicators 5 and 7, which are 1.96% and 2.53%, respectively. The ICR states that this target was modified and the new target was defined as reducing transmission losses by 14 MW in 2014. The ICR claims that this target was achieved, but does not provide any evidence on how this value was calculated. The project team subsequently provided information that reduction of peak load capacity requirements by 14 MW was originally defined by project consultants who when developing the project feasibility study, defined how much of peak load capacity could be reduced through investments under the Project. The project team further commented that with the construction of Dniester–Bar Transmission Line and the reconstruction of the Bar Substation, 1.91 MW of peak-load capacity and with the upgrading of substation controls in the Crimea 12.5 MW of peak-load capacity reduction were achieved. AS a result, the project team claims that total reduction in peak–load capacity amounted to 14.41 MW. 　 Technical transmission losses are defined as "natural losses from the resistive characteristics of all system components, especially conductors, as well as hysteresis losses from transformers." (Core Sector Indicators and Definitions, December 2012, p.105.) However, the project team commented that the project’s technical loss indicator was a "customized indicator". This was defined by UE as overall technical losses in the project area and this technical loss included additional factors related to emergency situations in the project area (not only substations that were under rehabilitations as part of the project, but all substations in the project area) and some other factors affecting losses. The baseline value for technical losses was 2.53 percent. The target value was 2 percent which was introduced in 2012.　The achievement was 1.64 percent. The ICR states that project-end measurements excluded two grids, namely the Crimean and Donbas Grids, where it was not possible to collect data due to the loss of Ukrainian control over Crimea and ongoing conflict in Eastern Ukraine. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 4 Objective To improve the quality of power supply. Rationale For Outputs please see Objective 1. Page 10 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) Outcome • Voltage quality was within acceptable range of +/-5% of the nominal voltage in all substations included in the project. Quality of power supply is directly related to the voltage stability of the electricity and closely related to the reliability of power supply. The rehabilitation works completed at the substations are expected to have a direct impact on the quality of power supply. There is only one indicator included in the results framework about voltage. The voltage was outside the accepted range of the nominal voltage, which is +/-5 percent, in several substations. These substations are not listed in the ICR. The target was to bring the voltage within acceptable range in all substations included in the project. It was informed by the project team that the UE had provided data about voltage variation at each rehabilitated substation which were within the acceptable range. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 5 Objective To ensure a secure and reliable operation of the high voltage power grid. Rationale Outputs • Establishment of a corporate-wide MIS in UE could not be achieved in this project. However, consultancy services were provided to define the terms for the MIS package to be implemented in the follow-on Second Power Transmission Project. • Further consultancy services were provided for capacity building at UE in procurement and project management and for annual auditing of project and company accounts. Outcomes • Completion of the rehabilitation works and the construction of a new transmission line resulted in a more secure and more reliable operation of the high voltage power grid, but this sub-objective aimed at improving the operation of the grid by improving the institutional capacity and technical capabilities of UE. Establishment of a corporate-wide MIS in the UE was the project output to this end. This activity could not be achieved under this project and it was moved to the Second Power Transmission Project. However, consultancy services for defining the terms for the MIS package to be implemented in the follow-on project Page 11 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) were provided. In addition, consultancy services were provided for capacity building at UE (which led to an increase in the efficiency of UE internal procedures) demonstrated itself in the UE's handling of FM and procurement activities. Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 6 Objective To facilitate the unimpeded operation and opening up of the electricity market. Rationale Outputs • Implementation of the Grid Code is excluded from the procurement package and WEM is deferred to the Second Power Transmission Project. Outcomes • Gradual power market opening could not be achieved because the new draft law targeted full market opening upon approval of the law by the Parliament. The law was approved by the Parliament and became effective on June 11, 2017. • Market rules for the implementation of Wholesale Electricity Market (WEM) were developed. The ICR claims that the target is partially achieved. However, the original target was to develop the Grid Code in 2008 and start regularly implementing it in 2010. This target was revised in September 2015 and it read “[d]evelopment of market rules, regulatory, and legal framework of new energy market model.” This activity was defined under Component E, but there was no procurement made under this component. When the GoU decided to harmonize the electricity sector legal framework with the EU 3rd Energy Package in 2015, the Grid Code which had already been prepared by the ENRC according to the EU 2nd Energy Package, could not be put into implementation until the approval of the new electricity law by the Parliament. Therefore, the objective of opening up of the electricity market could not be achieved under this project and it is expected to be achieved under the Second Power Transmission Project. However, the ICR claims that “[this] objective was achieved through financial and technical assistance to UE and the GoU in preparation of the regulations and provisions of the WEM and in shifting from the single buyer model to a new model of the electricity market based on bilateral contract market, balancing mechanism, and `day-ahead` and `intra-day` markets” (ICR, p.17). This activity was also included in the Hydropower Rehabilitation Project which had started before the Power Transmission Rehabilitation Project. The project team later commented that procurement had been made under the Hydropower Power Project to achieve this target and since both projects had supported the GoU’s Energy Program, the target of the relevant indicator had been deemed to Page 12 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) be met under the Power Transmission Project. The strengthening of UE's financial viability was targeted through the financial covenants in the Loan Agreement. A financially more viable UE would be operated more efficiently, which would result in the unimpeded operation of the electricity market. However, UE failed to meet the current ratio covenant, and it was later replaced by self-financing ratio. Therefore, it is not clear whether the financial covenants were instrumental in improving the financial viability and the efficient operation of UE. Rating Negligible PHREVDELTBL PHREVISEDTBL 5. Efficiency Economic Analysis A ‘with project’ and ‘without project’ approach was used to carry out the economic evaluation of the project. At appraisal, the project’s economic benefits were assessed under four categories: i) reduction in costs of maintenance and repairs; ii) reduction in ENS; iii) reduction in transmission losses; and iv) savings in electricity supply costs. First three categories were used at the project completion. Fourth category could not be used due to lack of relevant data. The calculations resulted in an economic internal rate of return (EIRR) of 14.0% at project completion, compared to the estimate of 18.2% at appraisal. The economic net present value (ENPV) at project completion was US$42.8 million at 10% discount rate and US$129.7 million at 6% compared to US$122.9 million at appraisal using 10% as the discount rate. It should be noted that the exclusion of the fourth category in calculating these figures due to lack of available date could have had a downward effect. The loss of transmission assets in Crimea and the slow-down of economic activity in Ukraine which caused a decline in the amount of electricity transmitted by UE could have had an adverse impact on the ENPV calculation, too. Lastly, a longer project implementation period had negatively affected the project efficiency. These calculations covered the activities completed under the A, B and C Components of the project which constituted around 97.5 percent of the actual project cost. Financial Analysis No project-level financial analysis was carried out during appraisal. Power transmission in Ukraine is a regulated activity, therefore, UE is allowed to earn a regulated rate of return on its investments taking into account prudent costs involved in their implementation and operation. This resulted in annual profits which permitted the UE to realize a substantial and increasing return on equity (PAD, p.16-17; ICR, p.30). Page 13 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) Operational and Administrative Efficiency The actual implementation period of nine years, which was partly caused by a slow start due to the poor performance of the consultant and partly by the conflicts in Crimea and Eastern Ukraine, was significantly longer than what was estimated in the appraisal which was four and a half years (PAD, p.51). The project was closed under budget, but some activities could not be completed. The Bank and UE were able to shift loan savings in the amount of US$20 million from partial cancellation of contracts in Crimea and Eastern Ukraine and from the devaluation of the local currency to rehabilitate additional substations (ICR, p.11). Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 95.80 Appraisal  18.20 Not Applicable 97.50 ICR Estimate  14.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The relevance of objectives is High and that of design is Substantial. The security, reliability, efficiency and quality of power supply are rated Modest, Substantial, Substantial and Substantial, respectively. Institutional capacity and policy reform activities which aimed at improving the operation of the high voltage system and opening up of the electricity market, respectively, fell short of their targets, because of the changing policy priorities of the GoU and the political instability in the country. These last two objectives are rated Modest and Negligible, respectively. The occurrence of the force majeure negatively affected the achievement of project objectives. The effect of other power transmission projects, which were implemented simultaneously, on the outcomes of the Power Transmission Project is not clear. (The ICR, on page 6, states that “[t]he PDO was broad ad included together with the outputs, the program-level outcomes, for which the project could only be partly held accountable, given the fact that the project was the one, among others, supporting an energy program carried out in Ukraine at the time.”) The ICR fails to provide sufficient evidence to establish a clear causal chain between the inputs, outputs, outcomes and the achievement of the project objectives. The ICR falls into the trap of judging the outcomes by indicators without supporting it with strong evidence and/or detailed explanation. Efficiency of the project is rated modest. Thus, the overall project outcome is rated Moderately Unsatisfactory. Page 14 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) a. Outcome Rating Moderately Unsatisfactory 7. Rationale for Risk to Development Outcome Rating Financial viability of UE stands out as a major risk that might materialize and negatively impact the project outcomes which is closely related to the economic activity in Ukraine and government’s commitment to the energy reform. During poor economic performance, the government might opt out to minimize tariff increases on essential services, including electricity, due to social concerns which in return would worsen the financial situation of UE. Inadequate financial resources might result in poor maintenance of rehabilitated substations. However, the GoU started tariff reforms and plans to bring tariffs to cost recovery level by 2017 (ICR, p.20). The annual cap imposed by legislation on the IFI financing amount which UE is eligible to receive might prevent UE from sustaining the benefits of the project. If not provided with enough financial resources, UE cannot continue with its rehabilitation and investment program (ICR, p.20). The possibility of an escalation of conflict in Eastern Ukraine poses another risk which might circumvent the ability of the GoU and UE to continue with the Energy Program. This might shift the budgetary priorities from the Energy Program and might also prevent UE’s from continuing with the rehabilitation program due to technical problems stemming from the continuous use of the system in non-conflict areas as was the case in 2014 and onwards (ICR, p.20-21). Continued commitment of the GoU to the energy program and specifically to the electricity market reform is crucial in maintaining the outcomes of the project. Approval of the Electricity Market Law has been a strong signal that the GoU is committed to reforming the electricity market. Not only would this open up the market, but also harmonize it with the European Union system. a. Risk to Development Outcome Rating Modest 8. Assessment of Bank Performance a. Quality-at-Entry The project benefitted from Bank’s extensive experience in Ukraine and the energy sector, including power transmission projects in the Commonwealth of Independent States and the EU member countries. Experience of the Bank in energy reform in Ukraine (Electricity Market Development Project) and lessons learned from similar power projects, such as the Ukraine: Hydropower Rehabilitation and System Control Project, and Page 15 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) Kazakhstan: Electricity Transmission Rehabilitation Project helped in building the project during appraisal. The project was strategically relevant with the main objective of the GoU’s Energy Program. The project was also consistent with the Bank’s sector priorities and the Country Assistance Strategy. Technical, economic and financial analyses were sound. Safeguards and fiduciary were adequate and mitigation measures were properly designed and consistent with the World Bank’s fiduciary role. Implementation arrangements were defined in detail. Noting that it was critical to have an implementing entity with advanced technical skills, strong project ownership, and commitment to results, UE had demonstrated such aspects under a prior power operation by the time of appraisal. However, because of the significantly larger scope of the Power Transmission Process, it was required that an experienced international consultant should be selected to assist UE in project management, scheduling, procurement and contract management. Selection of such a consultant under Terms of Reference acceptable to the Bank was a condition of loan effectiveness. On the other hand, the project objectives, especially those related to technical assistance, were broad and ambitious. An implementation period of 4.5 years estimated at appraisal was not realistic for such a rehabilitation project involving technical assistance components. At appraisal, the project indicators were not defined to directly monitor and evaluate the project progress and the achievement of PDOs, and new indicators had to be introduced as the projects progressed to reflect the scope of the project and its activities better. Risks, such as vanishing political commitment and deteriorating financial viability of UE, which caused delays, materialized during implementation, but there is not sufficient explanation in the ICR that the measures defined in the PAD were effective in mitigating these risks. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The project team was primarily based in Kiev throughout the project implementation. The team had a constructive relationship with the Borrower which contributed to timely identification and resolution of risks to the achievement of PDOs. Supervision missions were held regularly. Some of the missions had multiple agenda and were assisted by supervision missions from other projects implemented in the energy sector in Ukraine. Such joint missions created a synergy and improved the ability to target the issues, with which the Ukrainian counterparts were faced, with more sophisticated approaches. The key members of the project team, such as the TTL and economist/financial analyst/ procurement specialist, were with the project for most of the project implementation starting from Board presentation which had a positive impact on supervision quality. Although the mid-term review was held two years beyond schedule due to the slow start of the project, recommendations given in the mid-term review helped speed up the project implementation. Special attention was given to increase UE’s institutional capacity in managing procurement packages and handling Bank- financed projects in terms of procurement and financial management. Problems in procurement, caused by the low contract management capacity of the consultant, were overcome by a plan to mitigate procurement risks and to strengthen implementation capacity. Intensive training given to the project management unit of UE resulted in an improvement in its procurement capacity under the Bank’s guidelines. Page 16 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) Implementation of safeguard measures were supervised properly. Shortcomings in quality of safeguards reporting, especially those related to OP/BP 4.12 were noted on a timely basis and necessary support was given to UE for the implementation of the safeguard policy and the preparation of the detailed Land Acquisition Report. The Bank intensified its supervision over the implementation process after repeated failures of UE in reporting delays in the implementation of several contracts. As a result, a more frequent reporting obligation was introduced. The Bank was also quick to support UE in utilizing loan savings in the amount of US$20 million (accumulated because of partial cancellation of works due to force majeure and the depreciation of local currency) to procure two new contracts for the rehabilitation of additional substations. The Bank’s supervision performance was found satisfactory by the Borrower due to i) regular supervision missions and adequate skill-mix of the staff; ii) provision of timely responses to project issues; iii) actively involving the field office in maintaining close working contacts with local project counterparts; and iv) the application of realistic project performance ratings. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance Although the project was designed to support the GoU’s Energy Program by providing finance for the rehabilitation of aging substations, it took more than one year for the GoU to ratify the LA and pass it through the Parliament after the signing of the LA. (The LA was signed in November 2007, but ratified in October 2008 and approved by the Parliament in December 2008.) During the course of the project implementation there had been six government changes which had a negative impact on GoU’s commitment to the project. There were also frequent changes of management at the Ministry of Energy and Coal Industry (MoECI). As a result, the pace of the project slowed and, at times, there was lack of responsiveness from the MoECI to UE’s financial and organizational constraints. GoU delayed approving tariffs, processing of project titles and passports, and disbursements. There were delays in getting permission from the Ministry of Finance to increase the budget allocations to UE through which the Bank’s finance was provided for the project. This had caused no payment to contractors and hence delayed project implementation. Couple of years prior to the closing date of the project, there had been some efforts on both the UE and the Ministry of Finance sides to cooperate more proactively on solving the problems stemming from underfinancing. The GoU’s approach to energy reform agenda was inconsistent. The Grid Code was prepared and ready for implementation after the approval of the electricity market in 2013, but it could not be implemented due Page 17 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) to the introduction of a new electricity market legislation to the Parliament for approval. The GoU failed to maintain an acceptable pace of the energy reform. Political commitment was lacking and also there were problems in coordinating the stakeholders for the implementation of the WEM concept. It should also be noted that the occurrence of conflict in Eastern Ukraine and loss of Ukrainian control over Crimea adversely affected GoU’s commitment to the project. However, to expand the activities of the project and to build on its lessons and achievements, GoU requested the Bank to consider a follow-on project to further improve the reliability of the power supply in Ukraine and support the implementation of WEM. The Second Power Transmission Project became effective in June 2015. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance The investment part of the project which constituted 97 percent of the total actual cost was implemented by UE. UE struggled with handling the preparation of bidding packages, and the selection of the consultant took significantly more time than expected. These resulted in a significant delay in project start. Low initial contract management capacity of the consultant worsened the situation. Even by March 2013 the disbursement rate was minimal and signing of contracts was only starting to happen. The UE also failed to report delays in the implementation of several contracts. Specifically, the introduction of the MIS could not be achieved due to the failure of the UE in procuring and executing the contract for the MIS by the closing date, and this made the M&E implementation somewhat challenging. UE could not comply with the financial covenant in the Loan Agreement and, therefore, the current ratio was replaced with the self-financing ratio. The fourth restructuring enabling those amendments was approved on September 1, 2015. On the other hand, when the project started gaining pace after the launch of the bidding process and signing of first contracts, the UE’s commitment to and ownership of the project visibly increased. The company provided adequate management and staffing to achieve project outcomes. Towards the end of the project, UE managed to utilize the loan savings to procure additional equipment within a very short period of time. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization Page 18 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) a. M&E Design Although the objectives were stated clearly, they were broad and included outputs in the wording. The initial indicators could partly reflect the objectives. Reduction of ENS by 35 GWh per year and transmission losses by 14 MW were given as the only two quantifiable indicators in the results framework, but in “Table 9. Arrangements for results monitoring” (PAD, p.42) power market opening and implementation of WEM concept were given as the PDO indicators with baselines. These two indicators were later dropped because of the change in GoU’s policies. There was also one result indicator defined in the same table for each component. Frequency of reporting, data collection instruments and the party responsible for data collection were clearly defined in the results monitoring table. Appraisal clearly defined the responsibilities of UE in monitoring and evaluating project progress and in reporting, and also how the mid-term reporting would be carried out. However, the M&E framework had to be improved during the course of the project. b. M&E Implementation Four years after the start of the project, more indicators were added to the results framework to monitor project implementation and the achievement of PDOs. The indicators, especially for the investment portion of the project, were measurable in terms of numbers, timing and location. There were also baselines established for monitoring. Once indicators were established, sector specific measurements were used to monitor the progress. For example, the ENS analysis was performed by using the reliability module of PowerFactory - a software which uses a sequential method to evaluate the need for load shedding for each network configuration based on the failure statistics. UE and other project implementing entities submitted data through the lifetime of the project. The quality and reliability of reporting by the implementing agencies had to be reinforced by consultants and external auditors. After the cancellation of the consultancy contract by UE, the Bank’s team stationed in Kiev were more involved in the timely collection of data. While establishment of MIS at the UE was included as a project component which would improve the project implementing agency’s internal monitoring capabilities, this component could not be realized during this project and it is deferred to the Second Power Transmission Project. Lack of internal capacity to supervise and monitor transactions led to difficulties with project management and UE repeatedly failed in reporting delays in the implementation of several contracts. As a result, a more frequent reporting obligation was introduced by the Bank to better supervise the project implementation. c. M&E Utilization The project was faced with unprecedented changes during the course of its implementation, such as the conflicts in Crimea and Eastern Ukraine, frequent government changes and economic downturn. Under such difficult circumstances, M&E findings forced the Bank and UE to change the scope of works especially in technical assistance components. Information regarding the procurement and execution of contracts was used to follow the progress and to address the implementation issues as they arose (ICR, p.12). The findings of the project led to the follow-on intervention of Second Power Transmission Project which became effective in June 2015 before the closing of this loan. Page 19 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) M&E Quality Rating Substantial 11. Other Issues a. Safeguards The project was classified as a Category ‘B’ under OP/BP 4.01 (Environmental Assessment) and OP/BP 4.12 (Involuntary Resettlement) was triggered. Environmental Assessment OP/BP 4.01: Key issues related to the rehabilitation of substations included dust, noise, and disposal of wastes and used equipment in the construction phase, and electric magnetic field and noise in the operation phase. In addition to these key issues, access road construction and working camp wasters were identified as environmental concerns for the construction of a new transmission line between Hydro Pump and Bar Station. Public consultations were held at all three project regions where substations were later rehabilitated. UE prepared Environmental Management Plans and they were published on UE’s web site and also the English versions are placed in the InfoShop. The first review of the borrower’s compliance with OP 4.01 was carried out in May 2013 and full compliance with the safeguard was confirmed (ICR, para.54). Involuntary Resettlement OP/BP 4.12: This safeguard was triggered due to land acquisition for the footprints of transmission towers along the 330 kV Bar-Dniester transmission line, and rights of way below the line and in the safety zone. Less than 2.0 ha was to be acquired for approximately 280 transmission towers, spaced over 77 km. Each of the acquired parcels was very small, 52 m2 or 81 m2, depending on the type of transmission tower. A Land Acquisition Policy Framework (LAPF) was prepared by the UE and the document was placed in the InfoShop. However, during the May 2013 Bank mission, the quality of the Land Acquisition Report, which claimed that the provisions of the LAPF were fully complied with during the land acquisition process, was found unacceptable to the Bank. A local consultant was hired to help the Borrower to improve the quality of the report acceptable to the Bank. In March 2014, such a report was presented to the Bank and the relevant personnel at the PIU received an orientation with regard to the provisions of OP/BP 4.12 (ICR, para.54). Overall, the project was rated Satisfactory from the safeguards compliance point of view (ICR, para.54). b. Fiduciary Compliance Financial Management During the implementation of the project, special attention was given to the increase of UE’s institutional Page 20 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) capacity in managing procurement packages and handling Bank-financed projects in terms of procurement and financial management. The Borrower’s and UE’s existing institutional arrangements constituted the backbone of financial management (FM) of the project, which was supported by the purchase of additional software in 2014. Periodic Bank reviews of UE’s FM procedures were conducted. The Borrower’s FM team was adequately staffed and accounting and reporting systems were in place as well as the internal control systems and auditing arrangements. Although rated Satisfactory between 2010-2012, the project’s FM arrangements were rated Moderately Satisfactory until project closure because of below listed reasons (ICR, para.57): 1. Irregular Statement of Expenditure reporting, resulting in the inactivity of the special account for several months; 2. Delays with the submission of the revised Project Operation Manual; 3. Delays in submission of the audit reports (except for the timely submission of the 2015 audit reports in 2016); 4. The accounting system still to be put in active use, and needed fine tuning to allow automated generation of reports; and 5. Periodic delays in sufficient allocation of funds in the state budget. No misuse of funds was discovered during the implementation of the project (ICR, p.8). At the time of the writing of the ICR, all payments were made and the project audit was in progress (ICR, para.57). Two financial covenants were formulated in the Project Agreement which were i) annual current ratio of not less than 1.2; and ii) annual debt service coverage of not less than 1.5. UE was able to meet the first covenant during the implementation of the project except in 2014 and 2015. It was 1.3 in 2015 compared to the covenanted ratio of 1.5. On the other hand, UE had serious problems in meeting the current ratio due to changes in the national accounting system and a mismatch between the funds available to UE and the payments to be made at year-end. The current ratio was realized between 0.4 and 1.0 compared against the covenanted level of 1.2. Since UE was able to fulfill its payment obligations with some delay beyond the end of year, which is not reflected in the current ratio, the Bank agreed in 2015 to replace the current ratio covenant with self-financing ratio which UE had been successful in meeting (ICR, p.33-34). The ICR does not offer comments on external auditing, except in Annex 7: Summary of Borrower’s ICR and/or Comments on Draft ICR, where it reads “UE will also pursue improvements in their FM and accounting practices based on the auditors’ recommendations received during project implementation” (ICR, para.96). Procurement The Bank’s guidelines were followed for procurement and hiring of consultants. During the initial years of the project implementation, there were problems with the timely preparation of procurement packages because of the low contract management capacity of the consultant. This led to project delays. To assist UE with tackling procurement issues, a procurement capacity assessment was carried out by the Bank’s team in UE in February 2011. A plan to mitigate the procurement risks and strengthen implementation capacity was agreed with UE. During the following months, intensive training was conducted to help the PIU of UE improve its capacity in procurement under World Bank guidelines. This led to visible improvements in the PIU’s ability to handle Page 21 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) multiple contract packages. All contracts were subject to prior review of the Bank, and frequent implementation reviews were carried out which included procurement accredited staff. Starting from 2012 until loan closure, the project was rated Satisfactory for procurement (ICR, para.58). c. Unintended impacts (Positive or Negative) None. d. Other None. 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Attribution between a number of the project outputs and outcomes cannot be clearly established. Occurrence of the force majeure in Crimea and Eastern Ukraine, coupled with Moderately Moderately Outcome political instability in the Satisfactory Unsatisfactory country, negatively affected the achievement of the objectives. Efficiency is also rated Modest. Hence the Outcome rating is Moderately Unsatisfactory. Risk to Development Modest Modest --- Outcome ICR's rating of Quality at Entry is Moderately Satisfactory and the rating of Quality of Moderately Bank Performance Satisfactory Supervision is Satisfactory. Satisfactory Therefore, the overall ICR's Bank Performance rating is Moderately Satisfactory. Government Performance is rated Moderately Moderately Moderately Borrower Performance Unsatisfactory while the Satisfactory Unsatisfactory Implementing Agency Performance is rated Page 22 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) Moderately Satisfactory. Given the Outcome rating is in the Unsatisfactory range, Borrower Performance is rated Moderately Unsatisfactory as per the harmonized guidelines. Quality of ICR Modest --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons The ICR presents nine lessons of which IEG selects three as most important. Combining an ambitious sector reform program with significant investment activities in a fragile political economy carries high risks. In these circumstances a single focus infrastructure project may be the most efficacious choice. This risk is higher in countries where the borrower has a low track record of sector reforms. One of the objectives of this project was to support the GoU to achieve the opening up of the electricity market. This was a rather ambitious objective for a project which primarily aimed at rehabilitating a part of the aging transmission system. As a result of political instability and frequent government changes, GoU’s commitment to the sector reform decreased during the course of the project and both the implementation of the Grid Code and the WEM concept could not be achieved. The experience level of the procurement/project management consultant is crucial for a successful start and implementation of the project. In this project, the selection of the consultant company was slow and the company lacked sufficient experience in managing procurement in line with the Bank’s rules. This led to delay in concluding contracts which consequently resulted in a longer project completion period. On-the-job training is crucial in improving the implementing agency’s capacity to comply with the World Bank procurement and financial management standards and procedures. The World Bank worked closely with UE to assist in handling procurement and FM, which resulted in the upgrade of the project’s ratings in procurement and FM to Satisfactory and maintaining those ratings until the closing date. The support has also created a solid foundation for the implementation of the follow-on project, the Second Power Transmission Project. 14. Assessment Recommended? Page 23 of 24 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review POWER TRANSMISSION (P096207) Yes Please explain The Bank has been supporting Ukraine’s Energy Program through a series of projects, such as the Hydropower Rehabilitation Project, the Power Transmission Project, the Second Power Transmission Project and a possible Kyiv Pumped-Storage Plant Project, since 2005. An assessment of these projects would help IEG to better understand how a series of such interventions has been instrumental, or not, in achieving Bank’s development objectives. There might be some very important lessons to be learned, especially about the impact of these projects on demand side behavioral changes. 15. Comments on Quality of ICR The ICR is concise and informative. The Annexes are well prepared, especially Annex 3, but typos are common in Annex 7. The ICR provides a candid presentation of shortcomings in Bank and Borrower performance, and the results framework. It is consistent with OPCS guidelines. However, the ICR is more of an implementation narrative rather an outcome-driven explanation of the project. The ICR fails to avoid the trap of judging outcomes by indicators. The efficacy section could have benefitted from a discussion going beyond the indicators, and the M&E section, likewise, from a more detailed explanation of the use of M&E data. The discussion about financial strengthening and policy reform activities and their impact on achieving project objectives is confusing. Lessons are not clearly articulated and sufficient evidence and analysis are not presented. A discussion of the experience gained by implementing a project under a major force majeure situation could have been included in the Lessons Learned section. There were a few internal inconsistencies. The actual value achieved at completion for Indicator 6 on page iv is given as 139.8 MWh, whereas the same indicator reads as 155.8 MWh in Paragraph 66 on page 16. The Bank Performance rating is Satisfactory in the Data Sheet, whereas it is Moderately Satisfactory in Paragraph 88, p.22. There are inconsistencies in the date formats. a. Quality of ICR Rating Modest Page 24 of 24