ICRR 12729 Report Number : ICRR12729 IEG ICR Review Independent Evaluation Group 1. Project Data: Date Posted : 10/26/2007 PROJ ID : P050891 Appraisal Actual Project Name : Elec Sectr Ref US$M ): Project Costs (US$M): 70.30 44.73 Country : Russian Federation Loan/ US$M ): Loan /Credit (US$M): 40.00 37.93 Sector Board : EMT US$M): Cofinancing (US$M ): Sector (s): Power (80%) Central government administration (20%) Theme (s): Regulation and competition policy (29% - P) State enterprise/bank restructuring and privatization (29% - P) Infrastructure services for private sector development (14% - S) Rural services and infrastructure (14% - S) Access to urban services and housing (14% - S) L/C Number : L4181 Board Approval Date : 06/05/1997 Partners involved : Closing Date : 12/31/2000 12/31/2006 Evaluator : Panel Reviewer : Group Manager : Group : Robert Mark Lacey Peter Nigel Freeman Alain A. Barbu IEGSG 2. Project Objectives and Components: a. Objectives: The objectives of the Project, as stated in the Loan Agreement, were to assist in (a) refining and developing the Russian electricity sector reform program; (b) implementing the said reform program; and (c) commercializing the Russian electricity sector companies . The project was a Technical Assistance (TA) operation, and was to finance mainly the provision of consultancy services to various entities supervising and operating the Russian electricity supply industry . During implementation, there were substantial reallocations away from consultancy services and in favor of goods procurement (see below). The electricity sector reform program was also supported by a Structural Adjustment Loan (SAL). b.Were the project objectives/key associated outcome targets revised during implementation? No c. Components (or Key Conditions in the case of DPLs, as appropriate): The project was restructured in February, 2000. Between Board approval in early June 1997 and this date, no disbursements had taken place, and the project was rated as Unsatisfactory by all supervision missions. The reasons given for the restructuring were to enhance national ownership of the main elements of the project, and to improve the linkage to the infrastructure monopoly and electric power reform component of the Third SAL. The restructuring did not alter the project’s development objectives and was not submitted for Board approval but authorized by Management. Eight months after the restructuring, the project was rated Satisfactory, and disbursements had started by June, 2001. The components after restructuring were: Part A: Consulting services and goods procurement for the restructuring and commercialization of the nationwide holding company, Russian Unified Electric Systems Company (RAO “UES Rossii�), including the holding company itself and its subsidiaries, the Power Generating Companies, and the Regional Energy Companies. The component also financed support for the development and operation of a wholesale electricity market, public relations and legal services in the holding company and the Project Implementation Unit (PIU), located in RAO UES. The ICR does not provide the estimated cost of each component at restructuring, but at closure this one was US$ 34.88 million (78 percent of total project costs). Part B: Consulting services, procurement of office equipment, and training for the regulatory entity (initially the Federal Agency Commission, FEC, subsequently the Federal Tariff Service, FTS). This cost US$8.43 million, 19 percent of the total. Part C: Consulting services and training for the Ministry of Fuel and Energy (MFE) and its successors, US$1.42 million (3 percent of the total). After restructuring, the bulk of the technical assistance was directed towards the power utility holding company and its subsidiaries; there was also a reallocation of the loan proceeds in favor of goods procurement and away from consultancy services; multiple special accounts and international competitive bidding were introduced. There was a decrease in the previously approved amount for local shopping and a parallel rise in international sourcing . Later, there were two further reallocations of funds away from consulting services towards procurement of goods – one of US$2 million in August, 2002, and another of US$4.1 million in February, 2004. d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost. Total project costs at closure were US$44.73 million, about 36 percent lower than the US$70.3 million estimated at appraisal . Unfortunately, it is not possible to comment meaningfully on the sources of this difference, due to the lack of a cost breakdown in the ICR of the restructured components at restructuring. Part of the cost reduction may have been due to the restructuring itself, which reduced or eliminated some of the original components. Financing. The IBRD loan of US$40 million was to have financed 57 percent of the total estimated project cost. At closure, US$37.93 million of the loan had been disbursed, representing 85 percent of the total. The government’s contribution, estimated at US$12.1 million at appraisal, was in fact US$ 6.8 million, reflecting partly the reduced project cost and partly the lesser domestic expenditure share in procurement after restructuring. A contribution of US$18.2 million was expected from other donors at appraisal . In the event, this was not forthcoming. In February 2004, US$1.52 million of the loan, allocated for Parts B and C, was cancelled at the request of the Ministry of Finance since less funding than originally thought would be required. A further US$0.37 million were cancelled in September, 2005. At closure, US$0.21 remained undisbursed. The total amount cancelled was US$2.1 million. Dates. Although the project became effective as foreseen in December 1998, no disbursements took place until mid 2001. At the time of restructuring in February 2000, the closing date was extended by only one year, until December 31, 2001. A further two years’ extension was granted in August 2001, until December 31 2003, and then again, for one year, until December 31, 2004. When it became clear that many of the government’s priority actions would still not be completed by then – including procedures to attract and guarantee private investment, reorganization of the energy companies, and the development of guidelines for the operation of wholesale and retail markets in electricity – a fourth extension was granted until December 31, 2006, when the project was finally closed. 3. Relevance of Objectives & Design: Relevance of objectives and design was substantial . The project’s objectives were highly relevant to the needs of the Russian economy and power sector, and responded to the key themes of the Bank’s priorities in the Russian Federation, including sustaining rapid growth and improving delivery of social and communal services. The power sector reforms supported by the project were aimed at commercialization and at the attraction of badly needed private investment, especially in generation capacity . These remain priorities today. The technical design of the project was also highly germane . However, design provided for the implementation of the project over a three-and-a-half-year period. This would have been highly ambitious for such a complex operation in the best of circumstances . In the event, resistance to the reforms from powerful vested interests, and then the political, economic and financial crises which engulfed the Russian economy from late 1997 led to long implementation delays. Project implementation took nearly ten years to complete. Design, while responding to what was seen as a window of opportunity, did not take sufficient account of the power of opposing vested interests or of the prevailing political and macroeconomic conditions . 4. Achievement of Objectives (Efficacy): The degree of attainment of the project’s objectives is substantial . Objective (a) -- refining and developing the Russian electricity sector reform program – was substantially achieved. The blueprint for reform developed during project preparation was technically robust, and was taken up again when a reform-minded government finally adopted the power sector reform program in July, 2001. The project, however, had been restructured in February 2000 to make it relevant to changed circumstances and in order to increase national ownership . Objectives (b) and (c) – implementing the reform program and commercializing the electricity sector companies – are considered together since they are closely linked dimensions of implementation . They can conveniently be broken down into the following sub -objectives: (i) unbundling, privatization, restructuring, and commercialization of electricity sector entities and companies; (ii) development of a pilot electricity wholesale market; (iii) reforming the regulatory framework for the electricity sector; and (iv) strengthening the sector supervisory Ministries . Sub-objective (i) was substantially achieved, although not all the information required to make an assessment is available in the ICR . According to the latter, most of the preconditions for unbundling have been put in place and most vertically integrated entities are set, in principle, to disappear . RAO UES, in which the government owns 52 percent of the shares, and which possesses 100 percent of the national transmission grid as well as being a substantial shareholder in the large generating and regional electricity companies, is scheduled for dissolution by mid 2008. Prior to this, most of its generating subsidiaries are supposed to be spun off and privatized (the national transmission grid will remain state -owned. The ICR recommends that an evaluation be carried out two or three years after project closure to determine the extent to which unbundling and privatization have been completed . Subsequent information provided by the Region, however, indicates that the reform program has moved further than the ICR indicates – two generation companies have been fully privatized, and most of the remaining generation and distribution companies will be privatized within a few months . The Region also confirms that there are definite plans for dissolution of RAO UES by July 2008, which is verified by information in the company ’s website. There has been important progress in restructuring and commercialization : RAO UES has introduced modern business practices, and is now profitable (though profits did decline during 2006, as tariffs failed to keep pace with increasing operating and investment costs ). All revenues are now collected on a cash basis, compared with only 16 percent before the project. Methodology and techniques for appraisal of the market value of business performance and assets of both RAO UES and the regional power supply companies have been developed and successfully implemented in the course of the companies ’ restructuring. This enables the enterprises to operate for profit on a business basis, and has been accompanied by parallel mechanisms for attracting and guaranteeing private investment . The corporate image of RAO UES and its subsidiaries was assessed, and amendments to the Federal law on joint stock companies were prepared to cover the conditions prevailing in “natural monopoly� sectors. RAO UES shares, as well as those of some of its subsidiaries, are already traded on Russian and European stock exchanges . Sub-objective (ii) was substantially achieved. A pilot wholesale market for electricity began operating in 2003. The Operating Center for managing and improving the operation of both wholesale and retail markets was established in 2005. According to RAO UES, 21 percent of all electricity sales in 2005 took place in the free trading segment of the wholesale market, and 79 percent in the regulated segment. The proportion traded freely was expected to increase significantly in the following years since all electricity produced by new generating companies would be sold in that segment of the wholesale market . By contrast, accomplishment of sub-objectives (iii) and (iv) was negligible. The regulatory framework and institutions in the sector are still weak . On the positive side, such institutions are operating, though with limited autonomy. An information technology system has been put in place to exchange information and enable common methods of tariff setting on a national basis. However, the fundamental principles of independence and transparency have not been incorporated into the governing legislation . Like all the project’s objectives, regulatory reform was supported, not just by this operation, but by Structural Adjustment Loans as well. In this case, though, the more limited technical objective of helping the regulator design an improved operating framework, even within the limits of existing legislation, was not achieved. The institutional history of regulation during the project period was checkered . The Authorities abolished the previous regulatory agency, the Federal Energy Commission (FEC), and proposed to transfer its competencies to a Ministry . Only after intensive dialogue with the Bank was the successor institution, the Federal Tariff Service (FTS), established, albeit with a more limited mandate and less authority than its predecessor. While the ICR reports pressure on the Government’s part to keep tariffs low, the Region has communicated that they are currently set at the “financial cost recovery level, � though there is no evidence to this effect in the ICR. In any event, the lack of a robust and independent regulatory mechanism does not bode well for the sustainability of cost -covering tariffs. Support to the supervising Ministries was ineffective . The funds were spent on irrelevant policy studies which were not used for decision making. The Ministry was, in any event, abolished during the Russian governmental restructuring in March, 2004. The regulatory shortcomings are serious. Previous experience, reflected in IEG publications, has indicated the importance of having a strong and well-functioning regulatory framework in place before the privatization of “natural monopolies.� Despite, therefore, the substantial achievement of sub-objectives (i) and (ii), the overall rating for attainment of objective (b) is modest, while that for (c) taken alone (the commercialization of electricity sector companies ) is substantial. Overall, efficacy is rated as substantial . 5. Efficiency (not applicable to DPLs): As this was a TA operation, calculation of an ERR is not possible . However, the tying up of significant IBRD funds in this project during long periods of inactivity was highly inefficient . These funds carry a substantial opportunity cost, and could have been used for other projects in the Russian Federation or elsewhere. ERR )/Financial Rate of Return (FRR) a. If available, enter the Economic Rate of Return (ERR) FRR ) at appraisal and the re- re -estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal % % ICR estimate % % * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Despite the over-ambitious design and implementation timetable, especially against the turbulent background of prevailing country conditions, project relevance can be rated as substantial . Preparation was of a high technical quality, consistent with country and sector needs and Bank priorities, and the design demonstrated its durability, albeit with restructuring . Efficacy, with two of the three objectives substantially achieved and the third modestly, is assessed as substantial . Efficiency is modest, in light of the inactive tying up of substantial Bank funds for a three-and-a-half years. This amounts to an overall Moderately Satisfactory rating for project outcome. a. Outcome Rating : Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating: The reforms and restructuring carried out within RAO UES and its subsidiaries are supported by all relevant sector stakeholders and by the government and are likely to be sustained . On the other hand, the regulatory reforms have not only failed to achieve their objectives, but there is also a danger that the FTS may lose even its limited autonomy. Unless addressed, regulatory weakness could undermine the future course of the reform program. a. Risk to Development Outcome Rating : Moderate 8. Assessment of Bank Performance: Quality at entry was Moderately Unsatisfactory . On the positive side, the task team responded immediately to what was perceived as a window of opportunity, when the Bank was invited by the Government to support power sector reform for the first time in Russia . What was expected to be a rapidly implemented program was backed by a SAL and by this accompanying TA project . The TA was prepared in record time (three months from government request to Board approval). Technical standards of conceptualization and institutional capacity analysis were, nonetheless, high . Soon after Board approval, though, it became evident that the power sector reform program had not yet gained the necessary degree of consensus to be effectively launched . Resistance from vested interests was powerful enough to prevent start up, and then the macroeconomic and political crisis struck . While perhaps the full force of this crisis could not have been foreseen at the preparation stage, it followed so closely after Board approval that some awareness of what might have been forthcoming could reasonably have been expected of the task team. Design was, in any event, highly ambitious, and it is doubtful if such a complex reform program could have been implemented in three -and-a-half years even had background circumstances been more propitious. After the reform effort started in earnest in mid-2001, it still took five-and-a-half years to reach project closure with an uncompleted program . The lessons of previous complex TA operations in other countries were not sufficiently taken into account . There was, moreover, limited local ownership of the project, since enhancing such ownership was one of the main objectives of the 2000 restructuring. M&E design was very weak, with this task being left largely to the Borrower (see Section 10 below). Supervision was Satisfactory. The task team played a key role in achieving the consensus that was finally reached on the structural framework for sector reform . The development of this framework, and its full acceptance by the Borrower, necessitated a redefinition of the TA to be financed by the project, and this was done through the 2000 restructuring and subsequent adjustments. The patience and flexibility of the task team and its persistence in supporting the government ’s reform efforts contributed to a much better result than looked likely at the beginning of this decade . These achievements outweigh shortcomings such as insufficient attention to implementation and procurement issues, taking account of the size and complexity of the operation and the lack of experience and capacity of the implementing agencies. It is surprising, for example, that at the time of restructuring, only a year ’s extension to the closing date was thought to be enough. Moreover, the high rate of turnover of Task Team Leaders (seven in the nine-and-a-half years between approval and closure) led to lack of continuity and different approaches to the supervision of project implementation . at -Entry :Moderately Unsatisfactory a. Ensuring Quality -at- b. Quality of Supervision :Satisfactory c. Overall Bank Performance :Moderately Satisfactory 9. Assessment of Borrower Performance: The government’s commitment to the project’s development objectives, although strong at the time of conception, fluctuated considerably in the wake of the crises, and resulting frequent changes of administration. Successive governments failed to deal effectively with resistance to reform from pressure groups both within and outside the administration . Reluctance to liberalize pricing and to give the regulatory entity real teeth continues to reflect a political priority to keep tariffs artificially low . Decision making was slow. Only in the later stages of implementation did the government display ownership and consistent commitment. The performance of the implementing agencies, RAO UES, FEC/FTS, was mixed. RAO UES was a champion of reform, and its leadership was well organized and efficient, backed up by a supportive management structure. The regulatory agencies, FEC and FTS, performed their tasks satisfactorily, but both were severely constrained by the inadequacies of the legal and policy frameworks . The supervising Ministries were frequently opposed to the reforms . Their lack of ownership of project objectives is reflected in the continuous shift of management responsibilities and the assignment of project tasks to low level officials . a. Government Performance :Unsatisfactory b. Implementing Agency Performance :Moderately Satisfactory c. Overall Borrower Performance :Moderately Unsatisfactory 10. M&E Design, Implementation, & Utilization: M&E design, implementation and utilization were inadequate . The Technical Annex to the MOP (as was customary for TA operations linked to SALs, there was no PAD) states that the policies and procedures for monitoring project implementation and the achievement of its objectives were to be the responsibility of the government. The indicators were to be agreed with the Bank. Consequently, until 2005, there was no explicit system for evaluation and no criteria for assessing progress . Monitoring the outcome of the different consulting assignments was left to the implementing agencies . Even the monitoring indicators and data collection introduced in 2005 were sparse and barely adequate. The results matrix included in the ICR contains the minimum necessary to permit internal Bank monitoring of project implementation . a. M&E Quality Rating : Negligible 11. Other Issues (Safeguards, Fiduciary, Unintended Positive and Negative Impacts): Environment: Not applicable Resettlement: Not applicable Fiduciary: There is no discussion of financial covenants in the Technical Annex to the MOP . The government’s performance regarding financial management, procurement and reimbursement was generally satisfactory. However, the audit of Part C of the project, which was pending for 2003 and 2004, had still not been completed by closure. 12. Ratings : 12. ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Moderately Two of the three development Satisfactory objectives were substantially achieved, and one to a modest degree. Implementation took nearly three times as long as anticipated.. Risk to Development Moderate Moderate Outcome : Bank Performance : Satisfactory Moderately High technical quality in design was Satisfactory marred by unrealism. More account should have been taken of vested interest opposition and of the unstable and deteriorating background. M&E was poorly designed and implemented. Supervision is to be commended for persistence and flexibility, though implementation problems were exacerbated by high TTL turnover. Borrower Performance : Moderately Moderately Of the four entities involved, only Satisfactory Unsatisfactory one performed satisfactorily. Fortunately this was the critical entity. Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate . 13. Lessons: 1. The timing of a TA project and the reform program it is meant to support must be carefully synchronized. It is of little value breaking preparation time records for an operation which doesn ’t disburse for another four years. This is not only imprudent but also costly for the Borrower, who must pay commitment fees and interest. 2. Linked to this, is the need for a careful assessment of the strength and sustainability of political support for the program and the degree of national ownership of the TA project and its design . 3. By 1997, the year of the project’s preparation, enough experience had been garnered with TA projects, especially large ones supporting policy reform, to justify the utmost care and attention being paid to implementation arrangements. 4. Large and complex TA projects, with multiple implementing agencies, are especially vulnerable to implementation difficulties. 5. Continuity of staff, especially team leaders on both Bank and implementing agency sides, is essential for smooth implementation. 14. Assessment Recommended? Yes No Why? Yes. In view of the differences in rating between the ICR and IEG, and to bring the links between the reform program and the project into sharper focus. Also, it would be important to see whether the promised reforms, notably unbundling and privatization, are actually carried out . 15. Comments on Quality of ICR: The ICR contains a great deal of information and analysis, and does well in outlining the highly complex context and background to this project. Its two main faults are: (i) that it does not contain a clear summary of the sector reform program itself so that the reader is left guessing as to its overall shape and main components; and (ii) that it does not contain all the information pertinent to assessing the outcome of the project . Key additional data had to be supplied by the Region or garnered from external sources. There is considerable incongruity between the textual analysis and the project outcome and performance ratings. The lack of M&E indicators, either at appraisal or at restructuring, is not the fault of the ICR . Nonetheless, it would have been useful to have provided the cost breakdown of the project after restructuring and the extent of the reallocations between consultancy services and goods procurement made at that time. The document would have benefited from careful editing. There is room for improvement in both organization and clarity. At times, terms such as “model� and “transition� are used in a confusing way, and on occasion it is necessary for the reader to consult other sources to find out what has occurred . This is particularly true for Annex 2. Some acronyms are not included in the Abbreviations and Acronyms Table or explained elsewhere . The discussion of the FEC/FTS issue is fractured and initially unclear . a.Quality of ICR Rating : Satisfactory