ICRR 12238 Report Number : ICRR12238 ICR Review Operations Evaluation Department 1. Project Data: Date Posted : 09/27/2005 PROJ ID :P044728 Appraisal Actual Project Name :Ozone Depleting Project Costs 32.7 27.7 Substances Phase Out US$M ) (US$M) Project Country :Ukraine Loan/ US$M ) Loan /Credit (US$M) 23.2 23.2 Sector (s):Central ): government US$M) Cofinancing (US$M ) administration; Other industry L/C Number : FY ) Board Approval (FY) 98 Partners involved : Closing Date 12/31/2000 12/31/2004 Evaluator : Panel Reviewer : Group Manager : Group : Keith Robert A. Fernando Manibog Alain A. Barbu OEDSG Oblitas 2. Project Objectives and Components a. Objectives The overall objective of the Ozone Depleting Substances Phase -out Project was to: Assist Ukraine in the rapid phase-out of Ozone Depleting Substances (ODS) consumption, in a manner consistent with international efforts and within internationally agreed timeframes . Specific objectives were: (i) Assist high consumption enterprises in Ukraine to make the transition to non -ODS substitutes before ODS supplies diminish. (ii) Provide technical assistance for phase -out in the halon sector. (iii) Provide technology transfer associated with low global warming potential (GWP) refrigerants for domestic refrigeration. and (iv) Provide institutional strengthening to the Ozone Office within the Ministry of Environmental Protection and Nuclear Safety (MEPNS) and MEP. b. Components (or Key Conditions in the case of Adjustment Loans ): A. Investment component . (Cost at appraisal: US$30.0 million; actual cost: US$25.4 million) (Investment support to help enterprises consuming ODS to invest in new technology that does not consume ODS .) B. Technical assistance component . (Cost at appraisal: US$2.0 million; actual cost: US$1.3 million) C. PIU (Project implementation unit ) support component . (cost at appraisal US$0.7 million; actual cost: US$1.0 million) c. Comments on Project Cost, Financing, Borrower Contribution, and Dates The project was financed by a GEF grant of US$ 23.2 million. The originally planned project duration was 2 1/2 years. The project period was extended 3 times for a total of 4 additional years, thus making the actual project period 6 1/2 years. The grant was practically all used (the remaining balance at closure was US$ 18,600). 3. Relevance of Objectives & Design : Overall relevance of project objectives was substantial. Objectives : The project's objectives are consistent with the Bank's and the Ukraine Government's stated priority to address key global environmental issues and to participate in and comply with environmental conventions . Ukraine was one of the larger consumers of ODS in eastern Europe . A number of ODS projects had been financed by GEF in different counties in the ECA region . Ukraine was one of the later countries to develop a program for reduced ODS consumption. Compliance with the Montreal Protocol on ODS needed an accelerated phase -out program by Ukraine if it was to be in compliance with the January 2000 requirements under the London Amendment . Design : A framework approach of establishing a mechanism for provision of grant funding to ODS consuming enterprises was appropriate, though the level of these subsidies may have been too high to screen out enterprises likely to perform poorly. There is no reference to the ground rules for provision of such grants including the percentage that grant funding would comprise in sub -project costs. The project's provision of technical assistance was necessary. Project implementation by the responsible ministry was appropriate, though the implementation unit was not given sufficient executive authority to make decisions and supervise project activities . 4. Achievement of Objectives (Efficacy) : Overall Objective : Assist Ukraine in the rapid phase -out of ODS consumption, in a manner consistent with international efforts and within internationally agreed time -frames . The project's overall efficacy was substantial. Specific objectives : (i) Assist high consumption enterprises in Ukraine to make the transition to non -ODS substitutes before ODS supplies diminish . Substantial. Eight out of the original 9 enterprise sub-projects were transformed to non -ODS consuming technologies. One sub-project was cancelled but additional subprojects were financed . The new technology significantly reduced ODS and enabled the firms to continue production activities after Russia terminated its exports of ODS. ii) Provide technical assistance for phase -out in the halon sector . Modest. A halon management plan was (ii) developed, capacity to responsibly manage halon stocks was developed . An additional investment sub-project was for a halon recovery and recycling plant . But implementation of a phase-out program is not recorded in the ICR . iii ) Provide technology transfer associated with low global warming potential (GWP) (iii) GWP ) refrigerants for domestic refrigeration . Substantial. A corresponding sub-project was successfully implemented and developed a product -line of low GWP refrigerants with environmental standards enabling access to European as well as domestic markets . (iv) MEPNS ) and MEP . Modest. The Ozone Office and its project implementation iv ) Protection and Nuclear Safety (MEPNS) unit was strengthened and was able, with further Bank assistance to manage the project . However, government financing of the office after the project is not assured and thus sustainability is questionable . 5. Efficiency : Substantial: An ERR was not calculated for the project at appraisal or completion on the grounds that a substantial part of benefits are externalities . Actual project costs at US$27.7 million were lower than the estimated costs at appraisal of US$32 7 million, while the project's physical targets were largely met . 6. M&E Design, Implementation, & Utilization: Monitoring and Evaluation was a major weakness, both in design and during project implementation . Design : The appraisal report has a short section on M&E and reporting requirements but mainly refers to monthly/quarterly progress reports with management data such as status of physical implementation, disbursements and procurement. There is brief reference to the monitoring of ODS being continued by the Ozone Office . A small table on environmental performance (Annex I, page 4) provides some indicators, but with no structured links to either the project's objectives or the components . Implementation and Utilization : The ICR has no mention of M&E. Routine management information appears to have been collected from the information contained in the ICR and would have been important for project management. Data on ODS would also have been useful for reporting project progress . A short logframe is provided in the ICR (Annex 1), and some interesting information is provided on achievements . However, the logframe only includes a few indicators and is not organized by objective and components . Thus, it is only partially useful in assessing project achievements . There is also no discussion of how certain key parameters (eg. ODS) are measured. 7. Other (Safeguards, Fiduciary, Unintended Impacts--Positive & Negative): Regarding the rating below of Likely for Sustainability , this takes into account several recent developments since preparation of the ICR: (i) the establishment of a new ODS regulating unit, including a nucleus of full-time staff,; (ii) the ongoing discussions in the environment ministry to cover other international conventions; and (iii) the new GEF/Bank project for methyl bromide phase-out continuing the Bank's presence in environmental regulation. 8. Ratings : ICR IEG Review Reason for Disagreement /Comments Outcome : Satisfactory Satisfactory Institutional Dev .: Modest Modest Sustainability : Likely Likely Refer section 7 Bank Performance : Satisfactory Satisfactory Borrower Perf .: Satisfactory Satisfactory Quality of ICR : Satisfactory NOTES: NOTES - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - ICR rating values flagged with ' * ' don't comply with OP/BP 13.55, but are listed for completeness . 9. Lessons: PIUs ) need to be given executive powers sufficient to minimize implementation 1. Project implementation units (PIUs) constraints . A PIU approach for project management was chosen, and the PIU was placed within the Ozone office of the responsible ministry. The PIU was, thus, within the organic structure of the implementing ministry for Ukraine's ODS program. This was sensible for a PIU approach, and is likely to better enable maintaining technical and administrative capacity for continuing the program after project closure . However, the ozone office and PIU were at a low level within the ministry. The ICR comments that this reduced the PIU's capacity to make decisions, to enforce standards and to get clearance for procurement and disbursement decisons . A higher level of executive authority would have been better. Over -reliance on grant funding may reduce the commitment and sustainability of government and involved 2. Over- agencies : The project virtually fully funded government activities and a large part of the funding for the implementing enterprises. For government, the ICR comments that there is no indication that government financing of the PIU budget will continue. For the participating enterprises, the most successful at implementing their sub -projects were those that contributed significant and timely counterpart funds . Those enterprises that relied more on government funding, and which were late in providing their funds, performed weakly . Project start-up could have been conditional on a minimum first financial allocation by government . A condition for enterprise participation might also have been up-front funding. 3. Project implementation periods need to be realistic : The original project closure date was only 2 1/2 years after Board approval. In the event, after extensions accumulating to 4 years, project implementation was 6 1/2 years. In part, the chosen short duration period was to achieve reduction of ODS at a pace to remain in line with the Montreal ODS protocol. However, duration objectives that are clearly not possible might have been better explained up -front to the global ODS program. 10. Assessment Recommended? Yes No 11. Comments on Quality of ICR: - Satisfactory Overall and provides a candid discussion of the project's performance . The analysis of Ukraine's cost-effectiveness in ODS phase-out in comparison with other countries is interesting . - Some areas where the ICR could have been improved are : (i) the log frame and performance indicators could have been developed much further, structured around the project objectives and components; (ii) discussion of M&E needs and performance should have been included; and (iii) several brief paragraphs might have been included at the beginning of the ICR with background on the ozone program and international protocols . Finally, (iv) there were several gaps in the report: for instance, acronyms, a staff cost table and index page numbers were absent .