IEG Report Number: ICRR14673 ICR Review Independent Evaluation Group 1. Project Data: Date Posted: 06/29/2015 Country: Thailand Project ID: P004649 Appraisal Actual Project Name: The Ozone Depleting Project Costs (US$M): 40.00 35.53 Subs L/C Number: Loan/Credit (US$M): 40.00 35.53 Sector Board: Environment Cofinancing (US$M): 0.00 0.00 Cofinanciers: Board Approval Date : 08/07/1994 Closing Date: 12/31/1999 01/02/2014 Sector(s): Other industry (100%) Theme(s): Pollution management and environmental health (100% - P) Prepared by: Reviewed by: ICR Review Group: Coordinator: M. Gary Costello April Connelly Christopher David IEGPS1 Nelson 2. Project Objectives and Components: a. Objectives: The project's Global Environment Objective (GEO) as expressed in the Trust Fund Grant Agreement (TFGA) signed August 8, 1994 (TFGA Schedule 2), "is to assist the Recipient in carrying out its Country Program so as to phase-out the use of ODS within its territory through, inter alia: (a) the introduction of appropriate technology measures; (b) the institutional strengthening of responsible governmental entities; and (c) the implementation of specific cost-effective priority investments to reduce consumption of ozone depleting substances". While a Project Appraisal Document (PAD) was not prepared for this project, the Memorandum to the Director (MOD p. 2), indicated that the project objective was to support the Government program to phase out Ozone Depleting Substances (ODS) by (a) establishing an efficient mechanism for executing ODS phase-out projects through local institutions, and (b) implementing an initial group of cost effective priority sub-projects. The project's GEO was revised as part of the 2002 project restructuring to reflect a shift in the project implementation modality from sub-project-by-sub-project to a sector plan approach. A Project Paper and an Amendment to the Grant Agreement were approved in January 2003. The revised Grant Agreement included a revised GEO which was to assist the Recipient in carrying out its Country Program so as to phase-out ODS consumption within its territory. This review adopts the GEO as articulated in the TFGA and the revised Grant Agreement. The project will be assessed against the original and revised GEO using a split rating methodology. b.Were the project objectives/key associated outcome targets revised during implementation? Yes If yes, did the Board approve the revised objectives/key associated outcome targets? No c. Components: Provision of Sub-grants for the carrying out of specific Subprojects .(approved: US$ 38.8 million; actual US$ 19.47) The original project design had one component to provide sub-grants to support large manufacturing enterprises to convert from ODS to ODS friendly technologies. According to the MOD (p.2-3), the Project would support a total of 45-55 sub-projects in solvents, refrigeration, air conditioning, foams, and recycling. Each sub-project included technical assistance to support participating enterprises with technology transfer, design, training and implementation. Sub-projects would be supported for the phase-out of ODS related to (i) solvents; (ii) refrigeration; (iii) insulating foam; (iv) compressors; (v) mobile air-conditioning and (vi) chillers. Revised Components: National Chlorofluorocarbon (CFC) Phase-out Plan (NCFCP) (approved: US$ 14.73; Actual US1$12.01); National Methyl Bromide Phase -out Plan (NMBPP) (approved: US$ 2.93; Actual US$1.67); Institutional Strengthening (IS) (US$ 1.54; Actual US$ 1.54) The Grant Agreement was amended to accommodate implementation of the National CFC Phase-out Plan (NCFCP) and the Institutional Strengthening (IS) project in 2002, and the National Methyl Bromide Phase-out Plan (NMBPP) in 2005. At the time of the amendment, the grant ceiling was increased from US$40 million to US$45 million. The NCFCP and NMBPP broadened the scope of the phase-out utilizing a comprehensive approach characterized by a focus on the main ODS sectors. Activities included: (i) sub-project ODS phase-out for medium and small enterprises; (ii) strengthening of the Program Management Units (PMUs) of the sector plans; and (iii) institutional strengthening and technical assistance for Department of Industrial Works (DIW). d. Comments on Project Cost, Financing, Borrower Contribution, and Dates: Project Cost. At appraisal, project costs were estimated at US$40 million. At the time of restructuring in 2002, the project received a budget increase of US$5.00 million.. According to Annex 1 of the ICR (p.28), the actual total cost of the project was US$35.5 million. The ICR attributes the lower project cost the use of competitive procurement and controlling cost at the sub-project level. Financing. The project was financed through a grant from the Montreal Protocol investment Fund. There were no other external sources of financing. The Grant disbursed 88% of the original amount and 79% of the revised amount. Borrower Contribution . At appraisal, the Borrower contribution was US$ 11.00 million. The actual Borrower contribution was US$ 5.62 million. No explanation was provided in the ICR for the smaller borrower contribution. Dates. The Grant was approved on 08/02/1994 with effectiveness on 06/07/1995 and project closing originally planned for 12/31/1999. The grant agreement was amended seven times. The amendments included six extensions of the closing date which added 14 years to the original closing date. The extended implementation period was a function of the umbrella grant modality applied to Montreal Protocol operations as per the World Bank's OP/BP 10.21, which allows projects the flexibility to take onboard new commitments and funding from the MLF in support of achieving MP phase-out targets beyond what is specified in the original grant agreement. The first two extensions to the closing date permitted the implementation of new sub-projects (12/31/1997 and 01/05/2000).The third (12/27/2002) was undertaken to accommodate implementation of NCFCP and IS activities from 2002-2010. The fourth (10/04/2005) was undertaken to accommodate implementation of NMBPP activities from 2005-2013. The fifth (03/13/2009) was undertaken to reallocate grant funds, revise percentage of expenditures to be financed and extend disbursements to all project activities (other than NMBPP). Finally, the sixth extension (11/18/2010 ) was undertaken to extend disbursements for project activities including those under the NCFCP in order to ensure sustainability of ODS phase-out. Restructuring. The project was restructured when the grant agreement was amended on December 27, 2002. The GEO was revised and the project implementation modality shifted from sub-project-by-subproject approach to a sector plan. According to OP/BP 10.21, stand-alone projects financed by the Multilateral Fund for Implementation of the Montreal Protocol required only RVP approvals. Hence the restructuring did not require Board approval. 3. Relevance of Objectives & Design: a. Relevance of Objectives: Original GEO: The project objective was clearly defined and was relevant to Thailand's social and economic development and environmental priorities at the time of appraisal and it remained relevant throughout the 19 year implementation period. Thailand Ratified the Montreal Protocol on July 7,1989 and has since ratified the Protocols four subsequent amendments. As a signatory to the Montreal Protocol, the project’s objectives were highly relevant to helping Thailand meet its obligations under the MP, particularly considering that Thailand lacked a comprehensive national phase-out strategy and there was a strong government commitment to phase-out ODS at the sector level. The World Bank Country Assistance strategy for Thailand, in place at the projects appraisal, identifies the project as part of its program to address the improvement of environment and natural resource management. Thailand's last Country Partnership Strategy (2002) addressed in some detail ongoing ODS work as well as replacing chlorofluorocarbons (CFC) chillers and helping to mitigate not only ozone depletion but also greenhouse gas (GHG) emissions. The CPS also identifies three clusters for engagement including Infrastructure and Climate Change (Cluster 3) and initiating preparation work for the hydroflorocarbon (HCFC) Phase-Out . More recently, the Bank's Country Economic Memorandum (2014) specified the importance of mitigating and coping with the impact of climate change through reduction of emissions and environmental degradation. The project's objective is consistent with Thailand's Seventh National Development Plan (1992-1996), which identified the need for environmental improvements in the country, and calls for investment in pollution controls. The project objective is also aligned with current country-level and global environmental priorities as reflected in the National Economic and Social Development Board's Eleventh National Plan (2012-2016) which includes a pillar on climate change and environment and includes issues related to city planning and “livability”. Rating: High Revised GEO: The Revised GEO addressed "consumption" of ODS which is defined by the MP as production plus imports minus exports. The Revised GEO is was aligned with country-level and global environmental priorities as reflected in the National Economic and Social Development Board's Eleventh National Plan (2012-2016) and the Bank's most recent Country Economic Memorandum (2014), in particular their focus on improved environmental quality along with increased attention to climate change challenges. Rating: High b. Relevance of Design: Original Design: The project's design adopted an umbrella approach to facilitate the processing and management of multiple sub projects under a single grant agreement. Project implementation arrangements took into account the analysis of institutional capacity and a comprehensive survey on ODS use under the Country Program. The approach of funding of sub-projects targeting large enterprise phase-out of ODS was adequate for achieving the objective of reducing the use of ODS. The ICR (p.9) notes that while the initial design was capable of phasing-out a defined amount of ODS consumption, its limitation was that other ODS using companies could continue or even increase their ODS consumption. In 1994 at the time of appraisal, there was no mandate for a results framework. The MOD (Annex 5) defined the process for sub-project eligibility including elements to be addressed at appraisal. The results were well defined and focused on achievements at the enterprise level. Reporting requirements were specified and required the enterprise to submit one sub-project progress report at the time when the proposed investment activities have been completed and the new installations are operating. Rating: Substantial Revised Design: The Grant Agreement was amended in 2002 which allowed the project to adopt a more comprehensive/national approach directed at specific sector ODS phase-outs and introduction of ODS friendly technologies. This national approach addressed the the limitation of the original sub-project modality. Moving from a subproject-by-sub-project focus to a "sector" approach, the project was able to focus on small and medium enterprises and introduce important new ODS phase-outs, namely reduction of Methyl Bromide along with additional reductions in CFCs, TCA and Carbon Tetrachloride. Two national ODS phase out plans were added to the project, NCFCP and NMBPP, both of which had a significantly larger scope as both employed a comprehensive approach to phase out the remaining use of ODSs within a specific target date. The NCFCP was the first national plan approved by the Executive Committee (ExCom) for which funding spanned multiple years. Funds were released in tranches upon independent verification of the country’s achievement of the agreed targets. Under the revised design, small and medium size enterprises in the aerosol foam and solvent (garment and textile, and manufacturing) sectors benefitted from training and subsidies provided for technological conversion. Additionally, institutional strengthening activities and support for the PMU were funded. The NMBPP aimed to eliminate the consumption of Methyl Bromide (MB) by 2013, two years ahead of its required phase-out date under the Protocol. Rating: High 4. Achievement of Objectives (Efficacy): (1) to assist the Recipient in carrying out its Country Program so as to phase -out the use of ODS within its territory . Rated: Substantial. The ICR (Annex 2) reports the following Outputs and Outcomes by Sector and Activity. Foam Sector: 20 investment sub-projects implemented (2 sub-projects cancelled), phasing out 434.85 ODP tons Refrigeration Sector: 20 investment sub-projects implemented (1 sub-project cancelled). The 20 sub-projects under the refrigeration sector covered the majority of the enterprises in the domestic and commercial refrigerator subsectors. 1,129.41 ODP tons were phased-out. Solvents Sector: 10 investment sub-projects implemented (1 sub-project cancelled). 96.02 ODP tons were phased-out. Halon (fire fighting): 2 investment sub-projects implemented. 717.40 ODP tons were phased-out. The ICR also reports that the sub projects enabled the government to enact the supporting regulations to ban the use of CFCs after the completion of the sub-projects and that the regulations played a vital role in sustaining the phase-out of CFCs in the manufacturing and servicing sector:  In 1997, the Ministry of Industry issued regulations prohibiting the use of CFCs in the manufacturing of household refrigerators;  In1997, the Ministry of Commerce issued regulations prohibiting the import of household refrigerators containing CFCs; and,  In 2004, the Ministry of Industry issued regulations prohibiting the use of CFCs in the manufacturing of commercial refrigerator. Outcomes Under the original design, the project implemented 52 sub-projects which fell within the at appraisal estimate range of 44-55 subprojects. These sub-projects directly eliminated 2,378 ODP tons, falling short of the original target of ranged of 3000-4000 ODP tons. By project completion, using the broader national approach, the project had completely phased out the consumption of CFCs, halons, TCA and carbon tetrachloride. (2) Revised GEO: The revised GEO of the Project was to assist the Recipient in carrying out its Country Program so as to phase-out ODS consumption within its territory . Rated: Substantial. Two National Ozone Phase-out Programs were supported by the project to achieve this objective. The results of each program are reported separately. National CFC Phase out project (NCFCP). This program included sub-projects, technical assistance and capacity building activities to phase out the remaining use of ODS in the country. NCFCP Sub projects : The NCFCP identified the remaining use of ODS by small and medium-size enterprises in the aerosol, foam, garment and textile, and solvent sector. A total of 262 enterprises in these manufacturing sectors were supported with sub-projects to help them convert to non ODS technologies. Aerosol Sector: The Aerosol sector industry was completely converted. CFCs were phased out and regulations to ban use of CFCs were issued. Three aerosol manufacturing enterprises were eligible for financial assistance. One enterprise signed a sub-grant agreement with the GSB, but the sub-project was subsequently cancelled as the enterprise decided to stop manufacturing the products to be financed under NCFCP. The other two enterprises decided not to participate in the NCFCP because they had already converted to alternative technology on their own. Foam Sector: About 100 foam manufacturing enterprises that had never received any financial assistance from the MLF were identified during the preparation of NCFCP. 53 enterprises confirmed interest. 29 enterprises signed individual sub-grant agreement. 14 enterprises were covered under a group sub-project. The other 10 were either not eligible for financing, or cancelled. CFC Phase-out was completed and regulation banning the use of CFCs was issued as planned. Solvent sector-manufacturing using CFC-113: 4 sub-projects were planned. 1 project implemented. 3 chose not to participate. The sector was completely converted, regulations were established and CFC Phase-out was completed. Solvent Sector – manufacturing using TCA and CTC: 6 sub-projects were planned. Three sub-projects using TCA, and one using CTC were completed. Two enterprises using CFC did not want to participate. The sector completely converted and TCA phased out as planned. Regulation to ban the use of TCA was established. Solvent Sector – garment and textile: 214 garment and textile factories received subsidy for acquisition of ventilation system. The ventilation system provided to garment and textile industries proved that ventilation system could effectively reduce effect from 1,1,2-TCE used. NCFCP Technical Assistance : The NCFCP also provided technical assistance to the Mobile Air conditioning and refrigeration service sectors.  Technical Assistance was provided for the phase out of MDIs. The Allergy and Immunology Society of Thailand was appointed for data collection, development of phase-out strategy, and information dissemination. An FDA regulation was issued to ban import of MDI containing CFCs in Salbutamol, Futicasone, and Sodium Gromoglycate starting from January 1, 2006. The CFC MDI phase-out strategy was prepared and disseminated as planned.  Train-the-Trainer Program was established in the Refrigeration Servicing Sector. Training centers were established and provided training to refrigeration service technicians.  Certification of Refrigeration Service Technicians. 45 workshops were organized, providing training to 3,110 technicians from 1,734 refrigeration servicing shops (target provide training to 1,300 refrigeration servicing shops). Retrofitting procedures were put into the training curriculum and delivered to refrigeration service shops as planned to strengthen their awareness of CFCs phase-out.  A regulation was issued making Mobile Air Conditioning (MAC) Inspection mandatory. MAC inspection was operational at Headquarters to reduce CFC use by preventing reverse retrofit of MAC system.  A Train-the-Trainer Program in the MAC Servicing Sector was established. Training centers were established to undertake certification workshops for MAC service technicians and training was provided to technicians from MAC service shops as planned. 46 training centers were selected as authorized training centers (FOR?). Two training of trainer workshops were organized. The ATCs were provided with 62 sets of training equipment and 29 units of recovery and recycling machines.  Certification of MAC Service Technicians. 100 workshops were organized and resulting in the training of about 5,750 technicians from 4,050 MAC servicing workshops. Retrofitting procedures were put into the training curriculum and delivered to MAC service shops as planned. Exceeding the target of training service technicians from 2,750 MAC service shops.  Financial Subsidy for Procurement of Refrigeration Maintenance Tools. 1,714 vouchers were issued to certified refrigeration service shops ( 126.9% of the target). Some 1,643 vouchers (95.8% of total vouchers issued) were claimed and funds were disbursed. Refrigeration service shops received training and purchased a basic set of equipment with the subsidy as planned. CFC-12 consumption reduced as planned.  Financial Subsidy for Procurement of MAC Maintenance Tools. 3,751 vouchers were issued to certified MAC service shop which was 136.4% of the target. Of which, 3,533 vouchers or approximately 94.2% of total vouchers issued were claimed and fund disbursed. MAC service shops participated in training and purchased the basic set of equipment with subsidy as planned. CFC-12 consumption was reduced as planned.  Financial Subsidy for Procurement of MAC Recovery & Recycling Machines. 179 vouchers were issued of which, 166 vouchers or approximately 92.7% of total vouchers issued were claimed and fund disbursed. The financial subsidy was provided to purchase 179 sets was only 28.2% of the target. NCFCP Capacity Building :  Customs Training Program. Two training workshops were organized and 60 trainers were trained. The trainers subsequently trained 700 customs officers. Customs training on ODS control was put into the Customs training curriculum. Customs check points equipped with refrigerant identifiers. NCFCP Outcomes The NCFCP allowed Thailand to accelerate the phase out the remaining consumption of ODS ahead of the Montreal Protocol schedule. The ICR reports that in total 3,066 ODP tons of CFCs, 34 ODP tons of 1.1.1-Trichloroethane and 7.52 ODP tons of carbon tetrachloride were phased out as planned. Thailand was able to completely phase out consumption of these chemicals by 2010. The ICR (p. 12) notes that consumption of ODSs to be phased out under the NCFCP was audited and verification reports were prepared by an independent auditor on an annual basis. The verification reports were used to confirm Thailand’s compliance with the reduction targets as agreed with the MP Executive Committee. National Methyl Bromide Phase -out Plan (NMBPP) The NMBPP aimed to assist Thailand to phase out consumption of methyl bromide by January 1, 2013. It comprised both investment activities and technical assistance and training. NMBPP Sub-projects. Sub projects were financed to reduce MB consumption in three types of enterprises:  Type 1: Owner of storage facility with in-house fumigation of MB or use more than 50,000 tablets of phosphine per year. 13 of the 23 identified type 1 beneficiary enterprises signed sub-grant agreements to use of MB alternatives and improvement in pest management control. MB consumption for structural fumigation was reduced as planned.  Type 2: Owner of storage facility without in-house fumigation of MB or use less than 50,000 tablets of phosphine per year. 83 type 2 beneficiary enterprises strengthened their pest management capacity in their warehouses. 83 vouchers for basic fumigation equipment and 81 vouchers for IPM equipment were provided to the enterprises. 2 enterprises did not need basic fumigation equipment since they did not have in-house fumigation. MB consumption for structural fumigation reduced as planned.  Type 3: Fumigation servicing companies. 19 out of the 41 identified commercial fumigators participated in Type 3 sub-projects. 19 enterprises received vouchers for a set of electronic phosphine monitoring devices and gas masks. The remaining enterprises were not eligible for MLF fund (either owned by developed countries or established after the cut-off date). The capacity of commercial fumigators strengthened to use MB alternatives. By the closing date, the project had supported about 115 enterprises to employ the code of good practice for phosphine fumigation and integrated pest management (IPM) through investment sub-projects. The ICR notes that the investment activities and training provided to the beneficiary enterprises encouraged the enterprises to introduce the use of phosphine and Integrated Pest Management technology to reduce the reliance on methyl bromide for export of agricultural product. NMBPP Regulations, Technical Assistance , and capacity Building :  Accreditation system for storage facilities: The DOA established accreditation scheme to provide incentive to obtaining psytosanitary certification to accredited enterprises without fumigation. In 2008, DOA issued Notification “Accreditation scheme on manufacturing process for issuance of phytosanitary certificate for processed commodities”. The notification aimed to eliminate the use of fumigants (including methyl bromide) in processed commodities.  Train-the-Trainer: Eight Accredited Training Centers were established and equipped to provide training to concerned stakeholders. A train-the-trainer workshop was conducted to train representatives from the regional offices of the Office of Agricultural Research and Development (OARD) and the Plant Quarantine Stations, which were authorized by DOA as ATCs for training of methyl bromide users.  Certification of all fumigators: Ten workshops were implemented to strengthen the enforcement of the regulations on fumigant and pesticide trade. About 150 enterprises received training; 13 additional enterprises received individual training under individual subproject (voluntary). The DOA also provided series of training to fumigators to issue fumigation license. The Agricultural Inspection Sub-division undertook the monitoring activity to supervise agricultural input dealers for compliance with the laws and regulations on pesticides, fertilizers and other agricultural chemicals.  Capacity building on insect resistance control: An insect resistance control unit laboratory was under the Post-Harvest and Products Processing Research and Development Office to monitor tendency of insect resistance to phosphine. A study was carried out across Thailand in 2010 to determine insect resistance to phosphine.  Capacity Building of DOA Officials: Staff from the Post-Harvest and Products Processing Research and Development Office were trained on grain resistance inspection, monitoring and management. Officers from the insect resistance control unit attended training in USA in 2009. DOA conducted a fumigation visit to New Zealand to learn fumigation and MB recapture techniques, risk assessment of chemical pesticide and certification system. The ICR reports that the knowledge gained by staff was used in the operation of insect resistance control unit, management of insect pest in storage facilities, fumigation and recapture of fumigant techniques and certification system.  A licensing system and management information system was put in place to control import and use of methyl bromide. An MIS was established to facilitate effective recording and monitoring of import control and use of MB. It linked DOA’s Pesticide Regulatory Subdivision with plant quarantine stations around the country as well as registered MB importers. NMBPP Outcomes The NMBPP resulted in Thailand accelerating the phase out of methyl bromide consumption two years ahead of the Montreal Protocol phase out target. The ICR (p.16) reports that the project contributed to not only the reduction of MB for non-quarantine and pre-shipment (non-QPS) application, the consumption of which is controlled under the MP, but also to the reduction of MB use for Quarantine and Pre-shipment (QPS) applications, which is still exempt under the MP. As of December 2013, consumption of methyl bromide for non-QPS application was zero. The use of MB for QPS applications was reducded from 335 ODP tons in 2007 to 192.1 ODP tons in 2012, reversing a rising trend in QPS application that occurred from 2005-2007. With the global production phase-out of CFCs, halons, CTC, and TCA, the domestic supply of these substances was eliminated. Related ODS equipment was dismantled and verification systems were put in place. 5. Efficiency: Substantial A quantitative cost-benefit analysis was not carried out for this project as financing decisions for Montreal Protocol funded projects are based on the cost-effectiveness and opportunity costs of comparing different investments or interventions in various countries to reduce emissions of ODS into the earth’s atmosphere. Thus efficiency of the project is assessed on the basis of its cost effectiveness and project management efficiencies. Cost effectiveness At appraisal, it was determined that sub-projects with the most cost effective technology and with the lowest unit abatement cost of ODS which achieve ODS phase out in the shortest period of time would receive priority. Relative cost effectiveness was measured as "unit abatement cost". The purpose of unit abatement cost calculation is to give a relative ranking of projects across countries and within same sector. Lower unit abatement costs correspond to higher priority in cost effectiveness. The unit abatement cost is calculated at standard discount rates to facilitate cross country comparisons. The unit abatement cost reported by the ICR (p 19) at closure indicate that all three of the ODS reduction plans supported by the project were cost effective. :  The actual cost of phase-out in US$ per kg for pre-NCFCP, NCFCP and NMBPP was lower than the respective level approved by the ExCom.  For the pre-NCFCP sub-projects, the actual achieved unit cost of phasing out ODS of 52 sub-projects across all sectors was US$8.19 per ODP kg, which compares favorably to other ODS projects within the Bank’s East Asia and the Pacific portfolio namely Phase 1 of the Philippines (US$12.29 per ODP kg) and Malaysia (US$10.11 per ODP kg) ODS Phase-out Project.  For NCFCP, the actual achieved unit cost of phasing out Chloroflorocarbon (CFCs), 1,1,1-trivhloroethane (TCA) and carbon tetrachloride (CTC) across all sectors was US$4.21 per ODP kg, which compares favorably with the CE of Phase 2 of the Philippines (US$4.28 per ODP kg) and Indonesia (US$4.41 per ODP kg) NCFCP.  For NMBPP, Thailand was able to accelerate the phase out of methyl bromide consumption two years ahead of the Montreal Protocol phase-out target. Project Management Efficiency The project was extended 14 years beyond its original closing date but this was a function of the nature of financing of the Multilateral Fund and the long reduction / phase out targets of the Montreal Protocol rather than project inefficiencies. The project was designed as an umbrella grant agreement with a notional total grant fund amount that was estimated on the basis of the country program prepared in early 1990's. However, due to the nature of its cash flow, the Multilateral Fund decided to approve grant funds on activity-by-activity basis. Because the Multilateral Fund is replenished every three years, it can only release funds to the Bank, as an implementing agency of the Multilateral Fund, and to countries incrementally. The project's umbrella grant modality allowed for additional activities that were approved later to be added to the Project. In addition, in 2001 the Multilateral Fund decided to provide assistance to countries only on the basis of national plans and disbursements were aligned to the MP phase-out schedule. The project team also noted that ODS projects financed through the World Bank in other countries have had similar implementation periods. There were delays in the course of project implementation due to the following issues:  In the early years of implementation, the financial intermediary was unfamiliar with the World Bank guidelines in terms of enterprise procurement.  Proposed changes in the implementation modality required ExCom approval which resulted in implementation delays.  The East Asia economic crisis between 1997 and 1999 caused a tightening of credits which made it considerably more difficult for enterprises to secure funds to purchase equipment before claims for reimbursement could be made to Industrial Finance Corporation of Thailand.  The sub-grant agreement did not indicate a closing date resulting in some enterprises decided to delay the procurement of equipment until economic situations improved.  Some enterprises participating in NMBPP sub-projects were not able to provide documentation to demonstrate their eligibility due to inadequate record-keeping. Despite these implementation delays the project succeeded in enabling Thailand to meet its commitment to phase out CFCs in line with the Montreal Protocol schedule and to phase-out its consumption of methyl bromide two years ahead of the MP deadline. In addition, the final project costs of US$ 35.53 million were below the appraisal estimate of US$ 40 million. a. If available, enter the Economic Rate of Return (ERR)/Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation : Rate Available? Point Value Coverage/Scope* Appraisal No ICR estimate No * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome: Project Outcome under the original objective (59 % of actual disbursements ): The relevance of the original GEO is High considering the direct relevance to a number of current country-level and global environmental priorities. The relevance of design is Substantial considering the umbrella approach allowed projects results to be monitored in the aggregate and implemented in a cost-effective way. Efficacy is Substantial. Efficiency is rated Substantial considering that cost-effectiveness assessment was among the best when compared to ODS phase-out in other countries. The achievement of the original GEO was Satisfactory. Project Outcome under the revised objective (41% of actual disbursements ):Satisfactory; The relevance of the Revised GEO is High considering the direct relevance to a number of current country-level and global environmental priorities. The relevance of design is High considering the sector approach led to the phase-out of ODS among small and medium enterprises and the target for Methyl Bromide was met. Efficacy is Substantial. Efficiency is rated Substantial considering that cost-effectiveness assessment was among the best when compared to ODS phase-out in other countries. The achievement of the revised GEO was Satisfactory. Outcome rating for Initial Project: Satisfactory (5) X 59% = 2.95 Outcome rating for Revised Project: Satisfactory (5) x 41% = 2.05 Total Score 2.95 + 2.05 = 5 a. Outcome Rating: Satisfactory 7. Rationale for Risk to Development Outcome Rating: There are two main risks to the project development outcomes:  Ongoing demand for servicing CFC equipment results in continued use of banned CFCs.  Thai Government policy reduces importance of meeting MP obligations. These risks are considered negligible to low because they are mitigated by the following:  The Royal Thai Government (RTG) remains committed to comply with the remaining obligations under the MP, including the phase-out of HCFC. The HCFC phase-out project was approved by the Bank on April 7, 2014 will facilitate this process.  With the global production phase-out of CFCs, halons, CTC, and TCA, the domestic supply of these substances was eliminated. Related ODS equipment was dismantled and verification systems were put in place.  Legislative measures were taken to ban imports of CFCs, halons, CTC,and TCA and related enforcement measures were established. Similar regulations were put in place in 2005 to ban the use of CFCs and CTAs.  There continues to be demand for servicing the remaining CFC equipment but the limited availability and high price may provide an incentive for owners to retrofit or retire the equipment. Thus it is unlikely that enterprises will revert to ODS intensive processes. a. Risk to Development Outcome Rating : Negligible to Low 8. Assessment of Bank Performance: a. Quality at entry: The project took into account lessons learned from the Bank’s first experiences with MP operations. In countries where implementation was carried out exclusively by government agencies, implementation delays were common. The IFCT was selected to implement the investment component while the Department of Industrial Works (DIW) retained overall responsibility for supervising implementation and ensuring that ODS projects were consistent with the Country Program (CP). The ICR (p. 4) reports that the original GA was designed as an umbrella grant agreement, under which any additional sub-projects approved by the Executive Committee (ExCom), could be financed from the GA. The adoption of an umbrella approach was conceived to facilitate the processing and management of multiple sub-projects under a single GA. The adoption of an umbrella approach was conceived to facilitate the processing and management of multiple sub-projects under a single GA. Utilizing an umbrella agreement permitted adequate resources to be delegated to previously approved sub-projects and expected sub-projects over the first 2 years of implementation. However, one design weakness was the lack of a closing date for the sub-grant agreement which led to longer than anticipated implementation by large enterprise beneficiaries. The Bank team identified additional technical and commercial risks related to the introduction of new technologies not previously used in Thailand. Technical Assistance (TA) was to be provided to participating enterprises focussing on technologies which have been internationally tested and commercialized, and by requiring, in selected cases, technical cooperation agreements between the enterprise and the international technology supplier. Environmental risks associated with substitute chemicals were identified. Environmental Impact Assessments (EIAs) and local compliance certification were to be reviewed by the Bank as part of sub-project approval. Quality-at-Entry Rating: Satisfactory b. Quality of supervision: The Bank met its obligations to provide DIW and all other Project counterparts with ODS policy and technical advice. The ICR (p.22) reports that regular supervision missions as well as site-specific technical visits were conducted on average twice a year. The ICR noted that site visits provided the supervision team with a first-hand understanding of on-ground implementation experiences and were conducted as required and reported upon in annexes to the Project’s Aide Memoires. The Bank worked closely with the government and helped initiate activities which proved to be crucial to the project's success including: (i) introduction of the voucher system; (ii) development of laws and regulations; and (iii) flexibility to account for industry changes over time. The ICR (p. 22 ) reports that Bank supervision efforts were also responsible for (i) more aggressive reduction schedules as agreed with the ExCom under the NCFCP and NMBPP; (ii) formulation and implementation of policy frameworks and project implementation plans and (iii) ensuring that project outputs and outcomes were consistent with overall environmental country strategy. The Bank team monitored both safeguards and fiduciary aspects as required. Based on reporting in the ICR, however, safeguards supervision did not address potential worker layoff issues associated with specific enterprise ODS production phase-out. More attention could have focused on worker health and safety. The ICR (p. 23) also suggests that the Bank should have revisited FM implementation aspects when new FM requirements used in 2005 and two of the implementing agencies were also fiduciary agents to channel grant funds to beneficiaries. Different MTRs were planned for different parts of the project. The ICR (p.22) reports that the mid-term review of the pre-NCFCP and NCFCP was undertaken in September 2006. This report was not formally endorsed by Bank management due to concerns that the strong opinions expressed by the author could negatively impact client relations. For NMBPP, the Mid-term Review originally planned for 2011 was postponed due to the devastating flood event and was undertaken in 2013. The ICR (p. 23) notes that there was a lack of communication between the Bank and the client during 2009-2010 which contributed to implementation delays. Additionally, investment activities under NMBPP were delayed by 18 months while the Pest Management Plan was prepared by Department of Agriculture (DOA) and approved by the Bank, because the Bank had not informed the DOA of the requirement in a timely fashion. Additionally, the Bank should have updated the Results Framework continuously in the ISR for the use of both Bank management and the client. Quality of Supervision Rating : Moderately Satisfactory Overall Bank Performance Rating : Moderately Satisfactory 9. Assessment of Borrower Performance: a. Government Performance: The project benefited from sustained political will and commitment over the 19 years of implementation. Government entities at the national, provincial and municipal levels participated in project implementation assisting in the achievement of the GEO. Important government contributions included:  The phase-out of CFC, halon, TCA, CTC and methyl bromide consumption faster than prescribed under MP obligations;  Effective control of the import and export of ODS through a licensing and quota system, enactment of regulations to ban the use of CFCs in the manufacturing of domestic refrigerators and commercial refrigeration appliances  The ban on the use of CFCs and TCA in all manufacturing processes to support the sustainable phase-out of ODS in 2005; and  Thailand was the first developing country that put into effect regulations to prohibit the use of CFCs in the manufacture of domestic refrigerators and prohibit the import of CFC based domestic refrigerators. It was among the first countries to initiate a national/sectoral approach to phasing out ODS consumption. In addition, strict import regulations were put in place to prohibit the import of CFC based refrigeration appliances and vehicle inspections included refrigerant in Mobile Air-conditioning (MAC) systems. Government Performance Rating Satisfactory b. Implementing Agency Performance: Multiple entities participated in project implementation including DIW and DOA, as well as the Thai Military Bank (TMB), the Government Savings Bank (GSB) and the Industrial Finance Corporation of Thailand (IFCT) as financial agents. The DIW was the project's lead implementation agency and its PMU maintained adequate records as required. During the initial years of implementation, the DIW and IFTC carried out their responsibilities in an efficient and effective manner. However, the ICR (p.25) reports that after 2007 implementation of the project's remaining tasks faced considerable implementation delays. There were frequent changes of the Division Chief position who oversaw the National Ozone Unit (NOU), which exacerbated the situation, as the incoming chief typically arrived with little knowledge of ODS issues, creating a constant learning curve leading to implementation delays. The ICR notes that while staff rotation is normal, having five chiefs within 18 months was excessive. Additionally, overall implementation of the NCFCP was affected by the disbanding of the PMU and movement of its responsibilities to the NOU, as the experienced PMU staff left and the new staff, who were young and relatively inexperienced, faced a steep learning curve in implementing the remaining tasks. After an initial period of start-up delays the DOA’s PMU carried out its responsibilities as expected. There were some procurement shortcomings but they were eventually overcome. External audits were conducted annually by independent auditors and were submitted timely for DOA-PMU. The audit opinions were unqualified and the management letters did not reveal any major accountability or internal control issues. IFCT, TMB and GSB as financial agents prepared sub-project appraisal reports and Project Completion Reports (PCRs). The quality of the PCRs was in conformity with OPG requirements and the ExCom format respectively and documentation and filing system were systematically maintained. Implementing Agency Performance Rating : Moderately Satisfactory Overall Borrower Performance Rating : Moderately Satisfactory 10. M&E Design, Implementation, & Utilization: a. M&E Design: The project was approved prior to the Bank's adoption of the results framework evaluation methodology. However, the project was subjected to the MP’s sophisticated system of baselines, targets and M&E and its disbursements were performance-based. The design specified enterprise reporting requirements and if targets were not met, there was no additional disbursement from the MLF. b. M&E Implementation: Progress reports were prepared by DIW and DOA and submitted to the MLF Secretariat through the Bank. Implementation of the NCFCP and NMBPP was monitored and recorded through Annual Work Programs (AWP) and Biennial Work Programs (BWPs) that were submitted to the ExCom along with consumption verification reports. The ICR (p 12) reports that the AWPs and BWPs outlined the accomplishments and outcomes of activities undertaken in the previous year and the planned actions for the next year. AWPs and BWPs also allowed DIW and DOA to review and adjust their strategies and work plans during implementation. Consumption of ODSs to be phased out under the NCFCP was audited and verification reports were prepared by an independent auditor on an annual basis. The verification reports were used to confirm Thailand’s compliance with the reduction targets as agreed with the ExCom. In addition, verification reports included potential weaknesses identified within the period audited and recommendations to improve the existing system to ensure compliance with the GA. The NMBPP supported the DOA in the development and implementation of the MIS system, which linked the DOA’s Pesticide Regulatory Sub-division with Plant Quarantine Stations around the country, as well as with registered methyl bromide importers. c. M&E Utilization: According to the ICR, reporting by the Borrower was used by the Bank team during project supervision and more generally for monitoring and evaluation. Based on the M&E information, the Bank team provided recommendations and advice to address these issues or make adjustments as necessary. M&E Quality Rating: Substantial 11. Other Issues a. Safeguards: Under the Bank’s safeguards policy in place in 1994 when the project was prepared, the Project was categorized as a Category “C”. No specific safeguard actions were triggered at that time because its components offered positive environmental impacts. The ICR notes that with the development of the NMBPP, an Integrated Safeguard Datasheet was prepared and the Project's safeguard rating was reclassified to category “B”. Since methyl bromide and its alternative are pesticides and were used for managing pests that could affect agriculture or public health, the Bank OP 4.09 on Pest Management was triggered as was the Bank OP/BP 4.01 on Environmental Assessment (EA). The ICR (p.13) states that "overall, compliance with safeguards policies was satisfactory." Environment Over the life of the project, Bank environmental specialists formed part of the core supervision team, who monitored the Project’s environmental aspects and ensured that relevant health and safety issues were addressed in the design and implementation of each sub-project. The Bank team also ensured that all discarded ODS equipment was properly destroyed and received certification by DIW. Following the reclassification of the project to Category B, a Pest Management Plan (PMP) was prepared by the Recipient to use as an EA instrument in ensuring that the phase-out of MB consumption would be accomplished in a sustainable manner. The PMP was disclosed in country and at the Infoshop on November 24, 2010. The Mid-Term review in 2013 reviewed the objectives and targets set in the PMP and ascertained that project activities were complying satisfactorily with the provisions of OP 4.09 and BP 4.01 Annex C. Pesticides (fumigants) used in the project as an alternative to methyl bromide complied with both the national pesticide legislation and selection criteria in OP 4.09 and their use was fully justified under an IPM approach. Social Based on reporting in the ICR, it does not appear that any attention was given to worker layoff issues associated with enterprise production phase-out. Additionally, the ICR (p.23) suggests that the Bank’s safeguards work could have been more rigorous, with more emphasis on worker health and safety, and tighter reporting on safeguards supervision. b. Fiduciary Compliance: Procurement. The 1994 grant agreement stated that all procurement would follow Bank procurement guidelines according to the thresholds for International Competitive Bidding (ICB) and other procurement methods. Procurement was be carried out by IFCT to avoid the weak procurement capacity in the government agencies. When the grant agreement was amended to accommodate implementation of NCFCP and NMBPP, procurement for Parts B and C of the project was assigned to DIW and DOA. The project allowed direct procurement through a voucher system and pre-selection of suppliers. These procedures were considered efficient and economical to meet sub-project requirements. The ICR (p.25) notes that there were some initial procurement shortcomings but they were satisfactorily resolved. Financial Management According to the ICR financial management (FM) arrangements at entry were considered straightforward. The Bank formulated standard arrangements for reporting, external auditing, disbursement including special account and statements of expenditures. During implementation, post procurement and expenditure reviews were carried out on a regular basis by the Bank’s procurement and FM specialists. External audits were conducted on an annual basis throughout the life of the project period and no issues were identified. The audited financial reports were submitted on time for DOA-PMU. Financial management training was provided to DIW, DOA, IFCT, Thai Military Bank (TMB), and GSB and staff of the DIW and DOA regularly attended Financial Agent Workshops organized by the Bank’s Environment Department and Ozone Operations Resource Group. However, the ICR (p. 23) suggests that the Bank should have revisited FM implementation aspects when new FM requirements used in 2005 and two of the implementing agencies were also fiduciary agents to channel grant funds to beneficiaries. c. Unintended Impacts (positive or negative): The ICR (p.20) reports that ODS phase-out achieved under the Project contributed to significant indirect global environmental benefits as ODSs are considered greenhouse gases with greater global warming potential (GWP) than carbon dioxide (CO2). The CO2 equivalent impact of the phase-out achieved under the Project is calculated to be 36 million tons, equivalent to taking 7 million cars off the road permanently. d. Other: 12. Ratings: ICR IEG Review Reason for Disagreement/Comments Outcome: Satisfactory Satisfactory . Risk to Development Negligible to Low Negligible to Low Outcome: Bank Performance: Satisfactory Moderately There were shortcomings in supervision Satisfactory that contributed to implementation delays and shortcomings in the updating of supervision reports. Borrower Performance : Satisfactory Moderately IEG considers the delays in Satisfactory implementation at DOA and frequent management changes at NOU reasons for the rating to be downgraded. Quality of ICR: Satisfactory NOTES: - When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. - The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons: The ICR (pp. 25-26) identified 6 lessons learned. Some of them were statements of fact rather than lessons. The following lessons were selected, condensed and edited. Effectiveness of an integrated approach depends on attention to different stakeholder priorities : Technical and financial assistance from the MLF alone would not have been enough to achieve the project’s objectives. Commitment from the government and related stakeholders to the phase-out obligations was critical to ensure compliance with all obligations in a smooth and sustainable manner. Integration of regulatory frameworks, capacity building and technical and financial assistance should be considered as the optimal approach. Successful Implementation in the small service sectors requires more hands on management and incentives to participate. To deal more successfully with small enterprises, incentives should be provided to encourage them these workshops to urge them to participate in the program. It is important to establish simple eligibility criteria (as long as they are not against the relevant guidelines and rules) in providing incentive to these servicing workshops. Monitoring and evaluation over such long periods of implementation, requires regular updating of logframe/results framework. 14. Assessment Recommended? Yes No 15. Comments on Quality of ICR: The ICR is well-written and comprehensive. It covered all sections of the template with adequate detail. The quality of the evidence was good as was the analysis. The ICR adequately described the umbrella approach and how the process of sub-project approval and implementation was undertaken. Annex 2 presented detailed analysis of all elements of the investment and provided both outputs and outcomes for project activities including work undertaken under NCFCP and NMBPP. It also provided details on implementation challenges which occurred over the 19 years of project implementation and documented the need to amend the grant agreement to accommodate implementation of NCFCP, IS, and NMBPP. The ICR also covered with adequate detail the M&E design, implementation. and utilization. However there were some shortcomings. The ICR did not address important social safeguards related to the impact of the phase-out. The ICR did not adequately explain the difference between "use" and "consumption" of ODS substances which was the key difference between the original GEO and revised GEO. The Task Team Leader did respond to this inquiry with the accepted MP definition. The ICR also did not fully follow the split rating methodology to arrive at its outcome rating. Finally, the ICR should have included information on the nature of financing of the Multilateral Fund and the implications of the umbrella grant modality on the project implementation time frame. This information was subsequently provided to IEG in the task team comments on the draft ICR review and is critical for understanding the multiple extensions to the project completion date and the lengthy implementation period. a.Quality of ICR Rating : Satisfactory