Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) Report Number: ICRR0022235 1. Project Data Project ID Project Name P118064 UY OSE Sustainable and Efficient Country Practice Area(Lead) Uruguay Water L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-81830 28-Feb-2018 42,000,000.00 Bank Approval Date Closing Date (Actual) 05-Jul-2012 31-Dec-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 42,000,000.00 0.00 Revised Commitment 42,000,000.00 0.00 Actual 42,000,000.00 0.00 Prepared by Reviewed by ICR Review Coordinator Group Kishore Laxmikant Victoria Alexeeva Ramachandra Jammi IEGSD (Unit 4) Nadkarni 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The project development objective (PDO) is to increase the sustainability of the Borrower by improving the reliability and resilience of its water supply and sanitation systems, enhancing its efficiency, and strengthening its management capacity (Loan Agreement dated December 11, 2012, Schedule 1; Project Appraisal Document (PAD) para. 14). The Borrower was OSE (Obras Sanitarias Del Estado) - the National Water Supply and Sanitation Company of Uruguay. Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) For the ICRR, the PDO has been parsed as follows: Objective 1: To increase the sustainability of the Borrower (OSE). The efficacy would be assessed on the basis of achievements under the sub-objectives of (I) improving the reliability and resilience of the water supply systems of OSE; (ii) enhancing the efficiency of OSE; and (iii) strengthening the management capacity of OSE. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 07-Sep-2016 c. Will a split evaluation be undertaken? No d. Components Component 1: Investing in Reliable Water Supply Infrastructure (appraisal cost US$22.20 million; revised cost following restructuring US$33.10 million; actual cost US$35.65 million) This component was to finance construction, expansion and rehabilitation of water facilities. Sub- component 1: Aguas Corrientes Intake Structure and Electrical Systems: Construction of a new water intake structure and carrying out of improvements to the electrical system of the water pump at the Aguas Corrientes water treatment plant (WTP). The system design capacity would be 15,000 m3/hour. Sub- component 2: Construction of two WTPs in the cities of Durazno and Treinta y Tres. (PAD paras. 18 to 21). Component 2: Managing Water and Energy More Efficiently: (appraisal cost US$37.50 million; revised cost at restructuring US$37.50 million; actual cost US$31.30 million) This component was designed to finance activities to support OSE's flagship Non-Revenue Water (NRW) and Energy Management Programs. Sub-component 1: Non-Revenue Water Reduction: (i) Establishment of Districts of Measurement and Control (DMC) in Montevideo, Paysandú, Mercedes, Rivera, Salto, and Las Piedras-La Paz. (ii) Activities to strengthen institutional support for NRW reduction across Uruguay including: (a) establishment of standardized procedures for construction of water connections; (b) carrying out capacity building activities for OSE staff; (c) carrying out of annual evaluation workshops; and (d) provision of technical assistance to strengthen the NRW program. (iii) Carrying out of nation-wide improvements to OSE's metering system including (a) replacement of micro and macro meters; (b) carrying out a pilot program to test automated micrometers; and (c) establishment of a telemetry program. Sub- component 2: Energy Management: (i) Development of an Energy Management Plan (EMP) for OSE. (ii) Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) Replacement of pumps and underperforming equipment. (iii) Acquisition of monitoring equipment and hardware. (iv) Capacity building activities for OSE staff including a training program for energy efficiency. (PAD paras. 19 to 24). Component 3: Preparing for the Future: Management, Planning and Risk (appraisal cost US$11.10 million; revised cost following restructuring US$4.20 million; actual cost US$0.95 million financed from the Bank loan) This component was to finance management, planning and risk assessment activities to improve OSE's social reach, environmental management and economic efficiency. The advances made by OSE were to be captured under the BMEI jointly formulated by OSE and the Bank Project team. Sub-component 1: Corporate Management: (i) Establishment of an asset management program. (ii) Design and implementation of a logistics management model. (iii) Design and implementation of a knowledge and innovation management model. Sub-component 2: Risk Management and Planning: Support to OSE for developing a Long Term Strategic Plan including: (i) Carrying out a review of OSE's mission and vision statements. (ii) Carrying out of a strategic analysis of risks and operations. (iii) Development of risk maps. (iv) Carrying out of a climate vulnerability risk assessment. (v) Development of contingency plans for droughts and floods. (vi) Development of Water Safety Plans. Sub-component 3: Environmental Sustainability: (i) Provision of technical assistance to strengthen the UGA. (ii) Development of a sludge master plan. (iii) Development of a comprehensive strategic wastewater plan. (iv) Establishment of a water quality and quantity monitoring program in selected watersheds. (v) Implementation of a hydraulic and water quality management program at the Laguna Del Sauce watershed. (PAD paras. 25 to 28). Component 4: Knowledge Sharing and Project Management Activities: (appraisal cost US$2.0 million; revised cost following restructuring US$0.90 million; actual cost US$ 0.65 million financed from the Bank loan). This component was to finance: (i) Support for project management, supervision and engineering activities; (ii) Carrying out of external and internal dissemination of project results and major accomplishments; (iii) Carrying out of knowledge sharing activities with particular emphasis on South-South exchange. (PAD para 29). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The project cost at appraisal was US$84 million (PAD para. 8). During the project restructurings, the total cost was revised to US$75.70 million and there were reallocations among the components. The actual cost at project completion was US$66.76 million (as reported in the ICR Data Sheet). However, there are some discrepancies in the total project costs (US$68.55 million) as estimated from the costs of the project components reported in the ICR (Annex 4 Table 4.1). The significant reduction in project cost, as compared to the appraisal estimate, was due to cost reductions under Component 2 (due to dropping of two of the targeted cities for NRW reduction) and Component 3 (due to dropping of some activities and the carrying out of others by OSE utilizing its own financial and staff resources). Project Financing: The project was financed through an IBRD loan in the amount of US$42 million to OSE with the guarantee of the national government (PAD para. 8). It was fully disbursed. Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) Borrower's Contribution: The contribution agreed at appraisal was US$42 million (PAD para. 8). The actual contribution was US$24.76 million (as reported in the ICR Data Sheet). Dates: The project was approved on July 5, 2012 and closed on December 31,2019 with the original closing date extended by 22 months. Restructurings: The project had three restructurings. First Restructuring (September 2016): (Restructuring Paper, Section C. Proposed Changes and Annex 1 Results Framework). This was a Level II restructuring. There was no change in the PDO. The restructuring followed a period of a downturn in the national economy which affected the GoU's finances and also had an adverse financial impact on OSE. Consequently, a number of the originally planned activities were reduced in scope or re-defined to align them with OSE's available funds. The rationale for the changes was: (i) modifying some indicator targets to better align them with ongoing implementation progress; (ii) improving methodology to more accurately calculate energy savings and recovery of non-revenue water (NRW) (PDO level indicator); (iii) correcting baselines; and (iv) adjusting formulation and definition of some indicators. Specific changes to indicators and targets were the following:  Revision of the PDO indicator "NRW recovered by reducing real (physical) and apparent (commercial) losses" to "cumulative volume of NRW recovered under the project (m3)" defined as the sum of the water recovered each year for each of the systems supported under OSE's NRW program.  Revision of the calculation methodology for the PDO indicator "reach a satisfactory level in BMEI" to adjust it based on implementation experience. In the process 20 of the original sub-indicators were removed and 15 were added based on alignment with agreed activities and outputs to be undertaken under the project.  Revision of calculation methodology, and hence baseline and targets, for PDO indicator "energy saved in systems where the project is being implemented". The target value was reduced from 37,983 MWh to 13,640 MWh. Changes in the project's components were the following:  Component 1: Increase in the allocation from US$22.2 million to US$33.1 million since the contract prices came in higher than planned.  Component 2: Scope reduced. Dropping of the planned activities for (i) NRW reduction at two sites (Mercedes and Paysandú) and (ii) acquisition of the monitoring equipment and software related to energy conservation measures.  Component 3: Scope reduced including dropping of some activities and substitution by others as follows: Under Sub-component 3.1, dropping of the establishment of the knowledge and innovation management model and replacing it with capacity building activities for ozonation. Under Sub- component 3.2, development of risk maps, carrying out of a climate vulnerability assessment and development of contingency plans were moved to be implemented under the Water Safety Plans (WSPs) for each drinking water plant. Under Sub-component 3.3, (i) Dropping of the establishment of a water quality and quantity monitoring program and implementation of a hydraulic and water quality management program for the Laguna del Sauce watershed from the project scope because they were being developed by the National Water Directorate in collaboration with OSE (ICR para. 21); (ii) Dropping of the development of a comprehensive strategic wastewater plan because new priorities (protection of Rio Santa Lucia watershed) were emerging and straining OSE's sanitation Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) unit and its budget (ICR para. 21). Some activities were planned to be carried out under OSE's own budget. Second Restructuring (July 2017): (Restructuring Paper, Section C. Proposed Changes): This was to enable the extension of the closing date from February 28, 2018 to December 31, 2019 to allow time for completion of three contracts for the Treinta y Tres WTP, Montevideo NRW system, and Rivera NRW system. Third Restructuring (February 2018): (Restructuring Paper, Sections II and III. Proposed Changes) This was a Level II restructuring. Modifications were made to indicators related to NRW in the project supported systems. The revisions in indicators and targets were:  Revising the end target for the Montevideo system to reflect the drop of activities in two lots.  Dropping the indicator for the Las Piedras-La Paz system to reflect the lack of impact in La Paz.  Modifying indicators in the Rivera and Salto systems to reflect expected impact in those systems.  End targets for two PDO indicators "cumulative volume of NRW recovered under the project (m3)" and "energy saved in systems where the project is being implemented" were modified to reflect extension of the closing date and thus the additional gains under these indicators.  The target for energy saved in systems implemented under the project was increased to 20,780 MWh from 13,640 MWh established under the first restructuring. A split evaluation is not applied as (I) the PDO was not changed during implementation; (ii) the revision of the PDO indicator reflected the use of an improved methodology to more accurately calculate energy savings and non-revenue water reduction volumes, and (iii) most of the adjustments to indicators and targets made during restructurings were to better align them with project activities and outputs based on the implementation experience. 3. Relevance of Objectives Rationale Country and Sector Context: Access to potable water and adequate sanitation systems in Uruguay is high with 100 percent of the population having access but reliability of water supply can be affected by vulnerability to floods and droughts. OSE provides 98 percent of the urban population with household connections and continuous access to potable water, sewerage services to 43 percent of the interior of the country, and treats 60 percent of the wastewater collected. The population not served by OSE has adequate access to water supply (wells) and sanitation (septic tanks). Governance instruments for the water sector include a Water Law, water quality regulations administered by URSEA (Energy and Water Services Regulatory Unit), and guidelines under the National Plan for Integrated Water Resource Management. OSE is regarded as one of the most advanced water and sanitation utilities in the region. However, there is scope for further strengthening of its management and operations. The main challenges faced are in respect of: (i) improvement of resiliency and efficiency of infrastructure; (ii) capacity to plan for, and manage, risk, including the impact of floods and droughts; (iii) improvement of operational efficiency particularly reduction of non-revenue water (NRW) and high levels of energy consumption; and (iv) further strengthening of management capacity in the areas of corporate management, environmental stewardship, Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) and strategic planning. Towards this end, OSE had prepared and started implementing a Five Year Strategic Plan (2012 to 2017) with three pillars: (i) investing in reliable infrastructure; (ii) improving operational and commercial efficiency; and (iii) strengthening utility management (PAD, paras. 1 to 12). The OSE Sustainable & Efficient Project was aligned with these objectives. Alignment with Country Partnership Strategy: The project development objective is consistent with the latest Country Partnership Framework (CPF) that was prepared in December 2015 and covers the period FY2016 to FY2020. The CPF supports the Government of Uruguay (GoU) priorities reflected in the GoU's Five Year Budget Plan which includes an emphasis on protecting the environment with a special focus on water resources (CPF para. 40). The CPF includes three strategic pillars out of which Pillar 1 focuses on building resilience to economic and weather vulnerabilities. The CPF states that, based on the GoU's requests, the proposed support under the CPF will be focused on strengthening the efficiency of expenditures, improving selection and implementation of investment projects, and supporting state-owned enterprises (SOEs) (CPF para. 48). The CPF recognizes that improving efficiency in water use and addressing wastewater challenges are key challenges (CPF para. 58). The CPF specifically includes the project as an ongoing intervention under Pillar 1 (Building Resilience to Shocks), Objective 1 (increase the Efficiency of Public Investment and Strengthen the Efficiency of Selected SOEs) (CPF Annex 1). Alignment with national priorities: The project development objective is consistent with the GoU's priorities as reflected in the Five Year Budget Law that focuses on eight key objectives, one of which is protecting the environment, with a special focus on water resources. The GoU has prepared a National Plan for Integrated Water Resource Management (ICR para. 2) under which availability of reliable and high quality water resources plays a central role. The CPF states that the World Bank Group (WBG) will promote resilience of water related sectors to climate variability and climate change by supporting the GoU's plans for Integrated Water Resources Management and Development (IWRMD) (CPF para. 58). Prior Experience in the Sector: The Bank has been involved in the water and sanitation sector in Uruguay since 1988 and has financed a number of projects, including the OSE Modernization and Systems Rehabilitation APL Phase 2 project which included a number of components to strengthen OSE's management and operations. These aimed at improving OSE's transparency, accountability, operating efficiency, and attention to clients. The OSE Sustainable & Efficient Project built on the base established and results achieved under the earlier projects, with particular focus on addressing the issues mentioned above in this section. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Sub-Objective 1: Improving resilience and reliability of OSE's water supply systems Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) Rationale The efficacy of the project is best assessed based on achievements under the following sub-objectives, parsed out to emphasize progress across the various aspects of the theory of change: (i) improving the resilience and reliability of the water supply systems of OSE; (ii) enhancing the efficiency of OSE; and (iii) strengthening the management capacity of OSE. The theory of change (TOC) was that the construction of a new water intake of capacity 15,000 m3/hour at the Aguas Corrientes Water Treatment Plant (WTP) together with a supporting force main and equipment improvements in pumping stations would provide OSE with stand-by water intake capacity to supplement its existing capacity of 30,000 m3/hour. The existing capacity was assessed to be insufficient in case of unfavorable weather and climate events which could result in water supply service interruptions during OSE's planned or emergency maintenance activities. The additional intake capacity set up under the project would provide greater resilience to OSE's operations by increasing the redundancy margin (from 0% to 30%) and help OSE maintain a higher level of service by increasing the reliability of OSE's supply of drinking water to the population without significant interruptions. The two new WTPs at Durazno and Treinta y Tres would replace two existing WTPs that were (I) located in areas that were subject to flooding and (ii) operating with equipment and facilities in major need of improvement. The two new WTPs would be located in areas that were not subject to flooding and thus add to OSE's resilience to unfavorable weather and climate events. The WTPs would have new and modern equipment, which would increase the reliability of OSE's water supply operations in the two cities. In addition, installation of new equipment (pumps, motors, electrical systems) at selected OSE sites outside Aguas Corrientes would help further increase reliability of OSE's operations in delivering drinking water to the population. However, while the activities being undertaken were clearly linked with the PDO, the PDO outcome indicators adopted (number of beneficiaries benefiting from the rehabilitation works under the project and the number of piped water connections benefiting from the rehabilitation works under the project) were not directly linked to the concepts of improved resilience and reliability to be addressed under this sub-objective. Outputs: as reported in the ICR (Annex 1 Results Framework and paras. 34 to 37):  New water intake point (capacity 15,000 m3/hour) constructed and in operation at Agua Corrientes WTP  New WTP (capacity 700 m3/hour) constructed and in operation in Durazno city  New WTP (capacity 450 m3/hour) constructed and in operation in Treinta y Tres city  10 electric motors installed and in operation at Aguas Corrientes WTP  New equipment (pumps, motors, electrical systems) installed and in operation at selected OSE sites outside Aguas Corrientes (original target 251; revised target at restructuring 51; actual 51; achievement 20% of original target, 100% of revised target) Outcomes: as reported in ICR (Annex 1 Results Framework)  Number of people benefiting from rehabilitation works under the project (original target 1.86 million; actual 1.86 million; achievement 100%)  Piped household water connections benefiting from rehabilitation works under the project (original target 433,900; actual 433,900; achievement 100%) Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) Assessment: The targets in relation to outputs and intermediate results were largely met. In terms of the indicators adopted under the project, the outcomes were met, pointing to a substantial achievement of improved water supply systems. However, the indicators used do not fully capture the PDO aspects of resilience and reliability. The outcome indicators - number of piped water household connections and number of people benefiting from the rehabilitation works under the project - have only an indirect relation to the more encompassing concept of resilience to external shocks. This is acknowledged in the ICR (para. 72). The reliability target was not formulated in terms of hours of continuous water supply or reduced time for repairs and maintenance. The ICR reports (para. 38) that, based on beneficiary surveys in Durazno and Treinta y Tres, 97 % and 89% respectively of the sampled population expressed satisfaction with the water supply service. However, the ICR does not include a discussion of the observed impacts of the project investments in terms of demonstrating greater resilience and reliability. Rating Substantial OBJECTIVE 2 Objective Sub-Objective 2: Enhancing the efficiency of OSE Rationale The two key areas identified for increasing efficiency were Non-Revenue Water Reduction (NRW) and Energy Management. In regard to NRW, the TOC was that the efficiency improvements would best be accomplished by utilizing a District of Measurement and Control (DMC) approach that would initially be introduced in six selected cities (with the objective of expansion to other areas in Uruguay). The DMC program would require support for institutional strengthening as well as for equipment financing. Institutional support measures would include: (i) establishment of standardized procedures for construction of water connections; (ii) carrying out capacity building activities for OSE staff; (iii) carrying out annual workshops for evaluation; and (iv) provision of technical assistance to strengthen the NRW reduction program. In addition, the project would finance the carrying out of nation-wide improvements in OSE's metering system including replacement of micro and macro meters; carrying out a pilot program to test automated micrometers; and preparation of a telemetry program. The combination of institutional and equipment related support would result in increasing OSE's operational and financial efficiency through a significant reduction in NRW. In regard to Energy Management, the TOC was that OSE's ongoing energy management activities would need to be strengthened by institutional support and equipment financing. Institutional support measures would include: (i) development of an Energy Management Plan (EMP); and (ii) capacity building activities for OSE staff, including training programs in energy efficiency. Equipment financing would include (I) replacement of pumps and underperforming equipment; and (ii) acquisition of monitoring equipment and hardware. The combination of institutional support and equipment financing would result in increasing OSE's efficiency in energy management, resulting in savings in energy used. Outputs: as reported in the ICR (Annex 1 Results Framework and paras. 42 to 46): Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) Reduction of NRW is reported in terms of liters/connection/day (l/c/d).  Reduction in NRW in the Montevideo system (baseline 799 l/c/d; original target 635 l/c/d; revised target at restructuring 736 l/c/d; actual 696 l/c/d; underachieved by 61 l/c/d compared to original target; overachieved by 40 l/c/d compared to revised target)  Reduction in NRW in the Rivera system (baseline 668 l/c/d; original target 386 l/c/d; revised target at restructuring 483 l/c/d; actual 420 l/c/d; underachieved by 34 l/c/d compared to original target; overachieved by 63 l/c/d compared to revised target)  Reduction in NRW in the Salto system (baseline 579 l/c/d; original target 396 l/c/d; revised target at restructuring 483 l/c/d; actual 418 l/c/d; underachieved by 22 l/c/d compared to original target; overachieved by 65 l/c/d compared to revised target)  Reduction in NRW in Las Piedras-La Paz system (baseline 703 l/c/d; original target 400 l/c/d; revised target at restructuring 555 l/c/d; overachieved by 9 l/c/d compared to original target; overachieved by 164 l/c/d compared to revised target)  Number of electric motors installed and in operation in Aguas Corrientes WTP (original target 10; actual 10; achievement 10; achievement 100%)  Number of equipment (pumps, motors, electrical systems) installed and in operation outside Aguas Corrientes (original target 251; revised target at restructuring 51; actual 51; achievement 20% as compared to original target, 100% as compared to revised target)  Energy Management Program published and ready for implementation Outcomes: as reported in ICR (Annex 1 Results Framework)  The PDO indicator for NRW was " cumulative value of water recovered under the project" (original target 48.76 million m3; revised target at restructuring 81.90 million m3; actual 89.29 million m3; achievement 183% compared to original target, 109% compared to revised target)  The PDO indicator was "energy saved in systems where the project is being implemented" (original target 37,983 MWh/year; revised target at restructuring 20,780 MWh/year; actual 26,250 MWh/year; achievement 69% compared t original target; 126% compared to revised target) Assessment: In regard to NRW, although two of the originally targeted cities (Mercedes and Paysandú) were dropped from the project at the first restructuring in 2016, the overall target in terms of cumulative volume of non-revenue reduction was raised and the actual achievement exceeded both the original and revised targets. In part, this overachievement also reflects the inclusion of the additional NRW reduction resulting from the extension of the original closing date by 22 months. In regard to energy management, an Energy Management Program for OSE was prepared and was ready for implementation. In terms of energy saved, although the original target was substantially reduced at the first restructuring, the actual achievement was substantial (69%) compared to the original target and exceeded (126%) the revised target. Rating Substantial OBJECTIVE 3 Objective Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) Sub-Objective 3: Strengthening the management capacity of OSE Rationale Activities under the project were directed to help OSE prepare for the future by strengthening its capacities in the areas of corporate management; planning, and risk management; and environmental sustainability. Under the TOC, this was to be accomplished through a combination of development of studies and analytical tools, including supporting software and hardware, and capacity building through training and technical assistance. Regarding corporate management, the main tools to be developed were (i) establishment of an asset management program to help OSE deploy and operate its assets more efficiently; (ii) design and implementation of a logistics management model to help OSE improve its logistics efficiency, including sludge management, and (iii) design and implementation of a knowledge and innovation model to enable OSE to share information and experience with similar utilities in the region. With regard to planning, risk management, and environmental sustainability, the activities included (i) strategic analysis of risks and expectations; (ii) development of risk maps; (iii) carrying out of a climate vulnerability assessment; (iv) development of contingency plans for droughts and floods; and (v) development of water safety plans. The environmental sustainability related activities were expected to yield outputs including (I) Long Term Strategic Plan; (ii) a Sludge Master Plan; (iii) a Comprehensive Wastewater Plan; (iv) a Water Quality and Quantity Monitoring Program in selected watersheds; and (v) a Hydraulic and Water Quality Management Program at the Laguna Del Sauce watershed. The combination of activities and outputs was expected to result in further strengthening of OSE's management capacity, including operations, planning and risk management, and environmental sustainability. During implementation, some significant changes were made to the scope and/or manner of carrying out the above activities. Activities dropped were (i) design and implementation of a knowledge and innovation model for OSE and (ii) development of a comprehensive wastewater plan. Development of risk maps, climate risk vulnerability assessments, and contingency plans, were moved for inclusion in Water Security Plans as they were developed. Development of the water quality and quantity monitoring program in selected watersheds and the hydraulic and water quality management program for the Laguna Del Sauce were not carried out under the project since they were intended to be covered under other ongoing initiatives outside the project. Activities for strengthening of OSE's corporate management, originally to be supported by external technical assistance, were modified to enable them to be carried out by OSE utilizing its own resources. Outputs: as reported in the ICR (Annex 1 Results Framework and paras. 39, 40 and 47 to 54):  OSE Asset Management Plan for electromechanical equipment prepared and applied for water supply, wastewater collection and treatment in one region  OSE Logistics Model prepared  Ten OSE areas are using software for quality management  Water Security Plans prepared for different regions (original target 18, actual 19; achievement 110%)  Software for quality management installed and used in 10 OSE areas  Internal communications created and operational  Strategic Planning Process prepared and applied  Indicators Book published regularly  32 training events carried out on environmental issues, monitoring, control, wastewater treatment, and effluent control  Pilot studies prepared under sludge management program - including (I) prototype for bio-solid drying for agricultural application - this is planned to be included in the technical design of eight new Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) wastewater treatment plants (WWTPs) and (ii) matching supply of nutrient-rich bio-solids from WWTPs to demand for fodder for livestock Outcomes: as reported in the ICR (Annex 1 Results Framework) A single PDO indicator was included in the Results Framework to measure the project outcomes. This was the Business Management Efficiency Index (BMEI) that was jointly developed by OSE and the World Bank project team based on guidelines and templates used in the World Bank Global Practice and the International Water Authority (IWA). The BMEI was based on 100 defined activities. During the first restructuring in 2016, the BMEI was modified with the dropping of 20 of the originally defined activities and their replacement by 15 newly added or re-defined activities. The rationale was to better align the BMEI with the post-restructuring activities and outputs being supported under the project, and thus enable the BMEI to better reflect actual progress made by OSE under the project. The PDO target set under the BMEI was Satisfactory (representing a score of between 60 to 79 out of a possible 100). The ICR rates the actual achievement at project completion as Excellent (representing a score of at least 80 out of a possible 100). While the ICR does not provide an indication of the score actually achieved by OSE, the project team provided IEG additional detailed information in regard to achievement of outputs, impacts, and outcomes to supplement that provided in the ICR. Assessment: While the ICR reports that a number of the planned activities were completed (as reported above) and reflected in the BMEI, there were significant modifications in some of the activities. The ICR (paras. 25 and 26) acknowledges that these changes, including the dropping of the BMEI key indicator related to "operative knowledge and innovation management" limited and/or delayed OSE's plans to incorporate risk analysis in its Long Term Strategic Planning and limited the ambition of its wastewater management transformation. As discussed above in Section 2, under the first restructuring in 2016, the allocation for Component 3 (aligned with this theme) was significantly reduced from US$11.10 million to US$4.30 million and the allocation from the IBRD loan was reduced from US$5.55 million to US$0.95 million (ICR para. 47). Technical assistance and supporting software/hardware planned to be provided under the project were substantially reduced. The scope of the originally planned activities was significantly reduced and modified to enable them to be carried out by OSE from its own resources, including its own staff. Key planned activities and outputs that were dropped or modified included: (I) Development of a Strategic Wastewater Plan - dropped because of emerging new priorities and the strain on OSE's budgetary and staff resources (ICR para. 21). (ii) Knowledge and Innovation Management Model - dropped and substituted by capacity building activities on ozone technology for algae removal (ICR para. 22). (iii) Development of risk maps, carrying out climate vulnerability assessment, development of contingency plans - dropped because they were planned to be implemented as part of Water Security Plans (WSPs) for each WTP (ICR para. 22); in this regard, the ICR, together with additional information provided by the project team, reports that 19 WSPs were prepared under the project, which included the planned activities. (iv) Preparation of Water Quality and Quantity Program and Implementation of Hydraulic and Water Quality Management Program at the Laguna Del sauce watershed - dropped because they were being developed by the National Water Directorate in collaboration with OSE (ICR para.22); based on the additional information provided by the project team, these activities were later completed. The ICR (paras. 25 and 26) acknowledges that these changes, including the dropping of the BMEI key indicator related to "operative knowledge and innovation management" limited and/or delayed OSE's plans to incorporate risk analysis in its Long Term Strategic Planning and limited the ambition of its wastewater management transformation. On balance, taking into account the additional information provided by the project team, given the substantial achievements in most targeted activities, outputs, and impacts, the efficacy for Sub-Objective 3 is rated Substantial. Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project had a single development objective, "To improve the sustainability of the Borrower (OSE)" which was evaluated under three Sub-Objectives (1) improving the resilience and reliability of the water supply systems of OSE; (2) enhancing the efficiency of OSE; and (3) strengthening of the management capacity of OSE. For Sub-Objective 1, output targets were achieved and were clearly linked to the sub-objective of improving water supply systems. However, the PDO outcome indicators adopted were not appropriate for capturing the outcomes of resilience and reliability. For this reason, the efficacy for Sub-Objective 1 is rated Substantial with moderate shortcomings. For Sub-Objective 2, the output and outcome targets in regard to NRW reduction and energy efficiency were achieved or substantially achieved, and the efficacy rating for Sub-Objective 2 is Substantial. For Sub-Objective 3, while there were some reductions in scope, achievements of most of the planned activities, outputs, and impacts were substantial and the efficacy for Sub-Objective 3 is rated Substantial. Based on the above, and given the level of achievement, the overall efficacy is rated Substantial. Overall Efficacy Rating Substantial 5. Efficiency Economic and Financial Efficiency At appraisal, an economic cost-benefit analysis was carried out for Component 1 (Resilience and Reliability) and Component 2 (NRW Reduction and Energy Efficiency) that together accounted for about 71% of the total project cost. The indicator applied was the economic rate of return (ERR). The ICR reports (Annex 4) that the post-completion economic cost-benefit analysis was carried out using the same methodology as adopted at appraisal but updated to reflect actual results and costs and prices prevailing at project completion. The analysis covers project investments covering about 98% of the total project cost at completion (52% for the resilience and reliability component and 46% for the efficiency component). For the Resilience and Reliability component, the ICR reports an estimated post-completion ERR of 35.2% (compared to 39.9% estimated at appraisal) and for the NRW Reduction component, it is 77.6% (compared to 30.9% at appraisal). This includes a high ERR of 129.1% for the Montevideo-Las Piedras sub-component. The Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) project team explained that this was due the fact that, at completion, the actual project cost was substantially lower (63%) than the estimate at appraisal and the NRW reduction totals were higher due to an increase of 16% in connections as compared to appraisal estimates. For the project, the post-completion ERR is estimated at 44.4% compared to 34.9%% at appraisal. Administrative and Implementation Efficiency Project implementation duration: The project's planned implementation period was 60 months. Under the second restructuring (2017), the original closing date was extended by 22 months to allow for completion of three ongoing contracts (Restructuring Paper, Section C. Proposed Changes). Project cost: At appraisal, the estimated total project cost was US$84 million. The actual project cost as reported in the ICR Data Sheet was US$66.76 million. However, based on the actual costs of the project components as reported in the ICR (Annex 4 Table 1) was US$68.55 million. The significant reduction in the project cost at completion was due to reductions in the cost of Components 2 (NRW Reduction and Energy Efficiency) and Component 3 (Strengthening of OSE's Management Capacity). Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 71.00 Appraisal  34.90  Not Applicable 98.00 ICR Estimate  44.40  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives is assessed as High since the PDO was well aligned with the national priorities of increasing Uruguay's environmental sustainability, resilience to climate related shocks, and increasing the efficiency of state-owned enterprises, all of which are being supported under the Bank's Country Partnership Framework currently in effect. The project's overall efficacy is rated Substantial. For Sub-Objective 1 (improving resilience and reliability of OSE's water supply systems), the targeted outputs were achieved to deliver the outcome of improved water supply. However, since the PDO outcome indicator adopted was not suitable for measuring the project's resilience or reliability impacts, the efficacy rating for this sub-objective is Substantial with moderate shortcomings. For Sub-objective 2 (NRW reduction and increasing energy efficiency), the targeted outputs and outcomes were achieved or substantially achieved, and the efficacy rating is Substantial. For Sub-Objective 3 (strengthening OSE's management capacity), while there were some changes in scope and/or the manner of carrying out the activities, most of the planned activities, outputs, and impacts were Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) substantially achieved, and the efficacy of Sub-Objective 3 is rated Substantial. The project's efficiency, as measured by the Economic Rate of Return, exceeded the expectations at appraisal, and the rating for efficiency is Substantial. Based on these ratings, the project's outcome is rated Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome Technical risks: These risks are rated low. The technologies used in the project interventions are familiar to OSE and its should be capable of maintaining the outcomes without support from external consultants. Administrative and management risks: These are rated low. During project implementation, OSE demonstrated commitment to the project approach and activities by financing, from its own funds, activities that could not be financed under the project. The ICR reports (para. 97) that OSE has developed an Action Plan to use its own funds to carry on implementation and expansion of energy efficiency, NRW reduction, and environmental management activities initiated under the project. Financial risks: These are rated moderate. Sustainability of the project interventions and their expansion in future will depend critically on OSE's financial capacity. This depends upon the state of the economy and the GoU's policy in addressing unforeseen adverse events, including external shocks. Early on during project implementation, a downturn in the national economy led to financial issues for the GoU and OSE, and required a restructuring of the project, including major reductions in the allocated funds for some of the project components. OSE's ability to continue to maintain the gains under the project, and expand them further, would depend upon its financial capacity for which the GoU's support would be essential. 8. Assessment of Bank Performance a. Quality-at-Entry The project design benefited from experience gained from the Bank's long involvement with OSE and, in particular, the predecessor project (OSE Modernization and Systems Rehabilitation APL2) that included activities to help strengthen OSE's management capacity. The project design was influenced by the World Bank's raised benchmark for supporting interventions in higher income countries such as Uruguay which called for a more innovative and transformative agenda rather than infrastructure financing (ICR para. 88). The project's design in regard to the capacity building component was ambitious - a large number of activities was included under the component and supported by a substantial allocation (US$11.10 million) to cover technical assistance and software/hardware. A positive feature of the project design was the establishment of a Business Management Efficiency Index (BMEI), jointly developed by Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) OSE and the World Bank team, based on guidelines from the World Bank Global Practice and the International Water Authority (IWA). The BMEI was intended to help monitor the large number of activities under the capacity development component. During implementation, particularly following a period of financial difficulty in Uruguay that also affected OSE, the activities had to be dropped, substantially modified, or reduced in scope, with a drastic reduction in the allocated amount. As discussed earlier in Section 4, while some of the activities were completed in a modified form by OSE utilizing its own staff and financial resources, others were left to be completed using resources outside the project. Also, there were some shortcomings in regard to M&E design that are discussed below in Section 9. Quality-at-Entry Rating Satisfactory b. Quality of supervision The project supervision team was proactive during the early part of implementation. A Mid-Term Review (MTR) was carried out in November 2015 which preceded the first restructuring in September 2016. Key changes during restructuring included (I) a re-definition of some of the indicators to better align them with the scope of the project-supported activities; (ii) modification of scope and re-allocation of financial resources between components; and (iii) modification of the BMEI to better align it to capture progress under the activities supported by the project. However, the project team did not take the opportunity to modify the PDO outcome indicators under Sub-Objective 1 (resilience and reliability) to make them more relevant to the concepts of resilience and reliability. Following the first restructuring in 2016, a large number of the strategic and operational studies and capacity building activities were agreed to be carried out by OSE utilizing its own resources. While it is not clear from the ICR to what extent the project supervision team was proactively involved in following up on the progress and outcomes of the activities, additional information provided to IEG by the project team indicates a substantial degree of involvement in following up on progress during project implementation. The experience points to a need for continued diligence in following up on progress in achievement of project outputs and impacts during supervision. The project team had two TTLs during implementation. 15 supervision missions were carried out during implementation. The team was adequately supported by fiduciary, environmental and safeguard specialists. The team benefited from the location of the fiduciary and safeguard teams in Buenos Aires (which was 30 minutes away by air from Montevideo). The Environmental Specialist became a Co-TTL for the project which helped in overseeing the implementation of the environmental interventions which were a significant part of the project. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) 9. M&E Design, Implementation, & Utilization a. M&E Design M&E design was impacted by the large number of activities included, particularly under Component 3 (Strengthening OSE's Management Capacity). While the inclusion of a BMEI was, in concept, a positive feature of the M&E design, the definition of the activities included was based on a general framework and not fully aligned with those being carried out under the project. Consequently, the BMEI had to be substantially modified during the first restructuring in 2016. Also, as indicated in the ICR (para. 72), in regard to indicators, the PDO indicator in regard to "improving resilience and reliability of OSE's water supply systems" referred to the number of piped household connections benefiting from rehabilitation activities. The indicator does not address directly the objective of increasing resilience and has only a limited relation to the objective of increasing system reliability (e.g. increasing the number of hours of continuous water supply, reduced down times for equipment facilities repairs and maintenance, etc.). The NRW indicator had to be re-formulated during the first restructuring to make it more monitorable. b. M&E Implementation During implementation, a number of adjustments were made to the M&E system, including re-definition of indicators, changes in baselines and targets based on more relevant data, and modifications to the BMEI to align it better with activities under the project. These changes substantially corrected the design shortcomings of the M&E system and the system was able to generate most of the targeted data during implementation. One shortcoming that was not addressed throughout the implementation period was the need to modify the PDO indicators for Sub-Objective 1 to make them more relevant to capturing the outcomes under this Sub-Objective. The ICR reports that there were significant time lags between the information collected by the PIU from the individual operating and administrative units (ICR para. 76). c. M&E Utilization The M&E system was linked with OSE's operations and management system, drawing information from, and feeding information into, the overall system to facilitate policy making within OSE. The ICR also reports that the M&E system was used to identify problems and carry out remedial actions, but it does not provide details in this regard (ICR paras. 78 and 79). M&E Quality Rating Substantial 10. Other Issues a. Safeguards At appraisal, the project was classified as Category B requiring partial environmental assessment (EA) under the Bank's Environmental and Social Safeguards policies. Other safeguard categories triggered were: Involuntary Resettlement (OP 4.12); Natural Habitats (OP 4.04); Physical Cultural Resources (OP4.11); and Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) International Waterways (OP7.50), although the PAD noted that the project fell under the exception to the notification requirement mentioned in para. 7(a) of OP7.50. Environment: The ICR reports that no adverse impacts or issues that were not previously identified occurred during implementation. Moderate to low risks and potential impacts were successfully managed with the safeguard instruments developed under the project. Project environmental compliance was consistently rated Satisfactory throughout project implementation (ICR para. 81). Social: The ICR reports that, given the possibility that some involuntary resettlement may arise during implementation, OSE has prepared an Involuntary Resettlement Policy Framework (IRPF), but this was never required during implementation (ICR para. 83). The ICR does not provide any information whether an Environment and Social Management Plan (ESMP) was established for the project, including grievance redressing arrangements for handling complaints from the population. The ICR does not provide any information regarding the remaining safeguards that were triggered at appraisal: Natural Habitats, Physical Cultural Resources, and International Waterways. The ICR does not provide details of the safeguard ratings at the time of the last ISR prior to project completion. b. Fiduciary Compliance Financial Management (FM): The ICR reports that, overall, the FM system was rated Satisfactory. Required financial reports were provided in time. Audit reports, with unqualified opinions, were submitted timely to the Bank. Some issues during the early stages of implementation were slow implementation due to a lack of counterpart funds - the ICR reports that these were resolved during project restructuring. The ICR also reports that there were some non-compliance issues which were rectified, but it does not provide any detail in this regard (ICR para. 84). Procurement: The ICR reports that procurement performance is rated Satisfactory. World Bank rules and procedures were complied with except for minor deviations (not specified) which were corrected, and that there were no other issues in regard to non-compliance. The ICR does not indicate what were the FM and Procurement ratings in the last ISR prior to project closing. c. Unintended impacts (Positive or Negative) The ICR does not report any unintended impacts of the project. d. Other --- Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The ICR (paras. 98 to 101) lists a number of lessons. Specially relevant to projects being implemented in similar environments is the following: Lasting institutional transformation for major water supply and sanitation utilities requires a series-of-projects approach where successive projects build on, enhance and expand on the gains accomplished under earlier projects. In case of OSE, strengthening of the utility's operational and management capacity built on significant achievements that resulted from the Bank's earlier engagements with OSE. In addition, an IEG lesson that emerges from the experience under this project is that, when the project includes a large number of management strengthening activities, with analytical, strategic, and operational outputs, it is important to ensure that adequate attention is given to assessing the observed impacts of the outputs. This would require monitoring of results and their impacts as a part of ongoing project supervision. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is well-written and consistent with the Bank's ICR preparation guidelines. It provides a clear theory of change. The reporting is candid and the analysis is evidence-based. The ICR provides adequate information in the Results Framework and Annexes to support its findings and conclusions as to outputs and outcomes, but this could have been strengthened further by a discussion of the impact of the outputs on key areas of focus under the project - increasing resilience and reliability and strengthening OSE's management capacity. In general, while reporting outputs, the ICR does not provide sufficient information in regard to the observed impacts of the activities. For activities that were dropped from the project and included under other ongoing initiatives, the ICR does not indicate the extent to which these were actually accomplished (discussed in Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review UY OSE Sustainable and Efficient (P118064) Section 4 above). In regard to PDO outcomes, it could have provided an explanation of how the reported actual end-results (number of beneficiaries) had been calculated. The ICR does not include a discussion of the Safeguard OPs triggered and does not report the safeguard and fiduciary ratings in the last ISR filed prior to project closing. In addition, there are some discrepancies in regard to the reporting of project costs (as reported in the ICR Data Sheet) and those estimated from the costs of the individual project components reported in the text of the ICR (discussed in Section 5 above). The ICR provides a number of lessons learned from the project some of which are also relevant for other projects that are implemented in similar environments. Rating: On balance, the ICR is rated Substantial on the margin. a. Quality of ICR Rating Substantial Page 19 of 19