Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) Report Number: ICRR0022595 1. Project Data Project ID Project Name P145040 Sustainable Wastewater Management Country Practice Area(Lead) Georgia Urban, Resilience and Land L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-14912 31-Dec-2015 10,040,181.56 Bank Approval Date Closing Date (Actual) 12-Jul-2013 15-Oct-2020 IBRD/IDA (USD) Grants (USD) Original Commitment 10,052,155.00 10,052,155.00 Revised Commitment 10,040,181.56 10,040,181.56 Actual 10,040,181.56 10,040,181.56 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan Kavita Mathur Victoria Alexeeva IEGSD (Unit 4) Krishnamani 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objectives (PDOs) as stated in the Grant Agreement (page 5) and in the Project Appraisal Document (PAD, page 4) was "to promote sustainable wastewater management and pilot the implementation of wastewater treatment plants". b. Were the project objectives/key associated outcome targets revised during implementation? Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 27-Jun-2018 c. Will a split evaluation be undertaken? Yes d. Components There were two components (PAD, pages 4-5). 1. Technical Assistance (TA): The estimated cost at appraisal was US$2.6 million. The actual cost was US$2.5 million. Activities in this component included: (i) TA to develop the regulatory framework for improved wastewater treatment infrastructure; (ii) training staff on wastewater management and maintenance, sludge management, disposal and reuse of water resources, and study tours on best practice technologies; (iii) developing a policy for sludge management and final disposal; and (iv) preparing feasibility studies, engineering designs, construction supervision, monitoring and evaluation, and TA to the project executing entity - the Municipal Development Fund (MDF) and the United Water Supply Company of Georgia (UWSCG). 2. Investment Grants: The estimated cost at appraisal was US$9.9 million. The actual cost was US$8.8 million. This component aimed to improve the efficiency of the wastewater sector through: (i) rehabilitating two to three Waste Water Treatment Plants (WWTPs), and (ii) providing "upstream" investments in the water and wastewater systems, such as network rehabilitation/construction and customer demand management. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project cost. The estimated cost at appraisal was US$12.5 million. The actual cost was US$11.2 million. Project financing. The project was financed by the Swedish International Development Agency (SIDA) Grant of US$9.2 million. Following approval, SIDA allocated an additional grant of US$0.8 million to cover the appraisal financing gap (ICR, paragraph 12). With this, the total grant was US$10.0 million. This amount was fully disbursed. Recipient contribution. The recipient's contribution was estimated at US$2.5 million at appraisal. Their actual contribution was US$1.2 million. Dates. The project was approved on July 12, 2013, became effective on August 15, 2013, and was scheduled to close on December 31, 2015. The project closed about five years behind schedule on October 15, 2020. Other changes. The grant closing date was extended four times during the project lifetime. Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040)  The original closing date was December 31, 2015. With the amendment of the Grant Agreement and addition of funds, the closing date was extended from December 31, 2015, to June 30, 2018.  The closing date was extended by 20 months from June 30, 2018, to February 28, 2020, due to the implementation delays in rehabilitating the WWTPs caused by the poor performance of the contractor, and to allow sufficient time to complete construction and put into operation the two WWTPs financed under the project.  The closing date was extended by three and half months from February 28, 2020, to June 15, 2020, to complete all the remaining WWTPs construction works and the two TA activities (the National Sludge Management Strategy and the National Discharge Standards).  The grant closing date was extended by four months from June 15, 2020, to October 15, 2020, to address the delays caused by the disruptions due to the COVID-19 pandemic and the lockdown measures introduced in Georgia from March to mid May 2020 that suspended activities on project sites. There were three Level 2 restructurings during the project lifetime. The following changes were made through the first restructuring on June 27, 2018.  The ICR (paragraph 8) notes that at appraisal the final number of WWTPs, their location, and the design option for the cost-effective technology was yet to be finalized. The restructuring specified the target values for the number of WWTP to be constructed (two), their location (Telavi and Tskaltubo), and the technology for wastewater treatment.  The target values of some key outcome indicators were modified as follows: (i) the target for the "increased number of urban population with access to sustainable wastewater services was increased from 40,000 to 58,000; (ii) the target values for PDO and the Intermediate Results indicators relating to the WWTPs were increased to reflect the actual design parameters for the Telavi and Tskaltubo WWTPs based on the maximum WWTPs' capacity, and associated maximum number of the population that could potentially benefit from the connection to the two WWTPs in the long run; (iii) the target for the decreased discharges of untreated wastewater to receiving water bodies was increased from 5,000 to 13,700; and (iv) the target for the number of direct project beneficiaries was increased from 40,000 to 58,000 (ICR, Table 1, page 9).  Under component one, the sector reform elements were reduced in scope at the request of the MDF and the UWSCG. The preparation of the National Drinking Water Demand Management Strategy was also dropped from the TA scope.  A new TA activity was added. This related to training UWSCG staff by a "troubleshooter" in four seasonal scenarios and training UWSCF's staff in WWTP operations on sludge removal. The results framework was modified and some changes to the key PDO targets were reduced through the second restructuring on February 28, 2020. Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040)  The target for the number of beneficiaries was reduced to 30,000, to reflect the actual number of beneficiaries who would receive improved wastewater treatment services (that is, the number of people who were connected to the sewer network), as opposed to measuring the beneficiaries based on the WWTPs design capacities.  The target for the indicator associated with decreased discharges of untreated wastewater to receiving water bodies was reduced from 13,700 to 7,200. Split rating. The PDOs remained unchanged throughout implementation. However, given that the targets were revised rather substantially with the project restructurings and the project commitments remained the same, this assessment is based on a split rating of objectives, when 58% (US$5.8 million) was disbursed before the first project restructuring, 32% (US$3.2) before the second restructuring, and the balance 10% (US$1.03 million) after the second restructuring. 3. Relevance of Objectives Rationale Country and sector context. In the years before appraisal, the quality, coverage and maintenance of basic municipal infrastructure services had deteriorated in most cities, towns and villages in Georgia, due to insufficient resources for capital investment and maintenance, an inadequate regulatory framework and insufficient management capacities. About 35% of Georgia's population was served via sewerage collection networks. The sewerage pipes were decades old and in poor condition, with treatment facilities in only five of the 29 WWTPs. While many sewer networks collected waste water, the collected wastewater was released untreated into the waterways, posing environmental and public health risks (PAD page 2). The wastewater sector also underperformed with basic measures of sustainability, such as generating revenue and raising funds for asset upgrade and maintenance. Government strategy. The PDOs were well-aligned with Georgia's "Waste Water Management Strategy" approved on April 10, 2014. This strategy provided an action plan to: (i) reform policies and the regulatory framework for treating wastewater; and (ii) implementing cost-effective WWTPs that were aligned with the European Union (EU) standards, specifically for achieving "full wastewater treatment" (mechanical, biological and nutrients removal) by 2021, with an eventual six years "buffer" period until 2027. Bank strategy. The PDOs were well-aligned with the Bank strategy. The first focus area of the Bank's Country Partnership Framework (CPF) for 2014 - 2017 articulated the need for "strengthening public service delivery to promote inclusion and equity". Objective four of the CPF identified the need for "strengthening effectiveness and accountability of public administration". The PDOs were consistent with two pillars of the CPF for 2019-2022. While pillar one of the CPF articulated the need for "enhancing inclusive growth and competitiveness" through increasing economic participation of regions, pillar three highlighted the need for "building resilience" through enhancing management of natural resources and climate change. Prior Bank-experience. The Bank has financed prior projects in the targeted regions through the first Regional Development Projects in Kakheti Region and the second project in Imerati Region which financed basic infrastructure. The Bank has supported activities aimed at improving water service delivery through Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) the Regional Municipal Infrastructure project. The above-mentioned regional projects did not include activities aimed at managing waste water. This project, the first of its kind in the wastewater sector, complemented the other regional projects by aiming to improve public health and quality of life for residents in the targeted regions through providing wastewater treatment services and enhancing the capacities of the key sector organizations. Given that this project clearly responded to an identified need, the relevance of the PDO is assessed as substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To promote sustainable wastewater management. Rationale Theory of Change. The causal links between the project activities, outputs and outcomes were clear and the intended outcomes were monitorable. TA activities aimed at the institutional strengthening of the national service provider UWSCG - aimed to improve the regulatory framework of the sector. Operational and on-the- job training to UWSCG staff on wastewater management, disposal and reuse, were aimed at sustained delivery of wastewater. Under the assumption that the developed regulatory framework is implemented to ensure sustainability and environmental soundness of the operations, piloted technologies of wastewater treatment are scaled up and the trained employees remained in the sectors, the outputs of these activities are likely to contribute to the PDO of promoting sustainable management of wastewater services in the targeted areas. Outputs (ICR, pages 11-14).  The project held two study tours in Sweden and Latvia in October 2017. Thirteen technical and four managerial-level staff from the Ministry of Regional Development and Infrastructure (MRDI), the Municipal Development Fund (MDF), the UWSCG and the Ministry of Environmental Protection and Agriculture (MEPA) were trained on international best practices on WWTP operations and maintenance.  Twenty technical UWSCG staff (including engineers, procurement specialists and environmental and social specialists) were trained. The ICR (paragraph 19) notes that before the project, UWSCG did not possess either standard operating procedures or staff experienced in wastewater treatment, as no WWTPs were in operation at that time. There were no targets for this indicator. Two staff members were trained for managing wastewater under four seasonal conditions and sludge removal as targeted. Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040)  UWSCG plans for phasing out merit-based construction of treatment plants for five, ten and fifteen years were completed as targeted. The technology piloted under this project are currently used for the WWTPs under construction in Khashuri and Koubleti.  The National Wastewater Management Strategy (NWMS) was adopted by the Government. This output was however not attributable to this project. The ICR (page 14) notes that while this project contributed to the launch of the dialogue for preparing the NWMS and its implementation, the NWMS strategy that was adopted by Government Decree #638 on April 10, 2014 was financed under the Regional and Municipal Infrastructure Development Project (RMIDP-P110126). The target value of this indicator was achieved before the appraisal of this project and was accounted for as an achievement of the RMIDP. This strategy laid out a roadmap and action plan to reform the sector policies and establish a regulatory framework governing the wastewater sector in Georgia, and to implement cost-effective WWTPs aligned with EU standards and supporting technological choices appropriate for the national context. The purpose of the technical assistance was to further develop the regulatory framework for improved wastewater treatment infrastructure.  Although the drafts of the National Sludge Management Strategy and the National Discharge Limits document were completed, these were yet to be adopted by the Government when the project closed. The team confirmed that the adoption of these documents is expected to take another few months and that the team is in close contact with the Project Implementation Unit (PIU) and the relevant ministry officials to ensure that these documents are adopted by the Government.  The activity associated with the National drinking water demand management was not completed as envisioned.  The water and wastewater tariff did not increase by 25% as targeted. The ICR (page 14) notes that the UWSCG filed an application to the Georgian National Energy and Water Supply Commission (GNERC) for revising tariffs, but the request was declined. According to the Borrower's ICR (page 57), tariff increases were not planned in the near future. The ICR notes that the Government more recently decided to freeze or waive fees for communal services due to the COVID-19 crisis during 2020 and 2021. Outcomes (ICR, page 13). The two WWTPs rehabilitated in Telavi and Tskaltubo were in compliance with the European Union (EU) effluent discharges and sludge disposal standards as targeted. Given that many of the outputs were not completed and there is limited evidence that the project activities contributed to promoting sustainable wastewater management, efficacy of this PDO is assessed as modest. Rating Modest OBJECTIVE 1 REVISION 1 Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) Revised Objective To promote sustainable wastewater management. Revised Rationale No change was made to objective 1. The same progress applies as outlined above. Revised Rating Modest OBJECTIVE 1 REVISION 2 Revised Objective To promote sustainable wastewater management. Revised Rationale No change was made to objective 1. The same progress applies as outlined above. Revised Rating Modest OBJECTIVE 2 Objective To pilot the implementation of wastewater treatment plants. Rationale Theory of Change. The causal links between the project activities, outputs and outcomes were logical, and the intended outcomes were monitorable. Rehabilitation of the two WWTPs were aimed at increasing the number of beneficiaries with access to wastewater services, and to decrease the volume of discharges of untreated waste water to water receiving bodies. The intended outcomes were likely to contribute to higher- level outcomes of improving the quality of life and the environment for the residents and visitors in the project areas. Outputs (ICR pages 15 -16).  The two WWTPs were constructed (as planned) and connected to the municipal sewer network in Telavi and Tskaltubo municipalities. The WWTPs used a fit for purpose and cost-effective technologies of anaerobic ponds combined with trickling filters. The WWTPs were designed to respectively treat 4.3 million m3/year and 2.4 million m3/year of wastewater respectively.  The WWTPs removed 500 tones suspended solid pollution loads a year when the project closed, short of the original target of 800 tons. The plants removed 380 tones of Mass of Biological Oxygen Demand (BOD) pollution load a year, short of the original target of 450 tones. The plants Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) removed 200 tones of Dried Sludge pollution loads a year, short of the original target of 230 tones a year. Outcomes The project outputs were aimed at piloting the waste treatment plants and increasing the number of people in the project areas (beneficiaries) with access to wastewater services, and decreasing the volume of untreated wastewater discharged in the project areas.  30,000 people in urban areas who were connected to the sewer network and had access to wastewater treatment services benefited from project activities when the project closed (short of the original target of 40,000). 15,000 of the beneficiaries were females (original target 20,000). When the project closed, the wastewater treatment capacity of the WWTPs had increased to 58,000 beneficiaries who were to be connected in the near future. According to the clarifications provided subsequently by the team (email dated August 24, 2021), as of August 1, 2021, 43,928 beneficiaries (including 21,964 female beneficiaries) were connected to the sewer network, exceeding the original target of 40,000 beneficiaries and 20,000 female beneficiaries.  The volume of discharges of untreated wastewater in the project areas decreased to 7,200, exceeding the original target of 5,000. The first outcome was slightly below the target when the project closed (although the target was exceeded to date according to the information provided by the team), , and the second significantly exceeded the target. On balance, efficacy of this objective is substantial, with moderate shortcomings - before the first project restructuring. Rating Substantial OBJECTIVE 2 REVISION 1 Revised Objective To pilot the implementation of wastewater treatment plants. Revised Rationale The PDO did not change, but the targets for several indicators were increased during the first project restructuring. Outputs The outputs were the same as above under the original PDO 2. Outcomes Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040)  As of August 1, 2021, 43,928 people who were connected to the sewer network and had access to wastewater treatment services benefitted from project activities, lower than the revised target of 50,000 set at June 27, restructuring.  The volume of discharges of untreated wastewater in the project areas decreased to 7,200, but did not achieve the revised target of 13,700. Given that the revised outcome targets were met for the most part, efficacy of this PDO is assessed as modest after the first project restructuring. Revised Rating Modest OBJECTIVE 2 REVISION 2 Revised Objective To pilot the implementation of wastewater treatment plants. Revised Rationale The PDO did not change, but the targets for several indicators were reduced during the second project restructuring. Outputs.  The WWTPs removed 380 tones suspended solid pollution loads a year when the project closed, as per the revised target.  The plants removed 380 tones of Mass Biological Oxygen Demand (BOD) pollution load, as per the revised target.  The plants removed 200 tones of Dried Sludge pollution loads a year, as per the revised target. Outcomes.  48, 928 people (including 21,964 female beneficiaries) who were connected to the sewer network and had access to wastewater treatment services benefitted from project activities, exceeding the target of 40,000 and 20,000 female beneficiaries.  The volume of discharges of untreated wastewater in the project areas decreased to 7,200, as per the revised target. Given that the revised targets were met, efficacy is assessed as substantial, after the second restructuring. Revised Rating Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale Efficacy under original PDOs. The first PDO is rated modest, given that many of the outputs were not completed and there is limited evidence that the project activities contributed to promoting sustainable wastewater management. For PDO 2, first outcome was slightly below the target when the project closed (although the target was exceeded since then), and the second significantly exceeded the target. The second PDO is rated substantial. On balance, overall efficacy is substantial. Overall Efficacy Rating Substantial OBJR1_TBL OVERALL EFFICACY REVISION 1 Overall Efficacy Revision 1 Rationale The first PDO is rated modest, given that many of the outputs were not completed and there is limited evidence that the project activities contributed to promoting sustainable wastewater management. The second PDO is rated modest because the two outcomes (of reducing the volume of discharges of untreated water and increasing the number of beneficiaries with access to wastewater services) were well short of the revised target. Overall efficacy is modest. Overall Efficacy Revision 1 Rating Primary Reason Modest Low achievement OBJR2_TBL OVERALL EFFICACY REVISION 2 Overall Efficacy Revision 2 Rationale The first PDO is rated modest, given that many of the outputs were not completed and there is limited evidence that the project activities contributed to promoting sustainable wastewater management. The second PDO is rated substantial because on the evidence provided subsequently by the team. Overall efficacy is rated substantial. Overall Efficacy Revision 2 Rating Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) Substantial 5. Efficiency Two approaches were used to assess the economic justification for the project: (i) cost effectiveness approach to verify that the least-cost option was selected for the WWTPs based on feasibility studies and regional comparisons, and (ii) a cost-benefit approach. Cost effectiveness approach. The ICR (paragraph 33) notes that an analysis of the feasibility study confirmed that the option used for rehabilitating the WWTPs - combining anaerobic ponds and trickling filters - was the least-cost option - as compared to other alternatives (combining primary sedimentation with trickling filters or primary sedimentation and activated sludge). The team confirmed that trickling filters were considered to be the most affordable and were also easy to maintain due to the technology. The option used (which was compliant with the Georgian and the European Union (EU) effluent and sludge disposal regulations) was about 25% and 8% cost-effective as compared to the other options. The cost per beneficiary of around US$335 in this project was comparable to similar Water Treatment Investment projects, providing value for money. Cost-Benefit approach. As the project was demand-driven and investments were not known, a priori, an economic analysis was not conducted at appraisal. An Economic Internal Rate of Return (EIRR) was retroactively calculated by the ICR team for the appraisal and completion stages for activities associated with rehabilitating the WWTPs. These activities accounted for 79% and 77% of the appraisal estimate and actual cost respectively. The project benefits were assumed to come from operational cost savings of the UWSCG, benefits from decreased incidence of water-borne infections, revenues from temporary jobs created during implementation, and revenues from the TA activities (under the assumption that 25% of TA activities were performed by local personnel). The Net Present Value at 5% discount rate at closure was US$1,409,505.30, as compared to the NPV of US$2,904,016.58 at appraisal, using the same discount rate. The ex post EIRR was 7.04%, as compared to the retroactively calculated ex ante EIRR of 9.60% (ICR, page 44). The lower EIRR at closure was mainly due to project's 27.5 month delay compared to the initial timeline, due to the delays in starting construction and subsequently during implementation. Administrative and operational issues. There were delays in launching project activities due to the time-lag of twenty two months between the signing of the Grant Agreement (GA) and project approval. The ICR (paragraph 42) notes that according to the standard Bank procedures, the signature of the GA coincides with or takes place close to the date of approval by the Bank. However, in this case the Bank approved the project 22 months later in 2015, after the GA was signed in 2013, due to the delays in processing Bank approvals. The ICR notes that the Bank Team assumed that given the nature of the Trust Fund, no formal appraisal or a PAD was required. However this was not the case, and it took substantial time to finalize the appraisal. The project closed about five years behind schedule on October 15, 2020. These delays were due to the significant delays in completing the WWTP's project preparation and procurement processes, which contributed to delays in starting the rehabilitation of the WWTPs. Although the feasibility studies for the WWTPs were completed and the cost-effective technology was chosen, lack of consensus on the part of the stakeholders and beneficiaries, contributed to delays in launching tenders for the WWTPs, with a supervision consultant selected only in December 2016. The delays in preparation were exacerbated by delays during implementation for a combination of factors, including delays in acquiring construction permits; construction Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) began only in October 2017. Issues associated with the poor performance of the consultant led to the activation of the contract's dispute resolution mechanism. Further, the Municipal Development Fund (MDF) - the implementing agency was focused mainly on rehabilitating the WWTPs- its core mandate. Inadequate attention to sector reforms, led to non-completion of many institutional strengthening activities (such as the national sludge management strategy and the national discharge standards which were yet to be adopted by the Government when the project closed). In sum, efficiency is assessed as modest in view of the administrative and operational shortcomings, which led to significant delays and non-completion of many of the activities aimed at sector reforms. Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 79.00 Appraisal  9.60  Not Applicable 77.00 ICR Estimate  7.40  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Under the original outcome targets. With substantial relevance of objectives to the Government and Bank strategies, substantial efficacy, and modest efficiency, the overall rating is Moderately Satisfactory (4). Under the first revised outcome targets. With substantial relevance of objectives to the Government and Bank strategies, modest efficacy, and modest efficiency, the overall rating is Moderately Unsatisfactory (3). Under the second revised outcome targets. With substantial relevance of objectives to the Government and Bank strategies, substantial efficacy and modest efficiency, the overall rating is Moderately Satisfactory (4). A double split rating is applied based on the disbursement shares before and after the project restructurings in 2018 and 2020, when a disbursement share was at 58%, 32%, and 10% respectively (total financing was US$10 million). The overall outcome rating is Moderately Satisfactory, the weighted value is 4 (58%*4+32%*3+10%*4 = 3.7). a. Outcome Rating Moderately Satisfactory Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) 7. Risk to Development Outcome Political risk. There is substantial political risk to the sustainability of the project benefits. The ICR (paragraph 63) notes that earlier during implementation and then again during the COVID-19 pandemic, the Government did not increase the tariffs for water and wastewater treatment services (as well as for other utilities). This implies that further government subsidies will continue to be required to cover the Operation and Maintenance (O&M) costs of the WWTPs. Government commitment. According to the information provided subsequently by the team, the number of beneficiaries who are connected to the sewer network and receive access to wastewater treatment services due to the project activities is expected to increase in the near future. In terms of the regulatory issues, the National Sludge Management Strategy had yet to be adopted by the Government when the project closed. This poses serious environment risks. 8. Assessment of Bank Performance a. Quality-at-Entry The preparation of this project drew on the experience from the prior Bank-financed Regional Development and Region Municipal Infrastructure projects. Lessons incorporated at design included adopting an approach that combined infrastructure investments with technical assistance activities aimed at sector reforms, using least-cost solution as prescribed in the waste water management strategy, and using a phased approach to establish WWTPs. The implementation arrangements were appropriate, with the Municipal Development Fund (MDF) for Georgia in charge of executing the project. The MDF had executed prior-Bank financed projects (the Regional Municipal Infrastructure projects) and its staff had adequate experience in Bank's fiduciary policies (PAD, paragraph 20). Several medium risks were identified at appraisal, including risks associated with the weak implementation capacity and risks associated with the financial sustainability of the WWTPs. With mitigation measures, project risk was rated moderate at appraisal (PAD, paragraph 88). Appropriate arrangements were made at appraisal for safeguards and fiduciary compliance (discussed in section 10). There were significant shortcomings in identification and appraisal. One, while this project complemented the prior Bank-financed activities, the level of ambition was relatively high with respect to the sector reforms, given the country context. Further, while MDF had experience in implementing prior Bank- financed projects- its core mandate was on rehabilitating the WWTPs. Inadequate attention to sector reforms resulted in non-completion of many institutional strengthening activities (such as the national sludge management strategy and the national discharge standard document which was yet to be adopted by the Government). Two, it is not clear if the project was ready for implementation. Not all of the relevant stakeholders were involved in project preparation, which contributed to implementation delays. For instance, while the project aimed to increase cost recovery for the WWTPs Operation and Maintenance (O&M), the Georgian National Energy and Water Supply Regulatory Commission (GNERC), the agency for setting tariffs for public services was not involved when this project was prepared. Also, the relevant municipalities and their staff that oversaw construction permits Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) and environmental safeguards had not been identified or included as one of the key stakeholders at the preparation stage. and three, there were M&E shortcomings (discussed in section 9). Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The ICR (paragraph 61) notes that since the beginning of the project, an experienced engineer in the Bank team was involved in preparing the Terms of Reference (TORs) for construction works and associated supervision. This member regularly visited the WWTPs construction sites to ensure that the construction works adhered to the required technical quality. The Bank supervision team provided close support during the construction phase, especially in the last two years of implementation, through participating in site visits by local staff, which took place, first on a monthly and then on a bi-weekly basis in the last year. This helped in addressing the construction delays. The Borrowers ICR (page 61) notes that the team had excellent and efficient work style, full understanding of policies and procedures and that the Bank's expert advice contributed to the completion of works. The support provided by the supervision team aided in safeguards and fiduciary compliance (discussed in section 10). The team clarified that there were three Task Team Leaders during the project lifetime. There were shortcomings in supervision. The Bank supervision could have used the Mid-Term Review (MTR) for a comprehensive review project progress, including in terms of realism of achieving the results indicators. The key changes to the target were made only through the second project restructuring on February 2020, just about eight months before project closure on October 2020 (discussed in section 9). The shortcomings in M&E were not addressed during implementation (discussed in section 9). Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Several of the key indicators for component two activities, such as the number of WWTPs that were rehabilitated, the gender-disaggregated number of beneficiaries who were provided with wastewater services, and the biochemical indicators providing information on the quality of wastewater treatments were standard indicators used in the sector. The data for monitoring these indicators could easily be collected once the WWTPs were fully operational. The indicator associated with cost recovery for the Operation and Maintenance (O&M) of WWTPs was appropriate for monitoring sector reforms. Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) There were some shortcomings in M&E design. First, the indicator associated with "compliance per effluent discharges and sludge disposal standards of the EU requirements" was for environmental purposes and limited to the design of two plants. This indicator was clearly inadequate for monitoring the PDO of "promoting wastewater management in Georgia". Two, the targets for the number of beneficiaries had to be revised to reflect accurately the actual number of beneficiaries who were connected to the sewer network and hence would receive the treatment benefits from the WWTPs (as opposed to the design capacity of the WWTPs capacity and the associated maximum number of the population that could potentially benefit from the connection to the WWTPs in the long run). Three, one of the intermediate indicator - adoption of the National Drinking Water Demand Management Strategy - could not be attributed to this project, as the activities were completed under a prior Bank-financed project, and four, given that most indicators could only be monitored after the rehabilitation of the WWTPs, the methodology could have been refined to measure partial progress during implementation. b. M&E Implementation The deficiencies at M&E design were only partly corrected during implementation. While the Bank and MDF proactively reviewed and adjusted the results framework as part of project restructurings, the targets for indicators were only processed towards the end of implementation. The indicators associated with monitoring progress towards sector reforms (component one activities) were for the most part, not addressed. Neither were the activities outside the scope of this project (adoption of the National Drinking Water Management Strategy) removed during implementation. Further, the indicators were not revised to enable attribution for partial achievement during implementation and to remove activities that were not attributable to this project. c. M&E Utilization The ICR (paragraph 49) acknowledges that M&E utilization was limited given that some key results could only be measured at project closing. The ICR also notes that the rigid nature of the indicators, without clear definitions to measure intermediate progress contributed to the limited use of the M&E system. In sum, M&E is assessed as modest, in view of the shortcomings at design, which were not rectified during implementation. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as an Environmental Category B (partial assessment) project. Three safeguard policies were triggered at appraisal: Environmental Assessment (OP/BP 4.01); Involuntary Resettlement Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) (OP/BP 4.12); and Projects on International Waterways (OP/BP 7.50) (PAD, pages 13-15). No other safeguard policies were triggered during implementation. Environmental Assessment. The PAD (paragraph 47) notes that adverse environmental impacts were expected to be temporary and confined to temporary inconveniences to local residents during construction. Environmental Management Plans (EMPs) were to be prepared for each sub-project during implementation (PAD, paragraph 47). There was compliance with environmental safeguards (ICR, paragraph 52). The MDF prepared and publicly-disclosed an Environmental Management Framework (EMF) in 2013, and preliminary Environmental Impact Assessment (EIA) was carried out for the conceptual design of both WWTPs. With the development of detailed designs, site-specific EIA were prepared and publicly-disclosed for Telavi and Tsklutubo in 2018. The ICR (paragraph 56) notes that the project closed with an outstanding environmental mitigation measure, namely reinstatement of the WWTPs site (yet to be undertaken). This obligation was part of the environmental and social management plans approved by the Bank, as well as a condition of environmental permits issued by Ministry of Environmental Protection and Agriculture (MEPA) for operating the WWTPs. According to the Borrower's Completion Report, these activities are expected to be finalized within 12 months of construction of the WWTPs. The ICR notes that during implementation, adherence to occupational health and safety (OHS) standards for workers and communities became an issue at both work sites. The shortfalls were rectified as a result of close monitoring by the MDF, strong oversight by the technical supervision company, and support from the Bank supervision team. The ICR notes that the project closed without any OHS incidents. Involuntary Resettlement. The PAD (paragraph 48) notes that the temporary reallocation and loss of income or productive assets during construction were possible. A Resettlement Policy Framework (RPF) was prepared and publicly-disclosed at appraisal to address such issues. There was compliance with the safeguards on involuntary resettlement. The ICR notes that the project did not require any resettlement and hence there was no need for developing resettlement action plans. Projects on International Waterways. OP/BP 7.50 "Project on International Waterways" was triggered as the project intended to collect water and treat wastewater that would be released to the surface water bodies draining to the Black and Caspian Seas, shared by several countries. However, the project did not envisage new construction of WWTPs, but rehabilitation of existing WWTPs. Rehabilitation of the existing WWTPs was expected to collect waste water, which was being directly released to the surface bodies of the Black Sea or the Caspian Sea watersheds. As the design for rehabilitation of WWTPs allowed for water purification as per the World Bank, the EU, and the Georgian standards, the project was not expected to affect the quality and quantity of water flows to the other riparian's. Hence, an exception to the riparian notification requirement was approved by the Europe and Central Asia (ECA) Regional Vice President on April 10, 2013. b. Fiduciary Compliance Financial management. A financial management assessment of the MDF conducted at appraisal, concluded that the financial management arrangements of the MDF were deemed to be satisfactory. The Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) MDF had executed several Bank-financed projects and its staff had experience with the Bank's financial policies. The financial management risk was rated as moderate at appraisal (PAD, paragraph 38). The ICR (paragraph 59) notes that financial management was rated as satisfactory. There were no major issues, although there were some delays in submitting financial statements. The quality of financial statements were deemed to be satisfactory throughout implementation. The team clarified that the external audits were unqualified. Procurement management. The Municipal Development Fund (MDF) was managing the procurement of several Bank-financed projects. The procurement risk was rated moderate at appraisal (PAD, paragraph 43). Although there were some delays, these were resolved with support from the Bank supervision team. The ICR does not report of any case of mis-procurement. The ICR (paragraph 58) notes that procurement performance was rated satisfactory throughout implementation. The ICR does not discuss the rationale for this rating, given that ongoing challenges with respect to the relationships between the contractor and the MDF led to the activation of the contract's dispute resolution mechanism. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons IEG uses three main lessons from the experience of implementing this project, with some adaptation of language. 1. Institutional capacity building and technical assistance are crucial for project implementation in a weak capacity environment. Comprehensive training programs, relevant study tours on international best practices and implementation support by the Bank are essential for project implementation. This was especially the case for Georgia where the wastewater sector was underdeveloped at appraisal. The lesson from this project is that enhanced capacity building to Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) ensure the functionality of the rehabilitated Wastewater Treatment Plants is especially important in medium-size cities where capacity may be even more limited than in capital or larger cities. 2. It is important to be realistic about the level of ambition for sector reforms. This was the first Bank project in the wastewater sector, yet the level of ambition for the sector reforms was relatively high, leading to non-achievement of some of the intended results (such as increasing cost recovery ratio, supporting water demand management). A more gradual and phased approach aimed at sector reforms would be more appropriate, based on a realistic assessment of the willingness and capacity of key stakeholders during preparation. 3. The engagement of all stakeholders from the early stages of project preparation is necessary for facilitating project implementation. In this project, several of the key stakeholders, such as the Georgian National Energy and Water Supply Regulatory Commission or the relevant technical staff of the municipalities who oversaw construction permits, environmental approvals and other important procedures were not identified as one of the key stakeholders at the preparation stage of the project. This contributed to delays during implementation and rather limited results with sector reforms. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is clear. The evidence provided in the ICR's main text and annexes present a complete and robust evidence base to support the reported achievements. The ICR candidly discusses the shortcomings in Monitoring and Evaluation design and how they were only partially rectified during implementation. The ICR makes appropriate use of photographs to show the results before and after the project. The theory of change provided in the ICR presents a clear picture of the links between project activities, outputs and outcomes. The lessons provided in the ICR appropriately reflect the experiences and findings for the project and they are sufficiently linked to the narrative and ratings in the project. The information provided in the Borrower's ICR provides useful details. The main text of the ICR largely adheres to the recommended length. The ICR notes that procurement was rated "satisfactory" throughout project implementation. The ICR does not address the rationale for this rating and it is not clear how this rating was justified, given that the challenges with respect to the relationships between the contractor and the Municipal Development Fund (MDF) led to the activation of the dispute resolution mechanism during implementation. The quality of the ICR is substantial. a. Quality of ICR Rating Substantial Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Sustainable Wastewater Management (P145040) Page 19 of 19