Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) Report Number: ICRR0021724 1. Project Data Project ID Project Name P112073 BR MST Federal Integrated Water Sector Country Practice Area(Lead) Brazil Water L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-80740 31-Dec-2016 16,397,842.22 Bank Approval Date Closing Date (Actual) 12-Jul-2011 31-Oct-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 107,332,500.00 0.00 Revised Commitment 63,783,375.00 0.00 Actual 19,802,426.03 0.00 Prepared by Reviewed by ICR Review Coordinator Group Katharina Ferl Vibecke Dixon Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Project Appraisal Document (PAD) (p. ix) and the Financing Agreement of December 15, 2011 (p. 6) the objective of the project was “to support the Borrower to improve the coordination, and strengthen the capacity, among key federal institutions in the water sector toward an integrated approach.” b. Were the project objectives/key associated outcome targets revised during implementation? Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 30-Jun-2016 c. Will a split evaluation be undertaken? Yes d. Components The project included five components: Component 1: Water Resources Management (appraisal estimate US$44.21 million, actual US$4.82 million): This component was to support the deployment of tools and instruments to the National Water Resources Management System (SINGREH), enhance institutional development, reduce the disparities between the Borrower’s federal and states water management systems, identify actions to mitigate and adapt to climate change, streamline procedures, and establish criteria for ongoing monitoring and evaluation systems to increase efficiency and legal compliance of water resources guidelines and policies with the Borrower’s 1997 Water Law No. 9.433, which established the National Water Resources Policy. Component 2: Water, Irrigation, and Disaster Risk Management (appraisal estimate US$40.73 million, actual US$2.82 million): This component was to provide support for institutional strengthening to improve water infrastructure, irrigation and National Civil Defense System (SINDEC) activities, and to raise the overall assessment capacity of strategic existing water infrastructure and disaster risk management capacity, including floods and droughts and other hazards. Activities to be implemented were to include assessment of federal and state water infrastructure plans in the Northeast Region; development of criteria for selecting water infrastructure projects, preparation of state and national irrigation plans with incentives to optimize irrigation systems, creation of a risk management information system; development of a risk management plan, including risk identification, monitoring and early warning systems, and emergency response plans; development of information systems for water infrastructure; and increase of strategic planning for drought risk reduction and climate change. Component 3: Water Supply and Sanitation (appraisal estimate US$32.44 million, actual US$5.35 million): This component was to provide support to the implementation of the Borrower’s 2007 National Sanitation Law No. 11.445, the improvement of the quality of the provision of water supply and basic sanitation services and contribution to promote universal access to these services. Activities to be financed under this component were to include support to the implementation of the National Basic Sanitation Guidelines and of the National Information System on Basic Sanitation (SINISA), assessment of Growth Acceleration Program (PAC) investments in water supply and basic sanitation infrastructure, development of a national technical cooperation in the water supply and basic sanitation areas, development and expansion of a training, research and technology national network in water supply and basic sanitation, preparation of local and regional water supply and basic sanitation plans for states and municipalities; and review and evaluation of programs and actions included in the Multi-Year Plan (PPA) and PAC. Component 4: Intersectoral Coordination and Integrated Planning (appraisal estimate US$20.96 million, actual US$6.74 million): This component was to provide support to integrated planning, identify Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) areas of mutual interest, overlap, or conflict in the sectoral plans which impact and/or depend upon water, carrying out of studies and institutional improvements involving multiple sectors, and support water management and conservation measures preferentially in the São Francisco and the Araguaia-Tocantins river basins. Planned activities were to include improvement of water management and use associated with the Integration Project in the São Francisco River Basin, integration of planning and regulation in the water resources and water supply and basic sanitation sectors and in the energy and navigation sectors, implementation of methods to integrate river basin environmental assessment for planning in the hydroelectric, river navigation and water supply and sanitation sectors at the state and municipality level, and development and implementation of systems and methodologies to monitor and evaluate policies and public investments in the water sector. Component 5: Project Management, Monitoring, and Evaluation (appraisal estimate US$4.49 million actual US$0.67 million): This component was to finance (i) the management of the operational aspects of the project in order to coordinate, monitor, and evaluate all of the interventions undertaken to ensure that they meet the targets, timetables, and objectives originally specified and to ensure overall efficient administration including reporting, financial management, and auditing, and (ii) the effective transfer of knowledge and the use of best practices at the subnational level including a strong communication program, including workshops and seminars to inform the relevant stakeholders at the river basin, municipal, state, and federal levels. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The project was estimated to cost US$143.11 million. The project’s actual cost was US$26.81 million. Due to implementation delays resulting from weak institutional capacity and due to a substantial devaluation of the Brazilian Real, US$43.6 million was cancelled during the project restructuring in June 2016. The Bank team stated (May 31, 2019) that the remaining at project closure US$47.38 million was cancelled. Financing: The project was to be financed by a Bank loan in the amount of US$107.33 million and counterpart funds of US$35.77 million. Borrower Contribution: The Borrower was to contribute US$35.77 million. Actual contribution was US$7.00 million. The actual contribution was significantly lower than originally planned due to the government’s budget freeze in 2015 and beyond as a result of the country’s macroeconomic crisis in 2014. Dates: On June 30, 2016, the project was restructured to: i) revise two of the PDO indicators to reflect the activities being implemented. The third PDO was dropped, as it was not possible to be measured. A new indicator was proposed to reflect the integrated approach of the project; ii) drop, revise, and add several intermediate outcome indicators to reflect the revision of activities to be implemented. According to the Bank team (July 3, 2019) during the mid-term review, the revised indicators were agreed with the client. The final version of the revised results matrix was received by the Bank prior to the restructuring. However, when the restructuring package was prepared, due to a glitch in the Portal, the revised matrix in the system did not reflect all the changes agreed with the client. After restructuring approval in the Portal, it was not possible to modify the matrix anymore. Therefore, the revised matrix was registered in the first Implementation Status Report prepared after restructuring; iii) reallocate funds between components; iv) extend the project’s closing date by 22 months from December 31, 2016 to October 31, 2018 to allow for sufficient time to complete strategic activities necessary for the achievement of the PDO and the inclusion of additional Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) activities specifically geared toward mitigating the effects of the water crisis, particularly in the Northeast and Southeast regions of Brazil; iv) the Secretariat of Regional Development (SDR) was included under the Ministry of National Integration as a beneficiary of project's activities. 3. Relevance of Objectives Rationale According to the ICR (p. 6), despite past achievements at the time of project appraisal, Brazil was still facing significant challenges in improving its water resource management (WRM) due to freshwater resources being unequally distributed across regions. In the country’s northeast, the poorest part, and in its largest metropolitan areas, water scarcity was increasing and was expected to increase even further. Also, most large cities suffered from water pollution which had a negative impact on public health, the environment, and the cost of water treatment for downstream users. In 2007, the government launched the Growth Acceleration Program (PAC) which planned to invest US$52.7 billion in water sector infrastructure. Between 2011-2014 the government planned to invest US$33.2 billion in water-related infrastructure. However, these investments faced several challenges such as inadequate planning, limited institutional capacity, insufficient attention to feasibility and sustainability, and fragmentation between key entities in the sector resulting in inefficiencies and ineffectiveness. The objective of the project was in line with the Bank’s most recent Country Partnership Strategy (FY18-23) which aims under its third focal area “inclusive and sustainable development” to foster more efficient use of resources, and to focus on interventions related to areas such as water scarcity. Also, the Systematic Country Diagnostic which was conducted in 2016 stressed the importance of water for the livelihoods of the poor and the bottom 40 percent as well as the economic growth of the country. Furthermore, it emphasized that Brazil’s water infrastructure was not able to respond to extreme climate events. However, while there is clear alignment between the project’s development objectives and the country- and World Bank strategies, the relevance of the objectives is pitched at a level that does not adequately reflect a potential solution to a development problem. While acknowledging the difficulty of the operational environment, a shortcoming here was that the objective was not defined such that its achievements would be plausibly traceable to improvements in access to water and quality of water in urban areas envisioned to arise from strengthening coordination and capacity from key entities in the Brazilian water sector. These may be longer term targets but tracking them and identifying them is an important aspect of a successful development operation. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) EFFICACY_TBL Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) OBJECTIVE 1 Objective To support the Borrower to improve the coordination, and strengthen the capacity, among key federal institutions in the water sector toward an integrated approach: Rationale The project’s theory of change linked the production of outputs such as the deployment of tools and instruments to the National Water Resources Management System (SINGREH), enhance institutional development, reduce the disparities between the Borrower’s federal and states water management systems, identify actions to mitigate and adapt to climate change, streamline procedures, and establish criteria for ongoing monitoring and evaluation systems to increase efficiency and legal compliance of water resources guidelines and policies with strengthening water resources management of key federal water institutions. Furthermore, the project’s theory of change linked the production of outputs such as an assessment of federal and state water infrastructure plans in the Northeast Region, development of criteria for selecting water infrastructure projects, preparation of state and national irrigation plans with incentives to optimize irrigation systems, creation of a risk management information system with improving water, Irrigation, and Disaster Risk Management within key federal water sector institutions. Also, the project’s theory of change linked the production of outputs such as assessment of Growth Acceleration Program (PAC) investments in water supply and basic sanitation infrastructure, development of a national technical cooperation in the water supply and basic sanitation areas, development and expansion of a training, research and technology national network in water supply and basic sanitation, preparation of local and regional water supply and basic sanitation plans for states and municipalities with strengthening the capacity of key federal water institutions in providing water supply and sanitation services. Finally, the project’s theory of change linked the provision of outputs such as improvement of water management and use associated with the Integration Project in the São Francisco River Basin, integration of planning and regulation in the water resources and water supply and basic sanitation sectors and in the energy and navigation sectors with strengthening the intersectoral coordination and integrated planning function of key federal water institutions. Outputs (when one of the outputs listed below does not include a target it means that the ICR did not provide one). Strengthen the capacity:  ANA developed: i) a proposal of alternatives for improvement in water rights procedures in water bodies with unavailability of water due to intensive use; ii) a proposal of alternatives for improvement in water rights procedures considering future scenarios of uncertainty of water availability; and iii) definition of protection criteria for new reservoirs against backwater effects, iv) Hydrogeological Assessment of the Karst Aquifers of the São Francisco Basin and Proposition of Strategies for its Sustainable Use.  MMA developed: i) III National Formation Seminar on Environmental Education and Water Management, and the I Seminar on Environmental Education and Integrated Water Resources Management of the Doce River Basin; ii) the Water Resources State Plan for Goiás State; and iii) the Hydrogeological Cartography of groundwater resources of the Paraná State.  The Ministry of National Integration (MI) developed: (i) the National Water Security Plan - PNSH, ii) the Federal District Irrigation Plan, iii) the Strategic Actions Plan for the Rehabilitation of Federal Dams Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) – PLANERB; iv) the Identification of demands and alternatives for provision of capacity building on rural water theme; and v) updated the publications for to Civil Defense Technical Manuals.  The Ministry of Cities (MCid) developed (i) the Action Plan Proposal for the Reuse of Treated Effluents Policy and Water Loss and Efficient Use of Electrical Energy Management Project; provided Technical Assistance for the Regulation of Water and Sanitation, Services (WSS); and developed Methodologies and Formulation of Guidelines for Certification of WSS Services Providers Information. Improved coordination:  The Atlas Brazil for Depollution of River Basins via Urban Sewerage Treatment was developed.  The National Water Security Plan – PNSH was developed.  The Action Plan Proposal for Reuse of Treated Effluents Policy was developed.  The Diagnostics for preparation of the Economic-Ecological Macro Zoning Rio São Francisco Hydrographic Basin (phase 1) Outcomes:  The Inter-ministerial Management Committee (IMC) for federal water sector programs was formally established in 2012. The IMC met once in 2012, once in 2013, twice in 2014, and four times in 2015 (an average of twice a year) with the participation of all institutions involved in the INTERAGUAS prior to restructuring. The target of the committee being established and functioning, meeting twice a year at the Executive Secretariat’s level was achieved.  The PDO indicator “25 water sector activities and projects implemented by institutions participating in the project included in the GoB’s multi-year plan (PPA 2012-2015) following an integrated approach, as attested by an independent evaluation” was dropped during the restructuring. The ICR (p. 12) stated that it was not possible to assess to what extent this indicator was achieved since the activities were implemented by different federal government dependencies and were not disaggregated at the activity level. Rating Modest OBJECTIVE 1 REVISION 1 Revised Objective The objective of the project was not revised. Revised Rationale While the PDO did not change with the restructuring, the following changes were made and thus, the level of achievement has changed due to important amendments to the PDO indicators. These are as follows: Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) Strengthen capacity:  A quantitative and qualitative modeling of river stretches in basins considered critical was implemented by ANA.  An evaluation of ANA’s training actions for the National Water Resources Management System (SINGREH) was conducted. The aim was to consolidate data for ANA’s training actions for SINGREH, develop an impact assessment methodology and assess the impact of ANA’s training actions.  A diagnosis and evaluation of the National Water Resources Plan (PNRH) was implemented by MMA. The aim was to prepare a diagnosis of progress, achievements and results obtained with the implementation of the PNRH and assess PNRH’s impact on integrated Water Resource Management (WRM) and propose recommendations for the new PNRH.  An integrated technical planning unit was functioning within the Ministry of National Integration at project closure, achieving the target.  10 technical staff in the areas of hydraulic infrastructure, irrigation, civil defense and regional development were trained, not achieving the target of 100.  A manual for civil defense (responsible for risk and disaster management activities) was developed, not achieving the target to develop a total of four manuals for technical hydraulic infrastructure, irrigation, and regional development.  A strategic action plan for the rehabilitation of federal dam safety was established but was not implemented, therefore, not achieving the target.  A National Irrigation Plan was not prepared and completed as originally planned, therefore, not achieving the target. This indicator was added during the restructuring.  Four states irrigation plans were completed, not achieving the target of seven plans. This indicator was added during the restructuring.  The improvement of the rural water and sanitation program including technology and management innovations has not started yet, not achieving the target of the proposals being prepared. This indicator was added during the restructuring. 12  The number of professional specialists, members of the National Water Resources System (SINGREH) increased from 91 specialists in 2011 to 651 in 2018, not achieving the original target of 400 and surpassing the revised target of 120 specialists.  The National Water Resources Plan was revised (second revision) and adopted, achieving the target. This intermediate outcome indicator replaced the original indicator “information system established and functioning to monitor and evaluate the implementation of the National Water Resources Plan”.  14 water resources management institutions (at the federal, state, and the municipality level) were supported by the project, achieving the target of 14 institutions. This indicator was added during the restructuring.  Six water resources planning instruments were improved, achieving the target. The Ministry of the Environment produced: i) methodological roadmap to establish a municipal environmental zoning; ii) analysis of sub-surface waters in the state of Parana; and iii) the Water Resource Management Plan for Goias. Ana produced: iv) the National Water Security Plan; v) the National Sewage Atlas; and iv) the Hydrological Evaluation of Karstic and Fissure-Karstic Aquifers in the Sao Francisco Hydrographic Region. This indicator was added during the restructuring.  Three studies contributing to improve the analysis of water rights concession were finalized, achieving the target. This indicator was added during the restructuring. Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073)  Guidelines or drought preparedness and response developed within the Ministry of Integration (MI) were not developed, not achieving the target. This indicator was added during the restructuring.  23 critical basins of special interest to the country’s water resources management with qualitative and quantitative water balance were finalized for improved water resources management. (*Critical basins of special interest as classified by ANA in Portaria ANA N°62/2013).” The target of 23 basins was achieved. This indicator was added during the restructuring.  38 progress reports, financial management and audits of the program were prepared, surpassing the original target of 10 reports and the revised target of 14 reports. The target of this indicator was revised during the restructuring.  19 seminars, workshops or technical meetings were held, surpassing the original target of 10 and the revised target of 14 seminars, workshops or technical meetings. The target of this indicator was revised during the restructuring. Improved coordination:  During the restructuring the PDO indicator “25 water sector activities and projects implemented by institutions participating in the Project included in the GoB’s Multiyear Plan (PPA 2012-2015) following an integrated approach, as attested by an independent evaluation” was dropped as it was not possible to measure it.  A water indicators portal connecting sector-specific information systems, studies, and analytical work was implemented and is available to the public. However, the task of building electronic links to the other water-related federal institutional information systems had not been completed by the time of project closing, therefore, not achieving the target. This PDO was rephrased during the restructuring from the original version “A water indicators portal connecting sector-specific information systems, studies, and analytical work is implemented and available to the public”.  National and/or regional planning instruments developed in an integrated manner were not all completed (only the National Sewage Atlas and the National Water Security Plan were completed), not achieving the target of three instruments. This indicator was added during the restructuring.  One action strategy proposal was prepared, not achieving the target of two action strategy proposals being prepared (thematic areas were: “the reuse of treated effluents” and “water supply in the surroundings of large water infrastructure works”). This indicator was added during the restructuring.  The target of six metropolitan regions with water supply integrated planning developed was not achieved. This indicator was added during the restructuring.  The ecological-economic macro zoning of Sao Francisco river basin was completed, achieving the target. This indicator was added during the restructuring. Under the coordination of the MMA, this activity was generated with the participation and cooperation of numerous federal government agencies and states as well as the Hydrographic Basin Committee for the Sai Francisco River and the oversight of two federal commissions.  The number of PMU coordination meetings held increased from 10 meetings in 2015 to 24 meetings in 2018, not achieving the target of 28 meetings. This indicator was added during the restructuring. Improved coordination and strengthen capacity: Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073)  Hydrological evaluation of Karstic and Fissure-Karstic Aquifers in the Sao Francisco hydrographic region to improve knowledge of the aquifer system, provide management tools, and propose a joint integrated management plan.  A water resources plan for Goias State was implemented by MMA in coordination with Goias State aiming to improve its capacity.  Technical Assistance for the Basic Sanitation Regulations was implemented by MCid with the participation of Ana and sub-national regulators.  The National Water Security Plan (PNSH) was implemented by ANA and MI to define the main structuring interventions to ensure water supply for multiple uses and reduce risks associated with critical events (droughts and floods).  The National Sewage Atlas was implemented by ANA and MCid which represents a broad national diagnosis of the sewage collection and treatment systems of all the country’s urban centers and the quality of the receiving water bodies.  The Ecological-Economic MacroZoning of the Sao Francisco Basin was implemented by MMA with the participation of all project implementing agencies aiming to contribute to the planning, development and use of the territory in the Sao Francisco river basin.  A study on renewable energy in the integration of the Sao Francisco basin was implemented by the MI aiming to assess different options of renewable energy resources to operate in the infrastructure linked to PISF in order to select the most sustainable option considering technical, economic, and environmental criteria.  A technical, economic and environmental feasibility to supply water to the Pianco basin implemented by MI.  The WSS National Info System (SINISA) including all its sub-sectors (potable water supply, sanitation, solid waste, and drainage) was implemented, achieving the target.  The IMC met less frequently after the restructuring due to constant political and fiscal problems and a considerable demobilization at the management level due to political instability. However, the ICR (p. 12) stated that the IMC did not need to meet at the same frequency after project restructuring since the most critical decisions were already made before the restructuring. Therefore, the target of establishing a committee that functions and meets twice a year at the executive secretariats level was achieved. Outcomes:  The number of WSS service users that are being represented by regulators increased from 10 million WSS service users in 2015 to 36.2 million WSS service users in 2018, surpassing the target of 12 million WSS service users. This indicator was revised during the restructuring and the target was increased from 10 million users to 12 million users.  The number of WSS service providers which are implementing non-revenue water management and energy efficiency increased from 10 million users in 2015 to 20 million users in 2018, surpassing the target of 12 million users. This indicator was revised during the restructuring and the target was increased from 10 million providers to 12 million providers.  The number of service providers, users of the Sanitation Integrated Management System benefiting from the updating and revision increased from 12 service providers and users in 2015 to 33 in 2018, surpassing the target of 25 service providers and users. This indicator was added during the restructuring. Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073)  45 percent of project amount was executed in water sector integrated actions, surpassing the target of 25 percent. This percentage included the actual budget for all products that were produced in an integrated manner under the project (all activities under component 4 and the water resources management plan for the state of Goias. This PDO indicator was added during the restructuring. Revised Rating Substantial OVERALL EFF TBL OLD Rationale Achievement of the objective before the restructuring is rated Modest. Achievement of the objective after the restructuring is rated Substantial despite modest M&E quality since the project produced sufficiently good quality evidence to justify a Substantial rating. Overall Efficacy Rating Substantial 5. Efficiency Economic Efficiency: The PAD did not conduct a traditional Economic efficiency analysis and only stated (p. 16) that “much of the analysis to be undertaken during implementation would support improved water resources management and planning and development of water resources with better environmental, social, and economic consideration, and this will generate enormous benefits for Brazil.” The ICR did not conduct a traditional Economic analysis and stated (p. 17) that a cost-benefit analysis was not feasible for a technical assistance project since the project produced over 60 outputs with the largest 17 outputs accounting for 87 percent of the financing utilized, the wide variety of specific tasks and lack of precise comparators. Operational Efficiency: The project experienced implementation delays due to weak institutional capacity. The project’s closing date was extended by 22 months during the restructuring in June 2016. Also, the project’s disbursement levels were very low due to the devaluation of the Brazilian Real. Due to the delays and the currency devaluation, a total of US$90.95 million was cancelled. According to the ICR (p. 17) some of the originally planned outputs were cancelled, decreased or funded with alternative domestic sources. Efficiency Rating Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the objective at completion is rated Substantial, efficacy before the restructuring is rated Modest and efficiency is rated Modest. After the restructuring Efficacy is rated Substantial and Efficiency is rated Modest. According to IEG/OPCS harmonized guidelines, when a project’s PDO indicators are revised, the final outcome rating is an average of outcomes before and after the revision of objectives weighted by Bank disbursements under each set of objectives. In this project 49% of disbursements occurred before the restructuring and 51% after the restructuring. • Before the restructuring the outcome rating was Moderately Unsatisfactory (score of 3) and a disbursement weight of 0.49 the weighted outcome score is 1.47. • After the restructuring the outcome rating was Moderately Satisfactory (score of 4) and a disbursement weight of 0.51 the weighted outcome score is 2.04. • The combined weighted average outcome score is 3.51 which corresponds to a Moderately Satisfactory outcome rating. a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Political and institutional risks: The ICR (p. 21) stated that the managerial and technical staff at the four participating institutions consistently tried to proactively move project implementation forward and collaborate with each other despite being located in four different parts of the federal government. However, the project faced significant political challenges. When the president was impeached in 2016 changes at the top levels Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) of the federal government took place. Also, the MI was most impacted with top officials and all technical staff working on the project changing and the internal structure being reorganized twice during project implementation. According to the ICR (p. 25) the government institutions recognized the benefits of working in a more coordinated and integrated manner which was demonstrated in the recent institutional consolidation in the federal water sector. Also, the MDR has announced that it is going to implement several of the plans which had resulted from project implementation such as the national plans for water security and dam rehabilitation. This might have a positive impact on the sustainability of project outcomes. In regards to financial risks, MDR and ANA are having conversations with the Bank about continuing a partnership in water security and basic sanitation and setting up a new TA instrument to develop the new roles ANA may obtain once the executive order on basic sanitation will be approved by Congress. However, future work with the Bank will depend on fiscal conditions and adequate instruments. 8. Assessment of Bank Performance a. Quality-at-Entry According to the ICR (p. 24) the project was built on lessons learned from previous projects implemented in Brazil’s water sector. The Bank team stated (May 31, 2019) that the most important lessons pointed out the importance of focusing on integrating activities, specific studies and generation of knowledge, working with the states and other sub-nationals, taking advantage of the high institutional capacity of ANA. The Bank collaborated with all four implementing institutions during project preparation. The Bank identified the following risks as Substantial: i) weak institutional capacity within the executing agencies, especially in the ministries; and ii) the complexity of intersectoral partnerships and subnational agreements in the selected multistate river basins impairing the advance of necessary institutional arrangements and existing water management instruments. Mitigation measures were not adequate and weak institutional capacity resulted in implementation delays. The risk of slow disbursement was only rated as Moderate. However, slow disbursement became a major bottleneck during project implementation. Also, the size of the loan turned out to be significantly too large, especially for a technical assistance operation, resulting in a substantial size of unused funds. Furthermore, the design foresaw that project management units (PMUs) in each of the participating institutions were responsible for project implementation and coordination. These four PMUs were to be overseen by a high-level Inter-ministerial Management Committee (IMC) which was to be supported by a Project Technical Secretariat (PTS) within the National Water Agency (ANA). According to the ICR (p. 20) this institutional arrangement did not work out and resulted in poor coordination among the four PMUs since the PTS focused more on operational issues than coordination among the four PMUs. The project’s Results Framework had significant shortcomings such as unclear formulation of the PDO indicators or lack of measurability of one of the PDO indicators. Quality-at-Entry Rating Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) Unsatisfactory b. Quality of supervision According to the ICR (p. 25) the Bank team conducted regular supervision missions which were appropriately staffed and reported on implementation progress in a candid and comprehensive way. Even though the project had four different Task Team Leaders (TTLs), the Co-TTL and several key team members were based in the country office which allowed for a consistent and active engagement with the government. The ICR (p. 25) stated that the Bank team organized study tours for IMC members and project technical staff which had a positive impact on improving inter-institutional coordination and collaboration. Also, according to the ICR (p. 23) the Bank addressed procurement issues through intensive Bank support (see section 10 b for more details). The ICR (p. 9) stated that the Bank team informally revised most of the Intermediate Outcome Indicators during the Mid-Term Review in June 2015 in agreement with the implementing agencies. However, the Bank team failed to formally revise the Intermediate Outcome Indicators during the project restructuring in June 2016 and kept reporting on the informally changed indicators until project closing. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project’s objective was clearly specified and the theory of change and how key activities would lead to intended outcomes was sound and reflected in the Results Framework. The original Results Framework included a large amount of indicators (three PDO and 17 intermediate outcome indicators). The revised Results Framework included even more indicators (three PDO indicators and 26 intermediate outcome indicators). Several intermediate outcome indicators included more than one output (“technical hydraulic infrastructure, irrigation, civil defense and regional development manuals established”) making measurement challenging. Some of the original indictors had some shortcomings. All three PDO indicators were revised during the June 2016 restructuring. Two were modified to rephrase them to make them clearer and one indicator was adapted since it was not measurable. However, the changes were not included in the Restructuring Paper. According to the Bank team (July 3, 2019) during the MTR, the revised indicators were agreed with the client. The final version of the revised results matrix, as agreed with the client, was received by the Bank prior to the restructuring. However, when the restructuring package was prepared, due to a glitch in the Portal, the revised matrix in the system did not reflect all the changes agreed with the client. After restructuring approval in the Portal, it was not possible to modify the matrix anymore. Therefore, the revised matrix was registered in the first ISR prepared after restructuring. Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) The selected indicators encompassed all outcomes of the PDO statement. All PDO indicators and most intermediate outcome indicators lacked a baseline. According to the ICR (p. 22) baselines were to be established within the first six months of project implementation. Each of the Project Management Units (PMUs) were to be responsible for conducting M&E activities. Annual reports were to be prepared by the Project Technical Secretariat (PTS) to inform about component performance, identify implementation bottlenecks and derive lessons learned. b. M&E Implementation According to the ICR (p. 22) the planned M&E mechanism did not work out and neither the PTS nor the PMUs had the necessary M&E mechanism in place to conduct M&E activities. Also, progress reports were not submitted on a regular basis and were of inconsistent quality. During the project restructuring in June 2016 several of the indicators included in the Results Framework were rephrased or dropped while others were added. Also, according to the ICR (p. 9) most intermediate outcome indicators were modified during the project’s Mid-Term Review (MTR). The ICR (p. 22) stated that there were inconsistencies between the revised indicators agreed on during the MTR and indicators included in the restructuring paper and the following Bank Implementation Support Reports. c. M&E Utilization According to the ICR (p. 23) the monitoring of the indicators included in the original Results Framework informed the MTR and the project restructuring. The revised Results Framework was used to monitor progress towards the project’s objective by the Borrower and the Bank. However, the monitoring lacked consistency throughout project implementation. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was classified as category B and triggered the Bank’s safeguard policies OP/BP 4.10 (Environmental Assessment), OP/BP 4.04 (Natural Habitats), OP/BP 4.09 (Pest Management), OP/BP 4.36 (Forest), OP/BP 4.37 (Safety of Dams), OP/BP 4.10 (Indigenous People), OP/BP 4.11 (Physical Cultural Resources) and OP/BP 4.12 (Involuntary Resettlement). According to the ICR (p. 23) an Environmental Management Framework was developed, consultations were conducted with relevant stakeholders and communities that might be affected by project activities in the future. Also, all relevant documents were disclosed publicly. Also, a Social Management Framework including an Indigenous People Planning Framework, and a Resettlement Policy Framework were developed. According to the ICR (p. 23) Bank supervision missions conducted safeguards training for participating agency staff with a special focus on water resource security, safety of dams, irrigated agriculture, and Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) revitalization of the Sao Francisco River basin. All participating agencies nominated focal points to screen the social impacts of technical assistance activities and to ensure compliance with Bank policies and safeguards. The ICR (p. 23) stated that throughout project implementation the project ‘s safeguard performance was rated Satisfactory and the Bank team stated (May 31, 2019) that the project complied with all safeguards. b. Fiduciary Compliance Financial Management: According to the ICR (p. 24) the project’s financial management experienced some delays at the beginning of project implementation due to adjusting to the financial management model of the integrated system of financial administration to issue Interim Financial Reports. However, once this adjustment was made the project’s Interim Financial Reports were submitted in a timely manner and found acceptable by the Bank team. All audit reports had unqualified opinions and no instances of ineligible expenses were found. However, during the last year of project implementation the project’s disbursement was affected by the lack of counterpart financing resulting from the fiscal constraints that the government had implemented. Procurement: The ICR (p. 23) stated that the Inter-American Institute for Cooperation on Agriculture (ICCA) supported the project’s procurement. During implementation, the project experienced delays in setting up the arrangement between the IICA and the participating federal entities. The project experienced some procurement related delays in preparing Terms of Reference, assessing technical proposals, and estimating costs for consultant services based on Bank procurement requirements. The ICR (p. 23) stated that these issues were addressed by the Bank through intensive provision of support and training. The ICR stated that overall procurement was acceptable. c. Unintended impacts (Positive or Negative) NA d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) Moderately Bank Performance Moderately Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The ICR (p. 25-28) provided several useful lessons learned:  Being realistic about the size of a loan for technical assistance is important for both the Bank and the Borrower. In this project it the size of the Bank loan was significantly too large, especially, since only US$8.04 million was disbursed at the restructuring, already four years into project implementation. Also, given the political and economic situation, the risk was High that disbursement would not increase significantly.  When project implementation includes more than one autonomous ministry or several agencies, it is critical that that one entity with general authority takes on the responsibility to coordinate among all participating entities. In this project, the MPOG did not take on such responsibility which resulted in implementation challenges due to lack of leadership and clarity of responsibility for coordinating the program and its daily activities.  It is important that responsibilities for Financial Management and Procurement are defined during project preparation and capacity building activities are conducted if necessary. One of the reasons for implementation delays in this project was the complex and evolving procurement arrangements which were not implemented as defined in the project’s PAD.  Study tours can foster cooperation among institutions through learning about different approaches for institutions within and across government levels can effectively work together but also to build personal relationships between staff from different implementing agencies. In this project the study tours build stronger personal relationships among key stakeholders and fostered the collective commitment to achieve a shared goal. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provided an adequate overview of project preparation and implementation. The ICR was sufficiently candid and internally consistent and provided useful lessons learned. Since it was a technical assistance project the ICR did not conduct an Economic analysis and was not outcome driven. The ICR could have been more concise and provide more detail on the project's design and how the Bank team sought to deal with the various issues. There was only limited detail on Quality at Entry issues versus those related to Bank Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review BR MST Federal Integrated Water Sector (P112073) Supervision. The ICR would have also benefited from a more articulate linking of the project’s theory of change and how this related to the various project achievements. a. Quality of ICR Rating Substantial Page 17 of 17