Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) Report Number: ICRR0021796 1. Project Data Project ID Project Name P117617 MW-Shire River Basin Management Program Country Practice Area(Lead) Malawi Water L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-51250,IDA-H7750 31-Jan-2018 121,882,415.70 Bank Approval Date Closing Date (Actual) 14-Jun-2012 31-Jan-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 125,000,000.00 0.00 Revised Commitment 125,000,000.00 0.00 Actual 115,304,416.22 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ebru Karamete Victoria Alexeeva Ramachandra Jammi IEGSD (Unit 4) P127866_TBL Project ID Project Name P127866 MW: Shire River Basin Mgmt Program-GEF ( P127866 ) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) 6577999.48 Bank Approval Date Closing Date (Actual) 14-Jun-2012 Page 1 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 6,578,000.00 Revised Commitment 0.00 6,578,000.00 Actual 0.00 6,577,999.48 2. Project Objectives and Components DEVOBJ_TBL a. Objectives This is a fully blended IDA/GEF financed project that has both project development objectives and global environment objectives. The project was the first phase of a 15-year Shire River Basin Management Program (SRBMP) with an overall program development objective “to increase sustainable social, economic and environmental benefits by effectively and collaboratively planning, developing and managing the Shire River Basin’s natural resources”. The formulation of the project development objective (PDO) and Global Environment Objective (GEO) was identical in the Project Appraisal Document (page 6) and in the Financing Agreement (page 5), which were “to develop a Shire River Basin planning framework to improve land and water management for ecosystem and livelihood benefits in target areas.” This review will assess the project’s achievements by determining the extent to which it was able “to improve land and water management for ecosystem and livelihood benefits in target areas". b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? Yes d. Components The project had three components: 1. Shire Basin Planning (Appraisal. US$ 41.6 million, Actual: US$ 42.1 million.) Page 2 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) The component financed development of a Shire Basin planning framework that included a Shire River Basin Atlas, studies and spatial analysis products, Decision Support System (DSS) for basin planning; Shire River Basin Plan; as well as setting up a multi-sectoral Shire Basin institution to support long-term planning and management, technical assistance, infrastructure and incremental operational costs for coordinated basin planning and management, water resources information systems (30 meteorological and hydrologic monitoring stations and integrated hydromet data platform associated with a Flood Early Warning System, a drought warning and monitoring system. The component also financed operations of multi-sector Technical Team (TT) that was established to manage the project and carry out M&E functions. 2. Catchment Management (Appraisal. US$ 45.0 million, Actual: US$ 26.1 million). This component supported the rehabilitation and management improvement of priority catchments and protected areas for reduced erosion and improved livelihoods through an integrated and participatory approach. Activities included: (i) support for community-based natural resource planning and management, social mobilization and training in four priority catchments covering 133,000 ha; (ii) rehabilitation of targeted catchments via helping communities to implement soil and water conservation covering a targeted area of 33,000 ha. (iii) support income-generating activities, commercially oriented alternative livelihoods initiatives; construction of selected infrastructure (market places, feeder roads, bridges to access to markets), and provision of livelihood grants; (iv) implementation of infrastructure and management plans in Lengwe and Liwonde National Parks; community forest co-management in forest reserves; and support for zoning, patrolling and monitoring. 3. Water Related Infrastructure (Appraisal. US$ 59.0 million, Actual: US$ 52.9 million) This component financed the following infrastructure to improve the regulation of Shire River flows and strengthen climate resilience: ( i) upgrade of Kamuzu Barrage including installation of new gates, erosion protection works, upgrading of the weed collection system, construction of a new bridge, automation of operation; (ii) flood mitigation interventions: river bank stabilization, dykes, culverts, flood diversion structures etc.; and community awareness raising and implementation of adaptation measures, and ecological flood mitigation and climate resilient livelihoods in the Elephant Marshes; (iii) preparation of feasibility and design studies for water-related infrastructure works identified in the Shire River Basin Plan. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: Total project cost estimated at appraisal was US$145.6 million, and at project closing 84 percent (US$121.9 million) was spent. The reduced costs were partially a result of lack of Borrower funds as well as exchange losses due to fluctuations between the US$ and the SDR shortly after the signing of the Financing Agreement (the aggregate loss was estimated at US$9.7 million as of October 2018 in the ICR). Financing: The project was financed by IDA credit (IDA-51250) of US$93.75 million, IDA grant (IDA- H7750) of US$31.25 million, GEF grant (TF-12920) US$5.08 million; and Least Developed Countries Fund (LDCF) (TF12921) of US$1.5 million. At project closing, disbursement under IDA credit was US$84.7 million (90 percent of the original amount); IDA grant was US$30.6 million (98 percent), GEF and LDCF grants were full. Page 3 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) Borrower Contribution: Borrower committed US$11.3 million but did not provide any funds. Likewise, the planned beneficiary contribution of US$2.8 million was also zero at project closing. The ICR did not report on reasons for the lack of borrower funds; the project team subsequently stated that budget availability impeded contribution in cash. But there was an equivalent contribution in kind (salaries of staff working in the project) as well as cash compensation for resettlement/economic displacement impacts. Dates: The project was approved on June 14, 2012; and project effectiveness took about three months (September 7, 2012) The project planned to be closed on January 31, 2018 but closed one year later on January 31, 2019. The reason for extension was mainly to allow for full completion of the Kamuzu barrage rehabilitation. Restructurings: One level II restructuring that would not require board approval was made on August 17, 2017 that included: (i) modification of results framework (two of the original PDO indicators -vegetation cover change in selected catchments and downtime for hydropower stations on the Shire River- were dropped; a new indicator -improved operation of the upgraded Kamuzu barrage with adequate institutional arrangements- was introduced at the PDO level; In addition, the target for one of the PDO indicators - households in target areas reclassified to lower flood risk- was reduced from 20,000 to 2,780 households; (ii) closing date extension of one year; (iii) reallocation of disbursement categories; (iv) change in some legal covenants. A split rating is justified based on the PDO indicator revisions, as the target for the outcome indicator ’households in targeted flood prone areas reclassified to a lower flood risk’ was substantially reduced during the restructuring. 3. Relevance of Objectives Rationale The Shire River Basin remained the country’s single most important natural resource system, supporting not only energy (hydropower plants producing 98 percent of the country’s electricity), but also agriculture, industry, and water supply and sanitation sectors. However, decisions on the development and management of water resources of the Shire River Basin were often taken on an uncoordinated basis. There was no institutional mechanism to coordinate integrated investment planning and systems operations for the Shire Basin, and no modeling tools to support decision making. At the time of appraisal there was concern that future lake level changes could cause serious social and economic disruption to Malawi. With more intense climate variability, there were growing concerns that the existing Kamuzu Barrage built in 1965 would not provide enough buffering capacity to ensure continuous water flows. The Kamuzu barrage was used to regulate water levels in the Shire River to ensure sustainable water flow but the structural stability of the barrage was at risk due to significant unaddressed erosion affecting the foundation, with its regulatory capacity constrained due to the poor conditions of the manually operated gates, and by the lack of real time information to operate the dam efficiently, particularly during emergencies. In addition, the Shire River Basin had also become a hotspot of catchment degradation and siltation of river beds, reservoirs and floodplain wetlands, which affected irrigation canals, fisheries and hydropower generation. Thus, it was essential to address the root causes of the deteriorating environment and natural resources base in the basin to ensure sustainable growth and poverty reduction. Page 4 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) The project development objectives were relevant to the World Bank and the Government of Malawi (GoM)’s strategies. GoM's Water Resources Investment Strategy (2011) identified poor catchment conditions and deteriorating water quality as significant risks to water resources and associated infrastructure (dams, hydropower plants, irrigation systems), and recommended investments in catchment management in strategically important watersheds. Improving the management of water resources was and is still a top priority for the government. In this context, the Government conceived a long-term, multi- sectoral investment program and an enabling institutional framework to effectively and sustainably develop and use the Shire River Basin’s resources, to address the Basin’s evolving challenges, and to maximize its potential for the benefit of Malawi. The Shire River Basin Management Program Phase- I Project (P117617) was the first project to support the GoM’s long-term vision for the Basin. The project was consistent with the World Bank Group Country Assistance Strategy (2013-2016) for Malawi, and Malawi’s Growth and Development Strategy (2011-2016), both of which recognized water infrastructure and improved management of water resources as critical to economic growth. The Country Assistance Strategy highlighted the project as support for sustainable access to water through an integrated water management system, addressing increasingly conflicting economic and social demands on water resources, and enabling collaborative planning for growth (page 31). The PDO was also relevant to the overall development objective of the Shire River Basin Management Program. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To improve land and water management for ecosystem and livelihood benefits in target areas. Rationale Theory of Change: The project's Theory of Change (ToC) incorporated a multi-sectoral approach that covered a wide span combining large infrastructure works with small community level investments as well as policy and planning work and institutional capacity building activities at national and community levels. According to the ToC, the technical assistance activities that financed preparation of Shire River Basin Plan, water resources information systems as well as setting up a multi-sectoral Shire Basin institution aimed to support long-term planning and management. Capacity building for communities on participatory catchment planning and community forest management as well as investments on rehabilitation of targeted catchments via soil and water conservation techniques and upgrading of Kamuzu Barrage and flood mitigation interventions would Page 5 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) jointly contribute to improved land and water management in Shire River. These would in turn generate ecosystem benefits and livelihood benefits in the medium to long run. The design included a farmer incentive mechanism (community environmental conservation fund) as well as farmer field schools after mid-term review to motivate and train farmers on the adoption of land and water management practices. The outcome indicators to measure the achievement of the objective included:  Shire River Basin Plan developed by multi-sectoral Shire Basin Institution  Vegetation cover change as a percentage of baseline in selected catchments  Downtime for hydropower stations on the Shire River  Households in targeted flood prone areas reclassified to lower flood risk Indicators 2 and 3 were dropped during the restructuring in 2017. At project mid-term review, it was agreed that the project would finance installation of sediment monitoring stations at key points in watercourses in selected catchments. The stations were installed, but no measurements were carried out. The project's results framework did not include agricultural productivity, income or poverty changes to measure livelihood benefits. Outputs Shire Basin Planning Framework:  44 hydromet stations with accessible data in real time were established exceeding the target of 30 stations.  The National Water Resources Authority (NWRA) has been established to manage Malawi’s water resources, including in the Shire River Basin, meeting the target.  A geospatial database was developed and is in use, meeting the target.  Basin planning tools developed and used in the preparing basin plan, meeting the target. Catchment Management:  Rehabilitation infrastructure included soil and water conservation measures such as gully reclamation, marker ridges and contour cultivation on steep slopes, as well as conservation agriculture practices such as zero tilling and mulching; stream and water control, including check dams.  Vegetative protection measures on 4,166 km of water courses, significantly exceeded the target of 992 kms.  Rehabilitation of degraded community forests included improved forest protection to allow natural regeneration to flourish, fill-in planting where needed, and spacing where trees might be overcrowded.  305 village level action plans were approved meeting the target.  Area under sustainable land and water management was 25,212 ha, exceeding the original target of 24,460 ha. Water Infrastructure:  Community based flood early warning system was established and operational meeting the target.  Budgeted management plan was established for elephant marshes. Page 6 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617)  The rehabilitation of the Kamuzu Barrage included the replacement and automation of the 14 gates, downstream and upstream erosion protection, upgrading of the weed collection and handling system, construction of a new separate bridge and walkways to improve safety and to ease traffic flow during gate maintenance operations.  A network of 100 hydromet monitoring stations (75 meteorological and 25 hydrological) across the basin has been revamped to improve hydro-meteorological information for improved planning, operational decision support and flood forecasting. Modern hydromet equipment and associated software have been installed to enable access to hydromet data in near real time.  An improved operational decision support tool for Kamuzu Barrage has been developed and is currently in use.  A total of 34 different flood management interventions (dykes, flood proof roads, river bank strengthening, evacuation centers, etc.) were established. Livelihood Activities:  80 farmer field schools were established,  Enterprise micro-loans for alternative (non-farm) livelihoods were provided, total value of livelihood investment grants managed by targeted Group Villages (GVs) was 620.5 Million Malawi Kwacha (exceeding the original target of 220 Million Malawi Kwacha).  Community environmental conservation grants,  Rural infrastructure for market access (80 km of feeder roads, 11 bridges and 13 market centers).  366 Common Interest Groups (CIGs) were formed, trained and provided with low-interest micro loans (up to US$2,500 each) as start-up capital for various non-farm enterprises. The most common enterprises included goat rearing, piggery, fish mongering, grocery shops. Intermediate Outcomes Planning Framework:  National catchment management guidelines were developed and adopted at the national level and being used by various stakeholders, as reported by the ICR. Various plans at catchment, sub- catchment and micro-catchment/village level were developed to guide implementation of community- based land and water management interventions in four priority catchments covering a total area of 129,000 ha (about 5 percent of the basin land area) (ICR para. 27).  The multi-sectoral Shire Basin Institution- National Water Resources Authority- was established at the end of the project, and although the Shire River Basin Plan was developed by the multi-sectoral technical team, not the National Water Resources Authority as originally intended, the PDO indicator,” Shire River Basin Plan developed by multi-sectoral Shire Basin Institution” is deemed as achieved.  An operational decision support system for flood and drought forecasting was developed that makes use of near real-time hydromet data to calculate weather and rainfall-runoff forecast models to calculate levels and flows in 25 sub-catchments. The system is currently issuing automated alerts and warnings via SMS to stakeholders in the flood prone area of Lower Shire and to Kamuzu barrage operators. Land and Water Management Page 7 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617)  A total area of 35,385 ha (27 percent of the catchment area, and 1.3 percent of the basin land area) has been rehabilitated, including 10,173 ha of previously degraded community forests which have been regenerated. However, the project team stated that most of the catchment infrastructure originally planned (e.g. small dams, irrigation schemes) were dropped and funds reallocated to cover cost overruns in other critical activities such as Kamazu barrage.  446,650 people (compared to the original project target of 400,000) directly benefited from the project’s improved land and ecological management, as well as various livelihood enhancing activities linked to these interventions. About 50 percent of the beneficiaries were women.  At project close, about 77 percent of households within the targeted catchments were reported to be engaged in sustainable land and water management practices, compared to a baseline of 15 percent, exceeding the original target of 75 percent.  A Management Effectiveness Tracker Tool (METT) – a GEF tool was used as part of the M&E system to score changes in protected area management effectiveness in all targeted natural habitat sites under the project. At appraisal, METT score was 39 and the target at completion was 65. The results of the assessment conducted in March 2019 showed that for all national parks and reserves, the project achieved a METT score of 70 (compared to the target of 65).  Regarding land management, the ICR reported land use change results. The remote-sensed data showed that 56 percent of the land area rehabilitated under the project has registered a change in land use categorized as positive (land area categorized from either agricultural land to forest; from shrubland to forest/plantation forest, from marshland to forest/plantation forest and built areas to forest) (ICR para. 25).  The rehabilitation and upgrading of Kamuzu Barrage led to a gradual recovery of the water levels in Lake Malawi, i.e. the regulated level of lake Malawi by 40cm, thereby increasing inter-seasonal storage capacity of the lake. This was also partly caused by the use of real-time information by the operators to make decisions based on pre-agreed operating rules, thus a rigorous water saving policy has been applied to stop the decline of Lake Malawi’s water level. The PDO indicator: “Downtime for hydropower stations on the Shire River” was not measured, as it was dropped during the restructuring. Outcomes Although the vegetation index cover indicator was dropped, the ICR reported on Normalized Difference Vegetation Index (NDVI) and Land Surface Water Index (LSWI) – the two most commonly used indices for monitoring vegetation using satellite imagery. NDVI in the target catchments increased by 33 percent while LSWI increased by 66.7 percent during the period 2012-2018. For the national parks and forest reserves that benefited from improved management measures supported by the project, NDVI increased by 19.67% during the same period (ICR para. 25). These numbers significantly exceeded the original target of 10 % increase. The project fell short of achieving the PDO indicator, number of households in targeted areas re-classified to lower flood risk. A total of 3,155 households in targeted flood prone areas have been reclassified to a lower flood risk, compared to the project’s original target of 20,000 households. The reason for this reduction was reported by the project team as community-based flood risk management interventions took more time than expected. The ICR noted that (page 49, para. 13) the flood risk management interventions had multifaceted benefits due to the flood prone situation of the project area. Comparing the impacts of flood events pre- and post- Page 8 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) project implementation (in 2015 and 2019), the annualized damages and losses avoided due to interventions is calculated to be about US$1.1 million. In terms of other aspects for livelihoods development the ICR reported the following:  The ICR noted (para. 52 and 53) that household socio-economic surveys carried out in the project areas in 2014 and 2017 showed positive changes in average annual cash income from sale of crops. While, the changes cannot be fully attributed to the project interventions because positive changes were also observed in the non-project area, the Malawi Country Environmental Analysis (2018) showed a strong correlation between areas with highly degraded land and those with a high incidence of poverty, implying that increasing levels of land degradation would be contributing to the poverty of millions of Malawians. In addition, project activities under component B targeted small holder farmers in the poorest region of Malawi, where rural poverty rates are above 80 percent. The ICR also reported where it was observed from field missions that small livelihoods funds provided as an incentive to engage in catchment management for many farmers grew substantially as borrowers paid back their small loans with interest. These small loans were used for school fees, medical costs, farm inputs, and for small business development.  The ICR (page 47 para. 7) reported that on average 78% of farmer field school trainees adopted the recommended agronomic and crop protection technologies increasing their income on average by about 38.4%. The total benefits of farmer field school activities were estimated to be US$ 603,611 per annum. About 581 million was invested under the Common Interest Groups (GIGs) scheme to develop various production and livelihood activities (Table A5.3). These crop production and livelihood activities generated a net benefit equivalent to US$937,402 per annum and boosted the income of participating farmers, particularly women. The ICR did not discuss the APL program-level results indicators, which included change in poverty, change in high erosion area, and total hydropower generation in GWh. However, the project team informed IEG that the results achieved complied in full with the targets proposed as triggers for the second phase of the APL (as indicated I Pag A-6 of Annex I of the PAD). Consequently, government requested support for second phase of the program, which is currently under preparation. In sum, regarding ecosystem benefits, although erosion was not monitored, the ICR provided proxy evidence via NDVI and LSWI. In addition, though the results reported on livelihood benefits were not robust, there is some indirect evidence on improvements in livelihoods of beneficiaries. However, most catchment rehabilitation and flood risk management work could not be completed and thus, the PDO indicator, number of households in targeted areas re-classified to lower flood risk, was not achieved. Thus, the achievement of the objective is rated as substantial, but with moderate shortcomings. Rating Substantial OBJECTIVE 1 REVISION 1 Revised Objective Page 9 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) For ecosystem and livelihood benefits in target areas. Revised Rationale Theory of Change did not change because neither the structure and design of the project nor its objective changed. The target for the PDO indicator-number of households in targeted areas re-classified to lower flood risk-was reduced from 20,000 to 2,780. Thus, the project achieved this revised target, as a total of 3,155 households in targeted flood prone areas have been reclassified to a lower flood risk. All other outputs and results were the same. The achievement of the revised objective is therefore rated substantial. Revised Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project prepared the Shire River Basin framework and created the multi-sectoral water resources agency and completed the infrastructure investments (with some reductions in catchment rehabilitation and flood management works), and provided evidence that land and water management has improved. While there was evidence on ecosystem benefits, as there was less robust and indirect evidence particularly on livelihoods benefits, and the flood related outcome indicator was not achieved as compared to the original target. The achievement of the objective, for ecosystem and livelihood benefits in target areas, is rated substantial with moderate shortcomings before restructuring and substantial after restructuring. Overall Efficacy Rating Substantial 5. Efficiency Economic Efficiency. The ex-ante economic analysis was conducted for the following activities (i) Kamuzu barrage upgrade, (ii) flood management in lower Shire, and (iii) catchment management. A separate cost benefit analysis was carried out comparing “with the project” and “without the project” situations for the three types of activities. The assumptions for Kamuzu barrage were based on additional energy production from increased water levels due to the barrage upgrade. The economic internal rate of return (EIRR) ranged from 19.5 to 48.6 percent based on the estimated regulated water levels. The assumptions for flood management was based on a study (2010) that estimated flood damages in the Shire Basin and identified the main direct losses from flooding including damages to households, dwellings, local infrastructure, and agricultural crops. Accordingly, the analysis expected that the project would mitigate at least 60% of the non-agricultural losses in the project districts, with the economic value of average annual losses avoided in the amount of US$1.75 million. Based on Page 10 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) that, the EIRR of flood protection activities was calculated as 16 percent. Regarding catchment management activities, assumptions included an estimation of (a) improved productivity resulting from sustainable soil and water management practices and improved agronomic practice, (b) farm-level incremental benefits, (c) diversified and increased farm benefits from enterprise support and improved access to financial services and (d) avoided environmental costs to the electric company and to energy consumers. It was estimated that incremental yield increases for maize, horticulture and rice were 219, 50 and 145 percent respectively. The analysis also calculated benefits of crop diversification (oil, goats, chicken and honey). In addition, costs incurred by the power utility ESCOM due to erosion and sedimentation were included based on existing data. Based on these assumptions, the EIRR was calculated as 15.76 percent. All of these above mentioned EIRRs exceeded the opportunity costs of capital assumed at 12 percent, therefore showed substantial economic rate of returns. Ex-post analysis adopted the appraisal stage cost benefit analysis approach, with the exception that a consolidated cost benefit analysis was also done to assess the overall efficiency of the project. The ICR did not provide the details of benefit streams for catchment management activity calculations. However, the EIRR’s estimates were still higher than the opportunity cost of capital with 25.9 percent for catchment management, 37 percent for Kamuzu barrage upgrade and 12.9 percent for flood risk management. The overall EIRR for the project was 15.8 percent. Sensitivity tests using various cost and income variations showed in general a robust efficiency to changes (i.e. NPV became negative only when income declined as much as 30 percent). Administrative and Operational Efficiency. The project closing date was extended for one year. The multi- sectoral technical team (TT) established to manage the project was understaffed and unprepared for the task, but hiring of additional professional technical and project management staff was not implemented, consequently, the TT struggled to manage the multitude of activities leading to delays in project implementation. Project resources earmarked to support the operationalization of the National Water Resources Authority (NWRA) could not be fully utilized due to delays in establishing NWRA’s basic corporate governance structure. In addition, weaknesses in contract management led to time extension of contracts and cost overruns. Further, due to the capacity and time constraints within the TT, there was inadequate supervision and oversight of 31 consultants engaged under the project. There were also instances of delayed client feedback on reports produced by consultants, which also delayed subsequent tasks under these contracts, and in some cases compromised the quality of the products. On balance, based on the satisfactory economic efficiency and operational and administrative inefficiencies as described above that led to the project extension for one year, the overall project efficiency is rated as substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) Page 11 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome This evaluation carried out a split rating considering the PDO indicator revisions. Original project: The project’s relevance of objectives is rated high as a result of the relevant objectives to country and World Bank strategies. The efficacy of the objective is rated as substantial but with moderate shortcomings, as the PDO indicator measuring reduced flood risk was not achieved, and there was less robust and indirect evidence particularly on livelihoods benefits. Efficiency is rated substantial due to satisfactory economic efficiency while there were some operational and administrative inefficiencies. The outcome rating is moderately satisfactory. Revised project: The project’s relevance of objectives is rated high. Regarding the efficacy of the objective, the PDO indicator measuring reduced flood risk was achieved, as the targets got reduced. Thus, efficacy is rated substantial. Efficiency is rated substantial due to satisfactory economic efficiency while there were some operational and administrative inefficiencies. The outcome rating is satisfactory. Overall Outcome: When weighted average of the original and revised outcome rating is taken based on the disbursement share before (US$87.5 million) and after restructuring (US$34.4 million), the overall outcome rating becomes moderately satisfactory (0.72* 4 (MS rating) + 0.28 * 5 (S rating) = 4.28 (MS rating)). a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Sustainability of community structures is ensured by participatory planning process, as well as ongoing support to communities on soil and water conservation activities by the district council field staff. The ICR reported that during field visits, it was found that tree nurseries were still functional, protection and expansion of community forests was continuing, work to repair small gullies was occurring, and many farmers were undertaking small-scale soil and water conservation activities on their lands such as planting vetiver grass and strengthening contours on slopes. However, the ICR did not report on whether the communities had financial means to continue to maintain the structures and continue the sold and water management techniques. At the central level, the project supported the operationalization of the National Water Resources Authority that will facilitate multi-sector planning and management of water resources and provide continued support to watershed management institutions at community level. While at the time of the ICR there were some Page 12 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) remaining steps for its functioning, the government remained committed to continue the efforts started by the project. In addition, a follow-on World Bank project is currently under preparation to reinforce the gains and scale up the interventions to other river basins. 8. Assessment of Bank Performance a. Quality-at-Entry This was the first integrated river basin management project in Malawi, targeting critically important river basin, with a combination of investments in infrastructure, institutions and information, and watershed management. The project demonstrated new and innovative approaches for integrated catchment management, encompassing planning from catchment to village levels with community participation, capacity and institution building, soil and water conservation interventions, sustainable forestry and protected areas management and developing alternative livelihoods. The shortcomings are as follows: (i) The complex project design was to be managed through implementation arrangements via incorporation of a multi-sectoral technical team, but the capacity issues were not resolved, from the beginning although risk assessment rated the capacity risk as high. (ii) At appraisal, most of the project investments were not ready for implementation (except the barrage rehabilitation), despite the long project preparation period (28 months from concept to approval). (iii) Results framework lacked some of the indicators (biophysical and ecological), the systems for data collection were not in place and the project design did not include capacity building for biophysical and ecological monitoring and impact evaluation. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The ICR noted that (page 24) the Bank team provided sufficient guidance during the well-resourced (with many technical experts) Bank supervision missions, prepared detailed aide memoires and flagged issues candidly for management and government attention. The supervision also benefited from minimal changes in task team leadership during project implementation, as there were only two task team leaders; this contributed to continuity of the relationship between the Bank and the Borrower. The ICR noted the candidacy and accuracy of Implementation Supervision Reports, and the following examples were provided: the issue of 30 percent of project resources remaining uncommitted after four years of implementation was flagged by the Bank and support for the technical team was provided to prepare an action plan to commit the resources within six months. Three missions in 2017 and 2018 emphasized the institutional framework for water resources management and the sustained Bank engagement resulted in the appointment of the Governing Board for the National Water Resources Authority (NWRA) in November 2018. The ICR also noted that overall the Bank’s technical support and partnership was generally appreciated by stakeholders. A stakeholder survey conducted by the project M&E consultant at project close revealed that 82 percent of respondents were satisfied with the Bank’s support during project implementation. Also the Page 13 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) Borrower’s ICR stated that the World Bank provided the much needed support from the designing, planning to implementation of the project activities. During the support missions, the Bank provided valuable technical and expert support including quality control checks among many other areas. Therefore, the Bank contributed very positively to the project implementation. The project mid-term review was effective in terms of recommendations and revisions to project design, including introduction of catchment environmental conservation fund and farmer field schools, but the team left some issues with the results framework unaddressed; i.e. the lacking biophysical and ecological the systems for data collection was not completed. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The M&E system allowed for spatial data analysis and visualization to measure impacts, particularly of catchment rehabilitation activities. Nevertheless, the design could have benefitted from the inclusion of biophysical measurement on ecological monitoring such as reduction in land degradation, erosion, sedimentation loss. Also, in addition to the flooding indicator to better monitor livelihoods improvements, an impact evaluation could have been included to measure yields and income. The ICR also noted that (para. 66), the M&E system was designed to rely heavily on consultants and was not integrated with the participating departments and districts. b. M&E Implementation A consulting firm was contracted to carry out M&E functions on behalf of the technical team during most of the project period until January 2018. Thereafter, an individual consultant was engaged to provide M&E support. The monitoring of progress across all project components was detailed. However, the complexity of the project, involving many activities and multiple institutions (supported by different consultants), at national and local levels, posed challenges to the flow of quality and timely information. There were no beneficiary reports per se prepared during the last year of the project. However, the Government prepared an internal Project Completion Report in December 2018 that included a comprehensive consultation to asses beneficiary’s perception and satisfaction with the project outputs, which included key informant interviews and a qualitative data gathered from ten focus groups discussions held in 5 districts. The M&E design included a provision for biophysical monitoring at sub-catchment level, this activity was not done due to delays in installing the biophysical monitoring stations and capacity constraints at the Page 14 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) local level to manage these stations. Also, two indicators were dropped and target for one PDO indicator was reduced during the restructuring. c. M&E Utilization The information generated by the project’s M&E system was used for operational decisions throughout project implementation. Process and quality monitoring reports were routinely prepared, and findings used to adjust implementation processes. M&E Quality Rating Modest 10. Other Issues a. Safeguards Due to the Kamuzu barrage investments that was considered as could have significant adverse impacts, the project was considered Category A at appraisal, triggering eight safeguard policies: Environmental Assessment (OP/BP 4.01); Natural Habitats (OP/BP 4.04); Forests (OP/BP 4.36); Pest Management (OP 4.09); Physical Cultural Resources (OP/BP 4.11); Involuntary Resettlement (OP/BP 4.12), Safety of Dams (OP/BP 4.37); and Projects on International Waterways (OP/BP 7.50). The ICR noted that (para. 70) the project complied with all the applicable environmental and social safeguard policies of the World Bank. An overall project Environmental and Social Management Framework (ESMF) and an Environmental and Social Impact Assessment/Plan (ESIA/ESMP) for Kamuzu Barrage were prepared during project preparation, reviewed and cleared by the Bank. During implementation, subprojects (4 feeder roads, 12 bridges, 13 market facilities, 5 flood mitigation structures) prepared ESIA/ESMP which were reviewed and cleared by the Bank. The implementation and monitoring of the ESMPs was carried out by the PMU in collaboration with contractors, district authorities and communities and monitoring reports were submitted as part of project progress reports to the Bank for information. The borrower ensured that the required capacity for the management of safeguards was in place for the entire life of the project and safeguards instruments were prepared and implemented to completion for all subprojects. Safeguards issues for Kamuzu barrage – the largest infrastructure investment under the project – were generally well managed. Rehabilitation of the barrage involved the application of generic dam safety measures and was based on the advice of an international panel of experts and the recommendation of the Bank's quality assurance group. Dam Safety Plans (Instrumentation plan, Operation and Maintenance Plan, Dam Safety Plan and Emergency Preparedness Plan) were prepared in January 2019. Social Safeguards: The project was considered to have interventions that would potentially lead to the loss of land, assets, livelihoods and access to assets. In that regard, a Resettlement Policy Framework (RPF) and a Process Framework (PF) for the overall project were prepared, reviewed and cleared by the Bank and disclosed in January 2012. The operation’s subprojects did not entail physical relocation. However, there was economic displacement of 22 individuals, 4 institutions and 32 fishermen at both the barrage and Page 15 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) Shire River Basin Agency building. A project Grievance Redress Mechanism was established and utilized throughout project implementation. However, it was gauged as partially non-inclusive to all PAPs because larger business owners did not use it. By project closure one grievance remained unresolved and in the courts. b. Fiduciary Compliance The ICR noted that (para. 72-73), the project used consultative budgeting processes leading to detailed activity-based budgeting, which guided implementation on an annual basis. The project had an FM manual detailing policies and procedures in using proceeds of financing. The internal controls were generally effective demonstrated by only few observations of control and accountability in both external and internal audits. However, one district experienced problems in use of project funds leading to qualification of project audited accounts. At the community level, the project engaged Accounts Assistants to assist communities in financial management. Most of the communities maintained proper books of accounts and were able to report on use of funds supported by appropriate documentation. The ICR reported that (para. 72) compliance was consistently rated satisfactory during project implementation. Procurement: The ICR noted that (para. 74) the project generally followed the procurement procedures and guidelines. The project had a dedicated procurement team comprising a procurement specialist and two assistants. There was adequate oversight from the Ministry of Agriculture, Irrigation and Water Development’s Internal Procurement Committee. There was no declaration of any mis-procurement. However, as noted before, the Technical Team struggled to manage the large number of contracts under the project, leading to excessive extensions and in some cases cost overruns. c. Unintended impacts (Positive or Negative) No unintended impacts were reported by the ICR. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment The outcome of the original project is rated MS and the Outcome Satisfactory Moderately Satisfactory outcome of the revised project is S and the weighted average of the overall outcome is MS. Page 16 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) Quality at Entry is rated Moderately Satisfactory due to insufficient implementation arrangements, lack of readiness Bank Performance Satisfactory Moderately Satisfactory of investments and some issues in results framework that were not resolved during implementation Due to lack of indicators and measurements on land degradation and sedimentation Quality of M&E Substantial Modest loss, yields and income, to better measure livelihoods improvements Quality of ICR --- Substantial 12. Lessons The ICR provided some important lessons; the most relevant ones follow with some modification of language: Reversing watershed degradation via a community-led landscape approach requires significant investment in strengthening institutional and community capacity at all levels. The project experience showed that significant investment in capacity building and facilitation at the central, district and local levels is needed. The institutions need planning and decision-making systems and software as well as staff training. The project also showed that communities need adequate training and incentive mechanisms to comprehend and to implement various land and water management techniques and practices. Choices about project design and scope need to consider the capacity of implementing agencies, and project scope has to either match this capacity or advance capacity building for project management with the implementing agency. The project experienced several issues and delays in implementation due to shortcomings in the implementing agency’s capacity. The technical team was understaffed and unprepared for the task and hiring of additional professional technical and project management staff was not implemented. Any measures agreed to mitigate capacity constraints could be reflected in appropriate covenants in the financing agreement. Inter-agency collaboration on multi-sectoral projects requires strong government commitment and deliberate design choices. The project was successful in inter-agency collaboration partly because of clear government commitment expressed in a Letter of Policy to the Bank prior to appraisal, followed by the establishment of a multi-agency, multi-sector technical team drawn from all relevant ministries and departments to manage project preparation and implementation. In addition, the project was appraised at a time when Government policy was not in favor of stand-alone Project Implementation Units (PIUs) for delivery of projects. This policy therefore presented an opportunity to design implementation arrangements that fostered long-term institutional capacity of key land and water related institutions to perform their core mandate. Page 17 of 18 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MW-Shire River Basin Management Program (P117617) 13. Assessment Recommended? No 14. Comments on Quality of ICR The report was comprehensive and followed the guidelines. The ICR was candid in discussing design and implementation challenges. The ICR’s lessons were useful and based on the project's experience. The weaknesses were: the split of objectives in the Efficacy section left out the discussion of the achievements of the objective to generate ecosystem and livelihood benefits in target areas, particularly the livelihood benefits part. The efficiency section discussion did not include assumptions for benefits streams under the catchment management activity. Also, explanation behind the lack of contribution from the Borrower and the reason of the long lag between mid-term and project restructuring were missing. a. Quality of ICR Rating Substantial Page 18 of 18