Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) Report Number : ICRR0020673 1. Project Data Project ID Project Name ET-Tana &Beles Int. Wat Res Dev P096323 Project Country Practice Area(Lead) Additional Financing Ethiopia Water P154680 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-44570,IDA-57150,TF-95045 30-Sep-2013 70,000,000.00 Bank Approval Date Closing Date (Actual) 29-May-2008 31-Jul-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 45,000,000.00 3,484,679.00 Revised Commitment 50,099,353.96 3,484,679.00 Actual 46,321,999.26 3,484,679.00 Prepared by Reviewed by ICR Review Coordinator Group Ebru Karamete George T. K. Pitman Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The project development objectives stated in the Financing Agreement (p. 5) and Project Appraisal Document (p. 8) are: "to develop enabling institutions and investments for integrated planning, management, and development in the Tana and Beles Sub-basins to accelerate sustainable growth". The objectives were revised through a Level I Restructuring in March 2013 as : Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) " to develop enabling institutions and investments for integrated planning, management, and development in the Tana and Beles Sub-basins.” Subsequently the revised objective was restated as: “to develop enabling institutions and investments for integrated planning, management, and development in the Tana and Beles Sub-basins for integrated water resources management.” The date of the addition ‘integrated water resources management,’ is unclear but they were adopted in the ICR. As this clarifies the goal, the ICR version will be used in this Review. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 28-Mar-2013 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components • The project had four components: 1.Sub-basin Resources Planning and Management (Appraisal Estimate: US$17.61 million, Actual: US$23.10 million). This component aimed to develop the enabling institutional infrastructure and capacity for fueling and managing sustainable investments in the Tana and Beles sub-basins. A Water Resources Information System was to be established including a modem network of hydro-meteorological, groundwater, and environmental monitoring; special studies; as well as utilizing the system for management and development problems in the sub-basins. The component also included capacity building activities on resource planning and management supporting the Tana Sub-Basin Organization (TaSBO), Beles Sub- Basin Organization (BeSBO), and its parent Abbay Basin Organization in order to build an appropriate knowledge base, analytical capacity, and structured stakeholder consultation for “shared vision” sub-basin plans and management instruments. Restructuring. Development of a Decision Support System (DSS) was dropped during the March 2013 Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) restructuring. Instead the project used the advantage of advances made by the Nile Basin Initiative (NBI) DSS and the Eastern Nile Technical Regional Office (ENTRO) developed models. B. Natural Resource Management Investments (Appraisal Estimate: US$ 40.83 million, Actual: US$50.1 million). This component aims to finance critical investments to improve the natural resources management in the Tana sub-basin, due to urgency of the situation in this basin. Two sub-components were: 1. Watershed Development: Support for sustainable watershed development investments covering about 80,000 ha in the Ribb, Gumara, and Jamma sub-watersheds in the Lake Tana sub-basin underpinned by community- based planning and participation. Coordination with other related projects were planned (i.e. the proposed World Bank Financed Sustainable Land Management Project, proposed GEF/IFAD supported Community-based Integrated Natural Resource Management Project, and the GoF financed Rural Water Supply and Environmental Program). Financed activities were: (i) Livelihood improvement such as support for crop and livestock production within the three sub- catchments, improved rural access, upgrading of social infrastructure at kebele (village) level, water supply source development and irrigation (including groundwater pumping) development within the micro- watersheds, and the adoption of appropriate technologies and innovations. (ii) Natural Resources Management: support for soil and water conservation works (check dams, cut-off drains, gully improvements, stone faced bunds, hedgerows, steep and bedland improvements etc); and the protection of existing badly degraded areas and their improvement through the development of forestry and agro-forestry programs. (iii) Institutional Strengthening: to support woreda (district) and kebele level institutional capacity building, improvements to office infrastructure and provision of funds for training kebele staff, Subject Matter Specialists (SMSs) and Development Agents (DAs). At the regional level it will support a coordination unit, consultancy costs and monitoring and evaluation requirements. 2. Flood Management: supports community-based flood management and adaptation in the flood-prone areas around Lake Tana. Key activities were: (i)Training of trainers, who then trained volunteers to educate community groups; (ii) Participatory flood management planning and local flood risk mapping; (iii) Small- scale structural measures to manage flood impacts such as elevated access ways, drains, secure water supply, and stock watering points, food storage facilities and refuges; and overall flood preparedness at regional, local, and community levels with close linkages with the Water Resources Information System under Component A. Restructuring. Through the restructuring in March 2013, the activities, construction of 14.7 km access roads to hydrological monitoring stations and 77 km escape routes, were dropped after the detailed designs were completed because of higher cost, safeguard requirements, and time constraints:. C. Growth-Oriented Investment Facilitation (Appraisal Estimate: $US9.17 million, Actual: US$1.56 million). Two sub-components were: 1. Development Agency Support for relevant existing government bureaus (e.g. Regional Investment Promotion Agencies, Bureau of Trade and Industry) and to potential public and private entities for increased private sector participation and planning to create a growth zone in the Tana-Beles area. 2. Growth-oriented Investment Preparation, financed the preparation of investments in the Tana and Beles sub-basins, including surveys, pre-feasibility and feasibility studies, environmental and social assessments, designs, implementation arrangements, and analytical work and stakeholder consultation. Restructuring. The component was scaled down in September 2011 as a result of delay in implementation related to the complex nature of the component and its activities, as well as lack of Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) capacity and clarity within the government on the growth corridor strategy leading to delayed action on the agreed outputs. During the March 2013 restructuring all remaining activities under the component, except several completed ones, (i.e. endowment and potential study, stakeholder consultation, and implementation agency capacity-building support) were dropped and the remaining IDA fund was reallocated to Components A and B. D. Project Management (Appraisal Estimate: $US 2.24 million, Actual: US$2.20 million). The component financed project planning, coordination, management, quality oversight, technical supervision, procurement and financial management, as well as consolidated monitoring and reporting of project activities. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: Total project cost at appraisal was estimated at US$ 69.9 million, then it increased to US$ 70.7 because of additional operating and training costs for the closing date extension. Financing: The original IDA credit (4450-ET) was SDR 27.4 million (equivalent to US$ 45.0 million) and additional financing of SDR 3.7 million (equivalent to US$5.1 million) was approved in August 2015. IDA funds were fully disbursed. Government of Finland Trust Fund Grant (TF 095045) provided US$ 8.0 million, and the actual disbursement was US$ 3.5 million, the balance transferred to government through a bilateral agreement.. Borrower Contribution: The Borrower contributed US$ 5.4 million and communities contributed US$ 11.5 million as planned. Dates: The closing date of the operation was extended for 34 months from September 30, 2013 to end- July 2016 via five restructurings. This extension was necessary to catch up with initial delays and provide sufficient time to achieve the PDOs. Restructuring: The project went through two Level I restructurings on March 28, 2013 and August 25, 2015. • The first Level I restructuring did the following: (i) revised the PDO by deleting “to accelerate sustainable growth” part; dropped some the outcome and intermediate outcome indicators accordingly and added WB core indicators; (ii) dropped all Component C activities that had been already scaled down in 2011; (iii) reallocated the remaining IDA funds of US$6.9 million under Component C to Component A and B; (iv) reduced the scope of Component A-consultancy for the development of DSS was dropped and Eastern Nile Technical Regional Organization (ENTRO) DSS was adopted; and (v) reduced the scope of Component B - construction of 14.7 km of access roads to hydrological monitoring stations under Subcomponent B1 and 77 km of escape routes under Subcomponent B2. • The second Level I restructuring in August 2015 included an Additional Financing of SDR 3.7 million equivalent to US$5.1 million. This restructuring extended the closing date by 12 months to allow completion of the ongoing tasks. There were five Level II restructurings • The first level II restructuring in July 2010 extended the date for establishment of the BeSBO (Beles Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) Sub-Basin Organization) from December 31, 2009, to December 31, 2010. • The second Level II restructuring in September 2011 was after the mid-term review and, due to delays in achieving targets revised all the PDO indicators, scaling down targets for indicators. The US$1.15 million budget for Component C was reallocated to Components A and B. • The remaining three Level-II restructuring in March 2013, 2014, and 2015 extended the closing dates. 3. Relevance of Objectives & Design a. Relevance of Objectives Original Objective Modest The original project development objectives were modestly relevant to the country, region and sector strategies and needs, as they were unrealistically broad beyond the capabilities of the project (for details see next section). The focus of Project activities was in Abbay basin (Blue Nile basin), the largest basin in Ethiopia that included Tana and Beles sub-basins.. Lake Tana, the source of the Blue Nile, is a very valuable but ecologically fragile water resource, and at the time of project preparation. It was showing growing signs of stress that needed to be addressed in order to ensure sustainable development in a manner that optimized its socioeconomic benefits while protecting its valuable environmental and cultural resource base. The area had the potential for growth in multiple sectors, particularly smallholder agriculture, agro-industry, tourism, fisheries, livestock and energy resulting in improved livelihoods for the sub-basin’s 3 million residents and national economic growth. The project development objective was also relevant to the Government development agenda. The original PDO was aligned with the Government’s Poverty Reduction Strategy (2005–2010) that had identified the Tana and Beles Zone as the first of five planned growth zones in the country. The original PDO was also fully consistent with the World Bank Group’s Country Assistance Strategy (CAS 2008-2012) that aimed to help sustain growth and basic services by supporting the implementation of key elements of the poverty reduction strategy. It was also consistent with the World Bank’s Water CAS (2006) that identified strengthening water resource development and management as one of the priority responses to water resources challenges. Revised Objective High The revised objective was also highly relevant to the country, region and sector strategies and needs. The PDO continued to be relevant to the CPS (FY 13-16) at closing that aimed to increase productivity and competitiveness of agriculture sector through strong land and water management practices and institutions. When the part “to achieve sustainable growth” was taken out, the PDO became more realistic and relevant. Rating Revised Rating Modest High Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) b. Relevance of Design Original Design Modest The project design logic was weak and overly complex. 1. The project activities were not consistent with the over-ambitious project development objectives. Component C activities were included to finance productive investments to achieve sustainable growth. This was unrealistic, due to the limited scale and duration of the project, which, coupled with weak coordination and implementation capacity of the multiple agencies involved, and lack of clarity within the government on a growth corridor strategy caused delays. Together these design shortcomings frustrated integrated planning, management and development. 2 . The project tried to address two separate issues at the same time, i.e. integrated water resources development and accelerating economic growth, despite limited capacity. These multiple activities, i.e. river basin planning (Component A), natural resources management (Component B) and productive investments to facilitate growth (Component C) required exceptional technical and coordination capacity/arrangements and firm commitment that was lacking. Revised Design: Substantial The part on ‘achieving growth’ was taken out from the PDO, in line with very low achievements under component C, and this was replaced with ‘for integrated water resources management” for which Component A and B activities the PDO became more in line with the institutional capabilities. Rating Revised Rating Modest Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Original Objectives: “to develop enabling institutions and investments for integrated planning, management, and development in the Tana and Beles Sub-basins to accelerate sustainable growth”. Based on this statement, this Review considers two objectives: (i) develop enabling institutions and investments (ii) accelerate sustainable growth. Intermediate Outcomes are“ (i) integrated planning and management, development in the Tana and Beles Sub-basins”. Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) (I) Develop Enabling Institutions and Investments, rated Substantial. Rationale Outputs: Water Resources Information System Development • Hydrological Information System (HIS) / Basin Information System (BIS) were improved: • 30 hydrometric stations were upgraded/modernized and a hydrometric network established/strengthened. • 33 meteorological stations were modernized through rehabilitation of existing ones in the Tana and Beles sub-basins and rain gauges were installed. • 20 modern networks of groundwater monitoring stations (12 Tana and 8 Beles) were established for future groundwater development and management, and monitoring equipment was installed in 4 of the stations and made partially operational. • Hydrological information systems for flood management were developed, 19 flood early-warning monitoring stations established. • Weather Radar for Weather Monitoring • Weather radar, the first of its kind in the country, was procured and installed; training was provided to the National Meteorological Agency (NMA), Abbay River Basin Authority (ARBA), and Tana Sub-Basin Organization(TaSBO); and the operationalization was initiated. • Water Resources Planning and Management Models were developed and made functional using the Nile Decision Support System (DSS) and the tools developed by Eastern Nile Technical Regional Organization (ENTRO) and the weather, hydrological, and flood forecasting tools developed. • The Flood Management Information System (FMIS) was established and numerical weather prediction was models were completed. A detailed Topographic Survey of Lake Tana Flood Plain was conducted and flood risk maps were produced. While flood forecasting and an early warning systems are in place, it was not fully operationalization as project closing. Thus flood forecasting models could not take full advantage of the real-time monitoring. • Establishment of a water quality monitoring network: • Lake Tana water quality assessment was conducted (taking 150 samples from 25 sites) and river water quality assessment was conducted (taking 96 samples from 16 sites of major rivers of the Tana sub-basin). However, the water quality system was not functioning at project closing. Flood Management System • The Woreda Flood Management Groups (WFMGs) were established, trained, and flood management plans developed. • Installation of 11 flood warning sirens was completed; 8 community flood protection shelters were constructed; and 4 monitoring boats were procured for monitoring and service provision in the flood Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) season. Early warning indicators were installed for flood hazard severity level and local safe route demarcation, 77 km of roads for flood evacuation routes were designed to enable flood hazard relocation to dry and safe areas. Community training on flood hazard preparedness was provided. Resource Planning and Management Capacity Building • Tana and Beles Sub-Basin Organizations were established, equipped, and staffed with the necessary capacity (water resource development, socioeconomic, environment, administration, Geographic Information Systems (GIS) expertise, communication etc. specialists); staff were trained on different areas to strengthen their capacity and ensure sustainability of the Sub-Basin Organizations (SBOs) • 26,066 farmers, 2726 experts, and management team members were trained in improved agricultural extension services and on natural resources management, livelihood development, project planning and M&E. Outcome: The project substantially contributed to developing and strengthening the enabling institutions and investments. The ICR provided a long list of activities conducted by the project but evidence is lacking on whether the newly-created organizations were actually functional and able to make and implement decisions. It is also not clear if the systems developed were actually being used and that training led to enhanced knowledge of the stakeholders. • TaSBO and BeSBO was established, with operational with core staff appointed and office buildings with adequate knowledge base, analytical tools, and capacity for sub-basin planning in place. The initial integrated water resources management (IWRM) plans for Tana and Beles subbasins were developed. However, it was not clear if the IWRM plans were approved or being implemented. • M&E/MIS networks for soil erosion monitoring; hydrological gauging stations at the sub-watershed level were installed; sedimentation and hydrological data of micro-watersheds were collected and analyzed. Although an outcome indicator was achieved (fluvial sediment loading, whether this information being used in planning and decision-making is not clear. • Flood Management Information System (FMIS) as well as flood forecasting and early warning systems were developed, although it was not fully operationalization at the time of the ICR. • Land area where sustainable land management practices/improved natural resource management practices have been adopted as a result of the project, was 79,288 ha (slightly below original outcome target of 80,000). Rating Substantial Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) PHREVDELTBL PHEFFICACYTBL Objective 2 Objective II). Accelerate Sustainable Growth, rated Modest Rationale Outputs: Growth-Oriented Investment Facilitation • An Endowment and Growth Study (endowments, constraints, and opportunities for growth) for Tana and Beles was completed; stakeholder consultation was also undertaken, and capacity-building support was provided. • Livestock development: 9 new animal health posts were constructed. 129,300 animals have been vaccinated and 164,850 animals received medical treatment. 2850 households participated in enhancing production through using improved technology and varieties (animal fattening) and milk, egg, honey production. While egg and milk productions did not reach appraisal targets, project production was about 10% higher than without-project control areas: eggs 63/hen in control, 69 in project; milk production liters/cow/year was 569 in the control and 621 in the project. • Crop development and agricultural extension demonstrations in Farmer Training Center (FTCs) and individual farm plots: Crop production increased in the demonstration sites (FTCs); according to the impact assessment) production increase was at 21.5% for teff, barley at 21.7%, maize at 18.9%, wheat and finger millet at 28.5%, and potato at 88%. However, FTCs generally produce much higher yields than individual farm plots and these improvements represent the potential, not what was achieved at farm level. • 44,355 households adopted improved soil and water management practices out of the targeted 30,000 (147% achievement). • Self-help groups with 500 members were established and engaged in income-generating activities (bee keeping, weaving, tannery, tailoring, forestry, seedling production); about 189 members were trained in business management and booking skills.(30 – 35% are women); 10 flourmills were established in the 35 targeted kebeles. • Hillside land degraded by grazing were closed from free intervention of animals (70–75% grazing lands were closed). • Sustainable watershed development investment and land treatment was conducted on 79,288 ha of the targeted watershed out of a revised target of 63,355 ha (99% of original target of 80,000 ha and 121% of revised target • 54,159 ha of cultivated land was treated from a revised target of 44,355 ha (105% of the original target of 51,408 ha and 122% from the revised target). • 19,717 ha of degraded land was treated out of the revised target of 16,000 ha that includes degraded grazing land, hillsides, and bush lands and mixed-land use (63% of the original target of 30,977 ha and 123% of the revised target. • 1,089 ha gully land treated and rehabilitated out of a 1,000 ha revised target (79% of the original target of 1370 ha and 109% of the revised target). Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) • Forest development: 7,671 ha was covered with new plantations out of the revised target of 4,000 ha including 2,842 ha of community forests. • Irrigation infrastructure development: 14 Small Scale Irrigation (SSI) schemes on a command area of about 1000 ha have been rehabilitated and are benefiting 3300 households (12 of the schemes are handed over to the community). 14 irrigation cooperatives were established and provided with capacity-building support (office facilities, furniture training, and exposure visits). • 18 primary schools were renovated, 91 primary schools were provided with equipment, and 79,390 students have benefited from this service.15 health posts were rehabilitated and 36 health posts were provided with equipment. Access to health posts reported at 95%. • Water supply scheme management guideline was prepared and made available at the woreda and watershed levels. It is not known if these guidelines were utilized. • Village water supply schemes were constructed; 740 water points (rural water supply) to all 163 community watersheds (including hand-dug wells and springs) were constructed out of the targeted 657 water points (111.7%). • 67 km internal access paths and 89.70 km internal access paths were constructed and maintained respectively; 56 foot bridges were maintained, 70 fords constructed, 7 culverts, and 27 foot bridges were constructed. Outcomes: The ICR did not provide any project outcomes on accelerating sustainable growth in the region. The outcome indicator on percentage increase in private sector investments in Tana and Beles sub-basins was dropped. However, it is clear from the outputs above that there were numerous project and beneficiary investments that contribute to enhanced economic growth in the project area. How far the diverse small- scale interventions were part of integrated planning, management and development is unclear. In summary there were improvements in a few subsectors but, lacking an attempt to estimate their impact on regional growth that leads to a Modest rating for efficacy. Rating Modest PHREVDELTBL PHINNERREVISEDTBL Objective 2 Revision 1 Revised Objective Revised Objectives: “to develop enabling institutions and investments for integrated planning, management, and development in the Tana and Beles Sub-basins for integrated water resources management” There are two PDOs: (A) “to develop enabling institutions and investments for integrated planning, management, and development in the Tana and Beles Sub-basins;””, and (C) to enable “integrated water resources management.” (A) to develop enabling institutions and investments for integrated planning, management, and development in the Tana and Beles Sub-basins - rated Substantial based on the discussion of original objectives. Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) Revised Rationale Same as Original Objective. Revised Rating Substantial PHINNERREVISEDTBL Objective 2 Revision 2 Revised Objective (B) to enable Integrated Water Resources Management – rated Substantial Revised Rationale Integrated Water Resources Management requires: ensuring equal access to water resources by all water users including poor communities; as well as economically efficient and environmentally sustainable use of these resources. The ICR is not clear if the sub-basin organizations were representative of all groups in the basin, and if plans developed would provide equal access to water resources by all communities including the poor. Also, the ICR is not very clear if the plans developed by these organizations were fully integrated in the two sub-basins in an environmentally sustainable fashion. The is no pollution reduction. Although the ICR provided some data on improved land and water management practices by 168 communities, it is not clear how these few communities were selected. The ICR reported that: • 163 community watersheds were identified and plans were prepared for all community watersheds (each community watershed prepared their own comprehensive, multiyear, watershed development plans). 5,275 farmers were trained on watershed planning of which 1,801 were women. • Watershed teams at the community, woreda and kebele levels were established and their capacity strengthened including participatory integrated planning management. The outcome indicator, percentage reduction in sediment loads from targeted watersheds, was measured as 50 % according to the micro-watershed hydrological M&E report and impact assessment (significantly higher than the target of 10 %). The outcome indicator, percentage reduction in average loss of assets due to flooding around Lake Tana, was dropped. Although, the ICR reported that (p viii), there was no asset loss during the past rainy season, this is a very short period of time in which to conclude that this indicator was achieved. It is not clear if the project activities that established systems on flood management, are all operational, therefore it is not clear if they will help reduce flood damages in the future. In addition several other agencies were involved in supporting flood management so attribution is in doubt. The project team provided the following: Integrated Water Resource Management Plans. During project closure, although the sub-basin plan was discussed by stakeholders at different levels (kebel, woreda, regional, federal), it was not approved by the apex Abbay Basin High Council. However, the plan was approved just before the finalization of the ICR; even so, implementation was not started. Nevertheless, since the ICR there is evidence that the Government is continuing to finance the Abbay Basin Authority and the two sub-basin authorities through its annual budgeting process. The High Council has also provided guidance to the Authority to prepare implementation Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) guidelines as they implement the sub-basin plans. Revised Rating Substantial PHREVISEDTBL 5. Efficiency The ex-ante analysis included a cost-benefit analysis that was conducted for the soil and water conservation sub-component that also included livelihood activities (with a conservative 1 % yield increase over a 30-year period). The results of the economic analysis for the water and soil conservation activities showed positive returns: financial rate of return (FIRR) was 23.6% and economic rate of return (ERR) was 24.2 %. Another calculation was conducted for flood mitigation interventions based on flood damage estimates during the period 2005 to 2007 with 2006 being used as representative of a greater than average loss year. Future flood probabilities were estimated from the past 45 years of rainfall data. Three scenarios, under assumptions of l0%, 15% and 20% probablity, were used for damage prevention assessment and for an average loss per year; benefit-cost ratios were 1.37, 1.78 and 2.09 respectively for the three damage prevention scenarios. For all scenarios, the ERRs were higher than 12 percent discount rate. The ex-post financial and economic analysis was done only for the watershed and flood management sub- components and financial returns are estimated following the same methodology used at appraisal. These represent about half of project costs. The ex-post economic internal rate of return (EIRR) for the watershed sub-component is estimated at 17 percent with the economic net present value (NPV) of ETB 221.4 million (at 10 percent discount rate). However, both the economic and financial returns estimated at ICR are lower than the estimates at appraisal, primarily due to increase in operational costs. The ex-ante Flood Management subcomponent, was also projected to be economically viable, with the NPV ranging from US$8.7 million to US$20.3 million and the EIRR ranging from 12 percent to 41 percent. Ex-post financial EIRRs for the flood management subcomponent based on average loss scenario and assuming 10 percent, 15 percent, and 20 percent loss avoided is estimated at 21 percent, 24 percent, and 25 percent, respectively. Operational/Administrative Efficiency: There were significant operational and administrative inefficiencies. The closing date of the five-year operation was extended for almost 3 years (or 60%) to allow time to complete sub-projects that faced delays. However, the extension did not deliver all the expected results and Component C of the project was canceled and the scope of Component A and B was reduced. A number of investments remain to be completed (rainy season power supply for the weather radar; dredger commissioning and training). Project efficiency is rated Modest due to and significant administrative and operational inefficiencies. Efficiency Rating Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Original Project Outcome: Moderately Unsatisfactory.  Relevance of the original objectives and design are rated Modest, due to over-ambitious objectives that were not reflected in size, duration and implementation capacity of the project. The achievement of the objective “to develop enabling institutions and investments” is rated Substantial;  the objective “to accelerate sustainable growth” is rated Modest due to lack of outcome evidence. Efficiency is rated Modest due to administrative and operational inefficiencies. The combined outcome rating is Moderately Unsatisfactory. On a scale of 1 to 6, this scores 3. Revised Project Outcome: Moderately Satisfactory. Relevance of the revised objectives is rated High as the revised objective became more realistic; The efficacy of the objective “to develop enabling institutions and investments” is rated Substantial; the development objective “integrated water resources management”, is rated Substantial albeit with several shortcomings. Efficiency is rated Modest due to administrative and operational inefficiencies. The outcome rating is Moderately Satisfactory or a score of 4. Overall Project Outcome:  Moderately Satisfactory. According to ICR guidelines, the combined outcome rating of a restructured project is weighted according to the proportion of the Credit that was disbursed before and after the restructuring.  38 % of the loan was disbursed before the Level I restructuring and 62 % was after the restructuring. Applying these weights, the combined outcome score is 3.62. Rounding produces 4 or Moderately Satisfactory. a. Outcome Rating Moderately Satisfactory 7. Rationale for Risk to Development Outcome Rating The risk of sustainability of project investments is substantial, mainly due to some additional work needed to Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) fully commission various systems. Institutional risks are substantial. The capacity built to create the enabling institutional and an investment environment for integrated planning, management, and development in the Tana and Beles sub-basins is fragile below the River Basin Planning level. Without further nurturing and support the significant efforts to ensure sustainability of watershed development through participatory process and local ownership is at risk. In mitigation and to sustain this investment, the Government has approved a five-year staffing plan for the Abbay River Basin Authority and the two sub-basins. It is unclear how far integrated river basin management embraces water supply, irrigation, pollution prevention and fisheries. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry Designed to focus on economic development of part of one of the largest river-basins in Ethiopia, the Tana and Beles, resulted in an ambitious project that tried to optimize socioeconomic benefits while protecting its valuable environmental and cultural resource base. The inclusion of the growth component was over- ambitious, as it required a multi-sectoral approach for which institutional capacity and cooperation were lacking. The resulting project was more complex with components and sub-components containing a large range of planned outputs across several subsectors. The project reflected lessons from Nile Basin Initiative (NBI) Project, that was actively supported by the Bank though its facilitation role as well as through the administration of the Nile Basin Trust Fund. In addition, the project design drew from Eastern Nile Technical Regional Organization reports on Watershed Management (design, environmental and social framework, economic analysis, institutional arrangements, and implementation plan), as well as Eastern Nile Flood Preparedness and Early Warning Project design with specific sub-projects for Ethiopia, Sudan, Egypt, as well as for the Eastern Nile Region. A hydrology study (focusing on hydro-climatological and groundwater monitoring networks, as well as design-related sub-basin hydrological and hydrogeological studies) in addition to an institutional and capacity-building study highlighted existing situation. However, safeguards arrangements were prepared by carrying out a sound assessment of environmental and social impacts and guiding the project coordination units in preparing acceptable Environmental Impact Assessments and associated Environmental Management Plans as well as Resettlement Action Plans for Component B. However, risk assessment and mitigation was inadequate in not recognizing insufficient government commitment that meant that some activities were not likely to be implemented. Quality-at-Entry Rating Moderately Unsatisfactory Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) b. Quality of supervision Project implementation was initially delayed due to government staffing issues and limited capacity and experience in managing such IWRM activities and this frustrated Bank supervision, a task not helped by five changes of Task Manager over the eight-year project. Even so, Bank supervision team worked with partners such as ENTRO and the Government of Finland; supported the Government technically and helped build capacity such as bringing in expertise to assist in preparing term of reference for consultancies (such as HIS/BIS) and specifications for large procurements where there is no experience in the country (such as radar). Specific attention to procurement, financial management, and safeguards issues was ensured by assigning specialists to join implementation supervision missions to review progress and resolve implementation issues. Given the complexity of project objectives and very slow progress the Bank should have considered restructuring before Mid-Term Review in 2011. As it was, government’s intransigence delayed the Level-I restructuring until to 2013. Partly mitigating this the Bank approved Additional Financing to enable the project to complete its activities. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance    The Ministry of Water Irrigation and Electricity (MoWIE) and Ministry of Finance and Economic Cooperation (MoFEC) at the federal level and Bureau of Water Irrigation and Energy Development (BoWIED) at the regional level played a strong supporting role throughout the life of the project and provided appropriate guidance and support during all stages of the project. The Government also ensured timely provision of counterpart funds. Major shortcomings were related to the delayed submission of a proposal for a Level 1 restructuring. This resulted in continued delays to implementation of the project until 2013. Government Performance Rating Moderately Unsatisfactory b. Implementing Agency Performance The ICR noted (p. 26) the following results and shortcomings in implementing agency performance: The project had the National Project Coordination Unit (NPCU) at the federal level, Regional Project Coordination Units (RPCUs) at the regional level and the Tana and Beles Sub-Basin Organizations at the local level.  Weaknesses/shortcomings were reportedly mostly due to capacity-gap and staff turnover at the PCU as there was a lack of a full-time dedicated team with enough time and skills. Delays occurred in bringing onboard the full-time National Project Coordinator, procurement, M&E and environmental specialists. There was also a shortage of staff to ensure fiduciary compliance. Consequently there were Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) problems processing major procurements, contract management and timely follow-up and monitoring. Implementing Agency Performance Rating Moderately Satisfactory Overall Borrower Performance Rating Moderately Satisfactory 10. M&E Design, Implementation, & Utilization a. M&E Design The M&E indicators had some shortcomings as neither the effectiveness of capacity and institutional development nor integrated water resources management outcomes had adequate indicators. Nevertheless, the watershed development component of the project, which accounts for about 50 percent of the project cost, had a well-designed and functioning M&E system supported by an MIS to collect information units at different levels throughout the project area. The NPCU was responsible for the overall M&E implementation and to ensure the production of the necessary reports. b. M&E Implementation Progress was monitored regularly through field visits and reports (monthly, quarterly) by team comprising members of the MoWIE, NPCU, RPCU, and implementing agencies. The NPCU obtained regular information and progress data from the Regional Watershed Coordination Unit for Sub-component B1 and from the other implementing institutions for the remaining component/subcomponents. Capacity for operating the M&E system has been built at all levels and the system was in full use by the respective offices at the regional, woreda, and kebele levels. Integrated watershed baseline, follow-up, and impact assessment surveys were conducted to output indicators, evaluate implementation, and measure intermediate results and final achievements. c. M&E Utilization One of the key achievements of the project is the establishment of a web-based M&E /MIS, system specifically for the Watershed Development subcomponent. The system has been functional since 2010 to demonstrate the benefits of watershed conservation and management practices and facilitate the utilization of information produced for the benefit of all stakeholders. A particularly innovative aspect of the M&E was the focus on community monitoring of parameters such as water levels, stream widths, and turbidity. All this generated a wealth of information both on the scale and timing of interventions in each micro-watershed as well as their impact at a decentralized level. The M&E was also augmented with analysis using cloud computing software (Google Earth Engine) and other tools. Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) M&E Quality Rating Modest 11. Other Issues a. Safeguards The project was classified as Environmental Safeguard Category B under Environmental Assessment (OP/BP 4.01). Four other safeguards were triggered: Pest Management (OP 4.09), Involuntary Resettlement, (OP/BP 4.12), Safety of Dams (OP/BP 4.37), and Projects on International Waterways (OP/BP 7.50). There were no changes in the safeguards triggered at appraisal, and the Integrated Safeguards Data Sheet (ISDS) and Environmental and Social Management Framework (ESMF) remained unchanged during implementation.   The ICR noted that (p. 11), at project closing, an internal audit was conducted to assess compliance of the project with safeguards. While the audit did not identify any major safeguard failings during project implementation, it did identify some safeguards challenges. These included uneven application of ESMF screening criteria and inadequate implementation of mitigation measures in Small-Scale Irrigation schemes as well as limited efforts to provide training to the farmers on Integrated Pest Management. Safeguard compliance and management improved greatly during the last two years of the project when the NPCU hired a full-time safeguards specialist. b. Fiduciary Compliance Financial management. The project developed a Financial Management Manual that guided the financial management and internal control process. Although the project had financial management specialists at all levels to manage the project, NPCU was understaffed for the most part of the project, hence limiting the function of capacity building and providing training to the regions and woredas. The ICR noted that (p 11) that most audit reports reportedly were submitted on time with unqualified audit opinions. However, gaps were noted in areas of properly analyzing budget versus actual expenditure to determine variances, limited support by the MoIWIE internal audit function, delay in signing withdrawal applications, and delayed submission of interim financial reports (IFRs) in some instances with substandard quality. Procurement. The Procurement Plan was prepared at appraisal and it was revised during the project implementation. It included a large number of contracts (works, goods, consultancy) involving different methods of procurement and contract values.  Training and procurement clinics strengthened procurement capacity of the implementing agency. In general, the procurement performance was reportedly found to be consistent with the World Bank’s Procurement Guidelines and the Legal Agreements (ICR p. 11). There were moderate shortcomings in procurement, mostly associated with capacity, efficiency, and effectiveness in the procurement management, especially in dealing with contract management and supplier management that resulted in procurement delays Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) c. Unintended impacts (Positive or Negative) The ICR did not report any unintended outcomes and impacts. d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory --- Satisfactory Risk to Development Substantial Substantial --- Outcome Moderately Bank Performance Moderately Satisfactory --- Satisfactory Moderately Borrower Performance Moderately Satisfactory --- Satisfactory Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. 13. Lessons  The ICR provided several lessons. The  most important  follow with some modification of language:   Implementation capacity challenges need to be addressed up front during the project design period especially for multi-sectoral projects with multi-stakeholders The project was expected to deliver multiple outputs and outcomes engaging several implementation institutions that were not sufficiently developed and capacitated. Project experience showed that project scope as well as project time frame should be examined carefully and potential institutional capacity gaps should be addressed in advance before engaging in other development/investment activities.   Integrating livelihood development activities within natural resources management may foster positive incentives to the communities to be fully engaged and promote sustainable watershed Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review ET-Tana &Beles Int. Wat Res Dev Project (P096323) management. Linking farmland treatment with animal forage development, degraded land treatment and area closure with bee keeping and animal fattening, gully treatment with forage, and fruit and tree development have simultaneously promoted the community livelihood and reduced watershed degradation.   Post-project transition arrangements are critical for sustainability of project outcomes. Particular issues that need to be addressed include retaining the skilled capacity or staff; the exit strategy for financiers ,and direct government support that enabled active participation of the community and implementing partners at the local level. 14. Assessment Recommended? Yes Please explain A field level assessment would provide additional evidence on project outcomes and their sustainability after project closure. 15. Comments on Quality of ICR The ICR was quite comprehensive with candid articulation of implementation challenges as well as good formulation of lessons. However, the achievement of objectives was assessed by providing output level evidence rather than outcomes on economic growth and IWRM objectives: there was insufficient discussion of the original sub- objective ‘sustainable growth’. In addition, despite the overview on the list of restructurings in Section H and the reference to "catching-up with initial delays", the ICR could have been clearer on the drivers behind the 34-month closing day extension. This significant overshooting of the original target deserved a more thorough discussion. a. Quality of ICR Rating Substantial Page 19 of 19