Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) Report Number : ICRR0021506 1. Project Data Project ID Project Name P118316 Lake Victoria Phase II, APL 2 Country Practice Area(Lead) Africa Environment & Natural Resources L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-49730,IDA-H7100 30-Jun-2017 25,656,326.33 Bank Approval Date Closing Date (Actual) 13-Jun-2011 31-Dec-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 30,000,000.00 0.00 Revised Commitment 29,274,809.07 0.00 Actual 25,656,326.33 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ridley Nelson Stephen Hutton Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The original Project Development Objective (PDO) is not applied in this assessment because it was revised on June 10, 2012, only six months after project effectiveness, to be consistent with the Lake Victoria Environmental Management Project II, APL I (LVEMP II) supporting Tanzania, Uganda, and Kenya within the same Lake Victoria program with less than US$1 million disbursed at that date. (For reference, the original objective was to (i) improve the collaborative management of the transboundary natural resources of the Lake Victoria Basin (LVB) for the shared benefit of the Partner States; and (ii) reduce environmental stress in Page 1 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) targeted pollution hotspots and selected degraded sub-catchments to improve the livelihoods of communities, who depend on the natural resources of the Lake Victoria Basin (LVB).) The revised PDO was: to contribute to: (i) the improvement of the collaborative management of the transboundary natural resources of the LVB among the Partner States; and, (ii) the improvement of environmental management of targeted pollution hotspots and selected degraded sub-catchments for the benefit of communities who depend on the natural resources of LVB. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 15-Aug-2012 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components Component 1: Strengthening institutional capacity for managing shared water and fisheries resources (US$3.50 million at appraisal; US$1.23 million actual) This was to build capacity and enhance effectiveness of the two partner states’ national institutions in order to improve the cooperative management of shared transboundary natural resources in the Lake Victoria basin. It had two subcomponents: (a) harmonization of policies and regulatory standards; and, (b) ecosystem monitoring and applied research. Component 2: Point source pollution control and prevention (US$7.51 million at appraisal; US$6.52 million actual) This was to support investments to reduce environmental stresses on point source pollution, especially municipal wastewater, on rivers, wetlands, and lakes in the Lake Victoria basin portion of Rwanda and Burundi. There were two subcomponents: (a) planning for sewerage systems and wastewater treatment facilities; and, (b) promoting cleaner production technologies. Component 3: Watershed management (US$13.41 million at appraisal; US$13.20 million actual) Page 2 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) This was to reduce environmental stresses in the Lake Victoria basin through integrated watershed management, including the rehabilitation of degraded wetlands and riverbanks and the adoption of on-farm soil and water conservation on hillsides. It had three subcomponents: (a) restoration of wetlands and riparian vegetation; (b) rehabilitation of hillside areas for production and conservation; and, (c) community- driven development for improvement of livelihoods. Component 4: Project coordination and management (US$5.59 million at appraisal; US$4.18 million actual) This provided the resources necessary for coordination, collaboration, communications, and monitoring and evaluation (M&E). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Costs. Total Project Costs at appraisal were US$30.00 million. Actual at closing were US$25.66 million. Financing Financing from two IDA Credits, IDA-H7100 and IDA-49730 totaled US$30.00 million. Actual disbursed was US$25.66 million. Borrower Contribution The borrower was not expected to contribute any financing directly, although there was use of some existing capacity and resources in participating borrower institutions. Dates The project was approved June 13, 2011 and was effective December 29, 2011. The Mid-Term Review was December 11, 2015 and the original closing date was June 30, 2017. This was extended by six months to December 31, 2017 to complete some unfinished activities. 3. Relevance of Objectives Rationale The ICR uses the Revised Objectives throughout since the Original Objectives applied for only six months. This review does the same. (For a full discussion of the original objectives see the ICR Review for LVEMP II APL 1 covering Tanzania, Uganda and Kenya which used the original objectives for a sustained period.) This project is assessed as a two country project but in the light of the wider five country program, so there is significant overlap between this ICR Review and the one for LVEMP II APL 1 project covering Tanzania, Uganda and Kenya. The main environmental challenges facing the basin as a whole - not just Lake Victoria - as expressed in the ICR, para 7, were: (i) deteriorating water quality; (ii) declining lake levels; (iii) overexploitation of natural Page 3 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) resources; and (iv) resurgence of water hyacinth. A GEF financed transboundary Diagnostic Analysis identified the main environmental threats as deforestation, soil erosion, mining, and degradation of riverbanks. Point source pollution from municipal waste was also an increasing concern due to population growth and urbanization trends in both countries. Pollution from industrial sources was still relatively small in Rwanda and Burundi at the time of appraisal. A question might be asked whether the support for Rwanda and Burundi - neither country touching the shores of Lake Victoria - was as relevant as the support under APL 1 to Tanzania, Uganda and Kenya. While support for Rwanda and Burundi may arguably be of second priority for Lake Victoria itself, IEG concludes that the project was relevant for the following reasons. First, the objective of the APL projects was not simply to improve the environment of Lake Victoria but of the whole basin area including Rwanda and Burundi. Rwanda and Burundi are reliant on water bodies (including rivers, lakes such as Lake Rweru, and wetlands) that face environmental threats including water hyacinth infestation, encroachment of wetlands, loss of wetlands and very steep slope cultivation (ICR paras 5, 6, and 7). They also face sedimentation and pollution which threaten fisheries and pose downstream risks, as Rwanda and Burundi are the source of about 25% of the waterflow into Lake Victoria. There is a substantial and dense population in these two countries whose livelihoods are dependent on these resources. Second, Rwanda and Burundi are members of the East African Community which is fostering collaboration in many areas including trade, customs union, harmonizing policies, health and communicable diseases, immigration and labor, coordination on industrialization strategies, transport, tourism marketing, and links to other African regional organizations. The objective of collaboration should be seen in this wider regional collaborative context reflected in the Bank's strategy. In the medium to longer term, this collaboration on many fronts will offer a range of quid pro quos that, at this stage, may not be easy to quantify. The project objectives addressed most of the environmental concerns except mining, tackling three main areas: the need for a collaborative response, the need to reduce environmental stress in pollution locations, and the need to reduce environmental stress in land areas of the lake catchment, including those feeding into local lakes and wetlands that feed onwards to Lake Victoria. The project design covered the main issues, packaging the response in the three main areas of challenge, collaboration, pollution and land management, as shown in ICR Figure 1 on the theory of change. However, there are two areas that could have benefited from more attention. First, the relationship between the long- term nature of the outcome sought and what that meant for design of the land management interventions could have been addressed with more attention to the efficiency of land management technologies, R&D related to those technologies, farmer incentives, and the operational pathways to scaling up. Second, the longer-term strategy for nonpoint pollution sources from towns and cities could have been better identified. However, overall, the design was consistent with the objective and the longer term aim to tackle the basin as a whole and not just the immediate Lake Victoria surrounds, and to do so over a sustained multi-phase program. The design was consistent with the design of the APL1and its precursor LVEMP I covering Tanzania, Uganda and Kenya but was appropriately lagged behind the second phase of the support for Page 4 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) those three original partner countries. However, as with the APL1, APL 2 stretched the limits of implementation capacity for the two partner countries, Rwanda and Burundi. The project was fully consistent with the World Bank’s Regional Integration and Cooperation and Assistance Strategy for Africa (FY18 - 23). The aim of that strategy was to strengthen the management of regional Commons and cooperation among countries of the region on issues related to regional public goods. There was an expectation that joint participation in this project could strengthen capacity of the member countries to coordinate across a wider range of activities and policies. For Burundi, the PDO was relevant to the World Bank’s Country Assistance Strategy (CAS) for Burundi (FY 13 – FY 16). The ICR argues that the objectives are consistent with this CAS which articulates a strategic objective “to increase resilience by consolidating social stability”. The CAS notes Burundi's ability to harness its marshlands and lakes as a key to agricultural productivity and notes the concerns about pollution from coffee washing stations. The CAS also notes the importance of the broader cooperation within the EAC on tariffs and trade since Burundi's 2009 joining of the EAC. The objectives are also consistent with the Burundi government development strategy outlined in the second Poverty Reduction Strategy Paper. For Rwanda also, the PDO was relevant to the Country Partnership Strategy (CPS) for Rwanda (2014 - 2018) which drew from the government’s medium-term goals set out in its second Economic Development and Poverty Reduction Strategy. The CPS identified priorities for World Bank support under three themes. The project contributes predominantly to the second theme which was, “improving the productivity and incomes of the poor through rural development and social protection.” The CPS specifically aims (p.27), "to boost (agricultural) productivity built around soil conservation and land husbandry … including … progressive and radical terraces … and soil erosion control." The CPS also aimed to achieve," stronger integration of regional economies". In particular, the sustainable land management activities in the project contributed directly to this objective of improving resilience and productivity on degraded agricultural land through terracing and promoting alternative livelihoods. Relevance of the objective is rated Substantial given that it addressed important identified environmental challenges and the priorities set in the Bank and partner’s strategies. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Page 5 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) Objective 1: Contribution to improvement in collaborative management of the transboundary natural resources of the Lake Victoria Basin among the Partner States. Rationale The project design logic was essentially that the environmental status of the lake basin would improve if partners collaborated better on policies, investments and legislation to ensure a consistent and coordinated program, especially the management at the EAC level in the areas of: fisheries, water management including water release, pollution, water hyacinth and nutrient and siltation inflow. This design logic was generally sound. As briefly noted under Relevance, the rationale for cooperation/coordination by the these two countries with partners includes: (I) the framework of wider cooperation under the EAC; (ii) the benefits from bilateral cooperation between Burundi and Rwanda on water, land, and pollution management in neighboring and shared lakes and rivers including particularly Lake Rweru and wetlands; and, (iii) benefits for the three downstream partners touching Lake Victoria. Cooperation by Rwanda and Burundi with the three downstream partners was somewhat less critical on erosion control since this offers immediate private farmer benefits as well as external public good benefits. However, the still sub-optimal level of erosion control practices carried out by farmers of their own volition suggests that these private benefits are barely sufficient alone and may need some form of coordination on subsidized support to reward environmental services. There were two main design weaknesses related to this objective. First, with respect to institutional capacity and collaboration, there was not enough attention to the means compared to the ends. The PAD did not spell out clearly enough what the institutional weaknesses were likely to be in this project for these two partner countries and what the wider lake basin institutional weaknesses had been under the prior project supporting the original three partners of Uganda, Tanzania and Kenya and therefore what changes might be needed in institutional relationships, responsibilities, processes, and modalities of interaction. Second, the steps to reach longer-term scale could have been better specified. For example, there was no experimentation planned in land management to understand the profitability of erosion control measures that would influence adoption by farmers if subsidies were phased out and to understand the cost effectiveness of erosion control measures for reducing sediment movement and there is little evidence of knowledge sharing in this area. The first of these weaknesses was noted in the ICR (p.21) pointing out that, “Although the indicators measure the production of documents, it is the process of reaching agreement that reflects improved collaborative management”. Unfortunately, as in the case of LVEMP II, APL1 for Tanzania, Uganda and Kenya, indicators of this collaborative process in terms of the institutional linkages, team interactions, and the communications and exchange of data involved in that process are not sufficiently reported. This makes it difficult to assess how much the collaborative processes really changed as a result of the project. Nevertheless, the ICR correctly notes that, if properly implemented, the regionally endorsed and gazetted policies, acts, and regulations are expected to have an important long-term impact on the collaborative management of the natural resources of the basin. Page 6 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) A significant implementation and collaboration problem with this project, which did not occur in the LVEMP II APL I project, was that, after 2015, bilateral collaboration between Rwanda and Burundi became infeasible because of a breakdown of relations between the two countries, although they continued to collaborate at lower levels on some limited mutual activities and they both collaborated laterally with Tanzania, Uganda and Kenya, the other three partners in the program. The following paragraphs outline collaboration activities related to the main areas of intervention. On water management collaboration, the main progress was financed under the APL 1 project rather than under this project, but this project served as a platform for Burundi's and Rwanda's participation and their approval of harmonized policies and plans. A draft harmonized policy based on a water resources management plan to guide the future planning of water resources was adopted by the 10th meeting of the Sectoral Council of Ministers for LVB in 2012. Following this, an EAC Lake Victoria Basin Water Resources Management Bill was prepared in 2014 and is expected to be adopted by June 2019. The expectation is that this would lead further to adoption of national level policies by all partners. On fisheries management collaboration, this was relevant for Rwanda and Burundi because of fisheries in lakes and wetlands in these two countries, including the shared Lake Rweru. A draft harmonized policy on fisheries management and an agreement on a strategy and timeframe was submitted by the Lake Victoria Basin Commission for adoption by 2014. This was financed under the APL 1 but required input and endorsement from Rwanda and Burundi. This was achieved but with a four-year delay from the original target date. It was finally endorsed by the Regional Policy Steering Committee in March 2018. In addition, the Council of Ministers approved a number of other documents including a Fisheries and Aquaculture Policy for the East African Community, EAC Guidelines for the Establishment and Operation of Cage Fish Farming; an EAC Harmonized Fisheries and Aquaculture Border Inspection Manual; and Regional Fisheries Guidelines for Species Specific Licensing for Lake Victoria. The ICR reports also that the partner states had “operationalized” the interagency monitoring, control, and surveillance structure for the management of the fisheries resources. However, there is only some anecdotal evidence on the impact on fisheries in their largest shared lake, Lake Rweru. On pollution collaboration, regional standards for industrial and municipal effluent discharges for adoption by participating countries were planned to be put in place. This indicator was achieved and was gazetted in 2016. It is reported that this is a binding standard used by all five EAC partner states. However, it is not yet consolidated within national legislation. The ICR reports that the Sectoral Council of Ministers directed the Lake Victoria Basin Commission to rollout the standards to countries and urge the partner countries to mainstream them into national plans. This is underway, but at different stages in each country. In Rwanda and Burundi, there have been workshops on this topic and the standards applied to industries on effluent discharges are reportedly “regional adopted standards”. On management information systems collaboration, a system was developed and installed on the LVBC server. Both Burundi and Rwanda participated in data and knowledge exchange, but this was hindered by constraints on travel by international consultants during the breakdown in relationships between the two partner states. The type of data collected is not fully described in the ICR but it was web-based and focused Page 7 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) on output indicators at national level. Rwanda built their system on an existing Rwanda Environment Management Agency system. Burundi developed their own new system. In both countries there was public and interagency communication of project activities and results and in Rwanda a web site was developed. There were also a number of videos, communication reports, brochures, and a documentary film produced. The ICR notes that lessons were passed between national teams earlier on, and that a Burundi team visited and learned about models of lakeshore restoration and terracing from the Rwanda team. But such collaboration broke down later. At the regional level of collaboration and knowledge sharing, there is a question about the weakness of the collaborating agency, the LVBC. The failure of coordination of M&E is itself indicative of this weakness, particularly since both the APL 1 and APL 2 were preceded by an earlier project that had developed project M&E systems covering Tanzania, Uganda and Kenya. There is little evidence of improved regional performance on collaboration over time through the LVEMP I and the two LVEMP II projects. More attention to the operational efficacy and efficiency of LVBC could have improved overall basin collaboration. The achievement of this objective is rated, on balance, Modest, largely on the grounds of insufficient evidence on the changes in the processes of collaboration between the parties, the limited substantive role of the collaboration between Rwanda and Burundi as the upstream countries and Tanzania, Uganda, and Kenya as the downstream countries, and the collapse of collaboration between Rwanda and Burundi in the later part of the project. Rating Modest PHREVDELTBL PHEFFICACYTBL Objective 2 Objective Objective 2: Contribution to improvement in the environmental management of targeted pollution hotspots. Rationale The project design logic was fairly clear in principle but unclear in specifics for tackling effluent reduction in pollution hotspots. There is little evidence on the criteria applied for the selection of hotspots by size, volume, or pollution impact or on any ranking system. The quoting of the number of hotspots tackled, an output indicator, shows nothing about priority or scale. The ICR indicates qualitatively some of the main types of hotspots. These included latrines at schools, sewage collection and transport, waste sorting and recycling, on-site pre-treatment and the initial planning and design for waste investments. The project design would have been more able to contribute to longer-term program objectives if it had laid groundwork for future institutional enhancements in the sector to support later scaling up these interventions. Page 8 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) The project financed investments related to point source pollution in selected locations with an original target of 54 sites was later, in 2016, reduced to 41. The reduced target was achieved, with 42 urban pollution hotspots supported, mostly with investments to control wastewater pollution. All but one of these was in Burundi because this was the area with the most direct pollution impact on the lakes and tributaries. There was a target of investment in eight urban centers with point source pollution hotspots. This target was achieved, all of it in towns and districts on or near water bodies with eventual flow to Lake Victoria. There was a target of 34 education centers with improved on-site sanitation facilities. This was achieved but was only implemented in Burundi. In Burundi, the project supported public sanitation in three provincial capitals through latrines. The ICR reports anecdotal evidence in Gitega city that investments in latrines at 27 sites contributed to reducing pollution into two tributaries that flowed to the Lake Victoria Basin. In Rwanda, the project supported engineering designs for a sewerage system in one district of Kigali, constructed a wetland to treat effluent, and prepared sanitation master plans for three district centers. More positively, the project supported Cleaner Production Technologies for a target number of eight industries and harmonized effluent standards for five industries. As reported in the ICR for LVEMP II APL-1, this appears to have been the most successful activity in the project and with some chances of sustainability. In total, 33 industries adopted cleaner production technologies, 25 of them being in Rwanda. Reportedly, 30 industries in the two countries were in compliance with the harmonized effluent standards. About one third of industries that had been trained in adopting the cleaner technology. The ICR reports that from 2013 to 2017, reductions in waste production and natural resources use included: 147,024 cubic meters of wastewater, 30,072 megajoules of energy, 7,256 tons of solid waste, 22,437 tons of CO2 equivalents, 44,195 cubic meters of water use and 918,905 tons per year of materials use. However, since these are amounts treated or reduced, it is difficult to relate these to changes in water quality in lakes or rivers against any baseline figure. Estimates suggest that about US$3.3 million in costs was directly saved through these pollution abatement measures. No evidence is presented on health outcomes. Overall, despite some success with the Cleaner Production Technologies investments with industries, supported by some pollution volume reduction data, this objective is rated Modest, partly due to limited data on impacts and scale. Rating Modest PHREVDELTBL PHEFFICACYTBL Page 9 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) Objective 3 Objective Outcome 3: Contribution to improvement in the environmental management of selected degraded sub- catchments. Rationale The design logic here was that better land management by farmers and communities in the Lake Victoria basin areas would, in due course, reduce siltation and damaging nutrient flow into their own local rivers and lakes and eventually, through onward flow, into Lake Victoria. Again, while the logic was sound, the design and monitoring would have better supported longer-term program objectives if, in addition to supporting the limited area of land “under” improved management treatments, it had also included more support for determining what would be needed to achieve scaling up through wider adoption by farmers over the longer term. This would have called for more technology experimentation and demonstration and more assessments of farm profitability and erosion impacts to help project the likely future needs for subsidized support. As with the APL 1, there is limited outcome level evidence on environmental impact but some output level evidence. The main output indicator was the number of hectares “under sustainable land management” in the targeted sub catchments. The target of 13,000 ha was exceeded with a total of 15,352 ha under SLM practices. The Project Team reported that the measurement protocol to determine the area “under” sustainable land management was the area actually treated under a particular technology such as forestry, Napier Grass strips, progressive terracing, etc. In other words, this was not an aggregation of the total area of sub-catchments supported by community activities, these were treated area measurements. The outcomes are difficult to interpret because all land treated under the project had some farmed crop or forage or natural vegetation cover growing on it before the project and presumably this would have continued over the period of the project with a greater or lesser impact on erosion. Since, for example, one strip of Napier Grass on five acres is likely to have little impact on erosion control compared to a full coverage of strips at the recommended spacing, it is difficult to assess the incremental benefits for erosion or productivity without knowing more details on how the practices were implemented. Also, tree planting typically has complex and mixed benefits, reducing total annual water flow out of a catchment due to evapotranspiration losses, but reducing the peaks and troughs of flow, while having varying impacts on erosion depending on species and slope, some trees being better at holding soil than others and depending on management strategy. Therefore, it is difficult to estimate the likely impact of tree planting based solely on output data on the area of trees planted. A total of 15,352 ha under sustainable land management was achieved, somewhat above the 13,000 ha target. Within that, a total of 2,050 ha of degraded wetlands were rehabilitated, exceeding the wetlands target by about 50%. The target of 80% of farmers within the project intervention areas using improved farm methods was exceeded. These methods reportedly included: better seeds and seedlings, integrated pest management, soil fertility management, conservation tillage, agroforestry, inter-cropping with tree cover, green manuring, use of manure or compost and mulching, and liming. However, attribution and the Page 10 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) incremental impact is difficult to assess because without an established baseline, it is difficult to identify the additional adoption of improved practices caused by the project (as farmers may have been practicing some of these already), and without measurement it is difficult to have confidence that the practices would achieve the desired effects. The practices listed here appear more productivity focused than environmentally focused relative to the APL 1 which seemed to focus more directly on erosion control technologies but this may be due to differences in agroecological zone, differences in existing farm erosion control assets, or simply different labels e.g. conservation tillage may include Napier Grass strips or terracing. The M&E system was unable to provide much information on beneficiaries and the views of beneficiaries across all the project interventions but especially the land management activities..The focus was more on area treated. On beneficiaries by gender, an indicator added later, the ICR reports that, in Rwanda, 38% of the farmers cultivating with terraces and 32% using integrated pest management were women, and, in Burundi, 60% of the total farmers adopting were women. (This last figure is reported differently on page 46 of the ICR as 54% for Burundi and 36% for Rwanda.) At the level of Intermediate Indicators, a target was to achieve 60% of community natural resource management sub-projects with implementation rated satisfactory or better in the targeted sub-catchments. It is not entirely clear what methodology was used to rate this because the ICR (p.27) refers to “satisfaction rates”, and the rates achieved were 68% in Burundi and 82% in Rwanda. This was reportedly based on socioeconomic surveys conducted on a sample of 34 CDD subprojects in Burundi and 76 CDD/CMI subprojects in Rwanda. In Burundi, 49 km of a 50 m wide buffer zone around Lake Rweru, and 108 km along two major rivers, were restored with bamboo planting, supported by 4,200 ha of terraces, wood lots and agroforestry on 1,425 ha. There were also a number of other productive activities in this zone. In Rwanda, 278 ha of wetland buffer zone around the lake and 280 km along tributaries were restored. This included the construction of erosion control terraces. The ICR expressed concern about the sustainability of these because agricultural activities and cattle grazing within those buffer zone areas have not been restricted. The World Bank team is reported as seeking assurance from the governments that land use restrictions would be in place in these areas of riverside restoration. However, it is difficult to see how protection in such areas could be suddenly enforced by restrictions when the existing law has not been enforced for many years. The Project Team advised IEG that, since the investments had been made with community involvement, and since the investments such as tree planting and beehives offered benefits and, in the case of bees, would keep away cattle and goat grazing, there were incentives to sustain these areas beyond just the pressures of regulations. In Rwanda, flooding in 2016 caused damage which required remediation and restoration of some of the riverbank plantings which had not yet reached sufficient maturity. In both countries, some evidence is reported of increased productivity of crops from the improved land management although the source and the survey methodology for this evidence is not given. In Rwanda, an unreferenced survey reportedly found agricultural incomes increased by over 30%. It is difficult to interpret Page 11 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) what this means for meeting the environmental objectives. Incentives are important and typically, in Africa, better crop biomass and coverage leads to better soil protection, but not always, for example, when a high yield vegetable crop is taken off a fragile soil area on a steep slope which should be stabilized by permanent vegetation cover. Direct payments for labor intensive land management works became, temporily, an important source of income and social benefits but this windfall income will not be sustained unless there is further spontaneous adoption of labor-intensive technologies or further project support. The ICR reports three potentially negative issues with land management interventions. First, some sub- project investments were quite far from rivers and therefore unlikely to have much immediate erosion impact. Second, as commented on above, some activities were more production and income oriented than environment oriented. This may have helped farmer adoption but the project aim was to improve environmental management. Third, there was a serious disease outbreak of PPR (Peste des Petit Ruminants) that was introduced by the inadvertent project purchase of infected goats. This caused considerable economic loss to farmers supported by the project and there were high costs to rectify it. This was a serious negative unintended impact but clearly attributable to a project action. The direct project beneficiaries in the two countries, including CDD, CMI, and farmer field school beneficiaries (the latter only in Burundi) totaled 45,427 people (49.3% female), close to the original target. Overall, the achievement of this objective is rated Modest, mainly due to: limited outcome evidence; the scattered nature of the erosion control measures and thus the limited environmental impact on rivers and lakes, and the significant negative impact of the introduced goat disease. Rating Modest PHREVDELTBL PHOVRLEFFRATTBL Rationale All objectives were rated Modest in their level of achievement against the expected outcomes. Overall Efficacy Rating Primary reason Modest Insufficient evidence 5. Efficiency Page 12 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) The Efficiency analysis in the PAD did not attempt an aggregate economic rate of return. It focused on the internal rates of return (IRR) for three types of activity in Rwanda (watersheds, wetlands, and water resources and sanitation) which gave ranges of IRR between 13% and 26% and for one type of activity in Burundi (watersheds) which gave a range of 15 to 20% IRR. In the ICR, a small number of case studies were used for ex-post economic analysis in the same categories of activity used in the PAD. However, these are not claimed to be representative. A mix of data sources and assumptions was used. However, these are only IRRs to individual case studies and do not represent the project level. There are weaknesses in the data in these cases since data from actual reduced erosion, reduced waste discharges into rivers, or amounts of treated wastewater were not available. Data collected at completion for land management changes relied on interviews with a very small sample of farmers. In any case, many investment activities started late in the project cycle and so had benefits that were barely evident by the time of project closing. The benefit/cost ratios for the community level and farm level activities range from 1.1 to 3.0 with an average of around 1.5. The internal rates of return ranged from 7% to 13%. Given the risks typically faced by small farmers and the shortage of cash and credit, a benefit cost ratio that would be sufficient to get high levels of spontaneous adoption has generally been considered, as a rule of thumb, to be above 1.5. This suggests that a number of the investment activities at the farm level would still require grant support to achieve substantial spontaneous levels of adoption. There was a significant cost attributable to this project in livestock mortality with the death of 8,600 goats due to disease brought in with animals purchased by the project. While appropriate measures of amelioration were undertaken, the costs of this disease outbreak should be counted. This appears not to have been incorporated in any of the costings of sustainable land management activities, nor are there any project overheads within which to include it. The ICR estimated unit costs of sustainable land management practices in watersheds in Rwanda at about US$780 per hectare. This is at the upper range of comparable costs in the region with cheaper progressive terraces in Kenya costing as little as US$200 per hectare according to the ICR. However, with mixed treatments, comparisons on an area basis are difficult. The project did not collect enough data or do enough experimentation to draw conclusions about the most efficient soil protection practices for taking the program to greater scale so, for net benefit calculations, assumptions had to be made. The ICR (p. 30) attributes the lack of project data to the high costs of monitoring small ex post samples to assess productivity gains, and completion of some activities too close to project closing. On implementation efficiency, the project had a slow start and there were early procurement difficulties, resulting in a lagged benefit stream. Overall, Efficiency is rated Modest due to limited data, lack of project overheads in the limited number of case study IRR calculations, and the omission of the attributable costs of the small ruminant disease outbreak with the loss of over 8,000 goats and the need for a costly vaccination program to contain the outbreak. Page 13 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) Efficiency Rating Modest a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Point value (%) *Coverage/Scope (%) Available? 0 Appraisal 0 Not Applicable 0 ICR Estimate 0 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance is rated, on balance, Substantial. Efficacy is rated Modest in aggregate due mainly to weaknesses in evidence, partly a result of weak M&E. Efficiency is rated Modest due to limited evidence to support assumed benefit streams, lack of an aggregate rate of return, and no estimation of, or allowance for, the project’s inadvertent introduction of a small ruminant disease. Overall outcome is rated Moderately Unsatisfactory. a. Outcome Rating Moderately Unsatisfactory 7. Risk to Development Outcome There are a number of risks: 1. The relationship between Rwanda and Burundi could potentially make ongoing collaboration more difficult, particularly with respect to Lake Rweru and the feeding catchments. Moreover, overlaid on this risk are the risks of weak collaboration between the other three Lake Victoria partners, Tanzania, Uganda, and Kenya. This is a complex five-partner collaboration that needs high levels of commitment to be sustained. Page 14 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) 2. The ICR notes some concern for the lack of time to test sustainability of business plans in community- based investments. This would seem to be a correct assessment, but an unavoidable situation given the timeframe and the slow start. The efficiency analysis does suggest some modest private benefits for participating farmers, at least for some of the technologies, that would suggest some hope of sustaining investments already made. But whether there has been sufficient demonstration effect and extension staff training to reasonably anticipate a spontaneous spread of investments like progressive terraces or napier grass strips with less subsidy is by no means clear. 3 . The ICR notes the need for governments to enforce measures to protect investments by restricting agricultural activities in restored buffer zones. It seems very unlikely that the governments or local authorities will be in a position to restrict such activities that have been going on for many years. This issue needed to be tackled more comprehensively by looking at incentives for farmers in relation to existing legislation on land- use. 4 . The ICR correctly notes the risk of climate change on the basin as a whole and on the particular investments made in erosion structures and plantings. However, it also correctly notes that the interventions should improve resilience to these changes. The alternative livelihoods supported under the project are one aspect of this. Based on climate change projections, the climate change risk for Rwanda and Burundi is actually less than for Tanzania, Uganda and Kenya. 8. Assessment of Bank Performance a. Quality-at-Entry Two main sources of lessons informed preparation of this project. The first was the experience from the first Lake Victoria Environmental Management Project, the second was experience from the Lake Victoria Environmental Management Project II, supporting Tanzania, Uganda, and Kenya which had started implementation three years earlier. Lessons included lessons on community-based catchment rehabilitation and on the operation of Beach Management Units to control illegal fishing. One significant weakness in quality at entry design was the over-ambition of the Project Development Objectives. Another, noted by the ICR (p. 40), was the excessively broad focus on tackling pollution, spreading interventions too thinly. There were weaknesses in designing the project to match client implementation capacity and weaknesses in the Results Framework, especially in the formulation of the indicators. There was significant focus on the ends, the achievements of covenanted deliverable strategies, policies and plans, and relatively insufficient focus on the means, the intermediate outcomes that would lead to the collaborative processe and incentives as the essential drivers and mechanisms of enhanced collaboration. The ICR also notes that there should have been a baseline study to better inform the M&E system. This would have been particularly important for this project covering Rwanda and Burundi as these two countries had not had a prior phase like LVEMP I had which would have at least offered some closing date data as a de facto baseline. Page 15 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) As with LVEMP II APL I, the project design could have focused more on experimentation and demonstration, particularly in the land management interventions. Given that the interventions could realistically cover only a small portion of the total basin area, there was a need to move towards greater scale in the future by developing profitable solutions for farmers that would need less public funding. The project design could also have paid more attention to the social and legal aspects of managing vulnerable riverbank areas where sustainable management requires reduced pressures from cultivation and livestock, but where there has been unsustainable land use pressures for decades despite long-standing protective laws. On the issue of strategy and scaling up, there was a difficult strategic project design question about whether both projects should have concentrated their CDD/CMI land management activities to reach a critical mass erosion impact on a small number of selected community land areas or whether they should have spread more broadly and thinly across sub-catchments to optimize the spread of experimentation and demonstration nodes for wider impact in later phases. In other words, should the design have focused on maximizing the erosion control impact on a small selected area or spread more widely to maximize learning and demonstration. The strategy in this respect was not spelled out in the PAD and implementation seems to have fallen between these two extremes. The limited evidence, such as it is, cannot answer the question of which strategy would have been better but given the very small treatment coverage funded, it seems likely that the better strategy for a long term program would have been to initially maximize the learning and R&D and then move outwards to scale from those nodes of learning. The design would have benefited from more explicit strategic thinking on issues like this. Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision The Bank team carried out sound fiduciary supervision and gave useful technical support. The Bank’s role is recognized in the Borrower's comments (ICR Annex 5). The team organized a number of procurement clinics to build capacity. The ICR reports that there were prompt and regular communications with the Coordination Teams at national and regional level. During the project, a system of national TTLs was developed at the national coordination team level. This reportedly resulted in better communication with the Bank and faster decision-making, particularly for procurement “no objections”. This also seems to have been appreciated by the borrowers. The Borrower's comments note the issue of the World Bank changing TTLs. There were three TTLs up until early 2014 which the Borrower's note "affected implementation, but to a limited extent" (ICR p. 67). However, there appears to have been attention to handover and overlap between TTLs to make these transitions as smooth as possible. Page 16 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) Somewhat at odds with the community-based strategy applied in vulnerable riverbank areas under the project, the ICR (p. 27) notes that by project closure, the Bank had not received assurances that restrictions on agricultural and livestock activities in buffer zones would be in place. This appears to suggest an approach based more on legal compulsion than participatory community management. There are similar tensions between regulations and community management in lake fisheries with community beach/landing site associations which warrant further thought on implementation and design in any follow-on projects. There was a weakness in supervising the small ruminant purchasing which introduced disease. This arguably should have been flagged up front in the Environmental Assessment. With hindsight, there should have been more thorough veterinary inspection procedures but it is understandable that this was overlooked in a non-livestock project with only a single bulk procurement of small ruminants. The ICR notes with some concern the late MTR, four years into the project, with a slow restructuring that followed one year later. As with the parallel LVEMP II APL I project, supervision ratings were generous and did not flag the seriousness of M&E issues and the difficulties of measurement in assessing productivity, and environmental and community benefit achievements. Moreover, some of the changes in indicators still did not resolve the problem of measurement. The overall rating of Bank Performance is Moderately Unsatisfactory, mainly due to weaknesses in design, insufficient assessment of counterpart capacity and weak measurement of impact. Quality of Supervision Rating Moderately Unsatisfactory Overall Bank Performance Rating Moderately Unsatisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design As with LVEMPII APL I, there were weaknesses in the Results Framework indicators including confusion about the definition of some indicators such as “direct project beneficiaries”. The project included capacity building for M&E within the National Project Coordination Teams, in the Water Resources Departments, in the Lake Basin Authority and in a number of other government departments including those managing municipal discharges and environmental standards. The Results Framework covered mostly targets and measurement of outputs or intermediate outcomes, not very informative for understanding outcomes and impact. These included submissions of policies and plans for Page 17 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) approval at regional or national level, numbers of pollution hotspots, number of countries adopting regional effluent standards, numbers of industries adopting cleaner production technologies, numbers of hectares treated with improved land management, and numbers of beneficiaries. The indicators offered little evidence on the scale of pollution reduction and the reduction in erosion from agricultural land or the increase in farm yields or the financial profitability of land treatments for farmers with and without subsidy. As noted by the ICR (p. 37), indicator design was inadequate to measure some essential intermediate outcome changes, for example, the reduction in sediment load in rivers. There was also weak or nonexistent baseline evidence for many of the parameters. Despite an original intention to attempt some measurement of soil loss in watersheds, there was no construction of erosion traps to at least measure impacts of different treatments on sample erosion control plots. The ICR also points out that it should have been possible to make better estimates of the total volumes of waste managed based on site rates of use. Each country developed an MIS system measuring the achievements of project indicators. The project also included community-based M&E activities for self-monitoring of performance and impact at the local level. Geo- referenced data were collected, some of which is available on the websites. b. M&E Implementation The ICR points out (p. 38) that some of the indicators were weak in their ability to measure results and in their reduced ambition. For example, the indicator, “number of countries implementing the regional Water Resources Management Plan, the Updated Fisheries Management Plan, and the Sustainable Land Management Strategy” was replaced with, “Fisheries Management Plan updated”. The plan to harmonize a web-based MIS system at the basin level was delayed due to slow procurement and the time taken for testing and training. As a result, the national teams started to develop their own systems. Later, there being insufficient funds to bring these diverse systems into a single coordinated system, the national systems reported separately and the data was then manually input at the Lake Victoria Basin Commission level, largely defeating the harmonization aim. A proposed survey to assess farmer satisfaction with community subprojects was not done. The ICR reports that M&E specialists were experienced and had clear responsibilities but that there was only one in each country and with limited resources. c. M&E Utilization The annual work plans and the budget documents were based on information from the MIS system. The ICR notes that, when the 2016 flooding in Rwanda affected a number of areas where riverbank restoration was ongoing, the country teams were able to quickly assess the extent of the damage and budget the amount they would need to rehabilitate the affected areas. Page 18 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) Data from M&E was useful in the Mid Term Review when reallocation of resources between components was needed. Overall, M&E is rated Modest because of weak indicator design, because of concerns that it could not offer evidence on changes in water quality and sediment load, and because the monitoring systems were never really harmonized across countries. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was a Category B, calling for only a partial assessment. The applicable safeguard policies were: (a) Environmental Assessment (OP/BP 4.01); (b) Natural Habitats (OP/BP 4.04); (c) Pest Management (OP 4.09); (d) Physical Cultural Resources (OP/BP 4.11); (e) Forests (OP/BP 4.36); (f) Safety of Dams (OP/BP 4.37); (g) Involuntary Resettlement (OB/BP 4.12); (h) Indigenous Peoples (OP/BP 4.10); and, (i) Projects on International Waterways (OP/BP 7.50). The required environmental and social safeguard instruments were prepared and publicly disclosed. Arguably the environmental assessment should have noted the disease risk of livestock imports but it is not clear whether that was contemplated at the time of appraisal. The legal framework for environmental management was applied to mitigate potential negative effects. The rehabilitation or construction of sanitation facilities, wastewater treatment, watershed management and other CDD activities generated only moderate impacts. These were monitored. The main positive environmental impacts are related to CDD subprojects focused on natural resources conservation. Environmental and Social Impact Assessments were carried out for all major interventions of wastewater and sanitation. Environmental and Social Screening was carried out for CDD and CMI subprojects. Grievance address systems were in place in both countries. There were some issues for subprojects towards the end approaching the project closing date when there was insufficient budget to provide support. In Burundi, as the project came near to its closing date, support could not be given to some of the later women’s groups who were growing fodder for livestock. b. Fiduciary Compliance Financial Management In both countries, financial management was mainstreamed in the national system at the lead ministries. At the regional level, financial management was handled under the East African Community. Page 19 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) The main risks were related to staffing and obtaining necessary computer software. Audits were unqualified during the whole of the implementation period. Due to some concerns about fiduciary risk in Burundi, the ceiling for advances was reduced by half. Following concerns about financial management and CDD projects, in 2015, the project decided to pay directly the CDD providers rather than through intermediaries to reduce delayed payments to them. Procurement The ICR reports that procurement skills were a challenge throughout implementation but there were no major issues. In some cases, procurement was slow due to lack of familiarity with World Bank procurement requirements. This had an effect on the number of wastewater treatment investments in Rwanda, with only one wastewater treatment investment being completed. Some consultant contracts had to be canceled due to the political situation in Burundi during the last two years of the project. c. Unintended impacts (Positive or Negative) There was a negative impact due to the importing and distribution of diseased small ruminants that then required response through a vaccination campaign. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Moderately Outcome --- Unsatisfactory Unsatisfactory Moderately Moderately Bank Performance --- Unsatisfactory Unsatisfactory Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons The ICR has five lessons, three of them are similar to, or the same as, lessons from the LVEMPII APL 1 project. The following are partly derived from the most important of the ICR lessons but with adjustments of language. The last two are additional. Page 20 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) 1. The level of ambition stated in the project development objectives needs to be realistic and attributable even if the objectives are nested within a more ambitious, longer-term, vision, while measurable indicators need to be consistent with those objectives and not inconsistent with a counterfactual. In this case, the expectation of water quality improvement in the lake was unrealistic against a deteriorating trend. 2 . Collaboration should be measured not only through agreed end products such as frameworks, policies, and plans but as improvements at an intermediate outcome level in the following areas: institutional collaborative processes; modalities and pathways of communication, qualities of collaborative interactions; nodes of responsibilities; and extent and quality of sharing of data. In this case, the deliverable frameworks and policies were originally covenanted as outcomes but were not necessarily useful indicators of changed processes and capacities. 3 . In a regional partnership program, it is important for realistic planning to examine and understand the political economies and the incentives for individual partners within each sector to support the proposed reforms and investments. In this case, there were some unrealistic expectations related to partner incentives for collaborative action. 4 . In a long-term lake basin partnership program aimed at enhanced collaboration between partners and agencies, and involving many national players, it is important to ensure ownership of investments at the level of national and local authorities, sanitation companies, asset operators, and community groups. In this case, more could have been done in requiring formal agreements and operating budget commitments and, particularly for the Resource Efficient and Cleaner Production (RECP) activities, more attention could have been paid to performance incentives. 5 . In a long-term program aimed at environmental improvement over a series of projects, it is important to treat any one project as a phase in the pursuit of the larger vision and to ensure continuity of funding, staffing, strategy, and institutions. In this case, there were a number of discontinuities including, several years hiatus between the first and second project, and now, it appears again between the second and third (in the case of Rwanda and Burundi the first and the second). In the total five country program covered by this project and the project supporting Tanzania, Uganda and Kenya, there was lack of continuity in the M&E system, probably partly due to the hiatus between the first two projects. Bank planning of long-term programs needs particular focus on sustained support to allow each project to build on the prior project and to ensure cumulative learning. 6 . Proactive measures by the supervision team can reduce fiduciary risk in times of political crisis. In this case, there was political conflict in Burundi but the team adjusted quickly, carrying out the MTR off-site, and tightening financial controls which, the ICR notes, could have been taken even further with procurement review thresholds. 7 . Weak country veterinary systems and weak technical input and coordination by the Bank can lead to unanticipated consequences with livestock activities. In this case, in Burundi, procurement of goats was not subject to adequate disease inspection. Significant procurement of livestock in a project calls for veterinary input and tight supervision by the Bank. Page 21 of 22 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review Lake Victoria Phase II, APL 2 (P118316) 13. Assessment Recommended? Yes Please explain Yes, but only as a supplement to the other LVEMP II APL 1 project. The rationale is similar. However, the priority would be LVEMP II APL 1. 14. Comments on Quality of ICR As with LVEMP II APL 1, this project also was part of a very complex package of interventions with limited performance data. The ICR has substantial detail and in a few respects is stronger than that for LVEMP II APL1, but perhaps because the project itself was smaller and therefore easier to package. The ICR was generally candid about the weaknesses. More attention could have been given to understanding the changes in collaboration processes and linkages between the different layers of partnership institutions at both national and EAC/LVBC level. The ICR had to deal with limited outcome data due to weak M&E although more ICR field work might have filled some of the gaps, especially on land management. The problematic issue with diseased goats was adequately and openly covered. The ICR was open about methodology and sources of data. The lessons were thoughtfully formulated. a. Quality of ICR Rating Substantial Page 22 of 22