Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) Report Number : ICRR0020846 1. Project Data Project ID Project Name P104760 PE-Sierra Irrigation Country Practice Area(Lead) Peru Water L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-78780 31-Dec-2015 48,330,000.00 Bank Approval Date Closing Date (Actual) 27-Jul-2010 31-Dec-2016 IBRD/IDA (USD) Grants (USD) Original Commitment 20,000,000.00 0.00 Revised Commitment 20,000,000.00 0.00 Actual 19,969,590.10 0.00 Prepared by Reviewed by ICR Review Coordinator Group Ranga Rajan John R. Eriksson Christopher David Nelson IEGSD (Unit 4) Krishnamani 2. Project Objectives and Components a. Objectives The Project Development Objective (PDO) as stated in the Loan Agreement (Schedule 1, page 5) and in the Project Appraisal Document (PAD, page 7) was: "To contribute to increasing agricultural production and productivity in targeted areas of the Sierra". b. Were the project objectives/key associated outcome targets revised during implementation? Page 1 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components There were five components. Component A. Modernization and Rehabilitation of collective Irrigation. (Appraisal estimate US$15.92 million. Actual cost at closure US$16.21 million). This component financed pre-investment studies and the design, execution and supervision of subprojects to eligible Water Users Organizations (WUOs) improve water service delivery through rehabilitating collective irrigation systems. Activities included: (i) construction of new water intakes (from rivers or reservoirs) to supply existing irrigation systems: (ii) canal improvement: and, (iii) construction/rehabilitation of small water irrigation reservoirs. There were three eligibility criteria for WUOs: (i) The WUOs had to be legally established: (ii) The WUOs had to have the required technical staff for managing the collective irrigation network: and, (iii) collection efficiency of water tariffs of the WUOs had to be at least 50%. Component B. Irrigation Technology Improvement. (Appraisal estimate US$11.29 million. Actual cost at closure US$11.01 million). This component financed pre-investment studies aimed at supporting eligible farmers groups increase farm-level performance through improved on-farm irrigation systems. Activities included: (i) providing farm level equipment (such as, pipes, pumping units, filters, meters, pressure regulators, individual hydrants and rehabilitation of small irrigation reservoirs): and, (ii) providing equipment on the irrigation plots (such as, installation of sprinklers and drip systems, land leveling and gated pipes). Component C. Capacity Building and Support to Production and Marketing. (Appraisal estimate US$10.07 million. Actual cost at closure US$7.85 million). There were two sub-components. C1: Capacity building of WUOs. This component aimed at strengthening the institutional capacity of WUOs. Activities included: (i) capacity building program for the prioritized WUOs and a participatory diagnosis of WUOs: and, (ii) identifying sub-projects for Component A within the territorial jurisdiction of the selected WUOs. C2. Capacity building of Agricultural Producers and Business Groups. This sub-component aimed at strengthening the capacity of agricultural producers and business groups through: (a) raising awareness of farmers on the benefits of sub-projects: (ii) supporting the establishment of farmers' groups: and, (iii) providing technical assistance to farmers' groups and to formulate and implement viable business plans. Component D. Formalization of Water Rights and National Water Rights Administrative Registry. (Appraisal estimate US$3.32 million. Actual cost at closure US$3.02 million). This component aimed at providing technical assistance and equipment for: (i) formalization and issuance of water licenses in the selected WUOs: (ii) Formalization of the National Water Right Registry (RADA) for securing farmers' access to irrigation water in the selected WUOs: and, (iii) Installation of water measuring devices. Component E. Project Implementation Support. (Appraisal estimate US$6.23 million. Actual cost at closure US$10.05 million). This component aimed at providing technical assistance, training and acquisition of equipment: (I) administration, monitoring (including the baseline study), evaluation and auditing of the project: and, (ii) financing awareness raising campaigns for the WUOs of the new water law and on improved water management practices. Page 2 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. Appraisal estimate (including total baseline cost, costs associated with physical and price contingencies and Front-end Fee) was US$48.33 million. Actual cost at closure (there were no physical and price contingencies during implementation) was US$48.61 million. The actual cost of project implementation support (component E) and component A activities were 61% and 2% respectively, higher than expected at appraisal and component B, C and D activities were 4%, 32% and 11% respectively lower than expected at appraisal. The increase in actual cost of component E and A activities were covered through reallocation of funding between project components. Project Financing. The project was financed by an IBRD loan. Appraisal estimate of the loan was US$20.00 million. Amount disbursed at closure US$19.97 million. Borrower Contribution. Appraisal estimate was US$24.00 million. The actual contribution was about 8% more than planned at US$25.88 million. A contribution from the Local Farmer Organizations was estimated at appraisal at US$4.33 million. The actual contribution was US$2.29 million. Dates. Through a Level 2 restructuring, the project closing date was extended by 12 months for completion of ongoing activities. The project closed a year behind schedule on 12/31/2016. 3. Relevance of Objectives & Design a. Relevance of Objectives The Project Development Objective (PDO) was highly relevant to the government strategy at appraisal. Although the high nationwide annual growth rate (more than five percent per year) between 2001 and 2008 had contributed to reducing national poverty by more than nine percentage points at appraisal, 39% of the population was classified as poor. The incidence of poverty however remained disproportionately high at 75% in Sierra (highlands), where 80% of the households depended on the agricultural sector for their livelihood. Although the government had transferred the responsibility of operating and maintaining the irrigation schemes to Water Users Organizations (WUOs), WUOs in the Sierra had limited capacity and financial resources for undertaking irrigation infrastructure investments. The Government's poverty reduction and rural development strategy for the Sierra highlighted the need for increasing agricultural production both for internal markets and for exports. In 2006, the Government launched the ' Sierra Exportadora ' program, which set ambitious goals for agricultural production and exports, job creation, income generation and poverty alleviation throughout Sierra. This program highlighted the need for expanding the agricultural production of small farmers and identified irrigation schemes as potential economic growth poles because of their importance for the production of higher-value crops for exports and urban markets. The PDO was well-aligned with two pillars of the Country Partnership Strategy (CPS) for the 2007- 2011 namely, accelerating economic growth, widening its base and making growth environmentally sustainable and social development through strengthening human capital. The PDO was consistent with the current Country Partnership Framework (CPF), more specifically with its Objectives 3, 4 and 8 respectively: Facilitate absorption of skills and technology especially by small and medium-sized business: Page 3 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) Enhance the environment for sustainable private sector investment; and Strengthen the management of natural resources. Rating High b. Relevance of Design The statement of the PDO was clear and the causal links between the project activities, which combined irrigation investments with institutional strengthening of the WUOs and farmers groups, and their outputs and outcomes were logical. Activities included collective irrigation infrastructure investments and these activities in conjunction with technical assistance activities aimed at strengthening institutional capacity could be expected to improve the implementation of the Operational and Maintenance (O&M) Plans by the eligible WUOs in the Sierra. Activities at the farm level including providing farm level equipment in conjunction with capacity building support to the eligible agricultural Producers Groups could be expected to improve their capacity for implementing business plans in the Sierra. The intended outcomes were however unclear at design. In particular, while it is fair to assume that O&M plans are an important aspect of ensuring reliability and sustainability in the water supply, it is a stretch to assume that the plans by their very existence would result in sustainable increased yields. A combination of factors are necessary to ensure sustainable long term benefits of improved water supply. Inter alia this includes: clear financial benefit for recipients; cultural shifts amongst users; improved skills; and suitably targeted demand for new crops. Designing approaches to tackle this broad range of issues is necessary to justify the existing theory of change and to ultimately increase agricultural production and productivity in the Sierra region. Rating Modest 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective To contribute to increasing agricultural production in targeted areas of the Sierra. Rationale Outputs. • 42 water intakes, 164.9 kilometers (Kms) of canals and 14 small water regulation reservoirs were Page 4 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) rehabilitated at project closure. (ICR, page 26). No targets were set for this activity at appraisal. • 12 WUOs benefitted from the collective irrigation infrastructures that were rehabilitated at project closure. This exceeded the target of nine (ICR, Datasheet, Key Outcome Indicator Number One). • 18,759 Water Users and 14,770 hectares of land (ha) benefited from the rehabilitation of the collective irrigation infrastructure at project closure. No targets were set for this activity at appraisal (ICR, page 26). • 12 WUOs (out of 12) achieved eligibility criteria for component A at project closure. This exceeded the target of Nine (ICR, Datasheet, Intermediate Indicator Number One). • 12 WUOs satisfactorily implemented the Operation and Management (O&M) plans. This exceeded the target of Seven. • 4,011 users were trained at project closure. This included 3,264 people under the age of 29. Of these, 1,493 representing almost 50% of all users trained belonged to the community of indigenous people (ICR, page 10). No targets were set for this indicator. • 2,117 women were trained (accounting for about 22% of the total population of beneficiaries). Of these, 914 were from the Indigenous communities. No targets were set for this indicator (ICR, page 18). • 1,762.3 hectares of land in the project intervened areas had improved irrigation systems at project closure (ICR, page 16). Outcomes. An Impact Evaluation was conducted during implementation. The baseline for the evaluation was established in May 2015 and four field monitoring campaigns were undertaken (in October 2015, February 2016, June 2016 and October 2016. The evaluation entailed comparing WUO performance in 12 project intervened areas with control areas (that is, areas where there was no intervention). The main conclusions of the impact evaluation were: (i) Irrigation water flow increased and irrigation frequency (defined as the interval between irrigations due to collective irrigation systems) increased in the project intervened areas as compared to areas in the control group: (ii) As a consequence of water flow increases and frequency of irrigation improvements, 28% on average in the project intervened areas had more than one agricultural campaign per year.(see table below): (iii) The increase in agricultural yield as compared to the base line in the project intervened areas as follows in the right hand column: Table One. Results in Intensification of Agricultural land and Agricultural Yield Per WUO (page 13). WUO Intensification of land use (% of Area with Agricultural Yield: more than one agricultural campaign per Baseline /5th Monitoring Campaign year). (Kg/ha) Chonta 13 Ray grass 61,900 /75,000 Cajambamba 17 Avocado 10,000/12,000 Mashcon 17 Ray grass 18,5000/24,000 Mantaro 18 Not available Tarma 18 Vetch 4,000/6,900 C.de Huaylas 24 Alfalfa 11,238/12,500 Colca 0 Not available Cusco 0 White Maize 1,630/1,800 Page 5 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) Huancavelica 79 Maize 2,200/3,600 Huancabamba NA Yellow Potato 3,000/5,700 Juliaca 73 Oats Forage 3,200/5,700 Ayacucho 79 Quinoa 1,800/2,500 Simple 28% Unweighted Average Source: Results report of the 5th monitoring campaign. Note. The impact evaluation on which the table above was based reported data on two aspects of agricultural productivity. (iv) Irrigation efficiency improved by 72% in the project intervened areas as compared to 22% in the control group (ICR, page 16): (v) The percentage of farmers who switched to higher market value groups in the selected WUOs ranged from 25% to 100% as compared to in the control group (ICR, page 13). The ICR provides very little by way of data about the methodology followed in administering the survey. For instance, it is not clear how many beneficiaries were involved. Given that the impact evaluation did not look at other inputs, it is difficult to determine the extent to which this project contributed to the project development outcome. It is however reasonable to assume that increase in irrigation water and irrigation frequency contributed to crop intensification in project intervened areas and this in turn made a significant contribution to increasing agricultural production in targeted areas of the Sierra. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective To contribute to increasing agricultural productivity in targeted areas of the Sierra. Rationale Outputs. The following on-farm irrigation systems were installed at project closure: 107 Sprinkler systems, four Drip irrigation systems. Four Drip + Sprinkler systems, two Micro irrigation + Drip + Sprinkler systems and two Mircro-irrigation + Drip systems (ICR, page 29). There were no original targets for this indicator. • 1,684 producers groups benefitted from improved on-farm irrigation management carried out by the project. This exceeded the target of 1,000 (ICR, Datasheet, Intermediate Indicator Number Six). 1,969 ha of land benefitted from improved on-farm irrigation systems (such as through sprinkler and drip irrigation systems at project closure as compared to the target of 1,500 (ICR, Datasheet, Intermediate Indicator Number Seven). Page 6 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) • 87% of producers groups satisfactorily operated the improved on-farm irrigation systems at project closure. This exceeded the target of 75% (ICR, Datasheet, Intermediate Indicator Number Eight). • 136% of producer groups formulated a business plan according to project guidelines at project closure. This exceeded the target of 100% (ICR, Datasheet, Intermediate Indicator Number Nine). • 66% of sub-projects complied with the Pest Management Plan at project closure as compared to the target of 75% (ICR, Datasheet, Intermediate Indicator Number Ten). • 81% of producers' groups supported by the project satisfactorily implemented their business plans (a collective production and marketing strategy elaborated with the help of agribusiness consultants), at project closure. This exceeded the target of 80% (ICR, Datasheet, Key Outcome Indicator Number One). • 1,506 famers were trained at project closure (ICR, page 31). No targets were set for this indicator. • 2,980 water licenses were issued and 2,980 water licenses were registered at project closure. This was short of the target of 3,000 and 2,700, respectively (ICR, Datasheet, Indicators Number Eleven and Twelve). • 235 water measuring devices were installed at closure at the head of the irrigation units defined in relation with the formalization of water user rights as compared to the target of 250 (ICR, Datasheet, Intermediate Indicator Number Thirteen). • 1,506 farmers were trained, 9,542 Operation and Management Plans were formulated and implemented and 110 business plans were satisfactorily implemented at project closure. No targets were set for these indicators at appraisal (ICR, page 31). • The percentage of farmers who paid water tariffs increased from 50% to 80% between 2011 and 2016 and the total amount collected increased from Nuevos Soles (S/.) 1.52 million to Sls 2.22 million in the same period. No targets were set for this indicator (ICR, page 28). Outcomes. The following were the conclusions of the Impact Evaluation pertaining to the agricultural productivity in the project intervened areas as compared to that in the control group. The change in agricultural yield of rye grass in Chota WUO and Mashcon WTO 116.7 and 95.2 tons/hectare respectively at project closure as compared to 57.05 and 47.5 respectively in areas without the project. The corresponding figures for other WUOs in project intervened areas as compared to the control group: For Potato in Cajabamba province 22.0 as compared to 12.0: For Maize in Mantaro WUO, 14.4 as compared to 5.2: For Oregano in Colca WUO, 2.0 as compared to 1.5: For Organic Lettuce in Cusco, 15.5 as compared to 7.5: For Maize in Huancavelica, 7.4 as compared to 6.0: and for Quinoa in Juliaca, 0.5 as compared to 0.4. As discussed above, while it is difficult to determine the extent to which this project contributed to the project development outcome, given that irrigation is but one of the inputs in agricultural production, it is reasonable to conclude that the project activities probably made a significant contribution to realizing the PDO of increasing agricultural productivity in targeted areas of the Sierra. Rating Substantial PHREVDELTBL Page 7 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) PHREVISEDTBL 5. Efficiency Economic Analysis. A Cost-benefit was conducted both at appraisal and at closure for the irrigation infrastructure components which accounted for approximately 56% of the project cost both appraisal and at closure, using the same methodology. The potential benefits of the project were assumed to come from additional crop production in the targeted areas due to: (i) increases in the agricultural area under irrigation (mainly from improvements in water intakes, canal lining and the adoption of improved on-farm irrigation technologies): (ii) yield increases due to improvements in irrigation: (iii) a shift from low value crops to higher value crops. Costs included the costs associated with rehabilitation and modernization of off-farm (collective) and on-farm irrigation infrastructure and the costs associated with capacity building to improve irrigation management and support for agricultural production and marketing. The Net Present Value (NPV) using a 12% discount rate at project closure was US$30.2 million as compared to the NPV of US$16 million at appraisal. The ex post Economic Internal Rate of Return (EIRR) was 29.4% as compared to the ex-ante EIRR of 21%. Operational and Administrative Issues. There were initial delays associated with execution of project activities due to a combination of factors including, lengthy procedures for the approval and implementation of component A and B activities and problems associated with mobilizing counterpart funds from farmers (due to the enactment of regulations which reduced the counterpart funding from beneficiaries of modernized irrigation sub-projects) and failure on the part of regional governments to provide counterpart funding which were eventually covered by the central government. These issues were eventually resolved and all activities were completed, albeit with extension of the project closing date by a year. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 56.00 Appraisal  21.00 Not Applicable 56.00 ICR Estimate  29.40 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Page 8 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) The PDO was highly relevant to the Government and Bank strategy for Peru. Relevance of design was rated as modest in view of the unclear links between the outputs and intended outcomes. Efficacy of the objectives -to contribute to increasing agricultural production and productivity in the targeted areas of the Sierra is rated Substantial. While it is difficult to determine the precise extent to which the project activities made a contribution, it is reasonable to assume that the project made a significant contribution to realizing the PDOs. Given that the project was shown to be economically justifiable and the relatively minor administrative and operational shortcomings, efficiency is rated Substantial. The overall outcome is rated as Satisfactory, reflecting the substantial to high ratings in the constituent factors, with the exception of moderate shortcomings in design. a. Outcome Rating Satisfactory 7. Rationale for Risk to Development Outcome Rating Financial risk. Although the fee collected by WUOs for operating and maintaining the project activities showed an increase through time, it was not adequate for full cost recovery of the investments. The average tariff that the National Water Authority (ANA) in the Sierra was Nuevos Soles (S/.) 0.0064 per cubic meter. With a water consumption average around 8,000 per cubic meter and average collection rate in the 12 WUOs around 83% in 2016, the average irrigation revenue was a mere S/. 41 per hectare (equivalent to US$13 per hectare). The financial risk is rated as Substantial. Technical risk. Although the project activities provided technical assistance aimed at strengthening the management capacities of the WUOs and producers groups to implement business plans and operate and maintain the irrigation systems, it is not clear if there is adequate capacity for maintaining the irrigation systems. a. Risk to Development Outcome Rating Substantial 8. Assessment of Bank Performance a. Quality-at-Entry This project was prepared, based on the experiences of prior Bank-finance projects (Irrigation Subsector projects PSI1 and PSI 2) which highlighted the importance of institutional strengthening alongside investments in irrigation infrastructure. Lessons from prior projects were adapted to suit conditions on the ground. For instance, the required financial contribution of beneficiaries was lowered in view of the different socioeconomic environment of the Sierra. The implementation of three pilots in PSI 2 aided project preparation with information on the specific challenges associated with irrigation schemes in the Sierra. PSI 2 also had two gender pilots aimed at enhancing participation of women in water management Page 9 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) activities and this experience was used for incorporating the gender dimension of this project. This project also incorporated the recommendations of a study financed by the Bank (The Future of Irrigation in Peru) which supported the government in updating its National Irrigation Policy. Adequate resources were provided for project preparation and the Bank also provided a specialist with the required technical skills from the Food and Agriculture Organization (FAO). The implementing agencies chosen had experience with implementing Bank financed and other development partners' projects (discussed in section 9b). Several risks were identified at appraisal including substantial risks associated with obtaining counterpart funding from regional governments, difficulties of monitoring business plans and Operation and Management (O&M) plans in Sierra, underestimation of project costs and the implementing agencies’ ability to address procurement issues. Appropriate risk mitigation measures were adopted and appropriate arrangements were made at appraisal for safeguards and fiduciary compliance (discussed in section 11). As discussed in Section 3b, it is not sufficiently clear how the project was to deal with longer term issues around sustainability, productive opportunities for water users, skills gaps and required cultural change to ensure the improved water access would lead to increased agricultural production and productivity. In addition, there were shortcomings in M&E design (discussed in section 10a) including no baseline information or targets for several intermediate indicators to properly indicate progress against the intended benefits. In addition, while the impact evaluation was important in giving some indication of the changes in productivity, the compressed nature of this undertaking has significant methodological issues particularly if we assume many of the measures were recorded shortly after the installation of improved infrastructures and without any form of counter factual. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision Supervision missions were conducted every six months. The continuity of leadership was maintained with only one TTL throughout the project. Given the nature of the activities financed by the project, it was appropriate that the supervision team was led by a Task Team Leader (TTL) from the Water Global Practice (GP) and a co TTL from the Agriculture GP. The supervision team assisted the implementing agency in addressing the implementation issues in a timely fashion. For instance, when a large number of Irrigation Sub Sector Program staff who had been trained under the auspices of the project were fired in 2014, the supervision team aided in taking corrective measures in collaboration with the Ministry of Finance. This eventually led to the reinstatement of the staff. The supervision team also mobilized additional Trust Funds to develop a study of the sector that was helpful in examining the irrigation sector in the country. During implementation, restructuring the project would have helped to better align the PDO with PDO indicators and in establishing targets for the intermediate indicators that lacked targets at project appraisal. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Page 10 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) Moderately Satisfactory 9. Assessment of Borrower Performance a. Government Performance The government commitment to supporting irrigation technology improvement in the Sierra was high at preparation. The government also enacted national sectoral legislation (Law of Water User's Organizations 30157) aimed at participation of women in WUOs, by establishing that the WUOs should have at least three women members. In the wake of changes in regulation under Law 28585 in 2013 which greatly reduced the required counterpart funding from the beneficiaries from modernized irrigation sub-projects (from 20% to 0% for collective irrigation infrastructure and 20% only for the on-farm infrastructure), the government responded quickly in providing the additional funding needed for covering the lack of counterpart funds from beneficiaries and regional governments and this helped in speeding up implementation. The project steering committee which was expected to provide high-level guidance and oversight was mobilized and the committee helped in resolving issues such as the removal of staff (discussed in section 9b). The government did not provide harmonized guidelines regarding the financial counterpart funding from the beneficiaries. Government Performance Rating Satisfactory b. Implementing Agency Performance The Irrigation subsector Project (PSI in Spanish) in the Ministry of Agriculture (MINAG) was responsible for overall project implementation and coordination as well a direct implementation of components A, B, C, E and subcomponent D3. The National Water Authority (ANA) in the same ministry was responsible for implementing subcomponents D1 and D2 in collaboration with its local offices. PSI had experience with implementing prior Bank financed projects (PS! 1 and PSI II) and other donor funded projects and ANA had implemented the Bank-financed Water Resources Management Project and a similar project implemented by the Inter-American Development Bank (IDB). The implementing agency complied with safeguards and took corrective actions to achieve project results and this aided in completion of activities at closure, despite implementation delays in the initial years of the project. There were minor shortcomings. The implementation of activities could have been helped through better coordination of the implementing agency with other related government programs. Implementing Agency Performance Rating Satisfactory Overall Borrower Performance Rating Satisfactory 10. M&E Design, Implementation, & Utilization Page 11 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) a. M&E Design The two key outcome indicators at design - the percentage of producers' groups that had satisfactorily completed their business plans and the number of WUOs that implemented the Operation and Management Plans (O&M) plans- were not appropriate. It is not clear how these indicators could per se be expected to monitor increased agricultural production and productivity. There were no baseline or targets for several intermediate indicators. The intermediate indicators were disaggregated by gender. M&E design also envisaged a web-based, customized computer application for M&E facilitation. b. M&E Implementation The baseline for the key intermediate indicators were established in August 2013. Following the recommendations of the supervision team in 2014, a consultant was contracted to design an impact evaluation (comparing agricultural production and productivity in project intervened areas with production and productivity in areas which received no intervention). The baselines for the impact evaluation were established in May 2015 and this was followed by four field monitoring campaigns (October 2015, February 2016, June 2016 and October 2016) for monitoring performance through time). The decision to monitor performance through impact evaluation was appropriate, given that there were neither indicators nor baselines at design for monitoring performance relating to increased agricultural production and productivity. There were delays in M&E implementation due to delays associated with installing the M&E system. This was eventually rectified through the hiring of a consultant to design the M&E system. c. M&E Utilization The indicators were used for monitoring performance in terms of increase in agricultural production and productivity in project intervened areas vis-à-vis in areas that received no intervention. M&E Quality Rating Substantial 11. Other Issues a. Safeguards The project was classified as a "Category B" project. Other than Environmental Assessment (OP/BP 4.01), six safeguard policies were triggered: Natural Habitats (OB/BP 4.04): Pest Management (OP/BP 4.09): Physical Cultural Resources (OP/BP 4.11): Involuntary Resettlement (OP/BP 4.12): Indigenous Peoples (OP/BP 4.10): and, Safety of Dams (OP/BP 4.37). Environmental Assessment, Natural Habitats, Physical Cultural Resources, Safety of Dams and Involuntary Resettlement. The PAD (page 24) notes that the potential adverse impacts were expected to Page 12 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) be mostly localized and temporary in areas where irrigation schemes were to be rehabilitated. An Environmental Assessment (EA) was conducted at appraisal (ICR, page 25). Although investments were expected to be in areas traditionally dedicated to agriculture, the project preparation anticipated the possibility of a few subprojects entailing rehabilitation/improvement/ construction of small regulating reservoirs located upstream of the irrigation schemes. These activities could potentially lead to the inundation of high value ecosystems and partial temporary destruction of landscape including deforestation of small spots of riparian vegetation. The EA conducted at appraisal identified the locations and types of natural habitats and the EMF included measures for addressing Natural Habitats issues. The policy of physical cultural resources was triggered on account of the prevalence of archeological and culturally significant sites in the Sierra region. The policy on safety of dams was triggered as some of the irrigation subprojects that were to be implemented by the project could include the construction and/or rehabilitation of small dams and there was the possibility of potential damage to the existing dams. The project did not envisage land acquisition or physical relocation of populations. The EMF included provisions for compliance with the dam safety safeguard. An Environmental Management Framework (EMF) was prepared and publicly disclosed at appraisal (ICR, page 25). The EMF included measures for addressing environmental issues and issues associated with natural habitats, physical cultural resources, safety of dams and involuntary resettlement, during project implementation (PAD, pages 93-94). The ICR (page 9 and 10) notes that regarding environmental safeguards. although most reservoirs were protected at closure with a fence or signal indicating that access was banned, some of them were not fenced so there was a latent risk of people accidently falling in and drowning. The ICR provides no clear statement on compliance with environmental safeguards. The ICR notes that there was compliance with safeguards on natural habitats, physical cultural resources and safety of dams and there were no involuntary resettlement issues. Pest Management. This policy was triggered at appraisal as project activities on an estimated 1,000 hectares aimed at intensification of agriculture and this could potentially lead to increased use of pesticides. A Pest Management Plan was prepared and publicly disclosed to address issues associated with pest management. The ICR (page 10) notes that there was compliance with safeguards on pest management (ICR, page 10). Indigenous Peoples. The PAD (page 23) notes that the irrigated areas targeted by the project included indigenous people whose concentration was high in the Sierra and some project activities were anticipated to have direct impact on indigenous people. An Indigenous Peoples Plan (IPP) was prepared and publicly disclosed, which provided for screening and review of subprojects in a manner that was consistent with Bank safeguard policy guidelines (PAD, page 25). The ICR (page 10) notes that project activities were implemented following the strategic guidelines. Beneficiaries of the Indigenous Rural Communities Plan participated in 33 sub projects, accounting for a total of 5,472 beneficiaries cultivating 5.378 hectares (ha) of land. And a total of 704 IRC’s individuals benefited from the project, accounting for 42 percent of overall number of beneficiaries (ICR, page 10). b. Fiduciary Compliance Financial Management. The ICR (pages 57 and 58) notes that a financial management assessment was undertaken at appraisal. The financial management risk was rated as Substantial in view of the Page 13 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) implementing agency's performance in prior-Bank financed projects. A Financial Management Action Plan was prepared at appraisal (PAD, page 62). The ICR (page 11) notes that during implementation there were delays in submission of interim financial reports due to the high staff turnover at the implementing agency, but these were rectified over time. The ICR (page 11) also notes that the auditors issued unqualified (clean) for most of the periods audited during project Implementation. In 2013, the auditors issued a "qualified opinion" on the financial statement of the project. However, the implementing agency took necessary actions to overcome the problems and implemented the recommendations made by the auditors (ICR, page 11). Procurement. An assessment was conducted of the implementing agency's capacity to address procurement issues and a procurement plan was prepared at appraisal. The assessment concluded that the procurement risk was Substantial (PAD, 73). The ICR (page 11) notes that procurement management was overall deemed to be satisfactory, despite some minor issues, due to the lack of procurement staff with qualifications and experience in Bank's procurement procedures. c. Unintended impacts (Positive or Negative) --- d. Other --- 12. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- The financial risk is rated as Risk to Development Substantial in view of the Modest Substantial Outcome limited mechanisms for full cost recovery. There were moderate Moderately shortcomings in Bank Bank Performance Satisfactory Satisfactory performance both at design and at supervision. Borrower Performance Satisfactory Satisfactory --- Quality of ICR Substantial --- Note When insufficient information is provided by the Bank for IEG to arrive at a clear rating, IEG will downgrade the relevant ratings as warranted beginning July 1, 2006. The "Reason for Disagreement/Comments" column could cross-reference other sections of the ICR Review, as appropriate. Page 14 of 15 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE-Sierra Irrigation (P104760) 13. Lessons The ICR draws the following main lessons from the experience of implementing this project, with some adaptation of language. (1) Combining capacity building activities for WUOs and farmer groups' small improvements in infrastructure/equipment can be useful in increasing the performance of these institutions. In the case of this project, activities focused mainly on capacity building of WUOs and farmers groups. In combination with small improvements in infrastructure/equipment and capacity building contributed to project performance. (2) Implementing activities associated with a gender dimension can be useful for encouraging women participation. In this project, women participation was improved through disaggregating by gender the results of the project. (3) An adequate co-financing mechanism from the irrigation investment beneficiaries needs to be specified at appraisal. For this purpose, a socio-economic study of the project areas and its beneficiaries before approval is required to properly target the co-financing mechanisms. In the case of this project, delays in funding from beneficiaries contributed to delays, although this was eventually resolved through the government providing more than planned by way of counterpart funding. 14. Assessment Recommended? No 15. Comments on Quality of ICR The ICR is reasonably well-written and provides a good description of the project. It is candid particularly in discussing the issues that arose (such as the lack of indicators associated with increasing agricultural production and productivity). The description of the Impact Evaluation analysis in the ICR is brief and provides little information of the methodology (such as what was representative size of the control group that was compared with project intervened areas). This is a significant shortcoming, given that the PDO's performance in terms of increasing agricultural production and productivity was mainly assessed through the Impact Evaluation Analysis and the ICR should be a complete and comprehensive overview of the project. a. Quality of ICR Rating Substantial Page 15 of 15