Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) Report Number : ICRR0021085 1. Project Data Project ID Project Name P079165 PE Sierra Rural Development Project Country Practice Area(Lead) Additional Financing Peru Agriculture P127801 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-74430,IBRD-82460 31-Dec-2012 34,930,000.00 Bank Approval Date Closing Date (Actual) 24-Apr-2007 30-Jun-2017 IBRD/IDA (USD) Grants (USD) Original Commitment 20,000,000.00 0.00 Revised Commitment 39,149,331.78 0.00 Actual 39,149,331.78 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hassan Wally Vibecke Dixon Christopher David Nelson IEGSD (Unit 4) 2. Project Objectives and Components a. Objectives The Project Development Objective (PDO) as articulated in the Project Appraisal Document (PAD, p. 8) was to: "improve the assets and economic conditions of approximately 53,600 rural families in the selected areas of Apurimac, Ayacucho, Huancavelica, Junin, Huánuco and Pasco, and strengthen government capacity to implement an integrated Sierra development strategy." The Project Development Objective (PDO) as articulated in the Loan Agreement (LA, p. 5) was to: Page 1 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) "assist the Borrower in improving the assets and economic conditions of rural families in selected areas of the Borrower’s Apurímac, Ayacucho, Huancavelica, Junin, Huánuco, and Pasco regions, and strengthen government capacity to implement an integrated Sierra development strategy." The PDO was revised through a Level 1 restructuring carried out in 2013 to the following: "to assist the Borrower in improving the assets and economic conditions of rural families in selected areas of the Borrower’s Apurímac, Ayacucho, Huancavelica, Junín, Huánuco and Pasco regions." This Review will evaluate the project against the objectives as stated in the Loan Agreement; and a split assessment will be carried out due to the change in PDO and component 3 during the Level 1 restructuring in 2013. b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? Yes Date of Board Approval 28-Jan-2013 PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? Yes d. Components The PDO was supported by three components: 1. Rural Businesses (appraisal cost: US$15.61 million, actual cost: US$32.44 million). This component aimed to help rural business opportunities by the design and implementation of sub-projects that are proposed by groups of small producers to improve their production, market access and income. The activities supported included: providing pre-investment financing to create proposals with viable business plans and financing selected sub-projects through a competitive fund. The goal was to implement 620 rural business sub-projects. It was expected that most sub-projects would be in agriculture (crops, livestock and agroforestry), but non-agricultural sub-projects (aquaculture, tourism, services, handicrafts, small scale industry and agribusiness) were also eligible. 2. Community Development (appraisal cost: US$13.58 million, actual cost: US$20.17 million). This component addressed socioeconomic risk and insecurity among the poor who lack access to basic services and were likely to be left behind even in periods of broad economic growth. The component supported the design and implementation of sub-projects to diversify, expand, and improve the assets and production of rural families in areas that had limited access to markets. The main output was the implementation of about 875 community development subprojects with clearly defined and sustainable goals. Page 2 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) 3. Sierra Coordination and Project Management (appraisal cost: US$5.74 million, actual cost: US$13.67 million). This component ensured efficient and effective development planning in achieving the project development objective and other indicators and to feed that information back into enhancing project implementation. It also supported the Peruvian Government’s efforts to prioritize, strategize, coordinate and implement the execution of an integrated development program for the Sierra. Revised Components. With the Level I Restructuring of 2013, Component 3 was modified to "Project Management" and was no longer expected to support inter-institutional coordination in the Sierra region. The restructuring also resulted in expanding the geographical coverage of the project from 255 districts to 395 districts. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The total project cost at appraisal was estimated at US$34.93 million. Actual project cost according to the ICR (Annex 3) was US$66.27 million or 190% of the appraisal estimate. The main reason for the increase was the US$20.00 million that the project received in 2013 as additional financing. Actual total cost reported by the ICR (p. 2) was US$65.61 million. Financing. The project was initially financed through a Specific Investment Loan (SIL) worth $20.00 million from the IBRD. In January, 2013 the project was restructured and an additional financing worth US$20.00 million was approved. The actual amounts disbursed according to the ICR (p. 2) were US$19.83 million and US$19.32 million for the original loan and additional financing, respectively. The total amount disbursed was US$39.15 million. Borrower contribution. The borrower was expected to contribute $7.83 million of counterpart funds, and $7.10 million in cash and in-kind contributions from beneficiary families and producers. Actual amounts contributed were US$12.61 million and 13.86 million for the borrower and local communities respectively (ICR, p. 2). The total amount contributed was us$26.47 million. Dates. The project was approved in April 2007 and became effective 15 months later, in July 2008. The project was expected to close on December 31, 2012. However, it closed four years and six months later on June 30, 2017. This delay was to accommodate the expansion of project activities and allow time to disburse additional funds and implement activities. The project was restructured four times. The first was a level 2 restructuring on October 30, 2012, when the amount disbursed was US$17.24 million, in order to extend the closing date of the original Loan by six months from December 31, 2012 to June 30, 2013 to allow for full completion of 350 sub-projects. The second restructuring was a Level 1, on January 28, 2013, when the amount disbursed was US$18.24 million, in order to allow for changes in Project Development Objectives, modify the Results Framework (RF), change in components and costs, and extend the Loan closing date by three years from June 30, 2013 to June 30, 2016. Also, an additional financing worth US$20.00 million was approved at the same time. The third restructuring was a level 2 on June 28, 2016, when the amount disbursed was US$36.33 million, in order to extend the AF Loan closing date by six months period from June 30, 2016 to March 31, 2017 to allow additional time for Page 3 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) implementation of project activities; and reallocate funds between disbursement categories. The fourth restructuring was a Level 2 on March 31, 2017, when the amount disbursed was US$39.83 million, in order to extend the AF Loan closing date by three months from March 31 to June 30, 2017 to finalize completion of activities delayed by severe floods across the country. The Mid-Term Review was carried out on December 5, 2010 compared to a planned date on March 30, 2010. 3. Relevance of Objectives Rationale The Sierra region of Peru includes all areas of the Andes above 2,000 meters. It has a population of 10.6 million, of which about 60% (6.3 million) live in rural areas. About 75% of rural inhabitants are indigenous people, 82% are poor, and most are dependent on subsistence agriculture. The Sierra region suffers from slower economic growth due to "lack of economic opportunities and strategic productive alliances, socioeconomic insecurity and lack of productive assets and services, weak rural institutions, absence of strategic planning to maximize development impact and poor overall coordination of the existing development strategy" (PAD, p. 2). At project appraisal, objectives were in line with the Government priorities for the Sierra region which were emphasized in the Sierra Exportadora program (2006). The program emphasized expanding the production of small fanners, helping them identify and exploit commercial opportunities, and developing products that could compete in national and international markets. Also, objectives were in line with the Peruvian Peace and Development Plan (2003) for the Sierra region which aimed to increase production and productivity, particularly in agriculture and agri-business, among others. Objectives were also in line with the Bank's Rural Development Strategy for the Peruvian Sierra (2002) which was prepared in collaboration with FAO and the Peruvian Government. The strategy called for promoting sustainable rural economic growth to increase incomes and reduce poverty, and to develop productive alliances along with the creation of economic corridors, productive infrastructure, and private investment for productive activities, among others. Finally, objectives were in line with the Bank's Country Partnership Strategy (CPS, 2007-11) for Peru which called for widening the base of growth and supported sustainable growth and promoted a new social contract (nutrition aspects), strengthening public sector management, and managing decentralization. At project completion, objectives continued to be in line with the Government's National Plan (2016-2021) which aimed to close the infrastructure gap; and promote private investments and accelerate public investments; among others. The PDO was revised during the 2013 restructuring to remove support for the coordination of a regional development strategy-- a thematic area that was no longer needed at the project level given a number of government support programs that promoted decentralization. Hence, this change in the PDO allowed the project objectives to remain relevant for the government. Objectives were also in line with the Bank's Country Partnership Framework (CPF) 2017-2021 for Peru. The CPF featured rural development as a cross cutting theme among its three pillars which aimed to support productivity and growth; services for citizens across the territory; and Natural resource and climate change risk management (CPF, p. 1). The CPF's fourth objective called for enhancing the environment for sustainable private sector investments Page 4 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) an area which the Bank would support interventions to help small farmers and boost rural development (CPF, p. 39). The original and revised PDO statements were both broad and lacked an explicit connection to higher level objectives most notable poverty alleviation. The relevance of objectives at completion decreased slightly given the evolution in Government priorities for the Sierra region compared to appraisal time--almost a nine years difference in time. Based on the aforementioned information relevance of objectives is rated Substantial. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective Original PDO: to assist the Borrower in improving the assets and economic conditions of rural families in selected areas of the Borrower’s Apurímac, Ayacucho, Huancavelica, Junin, Huánuco, and Pasco regions, and strengthen government capacity to implement an integrated Sierra development strategy. As stated the PDO includes three objectives that will be evaluated separately as follows: Objective 1: Improve the assets of rural families in selected areas of the Borrower’s Apurímac, Ayacucho, Huancavelica, Junin, Huánuco, and Pasco regions; Objective 2: Improve economic conditions of rural families in selected areas of the Borrower’s Apurímac, Ayacucho, Huancavelica, Junin, Huánuco, and Pasco regions; and Objective 3: Strengthen government capacity to implement an integrated Sierra development strategy. Rationale 1. Improve the assets of rural families in selected areas of the Borrower’s Apurímac, Ayacucho, Huancavelica, Junin, Huánuco, and Pasco regions. Rated: Substantial. Outputs (pre-restructuring, 2008-2013) The following outputs were reported by the ICR (Annex 1, p. 41): • 17,303 families benefitted from the rural business sub-projects (target: 18,600). • 21.3% increased profit of beneficiaries of rural businesses (target: 25%). • Two groups of sub-projects were supported by the project, first were food security and resource Page 5 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) management sub-projects which included: inputs for staple crops, minor animals, poultry houses, sheds, water management improvements, small-scale water reservoirs and irrigation systems, and soil management techniques such as terracing. Second were production and productivity sub-projects which included: improving animal production and genetics through the introduction of improved genotypes of beef cattle, goats, and alpacas, boosting production of guinea pigs, and establishing agro-forestry systems. • 876 business plans were implemented (target: 620): 354 for animal production, 125 for agriculture, 219 for agro-industry, 109 for handcrafts, and 69 for other domains. Outcome The project financed 876 Territorial Development Plans (TDPs) in the first phase. A total of 17,303 households benefited from these investments which was lower than the target of 18,600. This outcome was assessed through measuring the increase of the principal productive assets of 75% of the beneficiaries, with a target of 30% at completion. This was done by comparing average values of assets (in US dollars) per household between 2011 and 2013, for land and house, livestock, household goods, vehicles, and other productive assets (such as tools and equipment), of a random sample of households. By 2013 the value of increase reported was 29.8%. However, this outcome indicator did not differentiate between assets from Business Plans (BPs) and Territorial Development Plans (TDPs). After restructuring the indicator was revised to reflect assets under the TDPs only which included productive assets such as infrastructure, equipment, machinery, livestock, plantations, and small vehicles. The increase in the value of productive assets using the new criteria exceeded the 30% target and was estimated to be 65.8% (ICR, p. 13 para 21). Based on the afore mentioned information, this outcome is rated substantial despite missing some output level targets. Rating Substantial PHREVDELTBL PHINNERREVISEDTBL Objective 1 Revision 1 Revised Objective Improve the assets of rural families in selected areas of the Borrower’s Apurímac, Ayacucho, Huancavelica, Junín, Huánuco and Pasco regions. Rated: Substantial. Revised Rationale Outputs (post restructuring, 2013-2017) The following outputs were reported by the ICR (Annex 1, p. 41): • 30,886 direct beneficiaries of Rural Businesses (revised target: 35,200). Page 6 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) • Two groups of sub-projects were supported by the project, first were food security and resource management sub-projects which included: inputs for staple crops, minor animals, poultry houses, sheds, water management improvements, small-scale water reservoirs and irrigation systems, and soil management techniques such as terracing. Second were production and productivity sub-projects which included: improving animal production and genetics through the introduction of improved genotypes of beef cattle, goats, and alpacas, boosting production of guinea pigs, and establishing agro-forestry systems. • 1,548 business plans were implemented (revised target: 1,174) with 672 additional plans implemented in the AF phase including: 293 for animal production, 165 for agriculture, 101 for agro-industry, 32 for handcrafts, and 81 for other domains. • 123 Rural business plans originally supported under the original loan received additional support under the AF (revised target: 140). • 99% of rural business sub-projects accomplished their disbursement and investment plans (revised target: 60%). Outcome The project financed 1,437 TDPs in total. A total of 56,837 households benefited from these investments exceeding the revised target of 50,000. By project completion, there was a 35% increase in net volume of sales (revised target: 20%) of families participating in Rural Businesses (based on sales of all 795 BPs financed from 2013 to 2017, including 123 BPs supported in both project phases); and based on a survey covering the total of 1,358 subprojects financed during the second phase (2013-2017) the ICR (Annex 1 p. 36) reported that 90% of sub-projects achieved over 70% of their objectives (revised target: 80% of sub-projects). Finally, a good sign of sustainability was that 82% of rural business were operational two years after completion (revised target: 70%, ICR, p. 41, Annex 1). Based on the afore mentioned information, outcome is rated substantial. Revised Rating Substantial PHEFFICACYTBL Objective 2 Objective Improve economic conditions of rural families in selected areas of the Borrower’s Apurímac, Ayacucho, Huancavelica, Junin, Huánuco, and Pasco regions. Rationale Outputs (prior to restructuring, 2008-2013) The following outputs were reported by the ICR (Annex 1, p. 43): •35,277 families (mostly indigenous subsistence farmers) benefited (target: 35,000). •864 community development sub-projects were implemented (target: 875) including: 98 for food security and 766 for production and productivity. Page 7 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) Outcomes The project supported a total of 753 BPs in the first phase (prior to AF). BPs had a commercial focus, mainly supporting animal production, agricultural production, basic agro-industry (or agro-processing), and other initiatives such as handicrafts and ecotourism. During the first phase sub-projects aiming to improve animal production received 40% of the total BPs financing. This outcome was assessed by measuring the increase in the net value of production of participating rural families, with a target of 20% at project completion. A random sample of 608 households in 200 sub-projects, 100 for TDPs and 100 for BPs was used to estimate the increase in the net value of production by comparing the average net value of production per family in 2011 and 2013, in 2011 US dollars. By 2013, an increase of 68.54% was reported, exceeding the 20% original target. Also, the same sample was used to estimate the increase in sales volume. The results showed that the average sales volume per household was 0.88 ton in 2011 and increased to 1.2 tons in 2013, representing an increase of 37.2% (ICR, p. 14, para 28). Improvement in economic conditions was also confirmed by calculating the average annual sales value per household (in 2011 US dollars) which increased by 53% percent, from US$ 2,112 in 2011 to US$ 3,225 in 2013 (ICR, p. 14, para 29). Further confirmation was through a propensity-score-matching analysis which estimated that the average annual sales value per household was US$625 higher in the treatment group than in the control group. The analysis was based on 157 households in the treatment group, compared to 566 households in the control group (made up of non-beneficiary households, whose proposals had been selected to participate in the second project phase, but who had not yet received any benefit); however, confidence level was below 90%. The ICR (p. 15, para 29) reported that most of the gains in sales value could be attributed to increased volume since price fluctuations did not appear to skew the results. Based on the afore mentioned information, outcome is rated substantial. Rating Substantial PHREVDELTBL PHINNERREVISEDTBL Objective 2 Revision 1 Revised Objective Improve economic conditions of rural families in selected areas of the Borrower’s Apurímac, Ayacucho, Huancavelica, Junin, Huánuco, and Pasco regions. Rated: Substantial. Revised Rationale Outputs (post restructuring, 2013-2017) The following outputs were reported by the ICR (Annex 1, p. 43): • 56,837 direct project beneficiaries (revised target: 50,000). • 36% of the Territorial Development Community Plans were led by women’s organizations (revised target: 20%). • 1,427 community development sub-projects implemented (revised target: 1,250) with 563 additional sub- projects carried out in the AF phase. These included: 374 for food security and 189 for production and productivity. Page 8 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) •The total number of BPs supported reached 1,548 against a target of 1,174, but the number of project beneficiaries was 30,886 households lower than expected target of 35,200. The reason might be the smaller size of producer groups presenting proposals during the AF phase (ICR, p. 17. para 32). Outcomes The project supported a total of 795 BPs in the AF phase. The BPs enabled organized groups of producers to upgrade their production and commercialization schemes, effectively boosting production and sales. During the AF phase sub-projects aiming to improve animal production received 43% of the total BPs financing. The results showed that net sales volume increased by 35% in 2017 compared to 2013, exceeding the target of 20%. The increase was calculated by converting the units sold in each type of sub- project into kilograms using standard coefficients. The sustainability of sub-projects was encouraging as the results showed that 82% of rural businesses were still operational after two years, exceeding the target of 70%. Further evidence on improvement in economic conditions was reflected in an improvement in the aggregated value of sales of BPs by 27%, from US$ 16.6 million to US$ 21.0 million (in 2013 US dollars). Finally, an end of project survey showed that 90% of beneficiaries confirmed that the project helped them to improve their household income (ICR, p. 17, para 34). Based on the afore mentioned information, outcome is rated substantial. Revised Rating Substantial PHEFFICACYTBL Objective 3 Objective Prior to Restructuring: Strengthen government capacity to implement an integrated Sierra development strategy. Rated Negligible. Rationale Outputs (prior to restructuring 2008-2013) • The Sierra Development Unit Ministers established and functioning (target: unit established). Outcome The project failed to achieve the intended outcome because it was very ambitious relative to the scope of the project, and required the coordination of a number of institutions that were beyond its mandate. While development investments in the Sierra region by the national government increased by 50% (target: 30%), however, the increment could not be attributed to the project (ICR, p. 18, para 38). The coordinating unit under component 3 failed to achieve any of its intermediate outcome targets; for example, the government’s rural development strategy for the Sierra was not updated; sectoral coordination committees were not established; and the integrated Sierra Development Program was not approved. In 2012, the Government Page 9 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) transferred the coordinating unit to the Ministry of Development and Social Inclusion and consequently the related sub-component and indicators were removed from the project as part of the changes introduced at the Level 1 restructuring in 2013. This objective was dropped at the restructuring. Rating Negligible PHREVDELTBL PHOVRLEFFRATTBL Rationale The two essential objectives were both rated Substantial, while the third objective was dropped after restructuring as it did not prove necessary to achieving the project objective. Overall Efficacy Rating Substantial PHREVISEDTBL 5. Efficiency Economic and Financial Efficiency ex ante • There was no detailed ex ante cost benefit analysis because the specific activities financed were dependent on the sub-project proposals made by community and producer groups during implementation. Preliminary estimates showed that if there were 53,600 beneficiary families, aggregate economic NPV and IRR would be US$11.7 million and 29%, while aggregate financial NPV and IRR would be US$4.0 million and 17%. • Some potential initiatives were analyzed, based on primary and secondary information. Benefits included increased net income and employment. Investment and recurrent costs included physical assets (facilities and equipment), working capital resources necessary to cover one production cycle) and services (training, technical assistance), contracted labor and/or family labor and without-project income likely to be forgone. • Aggregated project economic and financial NPV and IRR were estimated considering other project costs, namely: institutional strengthening, promotion and pre-investment costs. Benefits and costs relative to project management and the Sierra Development Coordination subcomponent were not included. Activities analyzed included: irrigated crops, livestock production, guinea pig production, aquaculture, community tourism, and lamb feedlot. ex post Page 10 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) • The ICR included a detailed ex post economic and financial analysis (EFA). The ex post aggregate EIRR was estimated at 39% which exceeded the ex ante EIRR estimate of 29%. • First project phase (2008-2013). The EFA relied on data from a sample of 36 sub-projects (of both components) financed between 2007 and 2013. Using a discount rate of 12%, the financial internal rate of return (FIRR) was estimated to be 44% and the estimated financial net present value (FNPV) was US$ 144.4 million. The economic internal rate of return (EIRR) was estimated to be 72% and the estimated economic net present value (ENPV) was US$ 246.6 million. Sensitivity analysis showed that the FIRR dropped to 12% under adverse scenarios including: an average decrease of 24% in benefits, an average increase of 160% in operational costs, and an average increase of 225% in labor costs. • Efficiency for the first phase is rated substantial. • AF phase (2013-2017). The EFA relied on data from a statistically representative random sample of sub- projects (40 Business Plans and 22 Territorial Development Plans) implemented from 2013 to 2017. The cost-benefit assessment for BPs estimated the FIRR at 13% and the incremental financial NPV was estimated at US$ 4.8 million, considering direct and indirect project costs. The EIRR was estimated at 32% and the incremental economic NPV was estimated at US$ 19.9 million, considering direct and indirect project costs. Sensitivity analysis showed that returns to BPs were highly sensitive to changes in gross revenues and therefore to changes in production and prices. A reduction of only 2% in income levels, for example, reduces the IRR to 8%, and an increase in labor costs of 3% renders the return on investment barely acceptable. • The cost-effectiveness analysis for Territorial Development Plans revealed a lower than expected cost- effectiveness ratio of 1.11 versus 1.59, using market prices, and 0.87 versus 1.25, with social prices. The incremental cost of the project at present value was estimated at US$ 23.9 million using market prices and US$18.8 million using social prices. At the cost of US$876 per beneficiary household, social costs were 30% lower than expected when the AF was prepared. The ICR (p. 20, para 46) attributed these results to the larger number of sub-projects that were implemented and a larger number of beneficiaries reached with the intervention. • Efficiency for the second phase is rated substantial. • Aggregated analysis for both project phases. The estimated FIRR and EIRR were 18% and 39%, respectively; thus the total investment in Business Plans yielded returns above a social discount rate of 8%. The incremental FNPV was estimated at US$15.2 million and the ENPV at US$45.3 million. The analysis relied on the 62 sub-projects analyzed under the AF phase as they were deemed representative of the universe of sub-projects implemented overall. Two assumptions were considered: in each sub-project category, the types of interventions and investments remained constant during the life of the project; and the socioeconomic profile of beneficiaries in both project phases was similar. Administrative and Institutional Efficiency The project closed four years and six months later than the expected date. This delay was to accommodate the Page 11 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) expansion of project activities and allow time to disburse additional funds and implement activities. The first phase of the project experienced delays in implementing numerous small, fragmented sub-projects. The first phase also experienced delays in hiring the environmental and social specialists to oversee mitigation and compliance issues. The project also experienced implementation delays due to complex administrative requirements and procedures. Most notable was that after the approval of the first loan, it took the Government 13 months to sign the Financing Agreement, and the project was effectively initiated more than two years after being approved. Also, during the first phase, the National Budget System categorized the project as a program and requested that every single sub-project be registered. This process took considerable time and was not anticipated at he design stage. Overall, efficiency is rated substantial as the ex post ERR was 39% which exceeded the ex ante estimate of 29%. However, there were some weaknesses on the administrative and institutional side. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  29.00 Not Applicable 0 ICR Estimate  39.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Pre-restructuring Relevance of objectives was rated substantial. Overall efficacy was rated substantial. The evidence points to the success of the project in improving the assets and economic conditions of rural families in selected areas. However, the project failed to achieve its target on strengthening the government capacity to implement an integrated Sierra development strategy. Efficiency was rated substantial. With substantial rating for all elements (relevance of objectives, overall efficacy and efficiency), outcome is rated satisfactory. Post Restructuring Relevance of objectives was rated substantial. Overall efficacy was rated substantial. The evidence points to the success of the project in improving the assets and economic conditions of rural families in selected areas. Page 12 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) Efficiency was rated substantial. With substantial rating for all elements (relevance of objectives, overall efficacy and efficiency), outcome is rated satisfactory. Weighted outcome Pre-restructuring: amount disbursed at restructuring/total amount disbursed X 5 (on a rating from 1 to 6 a satisfactory outcome is assigned 5) = 18.4/39.15 X 5 = 2.34 Post-restructuring: amount disbursed after restructuring/total amount disbursed X 5 (on a rating from 1 to 6 a satisfactory outcome is assigned 5) = 20.75/39.15 X 5 = 2.65 The overall weighted outcome is 2.34 + 2.65 = 4.99 which rounds up to 5 corresponding a satisfactory rating. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The ICR (page 31 &32 para 96) highlighted two important risks that could negatively impact the gains achieved by the project. First, the intervention at the sub-project level was short given that it covered only one production cycle. There is a risk that the sub-projects would underperform in the absence of further assistance post completion especially with the intense pace of implementation during both phases (and particularly in the AF phase). Second, producers in the Sierra region mostly sell their product to local markets which are volatile and uncertain. Only a limited number of sub-projects developed links to established supply chains that included formal buyers such as agro-processors. The main concern is that if target markets do not offer attractive prices that make up for opportunity costs, producers might suspend production seasonally or permanently in favor of other income-earning activities. 8. Assessment of Bank Performance a. Quality-at-Entry • The project preparation was exceptionally long with close to four years from 2003 to 2007. This involved ten Bank missions to Peru to inform official and authorities about the project. However, the main reason for the long preparation time was the "decision by the Government to fully complete execution of the trust fund granted by the Government of Japan before starting the new operation" (ICR p. 30, para 89). Page 13 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) • The project design combined elements of CDD and productive partnerships which represented an important and novel approach to policy and operations in Peru. With three levels of intervention (regional, sub-project, and household levels) design ensured a consistent strategic direction with emphasis on coordination of government investments across agencies; collective action by producers for production and marketing initiatives; and enhancing on-farm production and livelihoods. • However, design had two notable shortcomings. First, it underestimated the role of coordinating the Sierra development, which proved to be ambitious and beyond the project's mandate. Second, design underestimated the complexity of the legal and institutional framework which hindered implementation and caused significant delays. • Nine risks were identified at appraisal, with an overall moderate rating. While the risk of inadequate decentralization was identified at appraisal, the complexity of the legal and institutional framework was not anticipated. The ICR did not comment on the risks that materialized during implementation and the effectiveness of the suggested mitigation measures. • M&E design had some weaknesses the most notable being poorly linked indicators to the PDO and vague definition of some elements of the PDO (see section 9a for more details). Overall, Quality at Entry suffered from minor shortcomings relating to project design, risk, and M&E design. Therefore, QAE is rated moderately satisfactory. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The project lasted about nine years from effectiveness to completion. The ICR did not report on the number of missions conducted by the Bank throughout the implementation period. Nonetheless, the ICR (p. 30, para 92) reported that "the task team continuously assisted the project teams to improve implementation and track progress toward the PDO." The Bank worked closely with the Government to address a ten month delay after effectiveness. Quality of supervision benefitted from the presence of permanent Bank staff in the Country office. The project's M&E system benefited from the expertise of FAO M&E specialists who helped improve field data collection and management. The project also benefitted from the Bank's support to streamline procedures in the operations manual. The project's fiduciary management and safeguard compliance benefitted for hands-on training provided by Bank specialists. The preparation and approval of the AF was timely-less than eight months. The Bank worked with the Government to address delays in executing the AF through putting an emergency plan. This included putting an interim team at the Project Implementation Unit at the Ministry of Agriculture to fill a leadership vacuum and facilitate carrying out critical decision such as hiring consultancies. The Bank also supported the Government requests to extend the closing date of the project on three separate occasions. Overall, supervision provided effective support to the project and worked actively with the Government to address delays at different stages. Therefore, Quality of Supervision is rated Satisfactory. Page 14 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The PAD did not include an explicit theory of change. Nonetheless, the ICR (annex 7) outlined a clear theory of change that reflected the three levels of intervention (household, sub-projects and regional levels). The outline provided clear connections between project inputs, outputs and expected outcomes for the household and sub-projects level. However, this was less clear for the regional level. Design called for a Monitoring and Evaluation (M&E) unit to be established in the Southern Sierra Natural Resources Management Project central office. It would include a lead specialist and two assistants who would work closely with the component coordinators and regional offices to design and implement an effective M&E system. The PDO was to be assessed through four key outcome indicators. Two of these (Net value of production of participating rural families increased by 20%, and the value of project beneficiaries' principal productive assets increased by 30%) were directly linked to the PDO. However, the term assets was not clearly defined which made it difficult to measure the indicator. The "increased net value of production and increased value of assets" were expected to be the results of implementing both TDPs and BPs. These two activities were designed to benefit two different target groups through two different mechanisms (CDD and productive partnerships). Therefore, the two PDO indicators could have benefitted from a better definition that clearly specified which indicator referred to which type of intervention and key outputs. On the other hand, the other two key outcome indicators (the Sierra's economic production as a share of national GDP has increased- no target assigned, and the National Government development investments in the Sierra increased by 30%) were loosely linked to the PDO and could be impacted by different factors beyond the project control. The Results Framework (RF) also include seventeen intermediate outcome indicators to assess the different activities under the three components of the project. Most of these were measurable, however, baselines were recorded as zero for 15 out of 17 intermediate outcome indicators. b. M&E Implementation According to the ICR (p. 27, para 80) data was systematically collected, recorded and analyzed. The project had two management information systems for each of the project's two phases. For the first phase, a consulting firm was hired to carry out three surveys: a baseline (2009), a midterm review (2011), and a final evaluation (2013). However, the first survey had methodological shortcomings and was discarded by the Page 15 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) project team. The AF phase included a new online management information system that enabled performance monitoring by recording data on the results indicators. Project data were collected twice: before the intervention and at sub-project closing. Field data were used by consultants to validate estimates included in sub-project proposals and to make before- and after-project comparisons. Also, standardized forms were used and interviews were conducted in random samples of at least three beneficiary households. Information was also collected through workshops with community or producer organizations. The ICR (p. 28, para 82) noted that "the Bank team accompanied the data collection process and checked the accuracy and consistency of the database." In 2016, an additional survey was carried out and a final evaluation (the Propensity-Score-Matching impact evaluation for BPs supported during the first project phase), and a project systematization survey that focused on case studies were both carried out in 2017. Changes to M&E design after the Level I Restructuring of 2013. The restructured project explicitly linked each PDO indicator either to TDPs or BPs. To assess successful implementation of sub-projects, an intermediate indicator was elevated to the PDO level, and expanded to cover not only BPs, but also TDPs. In addition, the Results Framework was revised to improve some indicators and add disaggregation by gender and age. The indicator on net value of production was reformulated to capture the increase in sales based on volume not value to eliminate the influence of price variations. These changes were logical and rectified earlier weaknesses in the M&E design. c. M&E Utilization The project's M&E information was used to inform decision-making and make the case for additional financing at the end of the first phase. M&E utilization could have benefitted from demonstrating significant effects beyond the anticipated results, for example, to track complementary data on spillover effects and unintended outcomes. Finally, the ICR in many cases relied on M&E data from the project to demonstrate the achievement of the project outcomes. Overall, M&E suffered from design shortcomings which were mostly rectified during implementation. M&E implementation was generally sound and M&E utilization was good. Therefore, M&E quality is rated substantial despite design shortcomings. M&E Quality Rating Substantial 10. Other Issues a. Safeguards Page 16 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) The project was classified as a Category B. The following safeguard policies were triggered: Environmental Assessment (OP/BP/GP 4.01), Natural Habitats (OP/BP 4.04), Pest Management (OP 4.09), Physical Cultural Resources (OP 4.11), Indigenous Peoples (OP 4.10); and Forests (OP/BP 4.36). The project was expected to have potential site -specific adverse environmental impacts on human populations or environmentally important areas however, these would be reversible, and could be readily mitigated. Also, the project could involve natural habitats, but was not expected to lead to their loss, degradation or conversion. An environmental Mitigation Framework (EMF) was prepared and reviewed by the Bank. The EMF included a Pest Management Plan and guidelines for alternative designs and mitigation measures. The Level I Restructuring of 2013 did not trigger any additional safeguard policies. After the approval of the AF loan, the Environmental and Social Management Framework (ESMF) was revised several times to enable actions and procedures to reflect changes in the project. The ICR provided no coverage on impacts and mitigation activities. However, the ICR (p. 29, para 86) reported that "compliance with safeguards is rated satisfactory for most of the life of the project ." The was no implicit or explicit statement of compliance to any of the triggered safeguards policies. b. Fiduciary Compliance Financial Management. The Interim Financial Reports (IFRs) suffered from quality issues; and new FM personnel had trouble consolidating information for the IFRs in a manner acceptable to the Bank. There were delays in contracting an external auditor-which were flagged by the Bank. The ICR did not report on the status of external financial audits. Procurement. During the first phase of implementation, cumbersome procurement procedures for approving each subproject and recording it within the SNIP contributed to implementation delays. These problems were resolved in the AF phase. There were occasional delays in updating the Procurement Plan. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Page 17 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) Outcome Satisfactory Satisfactory --- There were issues with quality Bank Performance Satisfactory Moderately Satisfactory at entry. Quality of M&E Substantial Substantial --- Quality of ICR Substantial --- 12. Lessons The ICR included eleven lessons. The following three lessons are emphasized with some adaptation of language: • Building productive partnerships is one of the mechanisms that decision makers should bear in mind when it comes to boosting rural employment. The project's experience demonstrated that productive alliances can create jobs. The EFA revealed that project-supported Business Plans created over 1,000 new jobs. By stimulating production and sales, commercial partnerships sparked job creation at the farm level. The producers who benefited entered a virtuous circle of investment and reinvestment, in which new employment was a natural consequence of labor-intensive cropping and animal systems. Productive alliance projects should include mechanisms to stimulate partnerships with dynamic supply chains. Complementary market strategies that can contribute to economies of scale as well as differentiation (for example, via productive clusters or geographical branding) are critical to seize opportunities in better markets. Furthermore, it is important to anchor farm-level investments in more comprehensive value chain strategies. • M&E design should include well defined results indicators to ensure accurate assessment of project performance. Vague result indicators can be misleading. Originally, non-productive assets such as a house and household goods were covered in the first measurements for the assets indicator. Although the Project Appraisal Document referred to productive assets, they were not clearly defined and left open to interpretation. Unclear indicators can undermine project performance; in fact, if the project had continued reporting non-productive assets, it would have been difficult to assess the achievement of the outcome in the spirit of the PDO. A good practice is to use standard forms to develop detailed descriptions of each indicator that define key words, methods, formulas to monetize indicators. • Projects need to have realistic PDO outcomes to increase their chance of achievement by completion. The third PDO Outcome (strengthening capacity for Sierra development) was defined too vaguely and the indicators were beyond the scope of the project. During preparation, the project was under some pressure to play a pivotal role in coordinating rural development in the region. Although projects must respond to political priorities, yielding to pressure from decision makers and including unrealistic outcomes and components can create a serious problem in the end. Preparation teams should carefully assess the Page 18 of 19 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review PE Sierra Rural Development Project (P079165) feasibility of the proposed investments and outcomes, particularly for interventions that combine productive elements with efforts at the regional level. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is well written. It provided a good account of project activities and candidly reported on most shortcomings. Discussion of outcomes was logical and relied on the achievements of the project. The ICR also included a number of useful lessons, however, some could have benefitted from better formulation. The ICR could have improved on the following points: • Report of Safeguards with explicit statements whether the project complied with Bank's safeguard policies or otherwise. • Report on the status of external audit reports under Financial Management. • Provide more detail on the risks identified at appraisal and whether any of these materialized; and how well did the suggested mitigation measures work. a. Quality of ICR Rating Substantial Page 19 of 19