Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) Report Number : ICRR0021504 1. Project Data Project ID Project Name P106261 MX Sustainable Rural Development Country Practice Area(Lead) Additional Financing Mexico Agriculture P130623 L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IBRD-76520,IBRD-82160 31-Dec-2013 88,315,765.87 Bank Approval Date Closing Date (Actual) 24-Feb-2009 29-Jun-2018 IBRD/IDA (USD) Grants (USD) Original Commitment 50,000,000.00 0.00 Revised Commitment 86,800,513.19 0.00 Actual 79,051,759.27 0.00 Prepared by Reviewed by ICR Review Coordinator Group Hassan Wally J. W. van Holst Christopher David Nelson IEGSD (Unit 4) Pellekaan PHPROJECTDATATBL Project ID Project Name MX GEF Sustain. Rural Dev ( P108766 P108766 ) L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-93134 9,264,006.60 Page 1 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) Bank Approval Date Closing Date (Actual) 24-Feb-2009 IBRD/IDA (USD) Grants (USD) Original Commitment 0.00 10,500,000.00 Revised Commitment 0.00 9,264,006.60 Actual 0.00 9,264,006.60 2. Project Objectives and Components a. Objectives The Project Development Objective (PDO) stated in the Project Appraisal Document (PAD, p.6, para 17) and the PDO in the Loan Agreement (LA, p. 5) were identical and aimed to: "promote the adoption of environmentally sustainable technologies in agri-businesses." According to the PAD (p. 6, para 18) the Global Environment Objective (GEO) was to: "contribute to the goals of the National Strategy on Climate Change by reducing GHG (CO2) emission through the adoption of emission reduction technologies and the support to the implementation of the (Mexico) President’s Special Program for Climate Change (PECC), with special reference to the improved environmental sustainability of small and medium-scale agri-business." b. Were the project objectives/key associated outcome targets revised during implementation? No PHEVALUNDERTAKENLBL c. Will a split evaluation be undertaken? No d. Components The PDO was supported by four components. 1. Investments in environmentally sustainable technologies in agri-business (appraisal cost: US$151.04 million of which US$4.79 million would be funded by the GEF, actual cost: US$224.08). This component would promote investments in environmentally sustainable technologies in agri- businesses operating at the various stages of the production chain of agricultural products. Though Page 2 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) the project would primarily focus on existing agri-businesses, support to new agri-businesses could be considered on a case-by-case basis, mainly for the introduction of solar thermal systems. The GEF support would be primarily granted to provide initial capital investments and to remove technological barriers to improve energy efficiency. GEF funds would be provided as a matching grant to eligible agribusinesses for the initial capital investment to acquire generators which utilize biomass to generate energy, thereby reducing overall consumption of energy from conventional sources. 2. Investment and Production Support Services (appraisal cost: US$10.91 million of which US$1.73 million would be funded by the GEF, actual cost: US$0.03 million). In order to ensure quality at entry of investment sub-project proposals, this component would partially reimburse beneficiaries for the costs associated with business plan preparation for sub-projects, including the energy diagnostic when necessary. It would also provide beneficiaries with technical assistance for implementation of their proposed business plan, as well as training to integrate technologies promoted through the project in their farms and/or agri-businesses. 3. Institutional Strengthening (appraisal cost: US$3.59 million of which US$2.93 million would be funded by the GEF, actual cost: US$0.24 million). Activities to be financed by the project under this component would include assistance for policy development to address issues related to climate change and the environmental impact of sub-projects, in particular, institutional strengthening of areas within the Ministry of Agriculture, Livestock Production, Rural Development, Fisheries and Food (SAGARPA) that would address the targets outlined within the National Strategy on Climate Change and the President’s Special Program for Climate Change (PECC). This would include a series of inter-ministerial workshops to promote collaboration and knowledge sharing on climate change activities related to the project. Also, Fideicomiso de Riesgo Cornpartido (FIRCO-an autonomous unit within SAGRPA) would receive support to improve the capabilities of its regional offices with regards to the promotion and implementation of the proposed project and the coordination and execution of Programmatic CDM activities. 4. Project Management, Monitoring and Evaluation (appraisal cost: US$2.45 million of which US$0.95 million would be funded by the GEF, actual cost: US$0.27 million). This component would finance, in part, project management activities undertaken by the Project Implementation Team that within FIRCO will be responsible for project execution. The project would also finance the development and operation of a Project Monitoring and Evaluation System. In a further communication, the Bank project team explained that the actual amounts for components 2, 3 and 4 were lower than the appraisal amount due to the depreciation of the Mexican currency against the US dollar. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost. The project total cost was expected to be US$168.35 million. The actual cost reported by the ICR (p. 2) was US$358.21 million. The difference was due to an additional financing that the project received, and a higher contribution from the borrower and beneficiaries (see below for more details). Financing. The project was funded through an IBRD Loan of US$50.0 million, a GEF Grant of US$10.5 million. The actual amounts disbursed by project completion were US$46.80 and US$1.36 million for the IBRD loan and the GEF Grant, respectively (ICR, Annex 3). On November 22, 2012, the project also Page 3 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) received an additional financing totaling US$50.00 million and the actual amount disbursed according to the ICR (p. 2) was US$32.25 million, and US$10.0 million were cancelled. The total financing from IBRD loans and GEF Grant was US$110.5 million, the actual amount disbursed was US$88.32 million (ICR, p. 2). Borrower Contribution. The Government was expected to contribute US$28.0 million counterpart funds, and project beneficiaries were expected to contribute US$80.00 million. Actual amounts reported by the ICR (Annex 3) were US$33.12 million and US$143.34 million for the Government and the beneficiaries, respectively. Dates and Restructuring. • The project was approved on February 24, 2009. It became effective on January 29, 2010 for the IBRD loan and on February 8, 2010 for the GEF Grant. The Mid-Term Review was carried out on June 11, 2012- six months earlier than expected as the PAD (p. 13, para 39) stated that the MTR was expected to be carried by the end of third year of implementation. • The project's original closing date was on December 31, 2013. However, the actual closing date was 4.5 years later on June 29, 2018. • The project was restructured four times, all of which were Level 2 restructurings. The first was on November 20, 2012, when the amount disbursed was US$25.31 million, in order to approve additional financing to the project in the amount of US$50.00 million, and extend the closing date of the GEF portion of this blended project from December 31, 2013 to December 31, 2016), coinciding with the proposed three-year implementation period of the additional loan, revise the final targets of the result indicators to adequately reflect the impacts of scaling-up the project, and reallocate funds from disbursement Category 1 to Category 2 of the original Loan Agreement, following a request by the Government of Mexico substantiated by the fact that funding for activities in Category 1 would be provided directly from the budget of the Ministry of Agriculture, Livestock Production, Rural Development, Fisheries, and Food (Project paper, p. 1, para 4). • The second restructuring was on April 1, 2015, when the amount disbursed was US$52.23 million, in order to amend the Additional Financing Loan Agreement to specify "Commercial Practices" as an acceptable procurement method for "goods, works and non-consulting services" and "consulting services. • The third restructuring was on December 1, 2016, when the amount disbursed was US$72.36 million, in order to extend the closing date of the Additional Financing and the GEF Grant by 18 months from December 31, 2016 to June 29, 2018. • The fourth restructuring was on March 5, 2018, when the amount disbursed was US$77.64 million, to cancel US$10.0 million of Additional Financing Loan funds. Page 4 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) 3. Relevance of Objectives Rationale According to Mexico’s National Climate Change Strategy (NCCS-2007), agriculture continues to be an important source of carbon emissions (7% of total emissions), primarily through land-use change (e.g. deforestation to cultivation), tillage, synthetic fertilizers, and anaerobic decomposition of organic materials. To partially address this challenge, the Government of Mexico has prioritized increasing the competitiveness and environmental sustainability of agriculture and agribusinesses, in the context of climate change mitigation, through the promotion of energy efficiency (including the use of renewable energy) and biomass practices. At project appraisal, objectives were in line with Government's vision to promote environmentally and economically sustainable rural development through: sustainable utilization of natural resources, extended use of energy efficiency practices, and development of renewable energy sources, among other things (PAD, p. 1). Objectives were also in line with the Mexican National Development Plan (NDP) 2007-2012 (with the overarching theme of Sustainable Human Development) specifically through the second and fourth pillars, “Economic competitiveness and generation of jobs” and “Environmental Sustainability” respectively. Also, objectives were in line with the World Bank's Country Partnership Strategy (CPS) for Mexico (FY2008- 2013), which aimed to promote a more sustainable and equitable Mexico by reducing economic and environmental problems of small and medium-sized producers, and a foster a more competitive Mexico by encouraging the implementation of value-added and energy efficient practices, as well as the use of renewable sources of energy. Finally, the GEO was in line with GEF's Climate Change Focal Areas, in particular with the following strategic programs under GEF-4: SP2 “Promoting Energy Efficiency in the Industrial Sector,” and SP4 “Promoting Sustainable Energy Production from Biomass.” The GEF financial support would promote global environmental benefits in addition to national and state benefits in Mexico. At project completion, objectives remained in line with the Mexican National Development Plan (2013-2018), which focused on national prosperity and productivity by investing in sustainable development and clean energy in the agriculture sector, and focusing on small and medium-sized businesses, among other things. Objectives also remain in line with the World Bank's Mexico Country Partnership Strategy (CPS, 2014-2019) specifically the CPS’ "Green Growth" goal which aimed to reduce growth’s footprint and use natural resources more optimally, and the CPS' pillar "Unleashing Productivity" by increasing private sector innovation and upgrading infrastructure to decrease costs and promote competitiveness. Finally, the GEO remained in line with Mexico’s current National Climate Change Strategy that aims to reduce energy intensity through efficiency and responsible consumption, and seeks long-term growth through, adoption of clean technologies, among other things. The statement of objectives was simple and clear; however, it lacked a connection to the higher level objectives that the project sought to achieve such as reducing economic and environmental problems of small and medium-sized agricultural producers, and supporting a competitive Mexico by encouraging the implementation of value-added and energy efficient practices. The GEO, though relevant, was too broad and therefore lacked specificity as it referenced external sources such as "the National Strategy on Climate Change" and "President’s Special Program for Climate Change" without providing a clear connection to the project's GEO. Page 5 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) Conclusion. Based on the above mentioned information, relevance of objectives is rated substantial. Rating Substantial 4. Achievement of Objectives (Efficacy) PHEFFICACYTBL Objective 1 Objective PDO: promote the adoption of environmentally sustainable technologies in agri-businesses Rationale Theory of Change for the PDO. The theory of change is the relationship between activities, outputs from those activities and the final outcome (i.e. “promote the adoption of environmentally sustainable technologies”) in agri-businesses. In this case change is facilitated by a number of critical elements such as availability of technologies, willingness and ability of agribusinesses to adopt these technologies, climate- related Government-agencies working together effectively, and a sustainable and replicable approach. Activities and expected outputs. The project was designed to promote investments in environmentally sustainable technologies for small and medium agribusinesses. This was expected to encourage beneficiary agribusinesses to implement and operate investments efficiently. The project was also aimed at supporting TA, training and piloting of innovative energy technologies, and as a result pilots would be demonstrated, validated and available to the project. The project could also provide institutional strengthening of SAGARPA to implement the National Strategy on Climate Change, and for FIRCO to manage the project. These activities were expected to support the establishment of the Climate Change unit in SAGARPA; and, increase inter-ministerial collaboration, and knowledge-sharing on climate change. Outcomes. The adoption of environmentally sustainable technologies by small and medium scale agribusinesses, agribusinesses demonstrate an increased sustainability through kilo watts hours (KWh) of energy saved, mega watt hours (MWh) of energy generated/used from biomass conversion, and tons of CO2 emissions avoided. Also, SAGARPA formulated and monitored the implementation of national climate change mitigation policies in the agriculture sector. Expected Higher Level Outcomes pertaining to the GEO. Mexico’s agriculture sector could become environmentally sustainable, competitive and equitable if the goals of Mexico’s National Climate Change Strategy and its Special Program for Climate Change (PECC), and those of the advanced Kyoto Protocol commitments are also supported. From the above mentioned information, the causal links in the theory of change are evident, and the suggested activities are adequate to achieve the PDO and GEO (see the later under objective 2). However, there is uncertainty regarding the assumption that the climate-related agencies established to address climate change can work together effectively. If they cannot, the achievement of the higher level outcomes could be undermined. Page 6 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) Outputs Investments in Environmentally Sustainable Technologies in Agribusinesses • Project investments covered 30 Mexican States (Target; 30). • Sub-projects by beneficiary agribusiness size: Micro (1,058), Small (789) Medium (300) and Large (67) • Sub-projects financed: 419 Bio-digesters were financed (110.6%), 738 Photovoltaic systems (394.7%),162 Solar heating systems 37.0%), 214 Motor-generators (125.1%), and 700 high efficiency pumps (106%). In a further communication with IEG, the project team explained that high efficiency pumps were electric pumps that substituted mostly diesel pumps. Investment in Production Support Services • 2,286 energy-efficient and/or renewable energy sub-projects were prepared (dropped as an indicator by AF but still monitored by FIRCO) (achievement rate 105.4%). • 4 pilot projects carried out to demonstrate and validate other technological innovations that could be proposed under the project (achievement rate 100%). • 4 studies were financed by the project (target achieved). Institutional Strengthening • 994 agribusiness employees/representatives participated in training events organized by the project (achievement rate 107.0%). • 386 of SAGARPA/FIRCO staff participated in training events organized by the project (achievement rate 137.9%). • 17 Inter-Ministerial workshops were organized to promote collaboration and knowledge sharing on climate change activities related to the project, carried out (achievement rate 283%). • The General Directorate for Addressing Climate Change in the Agro-Livestock Sector was established within SAGARPA to support and/or to formulate, implement and monitor Climate Change mitigation and adaptation policies and programs in the agriculture sector (target achieved). Outcome • The overriding objective of the project was to promote behavior change within agribusinesses by demonstrating renewable energy and energy efficient technologies, disseminating the results through nationwide multi-media campaigns, and providing financial incentives, TA and training to agribusinesses to foster adoption of technologies. Page 7 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) • According to the ICR (para 26), FIRCO started promoting the technologies and the benefits of adoption well before effectiveness. The environmental and economic benefits of clean, efficient energy in the sector, the incentives scheme, and the types of technologies available, were already widely disseminated. Promotion was through demonstration days on the production units of large and medium-sized beneficiary agri-businesses where candidates could inspect the technologies in operation and discuss their features with owners and FIRCO technicians; and FIRCO’s dissemination campaigns via print, audio-visual and direct encounters with the agribusiness sector, federal and state agencies, academia and research institutions, as well as through the project website. • These promotion efforts resulted in 1,842 agribusinesses (85% of target) adopting 2,286 technologies (105.4% of target). The project reached 200% of the original PAD target for numbers of beneficiary agribusinesses, at lower than expected cost. The number of sub-projects financed exceeded the number of beneficiaries, and agribusinesses were clearly willing to contribute up to 50% of sub-project cost for each of several technologies. Micro, small and medium agri-businesses represented respectively, 95% of sub-projects and 92.5% of total investment financing. Large agribusinesses made up 5% and 7.5% of numbers and financing, respectively. • Conclusion. The above mentioned information indicates that there was strong private sector involvement in environmentally sustainable technologies and that the adoption of these technologies by agri-businesses increased. In the absence of other incentives to adopt environmentally sustainable technologies it is plausible to assume that their increased adoption since this project became effective was attributable to the project. On this basis the efficacy of this objective is rated Substantial. Rating Substantial PHREVDELTBL PHEFFICACYTBL Objective 2 Objective GEO: to contribute to the goals of the National Strategy on Climate Change by reducing GHG (CO2) emission through the adoption of emission reduction technologies and the support to the implementation of the President’s Special Program for Climate Change (PECC), with special reference to the improved environmental sustainability of small and medium-scale agri-business. Rationale Theory of Change. Same as Objective 1. Outputs The outputs mentioned for Objective 1 pertain to the GEO as well. Page 8 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) Outcome Outcome was assessed through two outcome indicators: the first is the reduction in CO2 emissions (CO2e), and the second is whether SAGARPA was successful in formulating a climate change mitigation and adaptation policies and programs in the agricultural sector and monitoring their implementation. Carbon dioxide (CO2) emissions. According to the ICR (Annex 1), the project supported investments resulted in a reduction of CO2 emissions by a cumulative 6,021,967 tons from the year of installation and adding each subsequent year of reduction up to the end of the reporting period which exceeded both the revised target of 1,987,500 tons and the original target of 770,000 tons. This over achievement was due to two main reasons: First, there was wide spread adoption of bio-digesters relative to other types of sub- project investments (such as energy-efficient chillers and photovoltaic systems). Bio-digesters have a much larger emissions reduction capacity than other types of technologies promoted by the project. Second, CO2e were calculated based on using a more accurate, technology-specific CO2e calculation methodology that included real-time field monitoring and used reliable databases. This resulted in a larger than projected CO2e reductions than previously estimated. Also, until 2014, the project measured CO2e only for the year after the technology installation. From 2014 on, the project measured cumulative CO2e, from the year of installation and adding each subsequent year of reduction up to the end of the reporting period. SAGARPA Performance. By project completion, SAGARPA established a General Directorate to address Climate Change in the Agro-Livestock Sector. Each agricultural sub-sector represented in the Directorate had its own programs aligned to national Climate Change strategies. The Directorate was tasked with monitoring climate-related agricultural risk in the cattle and cropping industries; establishing cross-sector synergies on Climate Change; reporting green house gas (GHG) emission reductions resulting from SAGARPA's programs (as measured by FIRCO) to the Ministry of Environment and Natural Resources (SEMARNAT, the national lead on climate mitigation policies and programs) and the Ministry of Energy (SENER); representing the sector at national and international climate forums; and sharing oversight of key, climate-related programs under SAGARPA which demonstrate strong mitigation contributions. According to the ICR (Annex 1), SAGARPA created the Bioeconomia Program which injected 1.0 million Pesos into SENER's Energy Transition Trust Fund (FOTEASE) to finance/influence agricultural renewable energy (RE) initiatives on a multi-year basis, while fostering inter-agency RE coordination. About 50% of this allocation was operationalized through FIRCO from 2010 to 2018 for project RE activities. SAGARPA also influenced policy-making and execution, e.g., through FIRCO's capacity-building seminars for development banks including FIRA (Agricultural Shared Risk Trust) and the Mexican Rural Financial Development Agency, following which, both institutions markedly increased their investments in agricultural renewable energy programs and projects. While the afore-mentioned developments are a step in the right direction, the results framework (RF) in the PAD lacked specific indicators to assess the activities under SAGARPA's General Directorate. Conclusion. Overall, the evidence provided in the ICR points to the success of the project in positively contributing to achieving the GEO although the achievement of the GEO is not rated in this review. Page 9 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) Rating Not Rated/Not Applicable PHREVDELTBL PHOVRLEFFRATTBL Rationale Only the achievements of the Objective 1 (the IBRD project’s PDO) are relevant to rating the efficacy of this project. Therefore, this project’s overall efficacy is rated substantial by this review. Overall Efficacy Rating Substantial PHREVISEDTBL 5. Efficiency Economic and Financial Efficiency ex ante • The PAD did not estimate an economic rate of return (ERR) for the project as a whole and for three logical reasons: (a) it was impossible to know the future mix of the many types of RE projects that would be financed by the project, (b) it was also impossible to know how many RE projects would be financed; and (c) the complexity of the system of energy subsidies in Mexico. • On the other hand, the long-term financial viability of agri-businesses likely to benefit from the proposed project support was assessed. Twelve agri-business models were analyzed during project preparation using the database that FIRCO constructed using activities financed under the Shared Risk Fund for Agribusiness Development (FOMAGRO) program. An internal rate of return (IRR) was estimated for each of the models. These models resulted in estimated IRRs in the range of 14% to 40% for energy efficiency projects; 17% to 37% for solar panel systems; and 3.5% to 7.7% for photo-voltaic systems connected to the electricity network (PAD, para 45). ex post • Economic Analysis. An ex-post economic cost-benefit analysis was included in the ICR. The economic value of the net CO2 equivalent emissions reduced (estimated by FIRCO based - according to the World Page 10 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) Bank project team - on accurate measurements of CO2 emissions by type of technology as a direct result of its collaboration with the Bank under this project) and valued at shadow prices based on the latest Bank guidelines for such prices. The potential value if GHG net emission reduction were estimated for three pricing scenarios, namely (a) a ‘baseline scenario’, not including the economic benefits of GHG emissions reduction; (b) a "low carbon price scenario", applying an assumed low economic valuation for the projected reduction in GHG emissions; and (c) a "high carbon price scenario", using a high assumed economic valuation for GHG emissions reduction generated through project interventions. The ERR analysis generated the following results: Baseline - ERR was estimated at 17%. Low shadow price - ERR was estimated at 33%. High shadow price - ERR was estimated at 65%. • A sensitivity analysis showed that these results were robust in the face of changes in costs and benefits. Estimated switching values with respect to cost increases and reductions of economic benefits were analyzed and presented in the ICR. In brief, the analysis showed that the project would remain economically feasible (i.e., an attractive investment for society in general), even if project costs were to increase by as much as 31% or if project benefits were to decrease by 24%. • Financial Analysis. A financial cost-benefit analysis was performed for the different categories of sub- projects implemented. All sub-projects proved financially viable with financial internal rates of return (FIRR) higher than the assumed discount rate of 10% and financial Net Present Value (NPV) higher than zero. The average FIRR was 36% and the FIRRs for various projects ranged from 12% for photovoltaic cells to 71% for energy efficient irrigation pumping. • Cost Effectiveness Analysis. The project supported 2,286 sub-projects. The solar thermal systems had lower costs for both the smallest and the largest capacities. In general, the lower unit costs were for the bio- digesters implemented on pig farms, cattle farms and TIF (Federally Inspected Type) plants, as well as heat recovery systems, high efficiency boilers and efficient water pumps. All had unit costs per ton of CO2 equivalent below US$20 dollars. According to the ICR (para 33) "comparisons of cost efficiency confirm that the project operated within sector-specific norms." • The ex post economic and financial analysis were thorough and provided good evidence for the validity of the efficiency of project investments. Administrative and Institutional Efficiency The project was approved on February 24, 2009. However, it was not until almost a year later on January 29, 2010 that the IBRD loan became effective. The GEF Grant became effective on February 8, 2010. The original closing date was on December 31, 2013 but the project closed 4.5 years later on June 29, 2018. The extension of the project closing date allowed the completion of additional technology investments, even though the underlying reasons were fiscal austerity and FIRCO’s efforts to disburse the Loan in the face of significant devaluation of the Peso (ICR, p. 13, para 38). The disbursement of funds from the GEF Grant suffered from delays due to the lengthy time (several years) to hire a fund administrator. Also, transfer of GEF resources to the fund administrator suffered additional delays due to changes in national budget policy. This affected GEF grant-financed project activities including training, field supervision and payments to the Implementation Team. Further delays were incurred until a seed capital revolving fund was established to allow up-front Page 11 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) payments by FIRCO to be later reimbursed by GEF. The GEF was used mainly to pay project management and supervision costs in the extension period; by the closing date GEF disbursements reached 88% of the target. Conclusion. The efficiency of the IBRD project is rated substantial despite delays faced by the project which were mainly due to factors beyond the control of the project's management. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0 Not Applicable 100.00 ICR Estimate  36.00 Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of objectives was rated Substantial. Efficacy was rated Substantial as the project succeeded in promoting the adoption of environmentally sustainable technologies in agri-businesses. It also exceeded its targets on several intermediate outcome/outputs indicators. Efficiency is rated substantial despite some shortcomings such as implementation delays. Conclusion. Based on a substantial rating for all three assessed parameters (Relevance, Efficacy and Efficiency), this project's Outcome is rated Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The ICR (p. 24, para 73) reported that the Risk to Development Outcome was low due to the following reasons: Page 12 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) • A favorable legal and policy environment in Mexico for renewable energy. The new government has expressed commitment to providing incentives to agriculture/agribusiness to invest privately in energy innovation and move to self-sufficiency. However, government turnover creates uncertainty about whether this supportive environment will continue, as ministries establish priorities. • Plans for the future O&M are in place. A high proportion of beneficiary agribusinesses had formal maintenance contracts with the supplier, and all beneficiary agribusinesses received O&M training specific to their investment, mandated under their Business Plans. After ten years, 85-90% of the technology sub-projects financed were expected to remain operational as the Beneficiaries’ investment combined with the cost savings associated with the project’s energy technology systems are strong incentives to keep them working. Also, some technologies like photo-voltaic and solar heating systems tend to have low maintenance requirements and a higher likelihood of sustainability. • The project promoted demand for rural energy technologies, helping to grow the domestic market for quality energy systems and specialized technical services. The project introduced clean energy practices into industrial processes and this created a new market for suppliers in the Renewable Energy and Energy Efficiency market, which helped to improve the availability of replacement parts. • Replication of, and demand for the promoted technologies beyond the project, is evident. Preliminary data showed that technology providers were supplying new private investments based on the project model. For example, in Sonora State, pig farms with project-financed motor-generators financed new equipment themselves to expand their capacity to produce electricity from renewable energy. 8. Assessment of Bank Performance a. Quality-at-Entry • The project was the first of its kind to be implemented in Mexico. As a climate smart/technology adoption operation, the project had high strategic relevance with important medium and long term implications. The PDO was clear, but the GEO was too broad. • Design was informed by previous rural development and renewable energy projects in Mexico and other countries of the LAC region. SAGARPA/FIRCO implemented a pilot US$21.7 million Renewable Energy for Agriculture Project (PERA) from 2000-2006, supported by a Global Environment Facility (GEF) grant of US$8.9 million. Key technical knowledge and lessons form that project were captured. Most notable lessons included: (a) an effective financial incentive scheme is essential for investments in renewable energy and energy efficient uses to have a high impact and prospects for broad replication, and (b) the productive uses of renewable energy and energy efficiency practices must be promoted through consideration of broader rural development rather than just concentrating on energy supply, among others. • Design was complex with ambitious expectations in a relatively short engagement period in the original design. While the investment was timely, and the policy/strategic context was supportive, institutional structures, financial arrangements, technical and operational capacity, markets and Page 13 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) agribusiness buy-in, all needed more time to develop (ICR, p. 23, para 70). The PAD was specific on implementation arrangements, but lacked details on the most desirable characteristics of targeted agribusiness. The ICR (p. 15, para 48) also reported that "FIRCO’s decentralized Management Units lacked the operational capacity to work with the project subsidy mechanism and needed training and experience on the job" which raised questions about readiness to implement. • Nine risks were identified at the appraisal stage. While predicting policy and regulatory changes after effectiveness was not possible, the PAD risk analysis by-passed additionality. Also, potential budget shortfalls were rated as a moderate risk with a vague mitigation measure that stated that budget was to be ensured through consultations with the client. The ICR (p. 15, para 47) reported that "the mitigation measures suggested for potential budget problems lacked substance." • M&E suffered from minor design weaknesses, but implementation had major weaknesses including lack of baselines and a control group to support formal impact evaluation (see section 9 for more details). Based on the above mentioned information, Quality at Entry is rated Moderately Satisfactory due to design shortcomings, limitations in risk analysis, and M&E weaknesses. Quality-at-Entry Rating Moderately Satisfactory b. Quality of supervision The project was implemented under a challenging fiscal environment. The Government’s fiscal austerity program (2014-2016) caused SAGARPA to tighten its budget and re-design its programs. This negatively impacted FIRCO's resources and consequently its operational capacity, due to the loss of professional staff. According to the ICR (para 71) the project benefited from regular supervision missions that were well staffed with fiduciary, safeguards, agribusiness and energy specialists. Also, M&E and agribusiness activities benefitted from FAO expertise. Bank fiduciary staff provided crucial assistance to address FM and procurement constraints. The Mid-Term Review was timely and according to the ICR (para 71) "diagnosed the critical issues affecting progress, and the benefits flowing from the project model and followed up both systematically." In 2012, the supervision team worked to address the Government's request for Additional Financing (AF). The AF was justified based on the satisfactory performance of the original project which was documented in the MTR report. There was also clear evidence of increasing demand by producers and agribusinesses for the innovative technologies promoted by the project which was in agreement with SAGARPA’s priorities to promote energy efficiency through investments in sources of renewable energy (Project Paper for AF, para 17). Supervision oversaw a smooth transition in the final year of implementation - when the Government continued to invest in the same technologies and types of sub-projects without World Bank support. However, supervision efforts to support M&E were mixed. On the one hand, supervision worked with FIRCO to launch and mainstream M&E, follow good practices, and ensure that adequate data was captured to support assessment of the project activities. On the other hand, supervision efforts with FIRCO to organize an impact evaluation were unsuccessful, and by the ICR finalization, the IE report was still in progress. Page 14 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) Based on the above-mentioned information, Quality of Supervision is rated Satisfactory by this review, despite some weaknesses. Overall, Bank Performance is rated Moderately Satisfactory which reflects Quality at Entry weaknesses and a strong supervision performance. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design • The PAD did not include an explicit Theory of Change - which was not required at the time the PAD was prepared. Nonetheless, the ICR (p.2) included a retrofitted theory of change that stated critical assumptions, and reflected the relations between activities, outputs, expected PDO/GEO outcomes, and higher level impacts. • The project's M&E system would according to its design be adapted to function within the existing M&E system managed by SAGARPA/FIRCO to monitor the existing FOMAGRO Program. The project M&E system would employ an adaptive management framework characterized by regular monitoring and evaluation, semi-annual reports on implementation progress, a comprehensive Mid- Term Review, and a final assessment. The PAD clearly described M&E arrangements, along with specific, timed evaluative/analytical deliverables. • The PDO was to be assessed through one outcome indicator: "number of small and medium-sized agri-business adopting environmentally sustainable technologies (renewable energy sources, energy efficient technologies and/or sustainable waste management and biomass conversion), and the GEO through two outcome indicators: "tons of CO2 equivalent avoided" and "SAGARPA is successfully formulating climate change mitigation and adaptation policies and programs in the agricultural sector and monitoring their implementation." • While the PDO outcome indicator was linked to the PDO, it was more of an output. The indicator was measurable and a baseline was to be established after effectiveness using agribusiness entry profiles. However, the Results Framework (RF) in the PAD lacked indicators to measure potential economic benefits of the project even at the intermediate levels. The first GEO indicator was directly linked to the GEO, but its initial target was low. The second GEO indicator could have benefitted Page 15 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) from a more nuanced approach to assess functionality of SAGARPA, rather than just satisfying the indicator with a simplistic Yes or No answer (ICR, p. 29). • The RF included 15 intermediate outcome indicators that were mostly quantitative in nature and some were outputs. The RF lacked social indicators and the economic benefits of the project were not captured. b. M&E Implementation • FIRCO managed the project databases. According to the ICR (p. 19, para 58) "semester reports were well-presented in a consistent, comprehensive format and kept the Bank informed, usually coinciding with supervision missions." • FIRCO used modern technology to accurately measure actual CO2 emissions. This was unprecedented in other Mexican ministries or agencies and according to the ICR (p. 19, para 58) was considered a best practice achievement of high value to stakeholders. • M&E implementation benefitted from FAO and the Bank's support. However, FIRCO failed to carryout a project baseline evaluation and there was no viable control group. • An evaluation study using simple before/after comparisons was arranged by FIRCO. It was only contracted in August 2018, and thus even preliminary results for the ICR were unavailable. • The project could have benefitted from the restructurings to revise the Results Framework to better capture the project's achievements. For example, the ICR (para 57) appropriately stated that the RF could have been revised to include: "core indicators; co-benefits, including social (since there were underlying equity goals); sustainability (O&M practices/arrangements, given that most investments were for equipment, systems and infrastructure); and/or energy cost reductions (as a component of competitiveness)" • M&E implementation suffered from major problems, most notable was the failure to establish baselines, lack of a control group, and failure to complete a final impact evaluation study. c. M&E Utilization • FIRCO’s quarterly M&E reports were presented in a consistent, comprehensive format. These kept the Bank informed as they coincided with supervision missions. The ICR relied in many areas on FIRCO's reports. • Accurate CO2e data strengthened the Government’s capacity for data-driven policy-making, strategy formulation, and official representation on climate change both nationally and internationally. • It was not clear from the ICR how the project's M&E was used to make informed decisions regarding project management. Page 16 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) •Finally, the ICR (p. 19, para 57) concluded that "SAGARPA had no expectations of evaluation, nor a methodology, and saw evaluation as more of a risk than an opportunity." M&E utilization could have benefitted from the completion of a final impact evaluation. Overall, M&E is rated Modest due to design and implementation shortcomings, and utilization was mixed. M&E Quality Rating Modest 10. Other Issues a. Safeguards The project was rated category B for potential environmental problems. It triggered two safeguard policies: Environmental Assessment (OP/BP 4.01), and Indigenous Peoples (OP/BP 4.10). In accordance with Bank policy, the Government prepared an Environmental Assessment (EA) of the project. No significant negative environmental impacts from project activities were envisaged. The project’s components and activities would all be focused on increasing the use of environmentally sustainable technologies (e.g. energy efficiency, renewable energy, and emission-reduction) in existing agri-businesses. As such, the project was expected to exhibit significant positive environmental impacts. Furthermore, improved capacity for environmental management by both the private and public sectors would also result in a positive impact on the environment in general. Potential adverse environmental impacts would be minor or non-existent, as they would be avoided or minimized through appropriate preventive actions and mitigation measures. Environmental. According to the ICR (para 61) the project's compliance was satisfactory throughout implementation due to FIRCO’s commitment and to agribusinesses’ acceptance of the environmental model, as well as the benefit of having the same Bank environmental specialist from project preparation through closing. FIRCO ensured that the approved sub-projects would not have negative environmental impacts. Also, it trained beneficiary agribusinesses in complying with applicable permits and licenses, as well as applying preventive, mitigation and corrective measures aligned to Bank rules while minimizing compliance costs. However, the involvement of multiple agencies in environmental issues and lack of clarity on environmental regulations at the federal, state and municipal levels complicated environmental compliance. The ICR (para 62) emphasized the environmental benefits of the project which included: significant reductions in GHG emissions; efficient treatment of solids from pig and cattle enertrprise; reduced water contamination due to improved manure and waste/residue management; by-products of the technology systems including organic fertilizers and compost, and cleaned waste waters were conveyed as organic fertilizer for irrigation via closed systems resulting in a reduction of the amount of commercial fertilizers used; compliance with national environmental laws; better quality of life for neighboring communities though improved quality of habitat surroundings, and improved working conditions for laborers due to reduced odors and environmental contaminants. Page 17 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) Social. The project successfully complied with the Indigenous Peoples safeguard (ICR, para 63). Indigenous participation through the agribusiness labor force was low, and none of the agribusinesses financed were indigenous-owned or managed. The ICR did not provide an explicit statement of compliance for the either of the two triggered social safeguard policies. b. Fiduciary Compliance Financial Management. Financial management was negatively impacted by human resource constraints at FIRCO. Financial management training for FIRCO staff at the state and local levels was delayed. The GEF grant-financed project activities suffered from delays because the funds flow mechanism could not function as designed due to changes in national budget policy. FM performance improved after adopting alternative mechanisms to accelerate disbursement through a revolving fund. Also, an administrative agency was contracted to manage consultants and training events. In November 2017, the Government sought a partial cancellation of US$10.0 million. By project closing, the undisbursed balances were US$7.75 million (AF) and US$1.2 million (GEF). According to the ICR (para 65) audit reports were of acceptable quality to the Bank and unqualified. Procurement. Procurement benefited from continuous and comprehensive oversight by the Bank. Also, the presence of a single Bank Procurement Specialist for the duration of the project was helpful. However, the Bank procurement rules were not well-understood, especially at FIRCO’s decentralized levels. While procurement benefited from the Bank-provided mentoring and training to FIRCO, procurement function was at times challenging. This was due to the need to adjust procurement modalities, processes and documentation as markets for the technologies developed and evolved. According to the ICR (para 69) "the project’s procurement challenges and their resolution contributed to the World Bank’s procurement reform, an important, positive institutional spillover effect." c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Page 18 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory --- Moderately Bank Performance Moderately Satisfactory --- Satisfactory Quality of M&E Modest Modest --- Quality of ICR Substantial --- 12. Lessons The ICR included five lessons. The following three lessons are emphasized with some adaptation of language. • To ensure a successful outcome, energy technology innovation projects in the agriculture sector need to factor in both demand and supply issues. Promoting agribusiness buy-in and adoption is just one side of an equation involving multiple, parallel development streams. Similar projects need to factor in: (a) market development, including appropriate technologies, (b) reliable, competitive local providers, (c) specialized advisory and maintenance services; (d) supportive, flexible project procurement which will help to establish technical, price and quality standards; (e) TA and training – public and/or private - focused on the installation, operation and long-term sustainability of systems; (f) dissemination of the economic, environmental and social results of technology adoption; (g) a parallel R&D agenda to resolve emerging challenges, develop databases and make the case for supporting the model; and, (h) ensure that the characteristics and needs of targeted beneficiaries are well-understood. • To ensure participation of beneficiaries, energy innovation initiatives can benefit from flexible targeting that involves a clear mechanisms for selection, support/incentives for investment and expected cost-sharing depending on size, production volume and technology. A comprehensive, upstream analysis is needed of the scale, activities, typology and motivation of potential beneficiaries to participate. Technically, interventions in renewable energy and energy efficiency can be equally viable for small or large agribusinesses, but economies of scale matter. Larger bio-digester technologies are more cost-effective and efficient, while solar has a greater potential to be modular to be efficient. Incentives also matter. While the incentive driving a large dairy operation to participate is likely to lead to a substantial increase in profitability, for a micro/small agribusiness, pumped water for cattle powered by a photovoltaic system might represent the difference between viability and failure. • To ensure accurate assessment of the project's impacts, a well designed impact assessment study needs to be done at the completion of the project. The project design needs to include adequate funding to carry out an impact assessment study, for example, through the Bank’s Development Impact Monitoring and Evaluation unit (DIME). The Bank can help public authorities commit to M&E (including by financing it with loan funds, not grant resources) and understand the benefits - social, political, technical and economic - of a data-driven assessment of public policy, strategy, plans and programs. Building a sound M&E system in key institutions is as important as executing the designed agenda. Page 19 of 20 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review MX Sustainable Rural Development (P106261) 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR is well written. It provides thorough coverage of the project implementation experience. It also gives good contextual background and clearly discusses various factors that impacted implementation. Discussion of outcomes is logical and relies on the project's achievements on the ground. The discussion of outcomes could have benefited from shedding light on the economic and social impact of the project. However, this was not possible because an impact evaluation was not available. Overall, the Quality of the ICR is rated Substantial. a. Quality of ICR Rating Substantial Page 20 of 20