Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) Report Number: ICRR0022865 1. Project Data Project ID Project Name P162659 RY: Smallholder Agric Production CLOSED Country Practice Area(Lead) Yemen, Republic of Agriculture and Food L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) TF-A5431 31-Aug-2020 36,000,000.00 Bank Approval Date Closing Date (Actual) 31-Jul-2017 30-Jun-2021 IBRD/IDA (USD) Grants (USD) Original Commitment 36,000,000.00 36,000,000.00 Revised Commitment 36,000,000.00 36,000,000.00 Actual 36,000,000.00 36,000,000.00 Prepared by Reviewed by ICR Review Coordinator Group Samjhana Thapa Christopher David Christopher David Nelson IEGSD (Unit 4) Nelson 2. Project Objectives and Components DEVOBJ_TBL a. Objectives As per the Grant Agreement (GA, 2017) and the Project Appraisal Development (PAD, 2017), the objective of the Project was “to increase the use of productivity and nutrition-enhancing agricultural practices by smallholders in targeted Project areas”. The project would target poor and food insecure households within seven of the most food insecure governorates in Yemen. Within the targeted governorates, district-level project sites would be selected based Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) on food insecurity and malnutrition levels, based on information that was collected from the 2016 Emergency Food Security and Nutrition Assessment (EFSNA) (ICR, paragraph 7). b. Were the project objectives/key associated outcome targets revised during implementation? Yes Did the Board approve the revised objectives/key associated outcome targets? No c. Will a split evaluation be undertaken? No d. Components To achieve the PDO, the project was structured using the following three components: Community Subprojects and Investments (Appraisal cost: US$29.89; Actual Cost: US$30.32) This component aimed to finance priority subprojects and investments to increase smallholders’ production, income, and nutrition. It was to finance civil works, goods, including livelihood kits and farm restoration start- up packages, consulting services, training, and capacity building (PAD, paragraph 29). As per the ICR (Page 9), the following activities were to be supported under each of the three sub-components, which were implemented through community-based and participatory approaches:  Strengthening Community land and water management: (a) water harvesting at the farm and micro watershed level in upper catchment rainfed areas, including rehabilitation of existing or constructing new water harvesting and storage; (b) water diversion for improved spate irrigation in the lower catchment of selected governorates; (c) roof-top and other type of water harvesting; (d) improving natural rangelands, including planting trees and establishing protection of structures.  Improving Animal husbandry, livestock production and animal health services (a) improving access to veterinary services in targeted communities; (b) carrying out vaccination campaigns to prevent major diseases in small ruminants; (c) improving animal nutrition; (d) enhancing animal husbandry and beekeeping; and (e) facilitating the development of new livestock activities, exclusively for poor women, returnees, internally displaced people or famers who lost their livelihood assets as a result of conflict.  Improving livelihoods and nutrition, and increasing the value-add of selected agriculture products in targeted areas: (a) provision of emergency agriculture livelihood kits to the Vulnerable Groups to restore crop production and generate income; and (b) increasing the value-add of key value chain agriculture products, through the provision of equipment, technical and business training to farmers to improve processing, packaging, and marketing, and promotion of partnerships with traders. Capacity Building and Extension (Appraisal cost: US$0.90 million; Actual cost: US$0.55). This component aimed to build the capacity of project beneficiaries involved in land and water management by introducing effective and climate resilient irrigation techniques, and in livestock production to protect assets; the piloting of Farmer Field Schools; promoting nutrition awareness; and capacity building for improved service delivery of farmer organizations, NGOs, and private sector services who were providers under the project. Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) Program Administration, Monitoring and Evaluation (Appraisal cost: US$5.21 million; Actual cost: US$5.13 million). This component would finance FAO’s management support costs, direct costs for management and implementation of the project, the cost of hiring a third-party monitoring agency, and project M&E. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project costs: The project cost at approval was US$36 million, and the actual project cost at completion was US$36 million (100 percent of appraised estimate). (ICR, annex 3) Financing: Per the Grant Agreement (GA,2017), grant from the Global Agriculture and Food Security (GAFSP) Multi Donor Trust Fund (TF-A5431) for US$36 million financed the project. Borrower Contribution: The project did not receive any contribution from the country Government. Dates: The project was approved on July 31, 2017 and became effective on August 10, 2017. The Mid-term Review (MTR) was conducted on March 07, 2019. The original project closing date was August 31, 2020, which was extended, and the project was closed on June 30, 2021. Restructurings: The project was restructured twice (August 06, 2020, and February 04, 2021). The details of the Level 2 restructurings are as follows:  1st Restructuring, August 2020: According to the Restructuring Paper, the first restructuring took place in July 2020 and made the following changes: (i) extend the project closing date from August 31, 2020, to February 28, 2021; and (ii) revise the project’s results framework to better reflect the type of interventions and update the target values for some indicators in view of the project progress and expected outcomes. The ICR stated that there was no change to the PDO or to the result indicators. In total, seven targets were revised (four targets were revised upwards, while for the remaining two, targets related to animal vaccines were reduced). The animal vaccine campaign was scaled down due to the World Bank requirement that project resources could not be used to implement any activity through the central government, which this review agrees was outside the control of the project. Details of the revisions were provided in the ICR (Annex 1).  2nd Restructuring, February 2021: Under the second restructuring, the project closing date was extended by four months (June 30, 2021). Due to difficulties in reaching project beneficiaries in the northern part of the country resulting from delays in obtaining permits for field activities, additional time was required to complete project activities. The project had also faced movement restrictions and fuel shortages due to COVID-19. Despite changes in the Results Framework during the project’s first restructuring, including revision of target values for few indicators, this review did not find evidence that the project had reduced its level of ambition. In fact, several of the revised targets made the project’s objective more ambitious. Thus, this review will not use a split rating methodology. 3. Relevance of Objectives Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) Rationale Country and Sector Context. Yemen is one of the poorest countries in the Middle East and North Africa region. Poverty is high in rural areas, where about two thirds of the country’s population resides and primarily depends on agriculture. The country is among the ten countries in the world with the highest rates of food insecurity and it imports 80 percent of the food it consumes (PAD, paragraph 5, 7). The country has faced conflict since early 2015 which has further exacerbated the situation of high food insecurity and rising poverty. In 2016, 17 million people - about 60 percent of the country’s population (41 percent before conflict) - were food insecure. Along with food insecurity, critical infrastructure in the country has been destroyed and socio-economic conditions have deteriorated. This has devastated the economy and in 2016 alone, Yemen’s GDP shrank by 40 percent (ICR, paragraph 1). Agriculture is a key source of livelihood and employment in Yemen. Prior to the outbreak of the conflict, the sector employed more than half (54 percent) of the workforce and was the main source of income for 73 percent of the population (PAD, paragraph 7). Despite the sector’s importance, productivity has always been low in the country due to limited availability of inputs (seed, fertilizer, machinery, irrigation etc.) for crops and animal health and husbandry services for livestock. This was made worse due to the conflict. According to the ICR (Paragraph 2), the conflict resulted in a shortage of inputs, damage to agricultural machinery, irrigation systems and storage facilities, and a breakdown of agriculture supply chains. This led to a drop in agriculture employment by 50 percent from pre-conflict levels (ICR, paragraph 2). Government Strategy. The project design was based on a proposal submitted by the Government of Yemen in June 2013 to the Global Agriculture and Food Security Program (GAFSP), which was approved by the GAFSP Steering Committee in September 2013. According to the PAD (Paragraph 11), the proposal addressed challenges faced by the sector, including food security, agricultural productivity, and climate resilience, which were the key features of the National Agriculture Sector Strategy (NASS) adopted by the Government of Yemen in 2012. The proposal was also in line with other national strategies, such as the National Food Security, National Water Sector Strategy, and Investment Program. Soon after, the country faced military conflict, which led to a halt in project preparation through 2014. During the crisis, the importance of the agriculture sector to address food insecurity and livelihoods was found to be even more critical. As stated in the ICR (Paragraph 6), in 2016 and early 2017, the original design of the project was modified, and the project was appraised with an aim to “provide the necessary investments for increasing agriculture productivity and restoration of livelihood activities of the vulnerable people who were impacted by the ongoing conflict”. Program activities were designed to target the poorest governorates/ districts. Thus, the project was highly relevant to the context of emergency faced by the country. Bank Strategy. The project was well-aligned with the World Bank strategy for Yemen at the time of appraisal and closing. The current World Bank Strategy for Yemen which is articulated in the Country Engagement Note FY20/21 (CEN, April 16, 2019, Report No. 136046-RY) has two key objectives, which are: (i) Continued support for basic service delivery and institutional preservation; and (ii) Extending support to livelihood, human capital, and basic economic recovery. The project contributes to both the objectives of the CEN, which according to the ICR, has “aspirations to address vulnerability and food insecurity through a balance of physical recovery and reconstruction with interventions that strengthen social cohesion and address the structural dimensions of fragility that is based on the project’s community based and participatory approach of physical investment and capacity building” (ICR, paragraph 20). Further, the project was also found to be in line with the Yemen Remaining Engaged in Conflict (RECA) Eligibility Note (December 17, 2020), which has focused on Bank interventions in human capital and basic economic Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) recovery, by working closely with international organizations like FAO, and in-country partners such Social Fund for Development (SFD) to implement basic service delivery projects. In summary, the project’s objectives were aligned with both the Government and World Bank strategies. The project objectives were relevant in addressing food insecurity and vulnerability faced by rural people, which were in line with both the Government’s National Agriculture Sector Strategy (NASS) and the Bank’s Country Engagement Note. Thus, Relevance of this project’s objectives to Government and World Bank strategies in Yemen at appraisal and at project closing is rated High. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective To increase the use of productivity and nutrition-enhancing agricultural practices by smallholders in targeted Project areas. Rationale Theory of Change. The ICR provided a retrospective Theory of Change (ToC) for the project (page 7). To meet the stated objective, project interventions included financing community-based water and land infrastructure related to new/improved irrigation/drainage services, on-farm water storages, and water saving technologies, including terrace rehabilitation. To improve animal husbandry, livestock production and animal health, community animal health workers would be trained, small ruminants and cattle would be vaccinated, and animal nutrition activities would be undertaken. To restore the conflict-affected farmers to resume crop and livestock production, seed distribution for crop and horticulture would be provided, along with the provision of dairy equipment and training for producers. Capacity building through Farmer Field Schools, and training packages including for nutrition would also be provided. These activities were expected to result in increased adoption of productivity and nutrition-enhancing practices by poor smallholder farmers and conflict affected households and assist them to re-engage in crop and livestock sectors to restore their livelihood and provide income for their basic needs. The ToC was found to provide a clear connection between project interventions and the expected outcomes. It also outlined the relevant assumptions that underpinned the achievement of the PDO which were: (i) the communities willingness to work with the project in identifying beneficiaries, and in adopting higher productivity and nutrition enhancing practices (Behavioral); (ii) menu of investments and assistance provided by the project would lead to higher productivity (Technical); and (iii) political and security situation would not deteriorate to the extent that activities could not be implemented (External). Outputs: Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) The project surpassed most of its output and intermediate outcome targets. Several indicators were disaggregated based on gender. The land and water sub-projects reached 22,660 HHs (Target: 26,000), and the construction and rehabilitation work was accomplished through cash for work (CFW) and community contracting approaches, providing US$3.5 million to 12,300 beneficiaries engaged in CFW and US$10 million to skilled community contractors (FAO Project-end report). The ICR reported the following outputs:  13,038 ha of land area provided with new or improved irrigation and drainage services (Target: 5,400)  3,873 ha of land area protected by wadi works (Target: 1,800)  76 new on-farm water storages constructed (Target: 24)  42 ha of terraces rehabilitated (Target: 70)  3,031 HHs benefitted from rooftop water harvesting (Target: 2,000) The Agricultural Assets and Services program reached 157,075 farmers (Target: 110,000). The number of women and women headed HHs benefitting from the program reached 38,000 (Target: 27,000). The various outputs were reported in the ICR, but the exact numbers breakdown was provided by the project team upon request from IEG:  59,640 farmers received staple seeds, backyard poultry and small ruminants  4,000 farmers received forage seeds  22,660 HHs benefited from the rehabilitation of community water infrastructure (Target: 26,000)  10,202 farmers received feed blocks  45,240 beneficiaries received improved veterinary services provided by community animal health workers (CAHWs)  15,333 beneficiaries benefitted from other project activities (the provision of fodder choppers, training and equipment to dairy, honey and horticulture producers, training and equipment support to community animal health workers) Additionally, capacity building and extension services were provided to 58,567 HHs (Target: 38,000)  2,100 people received nutrition training or awareness (Target: 3,000)  105 Farmer Field Schools were organized comprising of 2,100 farmers and training of 105 trainers  2,434 client days of extension services provided to farmers (Target: 2,400) Under the Animal husbandry, livestock production and animal health services, 82,008 HHs were reached (Target: 34,000)  84 community animal health workers equipped and trained  2.7 million small ruminants and cattle were vaccinated reaching 142,252 farmers (against a target of 130,000), and 28 cold chain facilities were set up (refrigerator, solar panels) for vaccine storage  Milk production increased (9% against a target of 10%)  17 private feed block producers supported Outcomes: Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) To increase the use of productivity and nutrition-enhancing agricultural practices by smallholders in targeted project areas, the following PDO indicators were measured.  Farmers adopting improved agricultural technology (Target: 10,000; Actual: 13,322) [Male: 7,000 (Target), 10,594 (Actual); Female: 3,000 (Target), 2,728 (Actual)]  Households supported to resume crop and livestock production (Target: 35,000; Actual: 59,641) The project exceeded both the PDO indicators and only fell slightly short on the target related to female farmers’ adoption of improved agriculture technology. Unfortunately, the ICR did not provide details on which agricultural technologies promoted by the project were captured by the first indicator. In communication with the IEG team, the project team mentioned that this indicator included the following practices: (i) farmers that adopted improved animal nutrition and animal husbandry practices, i.e., use of improved forage seeds and fodder choppers provided by the project; (ii) beekeepers using new plants and equipment introduced by the project; and (iii) horticulture farmers using improved seed/seedlings, equipment and training from the project. To provide evidence that the project's interventions and outputs led to the achievement of outcomes, the project utilized data from its M&E system. For example, one of the findings reported by the ICR (paragraph 41) was “because of the project’s interventions related to provision of inputs and knowledge, sorghum yields increased by 24 percent (average), and millet yields increased by 33 percent (average)”. These assessments done by the project compared the beneficiaries targeted by the project with a control group of farmers who were not assisted by the project, and there was evidence that the targeted area reported a higher average increase in yields (Project-end report). In addition, an independent third-party Beneficiary Survey was undertaken at project-end which documented qualitative feedback about the project’s benefits. According to the ICR (paragraph 33, annex 1), 88 percent of respondents expressed ‘overall satisfaction’ with the project and 87 percent indicated that the project responded to their priority needs. Other results of the Beneficiary Survey are reported in Section II.B of the ICR. In summary, the project met its target for PDO indicators and fell short only on a few output targets. The evidence provided by the ICR which was based on the project’s M&E data, project-end report and third-party beneficiary survey demonstrate that the project was successful in achieving its PDO in a difficult conflict environment. Thus, the efficacy with which objective was achieved is rated as Substantial. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale In summary, the project has mostly exceeded targets for its output indicators and has achieved both the PDO indicators. The evidence provided are found by this review to be sufficient. Therefore, the efficacy with which the PDO was achieved is rated as Substantial. Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) Overall Efficacy Rating Substantial 5. Efficiency Ex-Ante At Appraisal, the project undertook an economic and financial analysis (EFA) built on the following indicators: Economic Internal Rate of Return (EIRR), Net Present Value (NPV), and Benefit Cost Ratio (BCR). According to the PAD (Annex 4), the financial analysis included investments proposed by the project to calculate their financial returns and viability. The model assessed the following eight indicative project investments: (i) terrace rehabilitation; (ii) water harvesting; (iii) spate irrigation; (iv) water ponds; (v) water harvesting cisterns; (vi) increased livestock production; (vii) commercial and local laying poultry model; and (viii) beekeeping and honey production. Meanwhile, the project’s economic analysis focused on a non-conventional approach and assessed project benefits related to reduction in undernutrition and stunting built on literature on the economics of nutrition, and economic appraisals of similar recent projects. The benefits and costs were analyzed on lifetime earnings. The overall project EIRR amounted to 12 percent, with an estimated NPV of YER 1,054,316 million (US$4 217 million) at a 10 percent discount rate. The Benefit Cost Ratio amounted to 1.53. Ex-Post At project closing, an ex-post EFA was conducted for ten project interventions. The model included all project interventions included in the ex-ante EFA (and referred to in the earlier paragraph) except water harvesting cisterns for which the benefits were not calculated due to a lack of data. In the ex-post EFA, the model included additional investments made by the project on vegetable seedlings production centers, support to recovery of crop production (sorghum and millet), and the delivery of dairy production packages (ICR, Annex 4). A financial analysis was conducted on each intervention, which assessed investment, operational costs and annual revenues. These results were aggregated using the number of HHs that received the interventions, and accordingly for each intervention, a Financial Internal Rate of Return (FIRR) and Financial Net Present Value (FNPV) was calculated. To calculate the economic benefit, due to a lack of data, a conversion factor was used based on previous research from IFAD. Finally, the results of the model that covered all 10 interventions were aggregated to estimate the overall benefits through calculation of the FNPV, FIRR, ENPV, ERR, and BCR. As per the ICR (Annex 4), the financial and economic analysis used the following assumptions: (i) discount rate of 6 percent and 10 percent (at appraisal, the discount rate used was 10 percent but as per the ICR, the suggested discount rate for Yemen is 6 percent); and (ii) time frame analysis of 10 years and 15 years which were comparable to the ex-ante EFA. In addition, the following data were used: exchange rate (1 US$ =580 YER), average HH size (6.7 people/HH), and average landholding per individual HH (1 ha). Further, there were intervention-specific assumptions that differed between intervention types. The results under each of the scenarios were as follows:  Quantified ex-post EFA results using 6% discount rate. Under the time horizon of 10 years, the overall analysis showed a positive overall FNPV of US$ 22.68 million and ENPV of US$ 92.65 million, with FIRR and ERR of 13% and 38% respectively, and BCR of 1.31. The results of the overall analysis over 15 Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) years found a positive FNPV of US$ 51.43 million and ENPV of US$ 139.45 million, with FIRR and ERR of 17% and 39% respectively, and BCR of 1.37 (ICR, Annex 4, Table 4).  Quantified ex-post EFA results using 10% discount rate. Under the time horizon of 10 years, the overall analysis showed a positive overall FNPV of US$ 19.62 million and ENPV of US$ 160.11 million, with FIRR and ERR of 13% and 38% respectively, and BCR of 1.31. The results of the overall analysis over 15 years found a positive FNPV of US$ 61.04 million and ENPV of US$ 227.55 million, with FIRR and ERR of 17% and 39% respectively, and BCR of 1.37 (ICR, Annex 4, Table 7).  Sensitivity analysis for individual interventions. The EFA conducted a sensitivity analysis using 10-year and 15-year timeframes under both 6 percent and 10 percent discount rates. The analysis used a 30 percent, 20 percent, and 10 percent increase or decrease in variables (e.g., price and yield of sorghum grain, price of millet grain, sorghum fodder yield, price of eggs and local hen, price of honey and cane sugar, price of cowpea fodder, price of tomatoes and hot peppers seedlings, price of fertilizer). The analysis showed that the results of the EFA were robust as in eight out of ten project interventions, the ENPVs remained positive when there was increase in variables (prices) of 10 percent and 20 percent. Administrative Efficiency: The ICR (paragraph 50) stated that both the implementing agencies (FAO and SFD) were very competent with high quality staff and experienced field teams. Though the grant had to be extended twice, it was due to factors outside the control of the agencies which did not incur additional cost to the project. The activities were implemented efficiently considering the FCV context of the country. FAO’s fee of 5 percent to cover their administrative cost was reasonable. In summary, in a challenging FCV environment, the project was efficiently implemented, and had high FIRRs, ERRs and BCRs for both 10-year and 15-year period under 6 percent and 10 percent discount. The project’s Efficiency is thus rated as Substantial. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal  12.00  Not Applicable 0 ICR Estimate  38.00  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome The project’s objectives were highly aligned with both the Government and World Bank strategies, and therefore the Relevance of Objectives for this project was rated high. The project met its target for PDO indicators (except Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) for a slight under achievement on gender target for one of the PDO indicators), and there was evidence that the project achieved its objective to promote the use of productivity and nutrition-enhancing agricultural practices by smallholders in targeted project areas. Thus, the overall Efficacy of the project was rated Substantial. Efficiency was also rated Substantial with a high ERR. This review concludes that this project achieved its objectives with minor shortcomings and therefore its overall outcome is rated “Satisfactory”. a. Outcome Rating Satisfactory 7. Risk to Development Outcome The project was successful in building capacity of local agencies and communities to implement agricultural activities and provide services to poor and rural farmers in conflict environment. However, despite the project’s success, the ICR (Section IV.D) stated that sustainability of most of the activities implemented by the project continue to be a risk in achieving the project's development outcome. The mitigation measures discussed were either through continued support from FAO’s activities or by the upcoming World-Bank financed Yemen Food Security Response and Resilience Project (FSRRP) approved by the Board in April 2021. The new project was designed to continue implementing the activities of this project. The ICR identified the following activities that required continued support or further development:  Agricultural services such as farmer field schools, deployment of community animal health works and animal disease control program continued to need further technical support and capacity building.  Project activities that were supported through the restoration program such as the horticulture seedling and collection centers, women’s agro-processing activities were not financially viable yet, which needed further support to develop their business models.  Investments on community water and land infrastructure maintenance, including cold chain facilities for vaccine storage at the governate or district level required improved maintenance measures. In addition to the risks on sustainability of activities highlighted by the ICR, this Review would like to point out given that Yemen is still in conflict, the political risk continues to be high. The implementing partners have a good track record of working in volatile situations in rural communities that was proven by a successful completion of this report. Nonetheless, the high security situation of the country cannot be overlooked and is a risk to development outcome of the project. 8. Assessment of Bank Performance a. Quality-at-Entry The project was well-aligned with both Government and World Bank strategies in addressing food insecurity faced by the poor and vulnerable people in rural areas of the country. Despite an uncertain Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) environment, several factors of project design contributed to satisfactory implementation of project activities. The project design put in place appropriate implementation arrangements, with a UN Agency (FAO) working in close partnership with an effective local institution (SFD). FAO was a strong technical partner with experience in implementing emergency livelihood support, livestock, and food security and nutrition activities while SFD brought in the extensive experience of working with rural communities in the country with strong field presence. As pointed out by the ICR (Paragraph 68), SFD had a good track record of over 25 years of experience with project implementation outside of the government. It had competent staff and had experience working with many donors and were familiar with donor procedures and requirements. The project’s M&E system was well designed which was based on frequent assessments of field level activities and results collected by the project’s in-house team as well as by a third-party monitoring agency. There were some gaps in its Results Framework regarding indicator choices and targets, but other than that, the indicators were designed to capture project activities with several different outputs. Despite the breadth of activities, as stated by the ICR (Paragraph 64), “most of the activities were tried and tested over three decades of Bank investment in rural Yemen and were well adapted to the situation”. One key aspect of design was the focus on community-based approach for project interventions which would develop capacity at community levels and promote sustainability. Finally, considering the complex environment in which the project would need to operate in, as per the ICR, project preparation was supported by adequate budget and good mix of multi-disciplinary team. Quality-at-Entry Rating Satisfactory b. Quality of supervision According to the ICR (paragraph 101), the Bank put considerable effort and resources - a total cost of USD550,000 for three years that included 114 staff weeks – to closely supervise the project. Due to the security situation of the country, before COVID-19, Bank team met with FAO and SFD in third party countries like Egypt and supervision missions were held frequently. There was a continuity with a single task team leader engaged in the project from start to end. The ISRs prepared were candid in reporting progress and shortcomings and supervision documents included clear set of action and next steps that were agreed with the implementing agencies. During project implementation, restructurings were planned well to extend the project end date so that all planned activities could be completed. Targets were also revised and made more ambitious when more information was available from the project’s M&E about field level activities and outputs. It is important to note and recognize that the project was high risk and was being implemented under difficult conflict situation. As per the ICR (paragraph 69), situation in the field was closely monitored by the Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) implementing agencies, and Bank team were supportive in changing course when security situation changed. In summary, the Bank’s performance at entry was high. The project was relevant to both Government and Bank strategies. Implementation arrangements were well designed with strong technical and local institutions. The M&E system was designed to capture internal monitoring which was supplemented by third-party independent monitoring. The quality of supervision was also found to be high, with close and timely supervision provided by the Bank which was well resourced. The quality of Bank performance at entry and supervision is thus rated as Satisfactory. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design As stated earlier, the project’s M&E system was well designed with FAO taking the main responsibility for overall coordination of project monitoring. The project’s internal monitoring was supplemented by a third- party independent monitoring to track performance through collection of data and evidence and provide feedback to FAO on improvements in delivery of project activities. It also reported on lessons to facilitate learning within the project. The project design included important features such as a baseline survey to be conducted within the first three months of the project to assess needs as well as the end-project assessment to measure impacts. In addition, the M&E system benefited from the reporting requirements of GAFSP, the Trust Fund that financed the project. The Results Framework was developed to monitor progress and included relevant output and intermediate outcome indicators to measure the PDO. The Results Framework could have been strengthened by including an indicator to track nutrition related activities considering that nutrition-enhancing practices were key element of the PDO. Further, the PDO “Number of farmers adopting improved agricultural technology” did not comprehensively capture all practices or technologies adopted by the beneficiaries though this review agrees with the ICR (paragraph 80) that the intermediate outcome indicators provided support to towards of the PDO. b. M&E Implementation Over the project period, the M&E System ran smoothly except for some difficulties the project encountered occasionally on data collection. The project implemented large number of activities which were spread across geographical areas. There were also numerous local level organizations (FAO had contracted ten NGOs and worked with several other local Government offices) and information from all organization did not always come on a timely manner. Further, as planned during project design, Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) geospatial data on project interventions could not be collected due to Government’s concern about collection of geo-referenced data during security situation. Despite these difficulties, FAO managed to prepare monitoring reports every three months based on the M&E tool (Indicator Performance Tracking Table) that it had developed for the project, and reports were submitted to the Bank and GAFSP every six months. According to the ICR (paragraph 73), the M&E System captured data collected in the field by FAO and SFD community mobilization teams by following participatory processes consisting of focus group discussions, questionnaires, and interviews. In respect of cultural norms, community consultations were held separately with women, led by female community mobilization staff. In addition to the project’s own internal monitoring, a third-party company was hired, and they produced eleven Quarterly Reports, End-of-project Assessment and Beneficiary Satisfactory Survey (August 2021). c. M&E Utilization The project utilized various reports prepared by the M&E system to monitor progress and prepare follow up action plans during implementation. In addition, assessments (crop productivity, milk productivity survey) that were conducted supported with the project’s regular monitoring and were used in assessing ongoing project activities, identify solutions or change course when needed. In summary, M&E is rated as Substantial. The project’s M&E system was well designed and the decision for FAO to take full responsibility on M&E was sound. Periodic field level monitoring was undertaken by the project and timely reports were prepared by the implementing agencies, which were then supplemented by data collected by the third-party agency. During implementation, various assessments were prepared not only to measure progress but also to contribute towards improvements in implementation of project activities. The Results Framework had reasonable set of indicators though one of the PDO indicators could have been better articulated. M&E Quality Rating Substantial 10. Other Issues a. Safeguards The project was classified as a Category B (partial assessment) under World Bank safeguard policies. At appraisal, two environmental safeguard policies were triggered: Environmental Assessment (OP/BP 4.01), Pest Management (OP4.09). The project did not have plans to procure pesticides, but OP 4.09 was triggered because of possible changes in farming practices that may lead to increase use of agricultural chemicals such as fertilizers and pesticides. The project invested in construction or rehabilitation of infrastructure such as water harvesting structures (cisterns), terraces, on-farm water storage facilities for animal and domestic use and on-farm water harvesting through underground cisterns, construction of roof- top collection areas etc. but none of these interventions involved involuntary taking of land or loss of access to resources. Thus, social safeguard policies were not triggered. Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) Environmental and Social Safeguards: At appraisal, the project prepared an Environmental and Social Management Plan (ESMP) in accordance with Environmental Assessment Policy OP4.01. According to the ICR (paragraph 91), the ESMF developed at appraisal was robust and was implemented according to the plan. The ICR did not provide details on the types of activities carried out but upon communication with IEG, the Project Team indicated that all activities proposed under the ESMF were implemented. The project developed 17 site-specific ESMPs covering 106 subprojects under Subcomponent 1.1, 4 ESMPs for activities under Subcomponent 1.2, and 1 ESMP for activities under Subcomponent 1.3. As a result, there was a high satisfaction with the environmental and social safeguards related activities implemented by the project. As per the Beneficiary survey, 98% of the interviewed beneficiaries in all the targeted governorates did not report any negative health impact from project interventions. During implementation, the project was faced with two significant issues. First, ESMPs that were prepared by the implementing agencies were delayed from time to time due to internal clearances required within the implementing agencies (FAO and SFD). In some cases, it was found that work had started before the agencies received clearance of the ESMPs from the Bank which led to a temporary suspension of works. This happened during the early stages of the project due to misinterpretation of ESMF which was rectified and as a result, all ESMPs prepared thereafter were validated by the World Bank prior to the start of construction work. The second issue was related to a fatal incident that occurred in a project site in January 2019 when soil accumulated for land protection activity fell on a worker and suffocated him. This led to the Bank rating the Implementation Progress (IP) rating of the project to a Moderately Satisfactory on May 2019. As per the report of the Third-party Monitoring Agency, the investigation revealed that there were several shortcomings such as inadequate pre-work planning, hazard identification and risk management as well as insufficient safety training. Following this incident, as per the ICR, the implementing agency (SFD) established a dedicated Occupational Health and Safety (OHS) unit and a Consolidated Safeguard Corrective Action Plan (CSCAP) was prepared to classify risks of all sub-projects and identify OHS-related mitigation measures, including by strengthening their training and monitoring. As a result, no further incidents were reported. Grievance Redress Mechanism (GRM). The project established a Grievance Redress Mechanism (GRM) and were made accessible at the local level. According to the ICR, during project implementation, 325 complaints were received, and all were resolved within an acceptable timeframe. At completion, the Third- Party Monitoring found that two thirds of beneficiaries consulted were knowledgeable about GRM. b. Fiduciary Compliance Financial Management: The project was financed through a TF grant to FAO, and as a result, the financial management arrangements for the project was governed by the Financial Management Framework Agreement (FMFA) signed between the World Bank and FAO. This meant that the project would need to be compliant with UN Financial Regulations. At appraisal, in accordance with the OP/BP 10.00, an FM assessment was conducted by the Bank to ensure that the required capacity, fiduciary and accountability oversight existed within the implementing agency (PAD, paragraph 77). During project implementation, financial management was rated satisfactory or moderately satisfactory and interim Financial Reports (IFRs) were submitted on time. As per the third-party monitoring assessment, no reports of bribery or corruption were reported in the project. Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) Procurement: The project was implemented under FAO’s procurement procedures which falls within the Alternative Procurement Arrangements per the Procurement Framework Policy Section III (PAD, paragraph 81). At appraisal, Procurement Risk was rated High due to the security situation of the country. However, procurement activities did not face major challenges and were implemented well by FAO’s procurement specialists. For transparency and in promoting competition, the FAO and SFD publicized the bids locally. Inspections were held on goods at two levels- pre-shipment to warehouse, and on-site inspection by FAO staff. During project period, no mis-procurement were reported. c. Unintended impacts (Positive or Negative) --- d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The ICR included six lessons. This review has highlighted the following three with some adaptation of language that has broader applicability particularly in FCV contexts.  In the absence of Government services in FCV situation, partnership with institutions having high technical capacity and field experience was key to successful delivery of the project. The project’s key partners, FAO with a strong technical background and experience in implementing projects in conflict situation and SFD with a good track record of working in rural communities in the country complemented each other to implement the project in a fragile situation. Agencies like SFD whose implementation modality is based on a CDD approach of social mobilization, community contracting and cash for work allowed increased participation in project activities by rural people.  Clear selection criteria for targeting project beneficiaries helped promote transparency and ownership on project activities. The project followed a targeted approach to identify seven most food insecure governorates and twenty-one districts in the country based on data from the Emergency Food Security and Nutrition Assessment. Within the selected districts, poor farm HHs, conflict-affected farmers as well as women farmers Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) were identified with a clear criterion that were validated through a participatory approach of community engagement. The selection was led by the Implementing Partners and in close cooperation with local authorities.  M&E System based on participatory approach through community consultations and third-party monitoring was critical not only to collect information but helped build accountability. The project’s M&E that depended on extensive field level consultations, various assessments, along with GRM (Grievance Redressal Mechanism) helped build mutual accountability among the project beneficiaries, project’s internal M&E teams and the third-party monitoring agency. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provides adequate evidence on outcomes based on credible sources such as the project’s M&E data and third-party monitoring. Period assessments done during project implementation that measured activity-level outcomes (e.g., crop and dairy yields) as well as the results of the Beneficiary Satisfaction Survey that the ICR referenced were found as robust evidence in support of achievements reported. The ICR provided adequate discussion on the activities, outputs and intermediate results and on how they informed the achievement of the PDO. However, the document did not explain clearly which results were captured by one of the PDO indicators, which was clarified separately by the project team in communication with IEG. There were also some inconsistencies on data reported in the ICR. The lessons learned presented in the report had broad applicability for projects in FCV countries and were based on the project’s experience. Finally, the ICR followed OPCS guidelines, and was candid in acknowledging the shortcomings of the project’s M&E design, and few activities that were challenging to implement that did not yield satisfactory results. Overall, the Quality of the ICR is rated Substantial despite some minor shortcomings. a. Quality of ICR Rating Substantial Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review RY: Smallholder Agric Production CLOSED (P162659) Page 17 of 17