Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) Report Number: ICRR0022034 1. Project Data Project ID Project Name P090723 VN-Livestock Competitiveness&Food Safety Country Practice Area(Lead) Vietnam Agriculture and Food L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-46490,IDA-56920 31-Dec-2015 105,885,515.90 Bank Approval Date Closing Date (Actual) 22-Sep-2009 30-Jun-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 65,260,000.00 0.00 Revised Commitment 107,450,116.46 0.00 Actual 105,885,515.90 0.00 Prepared by Reviewed by ICR Review Coordinator Group Alexandra Christina Maria Vanessa Christopher David Nelson IEGSD (Unit 4) Horst Corlazzoli 2. Project Objectives and Components DEVOBJ_TBL a. Objectives The Project Development Objective (PDO) of the Vietnam Livestock Competitiveness and Food Safety Project (LIFSAP) as stated in the Financing Agreement dated December 10, 2009 included the following three elements: “(i) to increase production efficiency of household-based livestock producers, (ii) reduce environmental impact of livestock production, processing and marketing, and (iii) improve food safety in livestock product supply chains (mainly meat) in the Project Provinces” (FA, IDA 4649, page 5). The Project Appraisal Document (PAD) dated August 27, 2009 stated the same PDO (PAD, para 17). Page 1 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) The PAD (Annex 1) and ICR specified the project’s provinces and reiterated that these were located in Vietnam’s four “‘at risk’ production regions” based on a risk assessment conducted during project preparation.[1] The PDO did not change with the Additional Financing (AF) of 2015, and the Financing Agreement of AF from July 9, 2015 stated the same PDO (FA, IDA 4649 and IDA 5692, page 5) . Hence, the aforementioned PDO is adopted for the purpose of assessing the project’s achievements in this Implementation Completion Report Review (ICRR). [1] “The project intended to support up to 12 provinces located in four geographical production clusters: Thanh Hoa and Nghe An (Central North); Hanoi, Hai Phong, Thai Binh, Hung Yen, and Hai Duong (North); and Cao Bang (Northern Border); and Ho Chi Minh City, Long An, Dong Nai, and Lam Dong (South), which supply the Greater Hanoi and Ho Chi Minh City Metropolitan markets. These four clusters correspond to the four ‘at risk’ production regions, as per the risk assessment performed during the project preparation.” (PAD, para 20; ICR, para 22). b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components Component 1: Upgrading Household-based Livestock Production and Market Integration (Appraisal estimate: US$66 million, of which IDA US$53.8 million, Government funding US$1.9 million and private sector funding US$10.4 million; AF estimate: total US$43.69, of which IDA US$36.02 million, Government funding US$1.42 million and private sector funding US$16.25 million; Actual total at ICR: US$110.89 million, of which total IDA disbursed US$86.81 million, Government funding US$5.25 million, and private sector contribution US$18.83 million). This component aimed to increase the competitiveness of household-based livestock production, enhance food safety and hygiene along the meat supply chain, and enhance the environmental management of livestock waste (PAD, para 22). Key activities under this component were divided into three sub-components focused on:  (i) the promotion of Good Animal Husbandry Practices (GAHP) among farmers, extension officers, and veterinary staff to increase vaccination coverage for common animal diseases, improve biosecurity and waste management, and enhance data collection and compliance monitoring on GAHP standards and certification  (ii) the piloting of Livestock Planning Zones (LPZs) -an existing or proposed land area that local authorities designate for livestock production promotion- to improve disease monitoring capacity and promote the establishment of farmer cooperatives and partnerships with the private sector for better knowledge exchange and harmonized application of improved waste management practices and biosecurity investments, and  (iii) the upgrading slaughterhouses and meat markets to improve their hygienic conditions and waste treatment and management. Page 2 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) Component 2: Strengthening Central-Level Livestock and Veterinary Services (Appraisal estimate: US$4.3 million, of which IDA US$4.2 million, Government funding US$0.2 million; AF estimate: total US$4.36, of which IDA US$4.27 million, Government funding US$0.9; Actual total at ICR US$7.17 million, of which total IDA disbursed US$7.15 million and Government funding of US$0.02 million). This component was to strengthen the Ministry of Agriculture and Rural Development (MARD) Departments of Livestock Production (DLP) and Animal Health (DAH) in developing and monitoring the implementation of animal health practices, including an update the GAHP guidelines and strategic studies, training of trainers in GAHP, piloting of innovative approaches, the provision of equipment and incremental costs to monitor livestock breed and feed quality, waste management and environmental compliance, as well as disease surveillance and prevention, meat inspection at the provincial level, and collection and monitoring of zoo sanitary and food safety data (PAD, para 24) Component 3: Project Management, Monitoring and Evaluation (Appraisal estimate: US$8.7 million, of which IDA US$7.2 million, Government funding US$1.4 million; AF estimate: total US$4.83, of which IDA US$2.59 million, Government funding US$2.24 million; Actual total at ICR US$13.53 million, of which US$11.71 million total IDA and Government funding US$2.87 million). This component entailed the project implementation and coordination of various government agencies at central, provincial, and district levels, as well as monitoring and evaluation (M&E) of project activities and impact (PAD, para 25). e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost and Financing. At appraisal in December 2009, the total project cost for was estimated at US$79.03 million, of which US$65.26 million was expected from IDA credit financing, US$10.37 million from the private sector (i.e. the cost-sharing contribution of grant beneficiaries), and US$3.4 million from co- financing by the Government of Vietnam (GoV) (PAD, para 26). In October 2015, an AF of US$54.68 was approved, of which 44.68 million was expected from IDA credit US$6.25 million from the private sector US$3.75 million from GoV co-financing. Therefore, the total project estimate (both original and AF) amounted to US$133.7 million, of which the IDA credit amount (both original and AF) added up to US$109.9 million and private sector financing (both original and AF) to US$16.65 million and Government funding (both original and AF) to US$7.25 million (ICR, page 2 and page 57; FA, IDA 4649 and IDA 5692). According to the ICR (page 2), the actual total project cost at ICR was US$132.4 (99% of total estimate). Borrower Contribution. At appraisal the borrower contribution of the GoM was estimated at US$3.4 million and an additional US$3.75 at AF, totaling US$7.25 million (PAD, para 26). At project closure, the ICR reports an actual total Government co-financing of US$8.32 million (ICR, paras 23-25). Dates and Restructuring. The Project was approved on September 22, 2009 and became effective on March 10, 2010. It closed on June 30, 2019. The original closing date was December 31, 2015, which was extended through the AF approved in October 2015 and a restructuring for a closing date extension of six months in January 2019. The project underwent one AF and two restructurings: Page 3 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723)  In October 2015, an AF of US$44.68 million was approved to scale-up GAHP capacity-building activities by introducing a group/cooperative-targeted approach and expansion of improved management activities for meat markets under Component 1. It also aimed to increase assistance to the GoV for legal and policy dialogue and reform such as the adoption of the food safety guidelines as national standards and accreditation of the laboratory network under Component 2.  In November 2018, a Level-2 restructuring to modify intermediary results indicators and targets[1].  In January 2019, a Level-2 restructuring to expand the closing date by six months.[2] The PDO was not changed with the restructurings or throughout the project implementation. [1] Specifically, two Intermediary Results Indicators (IRI) of the Results Framework (RF) were modified: lower the end target value of “Groups of livestock producer households in priority production areas having received GAHP certification” IRI from 1,200 to 700, and appoint in the IRI the National Center for Veterinary Hygiene Inspection No. I as National Reference Center instead of the initially planned Animal Breeding and Feeding Center. [2] Specifically to allow for the completion of ongoing project activities due to accumulated delays since 2016 when the project was not sufficiently allocated implementation budget because of constraints placed by the GoV on IDA budget allocations. 3. Relevance of Objectives Rationale The project objectives were relevant at the time of appraisal and project closing and continue to be at the time of this ICR Review. Agriculture -livestock production in particular- has been a priority to the GoV, as the growth of smallholder livestock production has played a significant role in reducing rural poverty since the early 2000s, particularly for the country’s 7 million pig and 8.3 million poultry small-scale household producers that at appraisal accounted for about 70 percent of the total livestock production. By project preparation, meat production had increased from about 2 million tons in 2000 to 3.3 million tons in 2007 and was still experiencing rapidly expanding demand from a growing middle class in Vietnam. Alignment with Government Strategy: At appraisal, the project objectives were strongly aligned with the GoV’s 2008 Strategy for the Development of the Livestock Sector and 2003 Ordinance on Food Safety that had the objectives to meet the increasing demand for safe, quality meat and improve the conditions for household livestock producers to expand more small-scale “backyard” to a more intensive, commercial production. They were also directly linked to the Strategy’s key components of livestock disease control, food safety and enhanced livestock waste treatment. At project completion, the project objectives remained highly relevant to the GoV’s development priorities of a sustainable and competitive agriculture and livestock sector. These objectives were supported by the 2011-2020 Socioeconomic Development Strategy and related 2016–2020 Socioeconomic Development Plan, which aimed at developing “a new environmentally sustainable growth model based on improved Page 4 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) productivity and competitiveness, and investments in infrastructure development” (ICR, para 36). To date, the project objectives continue to also aligned with the GoV’s food safety concerns, outlined in the 2011- 2020 National Strategy on Food Safety and related Vision to 2030 approved by the Prime Minister in 2012. Alignment with World Bank Strategy: At appraisal, the project objectives were also strongly linked to the World Bank’s Vietnam FY07-11 Country Partnership Strategy (CPS), particularly to Pillar 1 “Improved business environment” by strengthening the competitiveness for household livestock producers and slaughterhouses, Pillar 2 “Strengthening Social Inclusion” by making basic services accessible to rural smallholder livestock producers, and Pillar 3 “Strengthening Natural Resource and Environmental Management” by introducing livestock producers, slaughterhouses and meat markets to waste treatment technologies to reduce environmental pollution. Moreover, the World Bank had 15 years of agricultural sector experience in Vietnam, with previous lending projects such as the Vietnam Agricultural Diversification Project that also included a livestock component (PAD, paras 10-12 and 27). It further had supported MARD in the development of Vietnam’s Food Safety and Agricultural Health Action Plan and commissioned a FAO study on the Livestock Sector Competitiveness in Vietnam. (PAD, para 11). Complemented by similar projects in the region, such as the Animal Feed Project in China and the East Asia Regional Livestock Waste Management (PAD, para 12), the Bank had a significant sector and country expertise at the time of appraisal of LIFSAP. At completion, the PDO remained aligned with the current FY18-22 Country Partnership Framework (CPF) and its three Focus Areas of (1) Enable Inclusive Growth and Private Sector Participation, which includes the promotion of private sector participation and agribusiness development and enhancement of trade competitiveness for the agriculture sector, (2) Invest in people and knowledge, which includes the reduction in malnutrition and improved access to health services and (3) Ensure environmental sustainability and resilience, which includes the reduction in GHG emissions and strengthened natural resource management. Based on the above-mentioned information, relevance of objectives is rated high given the clear ambition of the PDO, central contribution of the project to the objectives of the CPS/CPF, its continued relevance at completion, and the connection of the PDO to the World Bank’s higher-level objectives of poverty reduction and enhancing shared prosperity. Rating Relevance TBL Rating High 4. Achievement of Objectives (Efficacy) EFFICACY_TBL OBJECTIVE 1 Objective Objective 1: Increase in Production Efficiency of Household-Based Livestock Producers Rationale Page 5 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) According to the PAD, output from household livestock production made up about 70 percent of total livestock sector production in Vietnam at the time of appraisal (PAD, para 19). These small-scale producers had limited knowledge of safe and modern production practices resulting from weak veterinary and extension services and applied reactive rather than preventive disease control measures (PAD, para 5). Hence, the PAD identified that a key measure to enhance production efficiency was to promote Good Animal Husbandry Practices (GAHP) among household livestock producers, in addition to upgrading the processing practices at slaughterhouses and meat markets (PAD, para 41). In order to do so effectively, there was a critical need to strengthen the capacity of the animal health and veterinary services at the central and local level. A 2007 and 2010 assessment of the Vietnamese veterinary services by the World Organisation for Animal Health (OIE) had revealed that field services for livestock diseases were weak and there was a need to update the veterinary curriculum (PAD, para 7). Theory of Change. The Vietnam Livestock Competitiveness and Food Safety Project (LIFSAP) was designed to address the above-mentioned challenges and needs. Objective 1 was based on the Theory of Change (ToC) that supporting institutional capacity-building to MARD’s responsible departments (Department of Livestock Production -DLP- and Department of Animal Health -DAH) in updating GAHP standards and developing guidelines and regulations in various areas[1], combined with the modernization of equipment and financing of livestock breed and feed monitoring as well as disease surveillance capacity[2], would lead to enhanced veterinary and extension services. As a result, it was expected that the provision of these enhanced services to livestock producers would generate their uptake of GAHP, and combined with support to form GAHP collaborative groups, lead to greater farm-level production efficiency by shortening fattening times, increasing herds/flocks and reducing animal mortality rates. Similarly, it was expected that improved veterinary and inspection services would lead to higher compliance rates of livestock processors with national standards and acquisition of relevant certifications.[3] The outputs and outcomes supporting Objective 1 of are described in the following: Outputs (based on ICR page 18 and Annex 1)  232 (target: 100) GAHP collaborative groups were formed, comprised of 1,217 producers (Target: exceeded).  19 (target: 15) GAHP cooperatives were established, comprised of 552 producers (Target: exceeded).  25,472 (target: 23,107) farmers were reached with agricultural assets or services (Target: exceeded).  26,312 (target: 25,000) farmers adopting improved agricultural technology (Target: exceeded).  4,628 (no target) agricultural and veterinary extension workers were trained in GAHP standards and certification, production efficiency, disease control, prevention and reporting systems, livestock waste management, and food safety.  22,906 (no target) household livestock producers (21,983 pig and 923 poultry) were trained in GAHP.  43,076 additional (no target as indirect beneficiares) farming households outside the project benefited from GAHP trainings, an indication of significant spillover effects resulting from the project and increased capacity of the DLP and DAH workforce (ICR, para 45).  373 (target: 350) slaughterhouses (303 small and 70 medium to large) were upgraded and inspected (Target: exceeded).  572 (target: 500) meat markets comprised of 20,538 counters, were upgraded and inspected (Target: exceeded).  30 (target: 30) slaughterhouses obtained certification for Good Manufacturing Practices (GMP), Good Hygienic Practices (GHP) and Hazard Analysis Critical Control Point (HACCP) (Target met). Page 6 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723)  Vietnam’s undergraduate veterinary curriculum was amended to include courses on food safety along the livestock value chain, addressing a weakness identified by the OIE assessment (Target: met)  10,665 (no target) veterinary staff supervising slaughterhouses and meat markets were trained on regulatory enforcement and integrated risk management and meat inspection. Outcomes Improved Animal Health and Veterinary Services. The ICR provided evidence that the project-supported trainings of veterinary staff and extension officers enhanced the quality of support to livestock producers (see Outputs), and this led to improved outcomes in animal health and veterinary services. Att project completion, 715 groups of livestock producers had received GAHP certification – which require the proven uptake of specific technologies and practices and which are due to regular inspection-, demonstrating a significant compliance with GAHP standards (Target: exceeded, Target: 700). Moreover, fungal contamination in animal feed of project-supported household producers was reduced (no aflatoxin B1 was found - considered the most dangerous naturally occurring toxin with carcinogenic effect on human and animals). This in an indication of producers’ heightened awareness and knowledge of animal feed treatment, which can be related to improved extension and veterinary support in epidemiology, data recording and disease monitoring – all areas supported by the project. A shortcoming of the ICR is that it does not present the level of aflatoxin B1 in livestock feed before the project. Furthermore, the vaccination coverage for common diseases in animals owned by household producers increased to 93.7 percent by project completion (higher than the target of 90.8). Also, the project led to increased sero-surveillance monitoring capacity[4] for the most common diseases of pigs and poultry, allowing for a more effective determination of disease transmission and population immunity, in particular with regard to vaccine-preventable diseases (preventative rather than reactive disease control). This reduced contamination of animal feed and the increased vaccination rates among project-supported livestock farms can be linked to the ICR’s evidence on the reduced rate in animal mortality: at project completion, mortality rate in pigs had reduced from 15 percent to 10 percent (below the target of 11.8 percent) and similarly in chickens from 41 percent to 13.9 percent (significantly below the target of 29 percent). More details and further changes to livestock production efficiency and animal health are described in the next section. Improved Livestock Production and Processing Practices. The project also directly supported the adoption of improved practices in pig and poultry production (on-farm) and processing (off-farm at slaughterhouses and meat markets): At farm level, the project led to the formation of 232 so-called GAHP collaborative groups and 19 GAHP cooperatives , complementing the GAHP training targeting household livestock producers. These establishment of these group and cooperatives had the objective to enhance knowledge exchange and harmonize the adoption of improved practices, but also to increase producers’ purchasing and bargaining power with input providers to reduce the cost of animal feed as well as with buyers to secure market offtake. Beyond GAHP, producers received training and demonstrations on feed quality and balancing and on-farm biosecurity measures, among other topics. A shortcoming of the ICR is that it does not provide evidence on the adoption rate of the practices among the producers taught in these trainings. However, the reported outcomes of shortened livestock fattening periods and increased flock/herd sizes described in more detail below can be affiliated with improved on-farm technical skills and practices promoted by LIFSAP. Supplementing these trainings and addressing producers’ constraints to financial resources, the project also provided matching grants for investments in animal health (leading to the construction of 17,493 biogas digesters (ICR, page 67 para 15) and 142 post-biogas effluent treatment schemes (ICR, para 85), 93.7% vaccination coverage for common animal diseases (ICR, para 42), etc.), as well as the construction of Page 7 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) 1,608 composting facilities, and slurry treatment to increase their production efficiency and reducing environmental pollution from livestock production (outcomes on the latter are described under Objective 2). At processing level, the project supported upgrading and improvement of 303 small slaughterhouses (i.e., slaughtering 10 to 30 pigs per day) and 70 medium to large slaughterhouses (i.e., slaughtering more than 30 pigs per day) and 572 meat markets (see Outputs). This promoted their adoption of national standards to enhance hygiene, food safety, and waste management practices of meat slaughtering, butchering and meat handling. It specifically financed equipment necessary to comply with the national standards and provided meat inspectors, butchers and middlemen with training on these standards and guidance on how to obtain certification for Good Manufacturing Practices (GMP), Good Hygienic Practices (GHP) and Hazard Analysis Critical Control Point (HACCP). The ICR provides outcome evidence that the project-supported activities led to improved efficiency in livestock production and processing: at project completion, project-supported farms had a lower livestock mortality rate than at baseline, as captured by the respective PDO indicator. Specifically, for pigs the mortality rate reduced from 15 percent (2010 baseline) to 11.8 percent (in 2015) and 10 percent at project completion, meeting the target of 10 percent. Similarly, for poultry the mortality rate dropped from 41 percent (2010 baseline) to 31 percent (in 2015) to 13.9 percent at project completion, significantly exceeding the target of 29 percent) Other outcomes on production efficiency measured by PDO indicators were (i) livestock fattening periods which fell respectively by 14 and 15 percent for pigs and poultry in line with the target values, and that (ii) pig herd sizes increased from 25.6 animals per herd (2010 baseline) to 31 animals (in 2015) to 40 animals (target: 40) at project completion; poultry flock sizes doubled from 935 animals per flock (2010 baseline) to 1,400 animals (in 2015) to 1,825 animals (target: 1,800) at project completion. Moreover, the ICR described how GAHP collaborative groups and cooperatives increased production efficiency through joint purchases from input providers that led to 3 to 5 percent costs savings in animal feed expenses (ICR, page 19). It also stated that t links with buyers were established but does not provide details (ICR, page 19). Furthermore, at project completion, all slaughterhouses and meat markets operated in line with design and procedure requirements, and 30 of the slaughterhouses supported by the project had received certification for GMP, GHP, and HACCP, achieving the project target of 30 and demonstrating compliance with certification standards related to food safety and disease control. Piloting Livestock Production Zones (LPZs). The project piloted the concept of LPZs and implemented one LPZ scheme – described by the ICR as “a miniature and enhanced GAHP zone” (ICR, para 42). The project supported consultant services for spatial design planning of the LPZ, construction of basic public infrastructure (small access roads, electricity and water supply systems), improved biosecurity and veterinary services, as well as investments in livestock waste management and environmental protection (such as biogas digesters at the farm level or common central lagoons or pipe systems at the community level). The pilot scheme was to test if the LPZs’ geographic concentration of livestock activities, with a more harmonized application of (GAHP) production practices and access to shared services/common facilities would lead to higher productivity and competitiveness of livestock producers (and ultimately be a potential tool to incentivize livestock producers to relocate from urban to rural areas to reduce pollution and human health risks.) The ICR provided only limited information on outcomes of the LPZ pilot, referring to general benefits from “ease of extension and veterinary services”, common approaches and facilities, and deeper GAHP harmonization (ICR, paragraph 42c). One specific measured outcome demonstrated a higher efficiency of LPZ in terms of the proportion of vaccination for common diseases of pigs reaching 95.8 percent (compared to 93.7 percent in non-LPZ areas). At appraisal, three LPZ pilot schemes were planned (PAD, paragraph 57), but the ICR explained that environmental, economic, and technical risks had not been sufficiently considered, Page 8 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) leading to the piloting of one scheme “through which such risks could be re-examined further before scaling up” (ICR, paragraph 77). It further stated that the experience of the pilot LPZ would inform the new Livestock Development Strategy and has been reflected in the formulation of the Livestock Law and the Veterinary Law. Summary. Based on the above-mentioned assessment, the evidence in the ICR pointed to the success of the project in increasing the production efficiency of household-based livestock producers, despite some shortcomings in the demonstration of guideline dissemination and (continued) adoption of improved practices among the producers. Hence, the efficacy of this outcome for Objective 1 is rated Substantial. [1] Specifically, guidelines and regulations for (i) biosecurity, (ii) livestock waste management, (iii) quality of livestock feeds, (iv) sale and use of feed additives, (v) hygiene standards, and (vi) meat inspection. ICR Box 1 (page 20) provides some details on the GAHP handbook developed by the project. [2] This included carrying out serological surveys, which provide estimates of antibody levels against vaccine preventable diseases (VPDs) and is considered the gold standard for measuring population immunity due to past infection or vaccination. It is an important component of disease surveillance (Source: http://ncirs.org.au/our-work/serosurveillance) [3] The most relevant certifications were for Good Manufacturing Practices (GMP), Good Hygienic Practices (GHP) and Hazard Analysis Critical Control Point (HACCP). [4] Sero-surveillance provides estimates of antibody levels against vaccine preventable diseases (VPDs) and is considered the gold standard for measuring population immunity due to past infection or vaccination. It is an important component of disease surveillance (Source: http://ncirs.org.au/our-work/serosurveillance) Rating Substantial OBJECTIVE 2 Objective Objective 2: Reduce the Environmental Impact of Livestock Production, Processing and Marketing Rationale Theory of Change. The objectives of increasing production efficiency, reducing environmental impact, and improve food safety in the livestock value chain are strongly interlinked and share elements of the same ToC. For Objective 2 specifically, the provision of training to producers on improved waste and biosecurity measures -in combination with the provision of matching grants for the construction of biodigesters, composting facilities, slurry treatment, and implementation of biosecurity measures at the farm and communal level- were expected to not only increase production efficiency but also reduce the environmental impact of their livestock production. Similarly, the supported upgrading of small, medium and large slaughterhouses and meat market in their respective waste treatment and management practices was expected to lessen adverse environmental impacts. Moreover, helping medium and large slaughterhouses and meat markets to Page 9 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) comply with national environmental standards (and pass related inspections) was assumed to further drive a reduction in environmental impact. Many of the activities and outputs described under Objective 1 (capacity-building of producers and animal health and veterinary service providers, matching grants for improved livestock production, upgrading of slaughterhouses and meat markets) also support the achievement of Objective 2 and 3. The outputs and outcomes specifically supporting Objective 2 are described in the following: Outputs (based on ICR page 18 and Annex 1)  25,172 (target: 25,000) households adopted improved waste management measures, from 2015 baseline of 10,999 households (Target met).  6,371 (no target) households received technical assistance on upgrading their waste treatment system to reduce environmental pollution.  17,493 (no target, but PAD economic analysis assumed 3,000) biogas digesters were constructed.  1,608 (no target) composting pits were constructed.  142 (no target) post-biogas effluent treatment schemes were constructed.  303 (target: 310) small slaughterhouses upgraded in line with improved waste management practices, from 2015 baseline of 193 slaughterhouses (Target: not met).  70 (target: 40) medium and large slaughterhouses meet national environment standards, from 2015 baseline of 25 slaughterhouses (Target exceeded).  572 (target: 500) meat markets meet national environment standards, from 2015 baseline of 378 markets (Target exceeded). Outcomes Improved Environmental Practices in Livestock Household Production. The Project supported capacity- building activities for extension and veterinary staff and related training of livestock producers included technologies for improved livestock production waste treatment and management at the farm level. Furthermore, the upgraded the Department of Natural Resources and Environmental by financing equipment and incremental costs for monitoring of livestock waste and environmental compliance (at different levels of the value chain). The ICR claimed that these measures led to a reduction in methane emissions from manure as well as in GHG emissions from less use of traditional fuels and chemical fertilizers (replaced by bio-slurry). The ICR provided some evidence to support these statements, but the evidence was based on estimates in 2015 (at completion of phase 1 before the AF) expecting that the introduction of biogas digesters would reduce CO2 emissions (baseline greenhouse gas) from approximately 442 tons of manure per day, corresponding to 40,800 tons of CO2 per year (ICR, para 19). It was also expected that the use of biogas from project-supported digesters for cooking would reduce methane emissions, but the project-specific effects were not quantified. The ICR, however, did report that the project used mitigation measures to minimize negative environmental impacts at key stages of activities (design, bidding, construction, and implementation): The introduction of GAHP and construction of biogas digesters and post-biogas effluent treatment schemes reduced pollution levels generated by farms, slaughterhouses, and wet markets. The ICR reported that “environmental performance at 100 percent of the 235 slaughterhouses and 97.8 percent of the 378 wet markets supported by the project has been improved” (ICR, para 85) and met Vietnamese environmental standards by project closure. Moreover, the PAD stated that at appraisal only a limited number of household livestock farmers had waste treatment technologies, leading to groundwater and Page 10 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) surface water pollution at farm level. The ICR does not provide any evidence in the reduction of groundwater or surface water pollution. Compliance with Environmental Standards in Livestock Processing and Marketing. At processing level, the project supported 373 slaughterhouses to upgrade sanitation and veterinary procedures including improved waste treatment and management (Target exceeded, Target: 350) . As a result, at project completion, 70 medium to large slaughterhouses and 303 small slaughterhouses complied with national environmental standards. The PDO indicator target of 40 for medium to large slaughterhouses was largely exceeded, given the GoV priority for these facilities. A review of the project document revea that during implementation the Bank team convinced the GoV of the importance of environmental compliance by the large number of small slaughterhouses, leading to the near achievement of the end-of-project target of 310 small slaughterhouses. The ICR stated that the project “contributed to improved quality of posttreatment wastewater discharged into the environment, thereby reducing environmental pollution” (ICR, para 43), but the report did not provide specific evidence. At marketing level, the project supported upgrades -including in waste and wastewater treatment- of 572 meat markets comprised of over 20,000 counters, which led to compliance of all supported markets with national environmental standards, exceeding the PDO indicator target of 500 meat markets. Similar to slaughterhouses, the ICR refers to significant improvements in wastewater of these meat markets but does not state specific measures. Summary. Based on the above-mentioned assessment, the ICR provides evidence that the project has led to a reduction in the environmental impact of the livestock value chain (at the production, processing and marketing levels) and the compliance of project-supported beneficiaries with national environmental standards as targeted, despite some shortcoming in providing quantitative evidence of the specific effects. Overall, the efficacy of this outcome for Objective 2 is rated Substantial. Rating Substantial OBJECTIVE 3 Objective Objective 3: Improve Food Safety in Livestock Product Supply Chains (mainly meat) Rationale Theory of Change. The objectives of increasing production efficiency, reducing environmental impact, and improve food safety in the livestock value chain are strongly interlinked and share elements of the same ToC. For Objective 3 specifically, the project support in training livestock producers in GAHP -including food safety measures- and providing them with matching grants for production upgrades was expected to increase food safety at the production/farm level. The complementary training and financing to upgrade the meat handling equipment of small, medium and large slaughterhouses and meat markets and support to meet food quality and food safety standards in combination with strengthening meat inspection services was expected to enhance food safety at the off-farm segments of the livestock value chain. Many of the activities and outputs Page 11 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) described under Objective 1 and 2 also support the achievement of Objective 3. The outputs and outcomes specifically supporting Objective 3 are described in the following: Outputs (based on ICR page 18 and Annex 1)  National Center for Veterinary Hygiene Inspection No. I appointed to National Reference Center.  Food safety good practices included in the national veterinary curriculum.  70 (target: 40) medium and large slaughterhouses met national environment standards, from 2015 baseline of 25 slaughterhouses (Target: Exceeded).  373 (target: 350) small slaughterhouses upgraded to produce meat of improved quality and safety, from 2015 baseline of 235 slaughterhouses (Target: exceeded).  572 (target: 500) meat markets met national environment standards, from 2015 baseline of 378 markets (Target: exceeded). Outcomes Improved Food Safety Practices in Livestock Household Production. Food safety is an important element for competitiveness, as it determines whether food attributes comply with hygienic procedures/standards, increasingly demanded by (in particular urban) consumers. At farm level, the project supported livestock producers to improve hygienic practices and knowledge of food safety standards in the livestock value chain. The ICR reported farm-level outcomes based on 204 meat samples of beneficiary pig household producers with 100 percent negative detection of hormones or banned substances (ICR, page 69, para 23). Tests of animal feed at project-supported farms also had negative results. Compliance with National Food Safety Standards in Livestock Processing and Marketing. At institutional level, LIFSAP financed equipment and incremental costs at the Department of Health (and its provincial sub-divisions) to improve the quality of its services in disease surveillance/prevention and meat inspection at the provincial level as well as upgrade its capacity for monitoring of zoo sanitary and food safety data. Moreover, the National Center for Veterinary Hygiene Inspection No. I was appointed Vietnam’s National Reference Center with support of the project. With this support on food safety monitoring capacity, results on compliance with food safety standards were more reliable collected and analyzed than before. At processing level, the project support to upgrade equipment and increasing awareness on procedures for safe and hygienic slaughtering and meat handling led to a reduced and low microbial contamination on carcass samples and slaughter tools, according to the ICR. At project completion, all 373 (target: 350) project- supported slaughterhouses had upgraded their meat handling to improved quality and safety. Moreover, 30 (target: 30) slaughterhouses complied with national food safety/hygiene standards and received certification for Good Manufacturing Practices (GMP), Good Hygienic Practices (GHP) and Hazard Analysis Critical Control Point (HACCP) - the ICR does not specify if these are generally medium to large slaughterhouses. The ICR estimates that about 2.6 million consumers are indirect beneficiaries of these improvements in food safety at slaughterhouses (ICR, page 69, para 24). At marketing level, all 572 project-supported meat markets met the national meat quality and safety standards at project completion (target: 500). In addition to this PDO indicator, food safety monitoring at project- supported meat markets demonstrates that about 90 percent of pork meat samples met the microbial criteria for both Escherichia coli and Salmonella, and that 92 percent of chicken meat samples met the criteria for Salmonella and about 75 the E. coli requirement (ICR, page 69, para 25). The ICR estimates that about 1.8 million consumers are indirect beneficiaries of these improvements in food safety at meat markets (ICR, page Page 12 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) 69, para 25). This is good evidence of project results beyond the PDO indicators, which would be more meaningful if the ICR had provided the compliance levels at baseline. Summary. Based on the above-mentioned assessment, the evidence in the ICR pointed to the success of the project in enhancing the food safety in the livestock supply chain. Hence, the efficacy of this outcome for Objective 3 is rated Substantial. Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The achievement of Objective 1 -to increase production efficiency of household-based livestock producers - is rated Substantial as a result of positive impacts reported by the ICR, despite some shortcomings in the demonstration of continued adoption of improved practices. The achievement of Objective 2 -to reduce environmental impact of livestock production, processing and marketing- is rated Substantial due to meeting or exceeding indicators and targets, in particular the compliance of more than targeted slaughterhouses and meat markets with national environment standards. The achievement of Objective 3 -to improve food safety in livestock product supply chains - is rated Substantial as a result of sufficient outcome evidence, with more than the targeted slaughterhouses having upgraded their meat handling and received certification for Good Manufacturing Practices (GMP), Good Hygienic Practices (GHP) and Hazard Analysis Critical Control Point (HACCP). Based on this, and given that all targeted project outcomes were either fully achieved or exceeded at project completion, the overall efficacy is rated Substantial despite some shortcomings. Overall Efficacy Rating Substantial 5. Efficiency Scope of Analysis. At appraisal, a conventional cost-benefit analysis was carried out to estimate the Economic Internal Rate of Return (EIRR), using farm models for producers who were considered representative of the project beneficiary producers. The analysis was based on quantifiable investments related to project interventions at farm level (investments in production efficiency, supporting feed cost reductions from enhanced feed conversion and reduced livestock mortality) and quantifiable food safety and environmental improvements (prevented food poisoning treatment costs and worker illness productivity losses, and reduced fossil energy costs due to biogas use). The overall ex-ante EIRR was estimated at 17.4 percent with a Net Present Value (NPV) of US$19.9 million, assuming a discount rate of 12 percent and project lifetime of 25 years. Page 13 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) Economic Analysis. At closing, the ICR calculated the EIRR based on production survey data collected from project-supported pig and poultry producers and control groups (using incremental financial costs and benefits analysis to analyze the “with project” and “without project” scenario) to estimate quantifiable production benefits in terms of net profit per kg meat sold and economic benefits from biogas digesters (using cost-benefit analysis). The overall ex-post EIRR was estimated at 23.4 percent with a Net Present Value (NPV) of US$287 million, assuming a discount rate of 9 percent[1] and project lifetime of 25 years. The EIRR results remain robust also at different sensitivity scenarios of increased and/or reduced benefits: for example, at a reduction of 20% in benefits, the EIRR is estimated at 17.4%. Overall, the ex-post methodology appears sound. The PAD and ICR highlight that several economic benefits were not considered in the economic and financial analysis (both ex-ante and ex-post), which would lead to even higher EIRR. These benefits are related to increased food safety at the farm level (producers), processing level (slaughterhouses), and marketing level (meat markets), for which the project did not (plan to) collect data. Similarly, income increases of slaughterhouse and meat market operators were not considered in the ex-post analysis. Operational Efficiency. Regarding operational efficiency, the ICR reported that “the project team also demonstrated a higher level of implementation efficiency as they addressed emerging implementation challenges in a time and effective manner” (ICR, paragraph 49). However, actual costs for project management (Component 3) were 20 percent higher than assumed (US$11.71 million compared to US$9.79 million at appraisal/AF), indicating a lower than expected operational efficiency. Given the adequacy of the economic analysis, solid internal rates of return, and robust results from the sensitivity analysis, the project's efficiency is rated Substantial - despite shortcomings in operational efficiency. [1] The ICR states that the discount rate of 9 percent had been used for World Bank-funded investment projects in Vietnam. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 93.00 Appraisal  17.40  Not Applicable 93.00 ICR Estimate  23.40  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. Page 14 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) 6. Outcome Relevance of Objectives is rated High, as LIFSAP objectives were relevant to both the former and current national priorities and Bank sector strategies. The achievement of the PDO to increase production efficiency of household-based livestock producers, reduce environmental impact of livestock production, processing and marketing, and improve food safety in livestock product supply chains (mainly meat) in the project provinces is rated Substantial, given that all targeted project outcomes were either fully achieved or exceeded at project completion and additional evidence has been demonstrated for each objective. Efficiency is rated Substantial, given the project’s sound economic rate of return, robustness in the sensitivity analysis, and adequate methodology applied. As a result, the combined outcome rating is rated as Satisfactory. a. Outcome Rating Satisfactory 7. Risk to Development Outcome Two main risks to sustaining the DO in the longer term are:  (i) maintaining the application of GAHP among project beneficiaries and ensuring scale-up to the national level for widespread improvements in food safety, given their dependence on adequate monitoring and inspection capacity (both technical and financial) of the sector agencies. Hence, there is a need for continued budget to support activities established by LIFSAP, especially for district- and provincial- level monitoring capacity. These challenges can be expected to be (at least partially) addressed through a follow-up operation building on LIFSAP, which is mentioned as interest by the GoV in the ICR.[1]  (ii) the risk and unpredictability of animal disease and its detriment effects on the livestock sector, animal (and sometimes also human) health, and economic condition of producers and related supply chain actors. At the time of the ICR, Vietnam was experiencing an outbreak of African Swine Fever (AFS), which is highly resistant and led to a substantial mortality in pigs. Hence, especially smallholder livestock producers of pig were at risk of economic loss and potential exit of the livestock business. The ICR contended, however, that “relative resilience of project areas and beneficiaries, compared to other areas in the country, demonstrates the value of GAHP and increased biosecurity” (ICR, para 48). Project beneficiaries had lower incidences of AFS, as in Hanoi where data from October 2019 showed that “SF-affected households applying GAHP was lower (21.3 percent) compared to the overall percentage (38.9 percent” (ICR, para 99). [1] At the time of this Review, an estimated US$150million IBRD loan for the Vietnam Agri-Food Safety Project (AFSP, P171187) with the objective to “improve food safety management systems and infrastructure in targeted areas and reduce food safety risks in selected value chains” is in the lending pipeline (with expected Board approval date in December 2020). If the AFSP becomes effective, the sustainability of outcomes from LIFSAP are more likely to be sustained and scaled-up. Furthermore, sustainability of GAHP technical skills of household livestock producers and the adoption and continued availability of equipment Page 15 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) and practices to increase hygiene and waste management practices of slaughterhouses and meat markets, reinforced by an increased demand of growing middle-class consumers for safe food and quality standards. Furthermore, institutional capacity was built at different levels: (i) at the farm and communal level, the establishment of livestock planning zones (LPZs) and related infrastructure enhanced data recording and disease monitoring for better waste management and biosecurity, (ii) at the provincial and district level, livestock services were capacitated in improved monitoring of feed quality and environmental compliance and conducted in a pilot livestock identification system for pigs, and (iii) at the central level, support for livestock and veterinary services led to updated GAHP guidelines and training, improved data collection and monitoring capacity of livestock breed and feed quality as well as sanitary and food safety data and related waste management and environmental impact. 8. Assessment of Bank Performance a. Quality-at-Entry The World Bank Task Team ensured that the design of the LIFSAP and its objectives were fully aligned with the GoV’s agricultural sector priorities and the World Bank’s strategic involvement in Vietnam at the time of project preparation and appraisal. The team built the project design on the Bank’s long-term involvement in Vietnam’s agriculture sector and incorporated lessons from relevant Bank agricultural projects and relevant analytical studies on Vietnam’s livestock sector to address key constraints to livestock competitiveness and food safety. Given this experience, it is surprising that the project did not adequately anticipate the time needed/potential initial delays to build capacity of technical staff given the introduction of new concepts (such as LPZ, which was adapted to only one instead of three pilots during implementation) (ICR, para 77). Overall, component design was in line with the PDO, adequate for the capacity of the implementing agencies, and funded appropriately with the original activities of the project. The project design also incorporated lessons from previous sector projects in Vietnam, such as the importance of supporting adoption of improved livestock production technologies, need for private-public partnerships in veterinary service delivery to ensure effective veterinary coverage, an taking a decentralized approach to ensure ownership and capacity-building at provincial levels (PAD, para 27) Moreover, the World Bank Task Team put emphasis on designing an effective coordination mechanism using existing systems to generate institutional capacity-building for the time after project completion and ensure ownership by the GoV. The anticipated critical risks and respective mitigation measures described in the PAD were adequate, such as addressing (i) the risk of producer reluctance to adopt GAHP with intensive awareness raising and demonstrations, (ii) the risk of disease outbreaks with enhanced disease control reporting and procedural measures, (iii) the risk of unassertive meat inspectors with intensive training and external monitoring (PAD, page 12). Quality-at-Entry Rating Satisfactory Page 16 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) b. Quality of supervision The project was implemented largely according to its design, but it experienced significant initial delays and capacity constraints, which the ICR relateed to the novel approaches introduced by the project (ICR, para 71), such as GAHP, LPZs, biosecurity measures, food safety practices, and livestock waste management. The World Bank Task team was responsive in addressing these factors, for example by adapting the plans for the LPZ pilot to one instead of three to deeply examine the related environmental, economic and technical risks (ICR, para 77). Other factors were outside the control of the World Bank Task Team, such as the livestock market fluctuations due to food scares, restrictions placed by the GoV on IDA budget allocations, or the African Swine Fever (ASF) outbreak affecting pig production (ICR, para 74-76).. Project performance improved after the Mid-Term Review (MTR) – from temporary moderately unsatisfactory performance to moderately satisfactory after six months and later satisfactory performance. At completion, the project fully achieved or exceeded all targets in the results framework (RF). Supervision missions took place on a regular basis with adequate expertise in staffing, and included site visits and interactions with various project stakeholders. In addition, especially at early implementation the World Bank Task Team provided technical advice to the PCU when needed and brought international expertise on specific issues requested by the GoV (in collaboration with FAO). Task Team leadership was continuous with one change in Task Team Leadership at AF, which ensured continuity and a responsiveness to the client implementing agencies (ICR, para 94). The ICR and implementation status reports suggested that the World Bank Task Team was generally proactive in responding to new issues during supervision. For example, to address the significant delays in the first years of project implementation (ICR, para 33 and 89), the Mid-Term Review (MTR) recommendations were taken seriously and led to adjustments in the Results Framework, and the team did not shy away from restructurings when necessary. Quality of Supervision Rating Satisfactory Overall Bank Performance Rating Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design Theory of Change. Overall, LIFSAP’s M&E design and strategy was well-developed. A Theory of Change (ToC) was not (required to be) prepared at appraisal, but the ICR presented an ex-post ToC. This ToC is logical and convincing:  at farm level, the adoption of GAHP and improved practices with complementary on-farm infrastructure investments supported through matching grants and piloting of LPZ for improved biosecurity and waste management were expected to lead to better efficiency and productivity of household livestock production; Page 17 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723)  at processing level, the upgrading and improvement of slaughterhouses and meat markets as well as capacity-building of inspection services was expected to enhance hygiene, food safety, and waste management of meat slaughtering, butchering and meat handling throughout the value chain and to reduce the risk of animal disease outbreaks and unsafe food;  - at institutional level, capacity-building and training of both central and provincial animal health and veterinary services in GAHP standards and procedures, development and review of guidelines and regulations for on-farm and off-farm biosecurity, livestock waste management, hygiene standards, meat inspection, and certification was expected to lead to better veterinary services to livestock producers, improved animal health, and enhanced surveillance performance of the services. The project’s three intervention levels (farm, processing, institutional) are strongly inter- linked and share the objectives of more efficient, environmentally-conscious, and safer livestock production and processing and ultimately the reduction in animal disease in the livestock chain. Results Framework. Overall, the mix of PDO and intermediary outcome and output indicators for the different targeted livestock sub-sectors (poultry, pig) in the Results Framework (RF) design seems appropriate. Outcome indicators were focused on changes in on-farm production measures, levels of vaccinations and inspections, and producers and processors/markets meeting national environmental and food quality/safety standards. Output indicators were focused on establishment of standards and guidelines, trainings conducted, technologies promoted and adopted among farmers and processors. Also, the project had a large number of PDO (sub-) indicators in the RF given its three-fold PDO:  For the production efficiency objective, it identified three measures as (i) reducing livestock mortality rates, (ii) shortening of the livestock fattening period, and (iii) increasing the size of flocks and herds. This constellation of on-farm production measures is adequate and aligned with the appraisal context of food safety concerns and increased domestic demand for meat by a growing middle- class.  For the environmental impact objective, it also identified three measures to disaggregate between target beneficiary groups as the percentage of (i) livestock producers, (ii) slaughterhouses, and (iii) meat markets meeting national environmental standards. This is also considered adequate.  For the food safety objective the PDO indicators measured the percentage of beneficiary (i) slaughterhouses and (ii) meat markets operating at national food safety standards. These measures are adequate but could have been complemented by more intermediate results indicators, such as the specific equipment/capacity provided to these facilities and their use. The ICR rightly highlighted shortcomings in the appraisal RF in the partially unclear definitions of indicators and the sole dependence on the project’s M&E system without complementary data sources or an impact evaluation (ICR, para 78). The shortcomings on indicator formulations and definitions were largely adjusted during the AF preparations. But on the diversification of M&E data sources, the PAD actually does state that Component 3 would “finance technical assistance (TA) to carry out independent evaluations at mid term review and after the project is completed”, which were not conducted during project implementation (to be discussed more under M&E Implementation). M&E Arrangements. LIFSAP’s M&E design was based on the use of the national system, relying on data to be regularly collected by GAHP officers and veterinary services from farmers, slaughterhouses and meat markets. The PAD described that LIFSAP would M&E arrangements would be in line with the GoV’s Page 18 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) Aligned Monitoring Tool (AMT) established by the Ministry of Planning and Investment and key donors and finance “a simple” Monitoring Information System (MIS) on the basis of the AMT. Relying on the national systems is generally welcome to increase capacity and enhance changes of continuity after project completion, under the assumption that sufficient training is provided to ensure quality. According to the ICR, M&E staffing at the national and provincial level was adequate and M&E reporting was easy due to the project developing user-friendly reporting formats. b. M&E Implementation M&E Capacity. The PAD outlined M&E responsibilities in terms of data collection and monitoring across the various agencies involved in LIFSAP and planned for a consultant to provide technical assistance to the PCU for data collection planning. The ICR stated that the staffing for M&E was adequate and issues were addressed in a responsive and timely fashion (more details under Changes to RF/Restructurings). Yet, it also highlighted that the results monitoring capacity of the project implementation agencies had to be improved, especially at the provincial level/PPMUs. To address this, the project conducted various training sessions to PPMUs, which increased the quality of data collection and reporting over time. The project’s M&E rating improved from moderately satisfactory to satisfactory in the last two years of implementation. M&E Data Collection. Despite plans for establishing “a simple MIS” at appraisal, the ICR criticized that LIFSAP did not develop f a simple web-based MIS for key results indicators. This system would have allowed for the collection of and access to real time information. The MIS system would have generated time savings. Furthermore, a review of the RF at project completion reveals the lack of baseline values for the majority of results indicators (except the PDO indicators on production efficiency). Similarly, no (independent) impact assessment was conducted despite financing plans for technical assistance for “necessary surveys and independent evaluation reports” (PAD, page 42 para 10). The ICR contended that these shortcomings were addressed by the generation of “endline to baseline comparative data” using the project M&E system to also collect data on control groups to assess project impacts. However, it also explained that its overall M&E Quality rating is modest “due to the lack of the independent impact evaluation” (ICR, para 83) and it generated a lesson on the need for a robust M&E design and collection of at least two evaluations (ICR, para 103). Changes to the RF. The PDO did not change throughout project implementation, but a number of results indicators were amended for the AF to address some of the identified shortcomings in the M&E design. These changes were appropriate to provide clearer outcome measures (such as more precise wording or the clarification of applicability of national standards) and to adjust to the M&E capacity (such as the changes from measuring absolute versus percentage values). They also helped to demonstrate outcomes at a disaggregated level by type of livestock (pigs versus poultry). Some changes to the RF were already proposed during the 2013 MTR and noted in the MTR mission Aide Memoire, but no changes were implemented until the preparation of the AF in 2015. The ICR does not address why the 2013 MTR suggestions for the RF were not addressed and/or if the decision had been made to not move forward with them. Page 19 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) c. M&E Utilization The ICR briefly described how the M&E systems was utilized for project management or other purposes. It states that the M&E information was used to inform government decision-making on disease outbreak preparedness, rate of slaughterhouse and meat market upgrading, and the financing impact of project- supported biodigesters, among others. It is not described if the M&E system will be used or modified for the follow-on project. Overall, M&E Quality is rated Modest. This rating reflects the generally adequate M&E design, but shortcoming in the implementation to generate adequate baselines for several indicators and lack of a final assessment, as well as the limited utilization of M&E results. M&E Quality Rating Modest 10. Other Issues a. Safeguards Environmental Safeguards. The project was classified as Environmental Category B, as it was not expected to lead to any significant negative environmental changes. It triggered Environmental Assessment (EA) 4.01 and Pest Management 4.09 Operational Safeguard Policies. The PAD states that during project preparation an Environmental Management Framework (EMF) was prepared and found satisfactory by the Bank. However, no details on the EMF disclosure or public consultations with stakeholder in Vietnam are provided in the PAD or ICR. As mentioned in the Efficacy Section, the introduction of GAHP and construction of biogas digesters and post-biogas effluent treatment schemes reduced pollution levels generated by farms, slaughterhouses, and wet markets. While there is no detailed description on the monitoring and reporting practices by the Environment and Social Safeguard Officer of the Project Coordination Unit (PCU), the ICR highlights that the project built capacity at the responsible central and provincial institutions to better monitor and manage environmental compliance and livestock waste, using a comprehensive environmental quality monitoring database generated by the project. In addition, over 80,000 farmers received training to enhance awareness on GAHP, waste management, and biosecurity. The project was proactive in updating the EMF during implementation to comply with the Code Chapters of the World Organisation for Animal Health (OIE) and to react to environmental pollution resulting from the use of cleaning chemicals. According to the ICR and a review of the Implementation Status Reports (ISR), over time the environmental compliance improved from mostly moderately satisfactory to satisfactory throughout the last two years of project implementation. Social Safeguards. The project triggered the Indigenous Peoples 4.10 and the Involuntary Resettlement 4.12 Operational Safeguard Policies. In response, the Resettlement Policy Framework (RPF) and Ethnic Minority Policy Framework (EMPF) including the Resettlement Plan and Ethnic Minority Development Plan were updated and disclosed before appraisal. The ICR reports that the project led to no displacements and only minor land acquisitions with appropriate compensation and no complaints. It established a database of beneficiaries with disaggregated information on ethnicity, gender, and economic status, allowing to Page 20 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) demonstrate that ethnic minority people benefited more than expected from the project. Overall, no major complaints or grievances on environmental or social management were filed during project implementation. b. Fiduciary Compliance Financial Management. The PCU capacity for financial management (FM) was adequate and provided timely interim unqualified financial reports. Similarly, all annual financial audit reports were unqualified. FM supervision missions regularly confirmed the adequacy in FM staffing, accounting and internal control systems that consolidated FM reporting from 12 participating provinces and checked that disbursements were in line with the disbursement plan and guidelines. During project implementation, the FM rating improved from mainly moderately satisfactory to satisfactory ratings in the last 18 months of implementation, demonstrating capacity enhancements. Procurement. The PCU capacity for procurement was largely adequate after the implementation of actions agreed from the procurement capacity assessment conducted during project preparation. Good procurement planning and frequent technical support by the Bank procurement specialist led to a largely timely completion of planned procurement activities during implementation. Shortcomings were found mainly in procurement planning at the provincial level and some delays due to inadequate bid evaluation and approval processes. Procurement capacity and reporting at the central and provincial level improved over time, leading to satisfactory performance in the last 18 months of implementation. Overall, there were no major complaints during bidding processes and no mis-procurement occurred during project implementation. c. Unintended impacts (Positive or Negative) The ICR reports several unintended positive effects resulting from project activities, including (i) economic benefits for beneficiaries of biodigester investments, leading surplus gas used as fuel for cooking and water boiling with estimated savings of about US$15 to US$20 per month per household; (ii) unmeasured spillover effects of the GAHP promotion to neighboring non-beneficiary farmers who adopted GAHP; and (iii) unmeasured spillover effects of upgrading of markets for livestock traders that incentivized upgrading also among non-beneficiary fish and vegetables traders. Another unintended effect claimed by the project is its link to address antimicrobial resistance by promoting GAHP and thereby reducing the use of antibiotics. d. Other None 11. Ratings Page 21 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) Reason for Ratings ICR IEG Disagreements/Comment Outcome Satisfactory Satisfactory Bank Performance Satisfactory Satisfactory Quality of M&E Modest Modest Quality of ICR --- Substantial 12. Lessons The following lessons draw from the ICR with adjusted language: 1. Adjust (or phase) project interventions to stay relevant to country priorities and shape national strategies. The project implemented and tested several concepts that were new to Vietnam’s livestock sector (such as Good Animal Husbandry Practices -GAHP- or Livestock Production Zones -LPZ). While all concepts were relevant, the project adjusted the intensity and/or timing of implementing these according to country priorities to provide confidence and proof of concept of easier to convey concepts (such as GAHP) before moving on to pilot more complex and challenging concepts (such as LPZ). As such, it stayed relevant to country priorities, gained the trust of the government counterparts and other stakeholders, and consequently its results shape the development of the National Livestock Strategy and follow-on projects. 2. Invest in replicable models for scale-up. Projects should consider the replicability in terms of affordability of investments and routine operations and maintenance when selecting infrastructure upgrades and improvements. The ICR lessons state uncertainty around scale- up of the elaborate slaughterhouse model by the private sector after project completion. To maximize the probability of successful implementation models, projects have to consider private sector interest and financial ability for certain equipment/infrastructure, be prudent in the selection of supported infrastructure, and carefully monitor that procurement processes would be straightforwardly replicable by actors responsible for scale-up. 3. Incorporate long-term effects on food safety in project intervention design diseases calls for projects to anticipate and manage emerging issues, rather than responding only to immediate concerns. Projects should encourage client governments to put in place systems, practices, and knowledge that strengthen the resilience of the livestock production supply chain and fortify consumer trust and confidence that such systems are equipped to manage potential food safety hazards. 13. Assessment Recommended? No Page 22 of 23 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Livestock Competitiveness&Food Safety (P090723) 14. Comments on Quality of ICR The ICR is comprehensive and well-presented. It shows a good understanding of the technical content of the project activities and the logic of the project is explained well in the ex-post Theory of Change. It identifies and discusses critical issues and shortcomings, such as in the M&E design and implementation. The ICR also makes a good use of summary tables that underline its arguments and provide a good overview of project activities and findings. However, at times it provides detailed explanations on some issues such as Boxes on the GAHP Manual, beneficiary testimony, AFS outbreak, which are interesting but not essential and makes it a longer than the standard document. a. Quality of ICR Rating Substantial Page 23 of 23