Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005419 IMPLEMENTATION COMPLETION AND RESULTS REPORT TF-A5252 ON A SMALL GRANT IN THE AMOUNT OF USD (1.80) MILLION FROM THE JAPAN POLICY AND HUMAN RESOURCES DEVELOPMENT PROGRAM TO THE Socialist Republic of Vietnam FOR MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) June 28, 2021 Agriculture And Food Global Practice East Asia And Pacific Region Regional Vice President: Victoria Kwakwa Country Director: Carolyn Turk Regional Director: Benoit Bosquet Practice Manager: Dina Umali-Deininger Task Team Leader(s): Hardwick Tchale, Binh Thang Cao ICR Main Contributor: Ingrid Marie Pierre Mollard ABBREVIATIONS AND ACRONYMS AFF Agriculture, Forestry, And Fishery ARP Agricultural Restructuring Plan DARD Department of Agriculture and Rural Development DoP Department of Planning GAP Good Agricultural Practices GSO Government Office of Statistics IPSARD Institute of Policy and Strategy for Agriculture and Rural Development MARD Ministry of Agriculture and Rural Development M&E Monitoring and Evaluation MESMARD Monitoring and Evaluation in Support of Management in the Agricultural and Rural Development Sector MECARP M&E Capacity Building for Agricultural Restructuring Plan Implementation Project MIC Management of Investment Capital PAR Public Administration Reform PDO Project Development Objective PHRD Japan Policy and Human Resources Development TTL Task Team Leader VnSAT Vietnam Sustainable Agriculture Transformation Project TABLE OF CONTENTS DATA SHEET ....................................................................... ERROR! BOOKMARK NOT DEFINED. I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 4 II. OUTCOME ...................................................................................................................... 9 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 17 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 18 V. LESSONS LEARNED AND RECOMMENDATIONS .............................................................. 19 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 20 ANNEX 2. PROJECT COST BY COMPONENT ........................................................................... 26 ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ...... 27 ANNEX 4. SUPPORTING DOCUMENTS .................................................................................. 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name MARD M&E Capacity Building for Agricultural P159760 Restructuring Plan Implementation (MECARP) Country Financing Instrument Vietnam Investment Project Financing Original EA Category Revised EA Category Not Required (C) Not Required (C) Organizations Borrower Implementing Agency Ministry of Agriculture and Rural Development - Socialist Republic of Vietnam Department of Planning Project Development Objective (PDO) Original PDO Enhance monitoring and evaluation capacity of Ministry of Agriculture and Rural Development and selected Departments of Agriculture and Rural Development to monitor progress of the Agricultural Restructuring Plan Page 1 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) FINANCING FINANCE_TBL Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) Donor Financing TF-A5252 1,800,000 1,679,558 1,679,558 Total 1,800,000 1,679,558 1,679,558 Total Project Cost 1,800,000 1,679,558 1,679,558 KEY DATES Approval Effectiveness Original Closing Actual Closing 13-Apr-2017 26-Jan-2018 30-Jun-2020 31-Dec-2020 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 11-Jun-2020 1.09 Change in Loan Closing Date(s) Change in Implementation Schedule KEY RATINGS Outcome Bank Performance M&E Quality Satisfactory Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 01-Feb-2018 Satisfactory Satisfactory 0.00 02 26-Jun-2018 Satisfactory Satisfactory 0.00 03 04-Apr-2019 Satisfactory Satisfactory 0.53 04 12-Jun-2020 Satisfactory Satisfactory 1.09 Page 2 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) 05 28-Dec-2020 Satisfactory Satisfactory 1.52 ADM STAFF Role At Approval At ICR Regional Vice President: Victoria Kwakwa Victoria Kwakwa Country Director: Ousmane Dione Carolyn Turk Director: Martien Van Nieuwkoop Benoit Bosquet Practice Manager: Nathan M. Belete Dina Umali-Deininger Hardwick Tchale, Binh Thang Task Team Leader(s): Sergiy Zorya, Hanane Ahmed Cao ICR Contributing Author: Ingrid Marie Pierre Mollard Page 3 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES Context 1. Context at appraisal: Prior to the appraisal of this project, the Government had adopted the first phase of the Agricultural Restructuring Plan (ARP) for the period 2013 – 2015 to transform the sector towards a more sustainable growth path. The Government, through Decision No. 899/QD-TT approved by the Prime Minister on June 2013, started implementing the ARP which aimed to: (i) sustain the growth, raise the efficiency and competitiveness by increasing productivity, quality, and added values;1 (ii) satisfy the demands of consumers in Vietnam and boosting exports; (iii) improve income and living standards of rural residents, ensure food security (including nutrition security) in both the short term and the long term basis, contribute to the reduction of poverty ratio; and (iv) enhance natural resource management, reduce greenhouse gas emission and negative impacts on the environment. 2. Agricultural sector challenges: The agricultural sector in 2017 faced growing domestic competition for labor, land, and water from cities, industry, and services. Increasing labor costs was starting to hinder the sector’s ability to compete internationally as a low-cost producer of bulk undifferentiated commodities. Vietnam’s agriculture “needed to generate ‘more from less’. That is, it needed to generate more economic value—and farmer and consumer welfare—using less natural, human, and other resources. And it needed to increasingly compete on the bases of reliable supply, predictable quality, assured food safety, value addition and sustainability� (Vietnam Development Report, 2016). Change was required in Vietnam’s agricultural growth model, and structural patterns of production and supply chain organization. Collective action at the farmer level was limited, production structural patterns and supply chain organizations were fragmented, and vertical coordination was weak. The consequences were high transaction costs, lost opportunities for economies of scale, and insufficient incentives to produce and preserve higher quality produce and raw materials. Change was also needed in the framework of state management and policies in the sector service provision in the sector by the state to provide appropriate environment for private sector involvement in agriculture. (Vietnam Development Report, 2016). 3. The need for agricultural restructuring: The Government of Vietnam recognized the need to restructure agriculture and realign the functions of the state to address the above challenges. In June 2013, the Ministry of Agriculture and Rural Development (MARD) approved the Agricultural Restructuring Plan (ARP) for the period 2013- 2015, which called for a shift in sectoral goals beyond physical (output or trade) targets to include a broader set of indicators related to the triple bottom line of sustainable development i.e. defining sector goals in terms of economically, socially, and environmentally sustainable development. It laid a set of core principles to guide the sector development, the most significant of which are that: (i) agriculture will be market-led and consumer-driven, rather than state directed and production-led; and (ii) the role of the government will shift from being the primary investor and service provider to being the facilitator of investments and services provided by the private sector, community organizations, research institutions, commercial banks and others. 4. The need for an M&E system to help measure the performance of the ARP: The ARP called for the broad application of collaborative arrangements among government agencies, the private sector, farmer and community organizations, and the scientific community. The roles, approaches, and expenditures of the state in the sector will 1 By 2020, the growth target under the ARP was set at 4% per annum; the target for income growth was set at 2.5 times relative to 2008 level and 50% of the rural communes will have reached the standards set in the new Rural Development Program. Page 4 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) be restructured in order to help realize the goals for sustainable agricultural development and rural transformation. Implementing this change faced many challenges, including the weak Monitoring and Evaluation (M&E) of the sectoral performance. The structure of the institutional framework for policy design, monitoring and evaluation, and the means to review the efficiency of public programs and policies in the Vietnamese agriculture sector was complex. Sector monitoring covered reporting of quantitative information that were output oriented such as acreage planted, or quantities produced and focused mainly on the achievements of government set macro-targets for those indicators. No emphasis was placed on sector level performance based on outcome indicators capturing improvement of livelihoods, competitiveness of different crops within and outside Vietnam, effectiveness of public policy and investments and the sustainable and effective use of natural resources. The monitoring system during the time of project preparation was poorly linked to the evaluation of broader strategic level objectives as described in MARD annual and medium-term agricultural and rural development plans, as well as in the various sub-sector strategy documents. In addition, the capacity of staff to monitor qualitative indicators in the context of a market- oriented sector restructuring was absent to weak. 5. Building on existing M&E efforts developed prior to the project: As part of MARD’s efforts to enhance the sector M&E system, MARD had developed the Monitoring and Evaluation in Support of Management as part of the Agricultural and Rural Development Sector (MESMARD) project, which had two phases spanning 2006-2015. The MESMARD project was funded by the Swiss Agency for Development and Cooperation (SDC) and was implemented by the Department of Planning (DoP) of MARD from 2010 to 2015.2 The overall goal of this project was to support MARD to build a system that could effectively monitor and evaluate the outcomes of economic growth and the impact of MARD poverty reduction policies and program, through: (i) improved result-based planning, (ii) monitoring and evaluation of the sector and sub-sectors; and (iii) improved capacity, within MARD and selected provincial Departments of Agriculture and Rural Development (DARDs). This project received strong support from MARD’s senior management, including the Vice Minister. One of the lessons from the MESMARD was the need to have outcome indicators that would be relevant to measure the performance of the ARP – and to standardize the indicators, data collection and calculation methods to ensure accuracy and comparability of the indicators across provinces. This could not be done under MESMARD because it started before the ARP was approved in 2013. Before the implementation of the MESMARD, the M&E systems between MARD and DARDs were not well-integrated, and mostly use manual database. 6. The MESMARD project supported three general departments and eight technical departments within MARD, in particular DoP, and two pilot provincial DARDs of Hoa Binh and Vinh Phuc. The support focused on applying a results- based approach to sector and sub-sector plan formulation and implementation (linked to M&E) for its annual and five-year plans (2011-2015 and 2016-2020). Some of the achieved outcomes of the MESMARD project are: improvement in the quality of planning (better linkages between planning and budgeting), issuance of policies/strategies and laws with reference to the M&E systems, improvement in sector/sub-sector M&E indicator 2 MESMARD’s goal was to develop the results-based planning, monitoring, evaluation and policy making system for Agriculture and Rural Development (ARD) sector/subsector/field that is market oriented, further enhanced, used and well linked between MARD and provincial Departments of Agriculture and Rural Development DARDs. Total funding for the project was US$3,932,985 of which US$3,115,357 was financed by SDC and US$817,628 was counterpart funding provided by the Government of Vietnam. Page 5 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) sets, 10 online M&E databases were developed and are operating to assist policy design making process, provide training to planning officers in MARD and provincial DARDs about results-based planning and M&E, project officers and working group members benefited from lessons learned in other countries success and failures in implementing a results based M&E system. 7. The project has spread its positive influence and impact within MARD as well as other agencies beyond MARD. For example, the Danish International Development Agency project applied the plan templates developed by the MESMARD project in their 5 targeted provinces; and the Asian Productivity Organization benefited from the project practical experience particularly the Management of Investment Capital (MIC) database. The MIC is an online database system built in 2009 to support the management of capital investment projects. The MIC database contains information on 1,500 projects/sub-projects and 500 reporting agencies. The system is operated by DoP and updated by 300 agencies annually The Ministry of Planning and Investment studied the MIC system for designing a similar system; and many provincial DARDs in the nonpilot provinces requested technical support from the project in applying the results-based planning approach as well as developing M&E systems in their provinces. 8. While the MESMARD was designed long before the approval of ARP in 2013, it did not include the development of new indicators to monitor and evaluate ARP and the methodologies for data collection and analyses at local and national levels. The MARD had started the development of ARP M&E indicators, with the support from the Institute of Policy and Strategy for Agriculture and Rural Development (IPSARD). But this work required further financial and technical support. The objective of the MARD M&E Capacity for Agricultural Restructuring Plan Implementation Project (MECARP) was therefore to continue to further strengthen and modernize the sector-wide M&E system, building on the strong foundation established by the MESMARD project. 9. Theory of Change (TOC): The MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation Project (MECARP) was designed to assist MARD to build an evidence-based M&E system and capacity to implement its ARP for the period 2016-2020 and beyond. The project objective aimed at enhancing the monitoring and evaluation capacity of MARD and selected DARDs to monitor progress of the ARP. The project sought to achieve this by financing technical assistance activities to: (i) develop an ARP set of sector performance indicators and M&E plan for data collection and methodologies; (ii) pilot test the selected indicators in 8 Provinces; (iii) develop a roadmap for expansion of ARP M&E indicators and system to non-pilot provinces adopted by MARD; and (iv) train MARD, Department of Agriculture and Rural Development (DARDs), Government Statistical Office (GSO) and Institute for Policy and Strategy for Agriculture and Rural Development (IPSARD) staff on M&E and procedures for effective implementation and use of ARP M&E. Therefore, the project contributes to improving MARD decisions making to support the implementation of the ARP through more reliable data and monitoring of the progress of sector wide-developments and ARP implementation (see Results Chain in Figure 1 below).3 3 At appraisal, a Theory of Change was not developed. Figure 1 is an ex-post TOC developed to demonstrate the results chain during project implementation. Page 6 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) Figure 1: Results Chain Activities/Outputs Intermediate Project Level Long term Results Outcomes Outcomes - Develop M&E indicator set - Set of sector performance and Monitoring of the progress of sector wide-developments and ARP (quantitative and qualitative) to ARP indicators developed. MARD decisions making and polices development are based on measure agriculture sector PDO: Enhance performance, and ARP - M&E plan for data collection and monitoring and implementation. methodologies developed. evaluation capacity of Ministry of Agriculture - Develop and operationalize a - Pilot tests for selected indicators and Rural Development centralized M&E database conducted in 8 Provinces. and selected system for the implementation Departments of of ARP M&E. - Manual and protocols for data Agriculture and Rural collection approved. Development to monitor progress of the - Roadmap for expansion of ARP Agricultural M&E indicators and system to Restructuring Plan. non-pilot provinces adopted by available and more reliable data. MARD. - Measurable agricultural sector performance and ARP indicators adopted. implementation enhanced. information increased. - Staff of MARD and DARDs trained on the enhanced M&E - MARD, DARDs, GSO - Training of MARD and DARD system. and IPSARD staff trained staff working on M&E on M&E and procedures approaches, roles and function, - Feedback flows with private for effective indicators and methodologies. sector, academia, and research implementation and use of organizations ARP M&E. - Exposure to international best practices. - Responsibilities and procedures for M&E of MARD, GSO and DARDs reviewed and delineated 10. MECARP was directly complementary to another project supported by the Bank, the Vietnam Sustainable Agriculture Transformation Project (VnSAT – P145055). The VnSAT was approved by the Bank on June 30, 2015 with an IDA funding of US$238 million. The project aims to support the implementation of the ARP2 (2016-2020) by: (i) improving farming practices and value chains in the targeted project areas, and (ii) promoting institutional strengthening of relevant public agencies to effectively support implementation of the ARP2 for the period 2016- 2020 and beyond. The MECARP was to contribute to the achievement of the second part of the VnSAT objectives, by building synergies and focusing on strengthening of the M&E capacity of MARD, which is not financed by the VnSAT. The VNSAT became effective on December 3, 2015. It was originally supposed to be closed on December 31, 2020 but was extended by 18 months to June 30, 2022.4 The MECARP was conceived as a technical assistance facility to support the capacity building and institutional reform activities under Component A of VNSAT. 4 The total funding amount was revised to US$222.6 million with a cancellation of US$14.5 million. Page 7 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) Project Development Objectives (PDOs): 11. The Project Development Objective (PDO) is to enhance monitoring and evaluation capacity of Ministry of Agriculture and Rural Development and selected Departments of Agriculture and Rural Development to monitor progress of the Agricultural Restructuring Plan. Key Expected Outcomes and Outcome Indicators 12. The key expected outcomes of the project are as follows: (i) Enhanced existing M&E system for better monitoring of the progress of sector wide-developments and ARP implementation. (ii) Strengthened M&E human capacity and procedures at MARD and selected DARDs for effective implementation and use of the ARP M&E. 13. The intermediate results indicators in support of these outcomes are: (i) Measurable agricultural sector performance and ARP indicators by MARD’s M&E system adopted. (ii) Pilot tests for selected indicators conducted. (iii) M&E plan for data collection and methodologies developed. (iv) Manual and Protocols for data collection approved. (v) Number of MARD, DARDs, GSO and IPSARD staff trained on M&E and procedures for effective implementation and use of ARP M&E. (vi) Roadmap for expansion of ARP M&E indicators and system to non-pilot provinces by MARD adopted. (vii) Responsibilities and procedures for M&E of MARD, GSO and DARDs reviewed and delineated. Components 14. Component A: Support to the development of ARP M&E indicators and M&E system (Grant Financing: USD 995,500). This component was to enhance the existing M&E system for better monitoring of the progress of sector wide-developments and ARP implementation. This component was comprised of the following activities: (i) completion of M&E indicator set to include both quantitative and qualitative measures of agriculture sector performance, and ARP implementation. This includes indicator sets for the entire sector, six sub-sectors,5 and an indicator set capturing administrative reform. The indicators would follow the SMART6 criterion; (ii) support for the development, and operation of a centralized M&E database system for the implementation of ARP M&E; (iii) preparation of manuals and protocols for data collection, frequency of reporting, ARP M&E system operation, and outlining the roles and responsibilities of MARD DARDs, and relevant stakeholders; (iv) support for eight ARP pilot provinces to build M&E indicator sets (including support for data collection) to monitor sector-wide and ARP 5 The sub-sectors include: Crops, Livestock, Forestry, Fishery, Irrigation and Processing Industry. 6 SMART = Specific, Measurable, Achievable and Attributable, Relevant and Timely. Page 8 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) progress at the provincial level to feed into the centralized M&E database system; (v) preparation of the roadmap for expansion of the ARP M&E indicators, and system beyond project closure to the non-pilot provinces (55 provinces). 15. Component B: Capacity and institutional strengthening on ARP M&E implementation (Grant Financing: USD 430,000). This component was to strengthen M&E human capacity and procedures at MARD and selected DARDs for effective implementation and use of ARP M&E. This component included the: (i) review and delineation the roles and responsibilities of MARD units involved in M&E; and development of an effective collaboration mechanism on ARP implementation among different units under MARD, GSO, and DARDs with clearly assigned roles and responsibilities; (ii) the increase of awareness through information campaigns and workshops about the ARP M&E initiative to strengthen the flow of knowledge between central, local governments, and other stakeholders such as the GSO, IPSARD, etc; (iii) training of the MARD and DARD staff working on M&E on the approaches, roles and functions, indicators, and methodologies, and expose them to the international best practices; (iv) fostering a system of information and feedback flow with the private sector, agricultural commodity boards, researchers, and academia on their participation in the agricultural restructuring process. 16. Component C: Project management, monitoring and evaluation, and knowledge dissemination (Grant Financing: USD 374,500, including contingency). This component was to ensure effective project management, implementation, supervision, and knowledge management, including the project’s M&E. This component supported: (i) carrying out of the day-to-day project management activities, including reporting on the project implementation, procurement, and financial management; (ii) establishment and implementation of an effective M&E of the project; (iii) facilitate the knowledge management through analysis and dissemination of lessons learned from project implementation. II. OUTCOME Assessment of Achievement of Each Objective/Outcome (Efficacy) Rating: Substantial 17. The PDO was to be achieved through the support to the development of ARP M&E indicators and M&E system, capacity and institutional strengthening on ARP M&E implementation, and project management, monitoring and evaluation, and knowledge dissemination. The assessment of achievement of the PDO is based on a review of the outputs of the project against the targets in the results framework. As per the approved project results framework (RF), the project objective is assessed through two outcome indicators, namely: (i) measurable agricultural sector performance and ARP indicators in MARD’s M&E system adopted and (ii) number of MARD, DARDs, GSO and IPSARD staff trained on M&E and procedures for effective implementation and use of ARP M&E (number of beneficiaries in RF). Summary PDO results are presented in Table 1 and the achievement of intermediate outcome indicators are shown in Table 2. Page 9 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) Table 1: Status of the key performance indicators7 Indicator UoM Baseline Achievement End Target (2017) at project (2020) closing Measurable agricultural sector performance and ARP Yes/No No Yes Yes indicators in MARD’s M&E system adopted Direct project beneficiaries Number 0 68 60 Female beneficiaries Number 0 36 30 Table 2: Status of intermediate outcome indicators Intermediate Indicators UoM Baseline End of Project End Target (2017) Achievement (2020) A set of the ARP and sector performance M&E indicators Yes/No No Yes Yes developed 2: M&E plan with data collection method developed Yes/No No Yes Yes 3: Selected indicators piloted Number 0 93.3% 80% 4: Manual/Handbook and Coordination mechanism in data Yes/No No Yes Yes collection approved 5: Roadmap on scaling out the ARP M&E indicators and Yes/No No Yes Yes system to all remaining provinces (non-pilot provinces), approved by MARD 7The ARP M&E system has been established and approved by the Prime Minister in the Decision 678 / QD-TTg of May 19, 2017 and MARD issued the Decision No. 3449 dated on August 21, 2017 on the action plan for its implementation. The ARP M&E system and M&E database are designed as a web-based (online) system with 15 M&E indicators as the core and are connected to MARD’s Public Investment Monitoring Database (MIC). The M&E system aims to measure the performance of ARP implementation. Page 10 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) 6: MARD’s and DARDs’ staff trained on M&E system Number 0 250 60 7: Feedback from businesses, industry associations, Yes/No No Yes Yes researchers and scholars about the process of their participating in the ARP implementation. 8: M&E process and responsibilities of MARD, GSO and Yes/No No Yes Yes DARDs reviewed and enhanced 9: Progress reports submitted on time Yes/No No Yes Yes 18. PDO Indicator #1 – Measurable agricultural sector performance and ARP indicators in MARD’s M&E system adopted. The target was achieved. Several activities and outputs were implemented to contribute to the achievement of this outcome indicator. A set of 15 performance indicators (see Box 1) were developed for monitoring the performance of ARP. These indicators were approved by MARD following Decision No. BNN-KH dated February 29, 2019 (following an earlier Decision No. 678/QD-TTg dated May 19, 2017). The project developed a manual on implementing the set of ARP M&E indicators, which was approved by Prime Minister in Decision No. 816/QD-BNN- KH dated 08/03/2019 and it has been in use since. This manual defines each indicator and the responsibilities of relevant agencies in data collection, calculation and reporting for each of the indicators. Pilot tests for selected indicators were succesfuly conducted8 in eight provinces. The project team and GSO continuously supported the key officials of DoP and the eight pilot provinces to ensure proper estimation of the indicators based on standardized methodologies and data. This support was critical to ensure that the M&E system established would be well understood by key staff in DoP and the pilot provinces before the system was mainstreamed in MARD and all the 63 provincial DARDs. The project has strengthened the coordination between the GSO, MARD and DARDs for collecting data on the sector and the coordination mechanisms between the MARD and GSO have been institutionalized. The roles and responsibilities of MARD agencies and DARDs in the M&E system has been specified in the Decision 816/QD-BNN-KH. Based on the calculated and approved M&E indicators, the project has established a database for easy access by all stakeholders. The ARP M&E database is designed as a web-based (online) system with 15 ARP indicators as the core and are connected to MARD’s Public Investment Monitoring Database. The online reporting is making reporting faster and more accurate. However, for some officials in localities with low IT skills, the data processing for reporting is slow, so it is necessary to continue building their capacity. Box 1: List of indicators developed and approved under the MECARP 1. Growth rate of added agriculture, forestry, and fishery (AFF) value 2. Growth rate of income per hectare of cultivation land 3. Growth rate of income from livestock production 4. Growth rate of income from seafood production 5. Growth rate of income per hectare of production forest land 8 The project’s pilot provinces include: Vinh Phuc, Nam Dinh, Thanh Hoa, Ha Tinh, Lam Dong, Binh Dinh, Dong Thap, Phu Tho. The following non-pilot provinces participated in the discussions: Dien Bien, Hoa Binh, Ha Nam, Ha Noi, Quang Binh and Son La, Nghe An, Binh Phuoc and Ba Ria Vung Tau. Page 11 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) 6. Growth rate of income per hectare of salt production 7. Growth rate of income from AFF processing 8. AFF labor productivity growth rate 9. Ratio of AFF production produced in the form of cooperation and association 10. Ratio of AFF products produced under Good Agricultural Practices (GAP) or equivalent 11. Ratio of agricultural production area with applied economical irrigation technology 12. Ratio of sustainably managed production forests area with certificate 13. Ratio of farmers with agricultural training 14. Ratio of women among farmers with agricultural vocational training 15. Ratio of livestock production facilities using biogas and technology solutions for environmentally safe waste treatment Source: MECARP, Department of Planning, MARD 19. The roadmap for expansion of the ARP M&E indicators and system to the 58 non-pilot provinces was adopted by MARD and training courses on M&E system and guidance on the estimation of the ARP indicators have been rolled-out to all 63 provinces. With the support of GSO, up to now, MARD has developed the methodology for the calculation of 14 indicators (except the indicator 6 - growth rate of income per ha of land for salt production). The GSO calculates nine indicators (indicator 1 to 8 and indicator 15); 63 provinces have sent their data on six ARP indicators (indicator 9 to 14) to MARD for the years from 2017 to 2019 and 23 non-pilot provinces have been supported to estimate the results of six ARP indicators for the year 2020. The findings from the ARP M&E have shown that agricultural restructuring has created qualitative changes in agricultural production. A new set of ARP implementation M&E indicators for the period of 2021- 2025 was drafted and is being reviewed under at national and provincial level. In addition, indicators for assessing the performance of the Public Administration Reform (PAR) were established, approved and implemented. The PAR M&E indicators used to assess the performance of all Departments and agencies under MARD were approved as per the Decision 4859/QD-BNN-TCCB, dated December 10, 2018. These indicators were used to conduct the first PAR performance ranking for MARD’s agencies and publicly disclosed through Decision 535/QD-BNN-TCCB dated February 14, 2019. 20. PDO Indicator #2 – Direct project beneficiaries. This indicator was overachieved. The project has trained 68 key M&E staff (with a target of 60; 53 percent female) who will be responsible for the implementation of the ARP M&E system. Several other activities and outputs were also implemented to contribute to enhancing M&E capacity within MARD. The M&E staff responsible for the implementation of the ARP M&E system trained included staff from the Institutional Support Board, the ARP Standing Office and the DoP. The trainings focused on Training of Trainers covered topics such as: results-based M&E, results-based management, policy evaluation/analysis and setting-up and calculating M&E indicators, use of M&E database software, and indicators calculation and reporting. The project has also organized 32 training sessions for 1,650 staff from MARD units and 63 provinces. The trainings focused on data collection and calculation methods for the agriculture ARP indicators and the usage and management of its database. An instructional DVD was also produced, which explains the usage of the ARP indicators, shares experiences from the eight provinces in the M&E pilot and answers questions on common problems in data collection and calculation. The project has also prepared a Handbook on the 15 indicators for ARP implementation, which was approved by the MARD Minister through Decision No. 816/QD-BNN-KH dated March 8, 2019. Page 12 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) 21. The core M&E members in the eight pilot provinces have also formed working groups, among which they are exchanging information and experiences on the ARP indicators with provinces that were not included in the pilot. M&E working group members are leaders and officials from MARD units and DARDs, who have been trained on the use of the M&E indicators and operating the M&E system for the ARP performance. Responsibilities and procedures for M&E of MARD, GSO and DARDs were reviewed and delineated. To strengthen M&E capacity building and foster feedback on ARP M&E, the project has also organized conferences, seminars, consultation workshops, and trainings with the private sector, agricultural commodity boards, researchers, academia, MARD and DARDs staff. The project has successfully enhanced MARD M&E capacity and raised the awareness on the importance of M&E to ensure value- for-money from sector investments that align with the ARP objectives. 22. The project has supported MARD in organizing 20 conferences, consultation workshops on the ARP performance, the conference on national food security, agricultural mechanization promotion conference, conferences on agriculture and rural development policy communication., international workshops to learn experiences from countries in the region that have successful M&E systems. From the beginning of 2020, although the Covid-19 epidemic caused disruption in the project implementation, by mid-December 2020, the project also supported the Ministry to organize five national conferences with the attendance of 422 delegates from central and local agencies, some of which are at big scale and very important, such as: Conference on Review of 10 years implementation of the national food security program, National conference on agricultural processing and mechanization capacity assessment, Conference on "Women of Ministry of Agriculture and Rural Development participate in the agricultural restructuring implementation M&E... The National Conferences on the midterm review of the use of the approved set of indicators for M&E the ARP implementation were organized. The conferences were held in the North and the South for nearly 200 delegates (Source: The MECARP project completion report). The project also collaborated with the Vietnam M&E Network to organize an international workshop on monitoring and evaluation to share experiences in setting up and operating the M&E system to serve state management. It is a critical element to contribute to capacity and institutional strengthening on ARP M&E implementation. The speakers, sharing the experience at the Workshop, included M&E experts from the World Bank, representatives of the M&E network of Asia, the Pacific, Japan, Malaysia, Sri Lanka and Vietnam. The qualitative impacts of the impacts of MECARP interventions are highlighted in Box 2. Box 2: Qualitative impacts of the MECARP interventions The MECARP, funded through a Japanese PHRD grant has brought positive impacts in the quality of the implementation of the ARP. The grant has helped raise the awareness of the importance of M&E to ensure value-for-money from sector investments that align with the ARP objectives. The establishment of an on-line M&E database which is linked to the sector-wide M&E system (https://giamsatdanhgia.mard.gov.vn), including building the capacity of the M&E staff at MARD and in the provincial DARD offices has invariably raised the implementation effectiveness of the public sector investment projects aligned to the ARP objectives. The project has also helped establish the Public Administration Reform system to assess the performance of the MARD departments and provincial DARD offices – and this has raised the awareness and understanding of the ARP by all agricultural sector institutions. The grant has therefore contributed significantly to raise the profile of the ARP policies, mechanisms and developed indicators for measuring its implementation. This will invariably help to contribute towards the long-term sustainable development of the agricultural sector. “MECARP is a very practical World Bank project to support the Agriculture sector, all of its objectives are met and exceeded. The project has produced two basic things: (1) Building M&E system; and (2) Supporting staff to improve capacity so that they Page 13 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) understand, know how to measure and calculate data for the ARP M&E accurately � (Mr. Nguyen Do Anh Tuan, Director General of International Cooperation Department). “I am very excited with the results that M&E system has brought, because it helps reducing the workload for the staff in our Department. Previously, in order to make reports, we had to get data/information from other units, now, with the M&E system, we can get the data/information needed for report preparation directly from the system without having to ask for help from other units. This saves time while reducing workload, and the data is reliable and consis tent, so I like this system very much� (Ms. Tran Thi Loan, Head of Rural Industry Division, Department of Economic Cooperation and Rural Development ). Source: MARD Project Completion Report Efficiency Rating: Substantial 23. The project has achieved or overachieved all the indicator targets in the Results Framework while spending 96.3 percent of project funding. This indicates that the project has been highly efficient in terms of converting project resources into results. The project team also demonstrated a higher level of implementation efficiency as they addressed emerging implementation challenges in a timely and effective manner. While the project activities and outcomes do not lend themselves to a standard Economic and Financial Analysis (EFA), the project has helped to collect, synthesize and systematize the results of the implementation of the ARP.9 The M&E system has enhanced the implementation of the ARP – as implementers realized that systematic monitoring and reporting of results is mainstreamed. This has helped to enhance project implementation, and thereby leading to the improved performance of the ARP, and the agricultural sector as a whole. However, this is all difficult to quantify but was widely acknowledged by implementers at the national and provincial levels. Risk to the Sustainability of the Development Outcome Rating: Moderate 24. The sustainability of the development outcomes from this project depends on: i) the capacity of MARD and provincial DARDs to operate and maintain the system; ii) acceptance of the usefulness of the system by beneficiaries. Also, the usage and relevance of ARP indicators for results-based management by MARD and DARDs staff, the continuous improvement of ARP M&E processes and M&E web-based database within provinces and potential expansion to districts, and strengthening of M&E learning activities across all provinces including continuous training of staff involved in M&E of ARP both at national and local level. All these would require continuous allocation of adequate budget by MARD. The project has already developed a roll-out plan for all the other 58 provinces and MARD is committed to maintain this system as they are fully committed to measure and monitor the implementation of the ARP, and to be able to make timely adjustments to ensure the success of the agricultural restructuring process. Given the relatively high level of available capacity, coupled with the continuous training of staff, there is a high chance that this system will be maintained even after the project. Also, since the ARP M&E system developed under MECARP is integrated into the sector-wide M&E system, the provision of resources for the continuity of the system is guaranteed than if it continued to be a parallel system. Furthermore, MARD highlighted that they will be on the Page 14 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) look-out for grant funding to continue to support activities (such as training workshops) which may be ineligible for public sector financing. Overall Outcome Rating - Satisfactory 25. The overall outcome rating for this project is satisfactory. Overall, all the PDO and intermediate outcome indicators have been met and all quantitative targets have been surpassed. The ARP M&E indicators have been integrated into the MARD M&E system and the M&E manual has been developed. The ARP M&E web-based database integrates the 15 ARP indicators as the core and is connected to MARD’s Public Investment Monitoring Database. The M&E system has moved beyond the pilot phase in the eight provinces. It now helps MARD, its agencies, and 63 DARDs in reporting on their ARP performance. It is also used at some level for results-based management of the agricultural sector and ARP at national and provincial level. For example in Vinh Phuc Province, the growth rate of AFF value added was still low and unsustainable (based on results from indicator 1 and indicator 7 of ARP)10 therefore the Provincial People's Council issued: (i) Resolution No. 20/2020 / NQ-HDND dated December 14, 2020 on policies to support the restructuring of the agricultural sector for increasing income for farmers in the province in the period of 2021-2025; (ii) Decision No. 07/2021 / QD-UBND dated March 19, 2021 regulating the implementation of support policies under Resolution No. 20/2020 / NQ-HDND. The Resolutions focus on the development of an agricultural production to improve farmers' income by supporting: (i) crop structure change on paddy land; (ii) good quality rice varieties, key aquatic breeds with high economic efficiency; (iii) livestock and poultry epidemic prevention; and (iv) production of safe vegetables and fruits according to VietGAP. At national level, the results of indicator 1011 demonstrated that the application of GAP standards is still limited. Consequently, MARD has further encouraged production according to VietGAP, Global GAP, and management standards for agricultural inputs by approving (i) Decision 703 / QD-TTg on approving the program on developing the seed research and production for the agricultural restructuring in the period 2021-2030; (ii) Decree 109/2018 / ND-CP on promoting organic production; and (iii) Decision No. 885 / QD-TTg issued by Prime Minister dated June 23, 2020 on approving the Plan for the development of organic agriculture for the period 2020 – 2030. 26. When it comes to the PAR indicators developed, they have been used to assess the performance of all Departments and agencies under MARD and PAR performance ranking for MARD’s agencies and the rankings were publicly disclosed. The PAR database also synthesizes and calculates the PAR indexes to assess needed reforms or capacity building activities. The PAR system has also raised the awareness and understanding of the ARP by all agricultural sector institutions. 27. There are still some shortcomings that needs to be addressed to strengthen the current ARP M&E system and ensure its sustainability. The project has faced some difficulties in collecting data and quantifying some of the indicators. The indicator 6 (growth rate of income per one hectare of salt production) was not collected, as it requires 9 An EFA was not conducted for this project as the outcomes are all qualitative and do not lend themselves to the quantitative EFA. An EFA was also not done at project appraisal. 10 Indicator 1: Growth rate of added agriculture, forestry, and fishery (AFF) value and Indicator 7: Growth rate of income from AFF processing. 11 Ratio of agricultural production, produced under good agriculture process (GAP) and the equivalent has increased from 7.54% in 2017 to 8.67% in 2019 Page 15 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) a statistical survey and MARD has recently allocated some budget to undertake this survey. For indicator 9 (ratio of AFF production produced in the form of cooperation and association) some provinces do not fully understand it or know how to calculate it, so the project developed a standardized definition and methodology for calculating the indicators to make sure that they can all be comparable across provinces. Furthermore, both indicators 11 (ratio of agricultural production area with applied economical irrigation technology) and 13 (ratio of farmers with agricultural training) have their definition updated, as the latter are not always applicable to all provinces and quantification can be quite difficult. The level of usage of the web-based database is limited at provincial level, M&E staff mostly enter the results and upload reports after manual calculation; the system is not linked to districts to directly have access to results. The M&E capacity of some DARDs staff is still limited and some officials also have low IT skills, which leads to delays in reporting and processing of data. The web-based M&E system which has been developed under MECARP will help address this challenge as all M&E staff in DARDs can access the integrated M&E database through the link which has been created ((https://giamsatdanhgia.mard.gov.vn). Other Outcomes and Impacts 28. The project has helped raise the awareness of the importance of M&E to ensure value-for-money from sector investments that align with the ARP objectives. The operationalization of the M&E system and particularly the M&E training of MARD and DARDs staff have created a change in thinking and mindset about M&E in the agriculture sector. Given the benefits of the M&E system, it showed managers at the ministry and provincial levels the need for a result-based management approach12 to evaluate the performance of development objectives, plans and projects in the agriculture sector. MARD is committed to this approach, as a new set of ARP indicators for the period of 2021- 2025 has been drafted and is under consultation with relevant stakeholders at the national and provincial level. This approach will contribute towards the long-term sustainable development of the agricultural sector. 29. MARD’s departments have increased their efficiency thanks to easy access to online ARP data and information with the web-based database. It has reduced time spent accessing data and information to develop annual department reports. It also contributes to faster reporting with more reliable data and reduce staff workload. 30. This project has fostered close coordination between GSO, MARD and DARDs which was critical to improve efficiency and quality in collection and calculation of ARP indicators. Leadership and direction of GSO, MARD and DARDs have also provided an important legal basis for developing an M&E system for ARP indicators. It has also been essential to regularly mobilize and train provincial and local officials as well as to create consensus on ARP indicators and improve capacities to monitor these indicators. The project has also supported MARD to develop two Statistical Circulars on the System of Statistical Indicators and Regulation on Agricultural Sector Statistical Reporting System. MARD Minister approved the Circular on the System of Statistical Indicators in the Decision No. 16/2020/TT- BNNPTNT dated December 28th, 2020 and the Circular on Regulation on Agricultural Sector Statistical Reporting System in the Decision No. 16/2020/TT-BNNPTNT dated December 28th, 2020. 31. The project has much anecdotal evidence on the impacts on beneficiaries. For example, the project changed their perspective on project management, to focus on results/outcomes rather than outputs. This used to be the case before the project when most of the staff did not fully understand the project’s results chain. In some ways, due to the accountability for results, as a result of rolling-out the results-based M&E system, a number of guidelines were reviewed to facilitate speedy implementation in order to show results (as demonstrated in Box 3). 12 The ‘art’ of results-based management is defining outcomes that are measurable in a credible way and are used in decision making. Page 16 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) Box 3: Some positive impacts of the MECARP project The implementation of monitoring and evaluation of agricultural restructuring in the province shows that the promulgation of a set of monitoring and evaluation indicators for agricultural restructuring is very practical and has far-reaching effects. Although the implementation period is not long and the budget is limited, the results achieved are of great significance, helping the province: 1) Quantify the results of comprehensive restructuring in economic, environmental and social fields; (2) Establish a database system to serve the monitoring and evaluation of agricultural restructuring in the province over the years and in the coming time; (3) To train and form a contingent of management staff from the province to the district serving the monitoring and evaluation of restructuring; (4) Helping the province to have a more practical, in-depth and comprehensive view of agricultural restructuring issues, thereby recognizing the existing strengths and limitations and constraints as a basis for timely direction and management measures (in 2019, the Department of Agriculture and Rural Development advised the Provincial People's Committee to issue a policy to support restructuring the agricultural sector (Resolution No. 05/2019 / NQ-HDND dated 16) / 7/2019) which focuses on supporting limited areas such as: linking production along the value chain; production by good production processes, sustainable forest management (in 2020 through the policy, it has supported 5 tea branding projects; planting 384 hectares of pomelo, applying good practice for 1,292 hectares of grapefruit; granting certification of sustainable forest for 8,532 hectares of forest; supporting 14 projects. linking production with the consumption of key agricultural products) contributing to enhancing the value of agricultural, forestry and fishery products in the province, strongly promoting the process of agricultural restructuring in the province of Phu Tho); (5) The results of the implementation of the set of monitoring and evaluation criteria are used as the basis for the province to develop specific orientations and objectives to implement the restructuring for the period 2021-2025 and continue to have policies practical and effective in the coming time. (Mr. Tran Tu Anh - Deputy Director of the Department of Agriculture and Rural Development of Phu Tho province) III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME 32. The Covid-19 pandemic in Vietnam affected project implementation with the Directive No. 16/CT-TTg from the Prime Minister on the implementation of urgent measures to prevent and control the Covid-19 pandemic, which required social distancing and limited travel. A series of project activities could not be implemented as planned by the initial project closing date such as: (i) organization of conferences, seminars, training courses for 63 provinces and cities on the implementation of the reporting via the M&E database evaluation software; (ii) on- site guidance and data collection on using the sectoral ARP implementation M&E indicators; and (iii) mobilization of international and domestic consultants to carry out the final evaluation of the project. Therefore, the project was extended by six months (until December 31st, 2020). Thanks to the extension, the planned project trainings were completed. The extension also allowed for the workshop in Da Nang that had also been delayed due to heavy storms and floods in the central regions and the Central Highlands to be held. The project has also completed the rolling-out of the M&E system in all the 63 provinces and the final evaluation of the project was conducted. 33. Turnover of newly trained staff by the project at national and local levels may affect the pace of operationalization of the ARP M&E system as trained staff may not participate anymore in the rolling out of the M&E system. It was even more of an issue when the hand-over was not carried out adequately with the Page 17 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) new staff, who may not have been well briefed on the ARP indicators and what the M&E manual entails. However, the M&E trainings have increased overall MARD and DARDs’ M&E capacity. The trainings targeted key M&E staff and leaders from DARDs, staff from the Departments of Planning and Finance and representatives of the Provincial Statistical Office. It is good that leaders were trained, so they can fully support the M&E operationalization and change their thinking and mindset about M&E in the agriculture sector. Furthermore, the project supported the preparation of M&E Manuals (in Vietnamese) which will continue to be used by new staff. 34. Slow disbursement at the beginning. The project experienced slow disbursement at the beginning due to delays in establishing the systems of project implementation with DoP and in the eight pilot provincial DARDs. However, once the systems were put in place, implementation picked-up and only slowed down due to the Covid-19 related movement restrictions. The project has disbursed 96.3% of the grant, and about 3.7% (equivalent to VND 1.5 billion) from foreign exchange gains and other savings remained undisbursed and was refunded to the Bank on May 11, 2021. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME 35. M&E Design, Utilization, and Implementation: M&E design, utilization and implementation was Substantial. The project developed indicators were approved by relevant authorities and supported the provinces to collect the data to measure the indicators. The project M&E improvements have been incorporated in the overall sector M&E system, and the project has also supported the development of the software to enhance the integration and real-time utilization of the system at the stroke of a button. The project’s improvements have enhanced the overall performance of the agricultural sector M&E system. Furthermore, the ARP M&E database system is an integral part of the agricultural sector performance M&E database system with different modules for M&E of the sector plans, the sustainable development goals (SDGs), the sector and subsector strategies for 2021- 2030, with a vision to 2050. There are about 180 M&E indicators with updated data from 2016 to 2019 and part of 2020 in the system. 36. Bank Implementation Support: The World Bank team conducted five supervision mission during the three year implementation of the project. The team provided timely and effective support, especially during the project extension period and when the project needed to adjust some of its project activities. Having the same national co-Task Team Leader (Co-TTL) of the project for most of the project implementation was also critical to the project success. It was also essential to make the link with Component A of Vietnam Sustainable Agriculture Transformation Project13 (VnSAT), as the Task Team Leader (TTL) of VnSAT was also the co-TTL of MECARP. It ensured cohesion between the two projects and efficiency in usage of technical assistance. 37. Compliance to safeguards, procurement and financial management: There were no issues related to the project’s compliance to environmental and social safeguards as the project did not trigger any safeguard policies. The project team implemented the project in full compliance to the procurement and financial management procedures. There was no cancellation due to mis-procurement. There were also no audit issues throughout the project implementation. 13 Component A: Institutional Strengthening to Support Agricultural Transformation. Page 18 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) V. LESSONS LEARNED AND RECOMMENDATIONS 38. M&E capacity building of MARD and DARDs staff is the cornerstone of the ARP M&E system roll out in provinces and sustainability after project closure. Therefore, the impact of future projects could be enhanced if greater attention and support could be dedicated to the quality of capacity building activities at design, during implementation and at evaluation stage of the project. Continued provision of the budget (both recurrent and investment) is critical to ensure that continuous and regular implementation of the integrated M&E system, including training and capacity building of M&E staff. 39. Regular training of staff is critical: The training of staff on a continuous basis is a critical part of the M&E system roll-out and to ensure the sustainability of the M&E system after the closing of the project. It will also require continuous improvement of ARP M&E processes and M&E web-based database within provinces and potential expansion to districts, strengthening of M&E learning activities across all provinces, support to provinces to develop their own ARP M&E indicator and conducting regular data collection and statistical surveys for specific indicators. Therefore, Government continued support is essential, through Government’s own funding or grants from other development partners, towards the rolling-out and nationwide mainstreaming of the M&E system. The M&E system should continue to be upgraded and expanded to meet the requirements of M&E for ARP implementation in the period 2021-2025. 40. Strong coordination with statistical agencies such as GSO is important to: (i) ensure accuracy in the measurement of the indicators; (ii) standardized data collection and estimation methods for the indicators across provinces and (iii) maintain consistency and quality of the data collection and indicator estimation. GSO should also continue to provide on a regular basis appropriate training to M&E staff. 41. There is need for a longer time period of implementation to ensure complete mainstreaming of M&E in the implementation of the ARP. Experience from the project showed that it is difficult to achieve sustainable results at scale in a short time frame. A longer time frame (through several projects) and increased funding would be more appropriate, as it takes time to build M&E capacity of staff from national to local level, finalizing ARP indicator definitions and tools for data collection, improving the ARP indicators M&E manual, further improving and expanding ARP indicators web-based database, etc. MARD could also consider incorporating the M&E system implementation into other on-going or future projects in the sector to ensure continuity in mainstreaming results M&E system implementation in the agricultural sector. . Page 19 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: To enhance monitoring and evaluation capacity of Ministry of Agriculture and Rural Development and selected Departments of Agriculture and Rural Development to monitor progress of the Agricultural Res Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Measurable agricultural sector Yes/No N Y Y Y performance and ARP indicators in MARD’s M&E 05-Jun-2017 26-Jan-2018 30-Jun-2020 31-Dec-2020 system adopted Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Direct project beneficiaries Number 0.00 60.00 60.00 68.00 05-Jun-2017 26-Jan-2018 30-Jun-2020 31-Dec-2020 Female beneficiaries Number 0.00 36.00 30.00 36.00 Page 20 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) 05-Jun-2017 26-Jan-2018 30-Jun-2020 31-Dec-2020 Comments (achievements against targets): A.2 Intermediate Results Indicators Component: Support to the development of ARP M&E indicators and M&E system Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion A set of sector performance Yes/No N Y Y Y and ARP indicators developed 05-Jun-2017 26-Jan-2018 30-Jun-2020 31-Dec-2020 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion M&E plan for data collection Yes/No N Y Y Y and methodologies developed 05-Jun-2017 26-Jan-2018 30-Jun-2020 31-Dec-2020 Comments (achievements against targets): Page 21 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Pilot tests for selected Percentage 0.00 80.00 80.00 93.00 indicators conducted 05-Jun-2017 26-Jan-2018 30-Jun-2020 31-Dec-2020 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Manual and Protocols for data Yes/No N Y Y Y collection approved 05-Jun-2017 26-Jan-2018 30-Jun-2020 31-Dec-2020 Comments (achievements against targets): Unlinked Indicators Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion The roadmap for expansion of Yes/No N Y Y Y ARP M&E indicators and system to non-pilot provinces 05-Jun-2017 26-Jan-2018 30-Jun-2020 31-Dec-2020 is adopted by MARD Page 22 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Staff of MARD and DARDs Number 0.00 60.00 60.00 250.00 trained on the enhanced M&E system. 05-Jun-2017 26-Jan-2018 30-Jun-2020 31-Dec-2020 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Feedback flows with private Yes/No N Y Y Y sector, academia, and research organizations 05-Jun-2017 26-Jan-2018 30-Jun-2020 31-Dec-2020 Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Responsibilities and Yes/No N Y Y Y procedures for M&E of MARD, GSO and DARDs reviewed and 05-Jun-2017 26-Jan-2018 30-Jun-2020 31-Dec-2020 Page 23 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) delineated Comments (achievements against targets): Unit of Formally Revised Actual Achieved at Indicator Name Baseline Original Target Measure Target Completion Annual project progress report Yes/No N Y Y Y submitted on time 05-Jun-2017 26-Jan-2018 30-Jun-2020 31-Dec-2020 Comments (achievements against targets): Page 24 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) B. ORGANIZATION OF THE ASSESSMENT OF THE PDO Objective/Outcome 1 1. Measurable agricultural sector performance and ARP indicators in Outcome Indicators MARD’s M&E system adopted 1. A set of sector performance and ARP indicators developed 2. M&E plan for data collection and methodologies developed 3. Pilot tests for selected indicators conducted Intermediate Results Indicators 4. Manual and Protocols for data collection approved 5. The roadmap for expansion of ARP M&E indicators and system to non-pilot provinces is adopted by MARD Key Outputs by Component 1.Training courses organized (linked to the achievement of the Objective/Outcome 1) 2.Database designed and established Objective/Outcome 2 Outcome Indicators 1. Direct project beneficiaries Intermediate Results Indicators 1. Staff of MARD and DARDs trained on the enhanced M&E system. 1. Training courses organized and type Key Outputs by Component 2.Overall staff trained in all Provinces (linked to the achievement of the Objective/Outcome 2) 3. Working groups organized 4. Conferences organized Page 25 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) . ANNEX 2. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M)14 (US$M) 1. Support to the development of ARP M&E 995,500.00 1,027,136.00 103 indicators and M&E system 2. Capacity and institutional strengthening on ARP M&E 430,000.00 403,444.00 93.8 implementation 3. Project management, monitoring and evaluation, 374,500.00 302,203.00 80.7 and knowledge dissemination Total 1,800,000.00 1,732,783.00 96.3 14 At the closure of the grant account on May 11, 2021, US$120,441.79 which was the unused balance was refunded. This includes foreign exchange gains and unused funds. The reason for the difference between the actual disbursement value is that MARD Minister approved the total ODA budget in the project document in both VND and USD with temporary exchange rate of 22,415 VND/USD. Therefore, MARD could only spend under the ceiling budget in VND (40.3 billion VND). In fact, MARD disbursed VND38.8 billion/VND40.3 billion which is equivalent to USD1,732,473 with the exchange rate of 22,415 VND/USD as in the project document (the undisbursed amount is 1.5 billion VND equivalent to about 67,523 USD). More details for this explanation could be found in Annex 3, Point 1 Page 26 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) ANNEX 3. RECIPIENT, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS We have reviewed and much appreciate this report. This is a very comprehensive report on project performance from the start to the completion of the project. It has been well integrated from the project completion report, the final evaluation report and the findings from WB’s 5 supervision missions at the MECARP project. With an aim to further improve this report, we would like to share our comments for your consideration as you finalize the ICR: 1. Page 2: The total ODA cost is 1,732,473 USD: It should be revised that the total ODA cost is 1,679,558 USD as the amount of 1,732,473 was withdrawn by the Project to the designated account. However, on May 11, 2021, the project transferred to the WB an amount of USD 53,068.91 (the bank deducts the WB fee of 154.25 USD), so the Project's actual disbursement amount is 1,679,588 USD. The reason for the difference between the actual disbursement value is that MARD Minister approved the total ODA budget in the project document in both VND and USD with temporary exchange rate of 22,415 VND/USD. Therefore, we could only spend under the ceiling budget in VND (40.3 billion VND). Actually, we disbursed 38.8 billion VND/40.3 billion VND equivalent to 1,732,473 USD if application of the exchange rate of 22,415 VND/USD as in the project document (the undisbursed amount is 1.5 billion VND equivalent to about 67,523 USD). Response: The difference has been addressed in the ICR. Similarly, please update the disbursement value in the page 13. 2. Page 6: Results chain: Please check cross between the results chain in page 6 and the table on organization of the assessment of the PDO in the page 22: some contents are not consistent and different between immediate results and main outputs in these pages. 3. Page 9: The note at the end of the page 9: “The following non-pilot provinces participated in the discussions: Dien Bien, Hoa Binh, Ha Nam, Ha Noi and Quang Binh�. With these provinces, we understand that you mentioned about all discussions in the 4th and 5th missions from WB and discussions from the final evaluation team. If so, please add the following provinces in the list: Son La, Nghe An, Binh Phuoc and Ba Ria Vung Tau. Response: This has been corrected accordingly. 4. Page 10: “The project also conducted 5 out of the planned 7 conferences/workshops, including 2 regional conferences on coordination between statistical and agricultural sectors, especially to help improve the cooperation between MARD and GSO in standardizing the methodology for the calculation of the ARP indicators and consultation meetings on the report on assessment of food Page 27 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) processing capacity and mechanization in agriculture in the South and another conference to share experiences among the pilot provinces in using the ARP indicators� This is the implementation result of 2019. However, 2 unimplemented meetings of 2019 were moved to the beginning of 2020 and conducted in the form of online meetings. Since this is a project completion report, it is necessary to synthesize information from annual reports or information from project completion reports provided by the PMU. Therefore, this paragraph should be updated with inputs information from our completion report as below: “The project has supported MARD in organizing 20 conferences, consultation workshops on the ARP performance, the conference on national food security, agricultural mechanization promotion conference, conferences on agriculture and rural development policy communication., international workshops to learn experiences from countries in the region that have successful M&E systems. From the beginning of 2020, although the Covid-19 epidemic caused disruption in the project implementation, by mid-December 2020, the project also supported the Ministry to organize 5 national conferences with the attendance of 422 delegates from central and local agencies, some of which are at big scale and very important, such as: Conference on Review of 10 years implementation of the national food security program, National conference on agricultural processing and mechanization capacity assessment, Conference on "Women of Ministry of Agriculture and Rural Development participate in the agricultural restructuring implementation M&E... The National Conferences on midterm review of the use of the approved set of indicators for M&E the ARP implementation were organized: The conferences were held in the North and the South for nearly 200 delegates,…. (Source: The MECARP project completion report). Response: This has been incorporated. 5. Page 11: Overall Outcome Rating Please update more about the ARP M&E database system as this database system is a highlighted result of the project: The ARP M&E database system was expanded to be the agricultural sector performance M&E database system with different modules for M&E of the sector plans, the sustainable development goals (SDGs), the sector and subsector strategies for 2021- 2030, vision to 2050.There have been about 180 M&E indicators with updated data from 2016 to 2019 and some of 2020 in the system. Response: This has been incorporated 6. Page 11: “There are still some shortcomings that needs to be addressed to strengthen the current ARP M&E system and ensure its sustainability. The project has faced some difficulties in collecting data and quantifying some of the indicators. The indicator 6 (growth rate of income per one hectare of salt production) was not collected, as it requires a statistical survey and MARD has recently allocated some budget to undertake this survey.� These are not the shortcomings of the project but the difficulties. As in the project M&E framework, the target for the indicator 3 “Selected indicators piloted� is 80% equivalent to 12/15 Page 28 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) ARP indicators are tested. However, the project has collected information for 14/15 criteria equivalent to 93% (reaching 117% of the targets). At the same time, the ARP indicator 6 is required to collect data from survey, therefore MARD allocated its budget for this survey in 2020 and the data for this indicator will be available in 2021. Response: This has been corrected. 7. Page 12: “The project has also supported MARD to develop 2 Statistical Circulars on the System of Statistical Indicators and Regulation on Agricultural Sector Statistical Reporting System (to be submitted to the Ministry for promulgation in December 2020)� Please update as below: “The project has also supported MARD to develop 2 Statistical Circulars on the System of Statistical Indicators and Regulation on Agricultural Sector Statistical Reporting System. MARD Minister approved the Circular on the System of Statistical Indicators in the Decision No. 16/2020/TT- BNNPTNT dated December 28th, 2020 and the Circular on Regulation on Agricultural Sector Statistical Reporting System in the Decision No. 16/2020/TT-BNNPTNT dated December 28th, 2020�. Response: This has been incorporated. 8. Page 13: “Turnover of newly trained staff by the project at national and local levels has affected the pace of operationalization of the ARP M&E system as trained staff could not participate anymore in the rolling out of the M&E system. It was even more an issue when the hand-over was not carried out adequately with the new staff, which did not know the ARP indicators and what the M&E manual entails. The M&E trainings have increased overall MARD and DARDs’ M&E capacity, but the impact of trainings in none-pilot provinces was affected by the choice of who could attend trainings and seminars. The trainings in none-pilot provinces targeted mostly staff that were not the ones involved with data collection and calculation of indicators. The trainings targeted leaders from DARDs, staff from the Departments of Planning and Finance and representatives of the Provincial Statistical Office, who are not the ones who calculate the indicators and need to fully understand the indicators. It is good that leaders are trained, so they can fully support the M&E operationalization and change their thinking and mindset about M&E in the agriculture sector, but this training approach led to some inconsistencies in the understanding and calculation of indicators across provinces� There is no evidence for the assessment: “Turnover of newly trained staff by the project at national and local levels has affected the pace of operationalization of the ARP M&E system as trained staff could not participate anymore in the rolling out of the M&E system� It should be changed as follows: “Turnover of newly trained staff by the project at national and local levels may affect on the pace of operationalization of the ARP M&E system as trained staff may not participate anymore in the rolling out of the M&E system� There is no evidence for the assessment “The trainings in none-pilot provinces targeted mostly staff that were not the ones involved with data collection and calculation of indicators�. As we understand that WB has referred to the Final evaluation Report. However, in the Final Page 29 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) Evaluation Report, the consultants said that “There is still a situation that people who attend the training do not work, who work does not go to training�. This assessment does not mean “mostly�’. Moreover, the final evaluation team visited only 2 non-pilot provinces, therefore, we cannot use one or two cases as popular situation in non-pilot provinces. Response: These have been incorporated and some statements corrected/clarified. 9. Page 16: The M&E framework: It should correct the data and the date of completion: - The column “Actual Achieved at Completion�, the date “14 Feb 2020� seems to be incorrect, it should be 31 Dec 2020. - The indicator “Staff of MARD and DARDs trained on the enhanced M&E system�, the final target is 250, not 126. - The indicator� “The roadmap for expansion of ARP M&E indicators and system to non-pilot provinces is adopted by MARD�: MARD approved the road map in the Decision No,5430/QD-BNN- KH dated December 31st, 2020. Response: Corrected. Page 30 of 31 The World Bank MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP) (P159760) ANNEX 4. SUPPORTING DOCUMENTS (IF ANY) MARD, 2021. Project Completion Report. Report Prepared by the Department of Planning and the Project Management Unit. Hanoi, Viet Nam. Viet Nam Sustainable Agricultural Transformation Project (VNSAT), various Aide Memoires (Component A). MARD M&E Capacity Building for Agricultural Restructuring Plan Implementation (MECARP), various Aide Memoires. MARD and Swiss Agency for Development Cooperation (SDC). 2016. Monitoring and Evaluation in Support of Management in the Agriculture and Rural Development Sector – Phase II. Page 31 of 31