Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) Report Number: ICRR0022187 1. Project Data Project ID Project Name P123960 VN-Social Assistance System Strengthenin Country Practice Area(Lead) Vietnam Social Protection & Jobs L/C/TF Number(s) Closing Date (Original) Total Project Cost (USD) IDA-53540 31-Dec-2019 27,103,925.06 Bank Approval Date Closing Date (Actual) 22-Jan-2014 31-Dec-2019 IBRD/IDA (USD) Grants (USD) Original Commitment 60,000,000.00 0.00 Revised Commitment 32,981,691.10 0.00 Actual 27,103,925.06 0.00 Prepared by Reviewed by ICR Review Coordinator Group Katharina Ferl Judyth L. Twigg Soniya Carvalho IEGHC (Unit 2) 2. Project Objectives and Components DEVOBJ_TBL a. Objectives According to the Project Appraisal Document (PAD, p. ii) and the Financing Agreement of April 29, 2014 (p. 4), the objective of the project was “to support the government of Vietnam in strengthening the social assistance system by developing innovations in management and service delivery nationwide, and by piloting these innovations in the four project provinces.” Page 1 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) b. Were the project objectives/key associated outcome targets revised during implementation? No c. Will a split evaluation be undertaken? No d. Components The project included three components: Component 1: Strengthening the social assistance and poverty reduction system (appraisal estimate US$27.0 million, actual US$18.4 million): This component was to finance the development and testing of improved mechanisms for social assistance (SA) delivery and management, including the following: a) building a national database of social assistance beneficiaries and an integrated Management Information System (MIS) (POSAsoft) for nationwide use; b) developing a social collaborator network in the four provinces (Ha Giang, Quang Nam, Tra Vinh and Lam Dong) and supporting communication for development (C4D) activities at the national level and in the four provinces, including provision of remuneration for social collaborators; c) developing and implementing a grievance redress mechanism in the four provinces; and d) developing and implementing a monitoring and evaluation system in the four provinces and providing support for policy formulation at the national level and in the four provinces. Also, this component was to support the remaining 59 provinces in putting in place the necessary preconditions (i.e., guidelines, training, software and hardware) to use the strengthened system for social assistance management in a phased manner. Component 2: Launching a consolidated social assistance program (appraisal estimate US$30.0 million, actual US$6.2 million): This component was to test the mechanisms developed under Component 1, through the launching of social assistance consolidation, including the following: Component 2a): This sub-component was to cover a pre-identified financing shortfall for benefit payments for the Opportunity Program, a consolidated social assistance program that was intended to reduce fragmentation across multiple programs and agencies in the four project provinces for a period of three years. The financing shortfall between the government budget for the three existing programs for three years in the four provinces and the total estimated cost of the consolidated Opportunity Program was estimated to be due to expanded coverage of the Opportunity Program. This expanded coverage, under government Decree 136, extended benefits to pregnant women, children ages 0-3 years, and children ages 3-15 years from poor households who were not attending school. The program was to be implemented in the third year of the project, once MIS capabilities and other elements of the strengthened service delivery mechanism were in place. Component 2b): This sub-component was to provide the service fee of an independent payment agency (VN Post) for paying the cash transfers under the Opportunity Program and Decree 136 as well as paying the monthly allowance of the social collaborators. Component 3: Project management (appraisal estimate US$5.5 million, actual US$3.5 million): This component was to finance project management and capacity building support. A Central Project Management Unit (CPMU) and four Provincial Project Management Units were to be established to oversee the development and testing of the strengthened social assistance management and delivery system, Page 2 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) coordinate the consolidation of social assistance programs, manage and monitor the implementation of the Opportunity Program, and ensure appropriate fiduciary controls were in place. e. Comments on Project Cost, Financing, Borrower Contribution, and Dates Project Cost: The project was estimated to cost US$62.5 million. Actual cost was US$28.10 million for the following reasons: first, the number of additional beneficiaries of the Opportunity Program turned out to be much lower than estimated at appraisal; and second, the periods of benefit payment were shorter than planned at appraisal (see also the second bullet under “Dates” below). Financing: The project was financed by an IDA loan in the amount of US$60.0 million, of which US$27.1 million was disbursed. Borrower Contribution: The Borrower was to contribute US$2.50 million. Actual contributions were US$1.0 million due to underspending at the local level. Actual cost norms applied at the local level were lower than those projected at the national level. Dates: The project was restructured three times:  On September 5, 2017 the project was restructured to revise the Results Framework to provide more accurate intermediate indicators under Component 2.  On September 4, 2019 the project was restructured to cancel unused funds in the amount of US$21.0 million. Reasons for the cancellations were: i) the government was projected to use a 15 percent poverty rate in 2015. However, instead it used about 9 percent in 2015, 7.85 percent in 2016, and 6.70 percent in 2017 due to better than expected aggregate progress in poverty reduction. This had an impact on the number of households eligible for project benefits; ii) the Ministry of Finance (MoF) reduced the number of additional beneficiaries by approximately 2 percent since it agreed to support income-based poor households rather than poor according to other economic and social dimensions; iii) in some localities there was no consensus in supporting poor children who did not attend school and poor pregnant women, resulting in another reduction in the number of additional beneficiaries; and iv) the periods of benefit payment were shorter than expected (maximum of nine months for pregnant women instead of 12 months and nine months for children instead of 12 months).  On January 1, 2020 the project was restructured to cancel unused funds in the amount of US$2.64 million, for the same reasons as stated above. The project closed on its original closing date of December 31, 2019. 3. Relevance of Objectives Rationale According to the PAD (p. 1), Vietnam has been experiencing fast economic growth throughout the last four decades. This progress has lifted millions of Vietnamese out of poverty since the early 1990s. However, the Page 3 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) speed of poverty reduction has been slowing down, with the remaining poor population suffering from chronic poverty, especially minority groups living in isolated areas and households with low education and skills. In addition, Vietnam’s increased global integration and more mobile population have resulted in more opportunities for its population but also resulted in people becoming more vulnerable to economic and income shocks, as well as weather and health shocks. Furthermore, Vietnam continues to face the risk of an intergenerational poverty trap undermining its objective of sustainable poverty reduction. Despite the government’s efforts and significant progress made in achieving the Millennium Development Goals, education, health, and nutrition outcomes are significantly worse for children from poor households, especially for ethnic minority children, translating into the transmission of poverty to the next generation. The PAD (p. 2) stated that, at the time of appraisal, Vietnam's social assistance system (including about 12 cash-transfer programs) suffered from fragmentation of multiple poverty reduction and social assistance programs with overlapping objectives but separate budgeting and delivery mechanisms. The biggest cash transfer programs were: i) a monthly cash support to the disabled, orphans, single parents, and lone elderly without means; ii) a small monthly cash transfer to poor households to offset recent increases in energy prices; and iii) a cash transfer to children from poor households in school for nine months per year delivered through the education system (school principals). However, a benefit incidence analysis from 2010 showed that the main existing social assistance programs had limited coverage of the poor, considerable leakage to the non-poor, and low benefit adequacy. In addition, delivery systems for social assistance programs were weak, and other mechanisms for benefit payments and monitoring and oversight of social assistance were also inadequate. The objective of the project supported the government’s 2012 Resolution on "Major Issues in Social Policies of the Period 2012-2020," which aimed to ensure "guaranteed minimum level in income, education, health care, housing, clean water, and information and communication, thus contributing to gradual improvements in incomes and ensured safe, equitable and happy life for the people" by 2020. Specifically, the resolution emphasized the need to consolidate and modernize the management of programs and policies (including modernized payment mechanisms and an integrated beneficiary database) to ensure less fragmentation and overlaps of programs. Also, the project supported the government's overall objective of poverty reduction as laid out in the government's Resolution on the Directions for Sustainable Poverty Reduction (2011-2020). At project closure, the objective of the project was in line with the Bank’s most recent Country Partnership Framework (FY18-22) and its focus area “invest in people and knowledge” and objective 7, “improve integration and efficiency of social assistance, pension, and health insurance systems." The objective of the project was not sufficiently clear and lacked specificity in terms of what the project intended to do. It is noteworthy that the objective of the project did not specifically include consolidation and/or integration of fragmented programs, which had been cited consistently as a major shortcoming of the country's social assistance efforts. The relevance of objectives is therefore rated Substantial. Rating Relevance TBL Rating Substantial 4. Achievement of Objectives (Efficacy) Page 4 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) EFFICACY_TBL OBJECTIVE 1 Objective To support the government of Vietnam in strengthening the social assistance system by developing innovations in management and service delivery nationwide, and by piloting these innovations in the four project provinces Rationale The project’s theory of change envisioned that project activities such as the establishment of an MIS and digitalization of SA beneficiaries were to result in project outputs such as the development of a beneficiary registry and MIS. These project outputs were to result in the usage of a registry and MIS to manage SA programs, which was to result in the outcome of the efficient management of SA programs. Also, the project’s theory of change envisioned that project activities such as the development of Communication for Development (C4D) and the establishment and financing of a social collaborator network were to result in project outputs such as the development of a C4D action plan to support consolidation, improved uptake of SA programs, and behavior change communication as well as training and operations of a social collaborator network. These project outputs were to result in higher beneficiary awareness of programs’ processes and benefits and increased parenting knowledge and household support, which was to result in the outcome of improved SA delivery. Furthermore, project outputs such as the establishment of a feedback and redress mechanism, the selection and operation of an independent payment provider, and the inclusion of additional beneficiaries under the Opportunity Program were to result in outputs such as a functioning feedback and redress system, the delivery of consolidated payments on time, and the enrollment of additional beneficiaries. These outputs were to result in outcomes such as an increased number of feedback and complaints, increased payment satisfaction due to predictability, timeliness, convenience, and savings as well as increased SA coverage of vulnerable groups, which was to have a positive impact on the availability of evidence for program consolidation at the national level. Finally, the project aimed to align SA programs and reduce fragmentation by supporting the consolidation of three cash transfer programs into the Opportunity Program in the four project provinces. All of these outcomes were to result in the project’s objective of strengthening the social assistance system. Outputs:  The national program MIS was developed and is operational, covering all beneficiaries of SA programs and all members of poor and near-poor households, achieving the target of a national program MIS being developed and operational. The ICR (p. 20) stated that the MIS allowed for more frequent updates and increased flow of information, which had a positive impact on the planning process, and resulted in a decrease in administration costs as well as improved reporting speed from the commune to central level. Page 5 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960)  The project supported the digitalization of beneficiaries’ information under the POSASoft program. Activities included data collection and digitalization for SA beneficiaries and poor and near-poor households following the 2015 Poverty Census. In addition, 2016 data collection, standardization, and cross-checking were conducted in all provinces. When the project closed, 58 out of 63 provinces had finished digitalizing 100 percent of the 2018 data of beneficiaries. This achieved he target of a national beneficiary database having been created and made operational. However, only 50 percent of the provinces were able to update the 2019 data for SA program beneficiaries.  The project selected an independent payment provider (VNPost) to deliver payment of SA cash benefits (instead of government staff doing so). The aim was to reduce the workload of government staff, improving transparency, ensuring timely payments, and reducing time and cost for beneficiaries. By the time the project closed, 58 out of 63 provinces used this payment model.  While the project did not have any data on the reduction of inclusion and exclusion errors, it appears that the establishment of the beneficiary registry had a positive impact on identifying issues such as schools not returning money for students who had dropped out of school and payments to deceased beneficiaries. Also, the project established a formal feedback and redress system on an existing hotline that the Ministry of Labor, Invalids and Social Affairs (MOLISA) has been using to receive beneficiary feedback and grievances for other programs.  According to the ICR (p. 20) the project developed a network of social collaborators who supported beneficiaries in understanding the program, benefits, and responsibilities of beneficiaries in regards to childbearing, caring, and education. However, the 2018 Beneficiary Feedback Survey found that the social collaborators were not able to increase the beneficiaries’ knowledge of parenting and link beneficiaries to services. The ICR (p. 22) stated that even though 71 percent of beneficiaries knew their social collaborator, only 16 percent of beneficiaries remembered participating in parenting sessions or meetings with social collaborators encouraging the use of benefits to access maternal or pediatric health care services. The ICR speculated that the reason for the poor performance of collaborators was a low monetary incentive, resulting in high turnover and frequent retraining sessions.  18,909 individual additional beneficiaries were registered, not achieving the annual target of 22,000 beneficiaries.  The ICR (p. 29) stated that the project was unable to consolidate the three existing cash transfer programs under the Opportunity Program in the four pilot provinces due to frequent policy changes by different ministries, producing suspension of registrations and delays in payment during the transition period. Also, the programs used different budget mechanisms and benefit payment frequency. While the project ensured that payments were made as scheduled in the respective policy document, it was not able to develop an integrated monthly package ensuring a consistent amount being paid on a monthly basis. Outcomes:  96.5 percent of benefit payments were made on time, surpassing the original target of 75 percent and the revised target of 80 percent.  The percentage of beneficiaries satisfied with the payment process increased from 65 percent in 2015 to 93.7 percent in 2018, surpassing the original target of 75 percent and the revised target of 85 percent. Also, a 2018 Beneficiary Feedback Survey showed that 80 percent of beneficiaries were satisfied with VNPost, as beneficiaries benefited from fixed payment schedules, adequate facilities at Page 6 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) payment locations, and reduced transaction costs (travel costs, travel time, and waiting time). However, according to the ICR (p. 21), beneficiaries in remote areas still face challenges, including access.  Specific findings of the 2018 Beneficiary Feedback Survey included the following, which might be indicative of positive impact in some areas through strengthening SA: o In 2015 68 percent of beneficiaries were aware of eligibility criteria, which decreased to 58 percent in 2018. o In 2015 65 percent of beneficiaries were aware of the benefit, which increased to 89 percent in 2018. o In 2015 four percent of beneficiaries were aware of the feedback mechanism, which increased to 12 percent in 2018. o In 2015 the transportation cost to payment locations was 10,900 VDN, which decreased to 5,200 VDN in 2018. o In 2015 the average travel time to payment locations was 37 minutes, which decreased to 20 minutes in 2018. o In 2015 59 percent of children (ages zero to five years old) were monitored or had a check up, which increased to 65 percent of children in 2018. o In 2015 69 percent of children had routine immunization during the past 12 months, which decreased to 59 percent of children in 2018. o In 2015 83 percent of children (ages three to 15 years old) were enrolled in school, which decreased to 81 percent in 2018. According to the Bank team (September 14, 2020) in Vietnam, policy integration requires change in business processes, which needs to be carried out under issuance of a national level decree. Under the project, a special, inter-ministerial circular (between MOLISA, the Ministry of Education and Training (MOET), and MOF) had been issued that allowed the pilot to integrate several programs into the Opportunity Program. However, after the project pilot completed, these policies could not be integrated without further involvement of MOLISA at the national level to continue and/or expand the revised business processes that were allowed in pilot provinces under the inter-ministerial circular. Also, the Opportunity Program could not be continued without further legal decision by MOLISA in the pilot provinces. Although the piloting of consolidation of fragmented programs and expansion of coverage did not take place as envisaged, and therefore the Opportunity Program was not scaled up as planned, the project substantially strengthened the SA system through the development of the MIS and beneficiary database, as well as selection and operation of the independent payment provider. Achievement of the project objective is therefore rated Substantial, but with important caveats. Page 7 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) Rating Substantial OVERALL EFF TBL OBJ_TBL OVERALL EFFICACY Rationale The project achieved its objective of developing innovations by building the beneficiary registry and MIS platform. Also, the project was able to develop innovations in service delivery in project provinces by establishing a grievance redress mechanism, supporting Communication four Development (C4D) activities, and using an independent payment provider to deliver social assistance cash benefits. The project did not fully succeed in establishing a network of social collaborators and did not achieve the target for additional beneficiaries being registered. Also, the project was not able to implement the pilot as planned and did not chart a clear path forward for the consolidation of all the different benefits programs as planned. Taking everything together, the efficacy rating is Substantial but with important caveats. Overall Efficacy Rating Substantial 5. Efficiency Economic Efficiency: Neither the PAD nor the ICR conducted a traditional economic analysis. The PAD (p. 12) stated that conducting a standard cost-benefit analysis for system strengthening projects is challenging, since it is difficult to quantify the impact of systemic reforms. Instead, the PAD outlined several benefits of the project, including those to the government from improvements in the cost-effectiveness of the SA system and to beneficiaries from the Opportunity Program directly. These benefits included improved service delivery of SA, and better awareness of SA and poverty reduction programs. Also, benefits included more efficient program procedures, better trained officials, and reduced duplication of effort and resources across programs and agencies. The ICR (p. 24-26) also outlined the benefits of project implementation and provided data on potential annual savings from improved systems. The ICR estimated that due to the project, the government was to save a total of US$17.72 million. These savings were to result from preparing monthly beneficiary lists, reconciliation, and accounting and reporting, as well as savings on travel costs and waiting times. Also, according to the ICR (p. 26), the cost of developing the MIS and database was US$2.59 per household (US$1.43 per household, deducting the training costs related to POSASoft). Comparing these to the cost of developing Turkey’s integrated SA system (US$1.30 per household excluding training costs), it appears that this project’s costs were reasonable. Page 8 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) Operational Efficiency: The project experienced some delays in procurement and updating the registry with data of SA beneficiaries and of poor and near-poor households. This resulted in program administrators not being able to use the data for identifying potential additional beneficiaries. Despite these delays, the project closed on its original closing date. However, according to the ICR (p. 30), the project experienced high staff turnover and reduction at the local level due to the government’s administration reform (aiming to achieve a 10 percent reduction in staffing by 2020), which had a negative impact on project implementation. Furthermore, the project faced challenges with policy limitations at the local level and an insufficiently clear definition of program consolidation. These issues resulted in shortcomings in addressing program consolidation. The project realized cost savings in several areas and cancelled funds when it became apparent that program consolidation would not be scaled up as planned. However, the shortcomings in anticipating the steps that would be necessary for national-level program consolidation represented a missed opportunity to take the consolidation agenda forward. Efficiency is therefore rated Substantial, but with important caveats. Efficiency Rating Substantial a. If available, enter the Economic Rate of Return (ERR) and/or Financial Rate of Return (FRR) at appraisal and the re-estimated value at evaluation: Rate Available? Point value (%) *Coverage/Scope (%) 0 Appraisal 0  Not Applicable 0 ICR Estimate 0  Not Applicable * Refers to percent of total project cost for which ERR/FRR was calculated. 6. Outcome Relevance of the objective was Substantial, given its alignment with the Bank’s most recent Country Partnership Framework (FY18-22). Efficacy was Substantial due to progress on the MIS platform, beneficiary registry, and use of an independent payment provider but with important caveats with regard to not being able to implement the pilot. Efficiency was Substantial, but with important shortcomings, due to operational inefficiencies. Based on these overall moderate shortcomings with both Efficacy and Efficiency, the project’s outcome rating is Moderately Satisfactory. Page 9 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) a. Outcome Rating Moderately Satisfactory 7. Risk to Development Outcome Risk to the sustainability of project outcomes can be classified into the following categories: Technical/capacity: According to the ICR (p. 35), updating of SA program information was regularly conducted under the project. However, data for poor and near-poor households was only updated on an annual basis, resulting in a large volume of data needing to be updated each year at the district level. The project developed an offline solution so that data could be updated on the commune level, but some communes faced capacity constraints. According to the ICR (p. 35), the sustainability of the MIS is ensured by MOLISA’s Ministerial Circular from July 2019 (“Regulations on development, management, update and exploitation and use of SA and poverty reduction databases”), which supports the sustainable and effective use of the new system at the national and sub-national levels. In order to ensure continuous availability of data, it will be critical to build capacity at the commune level to allow for updating of data on a regular basis. Also, it will be important to keep up to date with any new technology, upcoming reforms, and needs of potential users at the national and local levels. Financing: While the project established a social collaborator network, this pilot ended in June 2018, and the government has since used an already-existing network of social collaborators from other programs (such as population and grassroots health support programs) rather than continuing a separate SA-specific network of social collaborators. The government has not been willing to support the additional financing that a dedicated network would require. 8. Assessment of Bank Performance a. Quality-at-Entry According to the ICR (p. 33), during project preparation the Bank fostered a partnership between MOLISA, the United Nations Children’s Fund (UNICEF), and the UK Department for International Development. The Bank team identified relevant risks such as the relatively low capacity of the implementing agency. According to the PAD (p. 12), this project was the first Bank-funded project implemented by MOLISA, and the ministry had limited prior experience in the technical and procedural management of operations of this scope, including on fiduciary aspects. In order to mitigate this risk, the Bank organized several training and study tours for government counterparts to visit countries that operated large SA programs. The ICR (p. 29) stated that during project preparation the Bank conducted several field trips and consultative workshops at the central and local level to identify provinces for the pilot program. According to the ICR (p. 33), the project design built on lessons learned from other Bank and donor-supported projects. The main lessons were that consolidation of SA programs increases Page 10 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) efficiency and effectiveness, and that SA programs need to be supported by a national beneficiary registry of poor and vulnerable households with strong MIS functionalities. The Bank team did not identify several risks that materialized. For example, the number of beneficiaries of the Opportunity Program was smaller and the periods of benefits payments were shorter than planned, resulting in a substantial cancellation of project funds (US$27.49 million). Also, the project experienced high staff turnover and reduction at the local level because of government administration reform that ordered a 10 percent reduction in staffing by 2020, resulting in contracted staff being let go and local staff having to take on additional tasks and carry out project activities such as learning to use the new MIS system and updating data into the system. Importantly, the ICR (p. 33) stated that the project would have benefited from a longer discussion of policy limitations at the provincial level in regards to what policies could be integrated under the Opportunity Program without involvement of the national MOLISA. As stated in the efficacy section, in Vietnam, policy integration requires change in business processes which needs to be carried out under issuance of a national-level decree. While a special, inter-ministerial circular (between MOLISA, MOET, and MOF) had been issued that allowed the pilot to integrate several programs into the Opportunity Program, when the project pilot completed, these policies could not be integrated without further involvement of MOLISA at the national level to continue and/or expand the revised business processes that were allowed in pilot provinces under the inter-ministerial circular. Also, the Opportunity Program could not be continued without a further legal decision by MOLISA in the pilot provinces. These political economy issues could have been identified if the Bank team had conducted a political analysis during project preparation. The Results Framework had several shortcomings in M&E design (see Section 9a for more details). Quality-at-Entry Rating Moderately Unsatisfactory b. Quality of supervision According to the Bank team (September 14, 2020), the Bank conducted 12 supervision missions as well as interim technical support missions when necessary. The project had three different Task Team Leaders (TTLs) throughout its implementation. One TTL and the fiduciary and safeguard team members were based in the country office, which allowed for continuous dialogue with the government and provided the opportunity to provide technical assistance to address implementation bottlenecks. The ICR (p. 34) stated that the Bank was able to provide a technical expert for the development of information systems and M&E activities. Also, the Bank team promoted adjustments in project management and enabled local governments, vendors, and service provider teams to build a new MIS and payment systems by providing training and technical support to the project teams. The Bank team modified the project’s Results Framework to ensure its alignment with project activities. According to the ICR (p. 34), after the Mid-Term Review, the Bank tried to expand project interventions to develop an integrated social protection system with the surplus of funds that arose due to fewer beneficiaries and shorter payment periods in the Opportunity Program. However, it was a moderate Page 11 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) shortcoming that the Bank team did not restructure the project once it became clear that it would not be possible to implement the pilot. Quality of Supervision Rating Moderately Satisfactory Overall Bank Performance Rating Moderately Satisfactory 9. M&E Design, Implementation, & Utilization a. M&E Design The project’s theory of change and linkages between key activities and intended outcomes were soundly reflected in the Results Framework. The Results Framework included two PDO and five intermediate outcome indicators. All indicators were measurable, had a baseline where appropriate, and included a target. However, according to the ICR (p. 11), the original Results Framework had several shortcomings: i) intermediate process indicators were used to measure institutional capacity and system strengthening, but the measurement systems for these process indicators were weak and were only established during the first year of project implementation; and ii) while the project aimed to influence national SA delivery systems, the activities under component 2 only focused on the four pilot provinces. Also, the objective was not sufficiently clearly stated and overly broad and not sufficiently specific. According to the PAD (p. 10), the project was to support a process evaluation, using two rounds of surveys combined with qualitative research during the course of implementation. The aim was to assess whether the project activities were being implemented as planned and the project was on track to achieve its objective. In addition, the project was to conduct internal spot checks for beneficiaries, monthly brief surveys for the payment service provider (VNPost), social collaborators, and MIS operator, household surveys (baseline and end-line surveys), and Beneficiary Feedback Surveys. MOLISA, the main implementing agency, was to be responsible for the project’s M&E activities. b. M&E Implementation According to the Bank team (September 14, 2020), the project put in place a sound M&E system to monitor implementation and evaluate results. Multiple tools used by various stakeholders were implemented to get information from various sources for timely decision making, including regular (monthly) third party assessments and spot checks, beneficiary and community feedback surveys, household surveys, and a comprehensive process evaluation conducted throughout project implementation. The Bank team stated that these M&E activities helped the CPMU to monitor whether Page 12 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) the planned innovation/reforms were implemented as intended, address implementation bottlenecks, and obtain timely feedback to draw lessons for scale-up and future reforms. Furthermore, the Bank team stated that the project conducted a review and assessment of the business processes for delivering SA program in project provinces to: i) get better understanding of local conditions to allow for further work being done until project closure. This was to ensure that the database, MIS, and other integrated system building blocks were fully aligned and effectively used by provinces; ii) draw lessons and recommendations to government in order to expand the project’s benefits to other provinces; iii) analyze business requirements by central and local offices in order to propose needed business process re-engineering to make the best use of new systems, simplify processes, and reduce administrative burdens for both beneficiaries and local staff. The findings of the assessment were discussed at a consultation workshop with participating provinces as well as others. The project had planned to conduct a second round of household surveys in June 2018. However, the ICR (p. 31) stated that due to implementation challenges related to social collaborators under component 2, it was decided to conduct a process evaluation at the end of project implementation instead. In September 2017, the project was restructured to revise the Results Framework to provide more accurate intermediate indicators under Component 2. A new indicator to track the number of individual beneficiaries registered under the Opportunity Program was added. The indicator on “share of benefit payments for the Opportunity Program made on time in each pilot province” was modified to “share of benefit payments on time,” and the indicator on “share of beneficiaries reporting satisfaction with the Opportunity Program in the project provinces” was revised to “share of beneficiaries reporting satisfaction with the Opportunity Program in the project provinces payment process." c. M&E Utilization According to the ICR (p. 31), M&E data was used to inform decision making throughout project implementation. Spot checks identified implementation bottlenecks in a timely manner, including issues with enrollment procedures for additional beneficiaries due to the lack of support from local officials and lack of identification and supporting documents. The baseline survey identified key gaps in service delivery before project implementation started. Furthermore, Beneficiary Feedback Surveys provided the team with feedback about new activities introduced by the project (e.g. such as social collaborators facing challenges to organize meetings for providing parenting support). According to the Bank team (September 14, 2020), the administrative reporting part of M&E functions is likely to be sustained after project closure and has led to improvements in regular reporting of the program. M&E Quality Rating Substantial 10. Other Issues a. Safeguards Page 13 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) The project was classified as Category C and triggered the Bank’s safeguard policy OP/BP 4.10 (Indigenous People). The ICR (p. 31) stated that ethnic minority groups faced barriers at every stage of SA delivery due to their location, language, capacity, and norms. Also, households of ethnic minority groups had lower SA coverage, since potential ethnic minority beneficiaries and beneficiaries had limited knowledge of the program and benefits. According to the ICR (p. 32), an Ethnic Minority Development Plan and a Social Assessment were prepared, and ethnic groups were consulted during project preparation. Also, the project conducted communications and outreach activities. Furthermore, benefits were delivered through the mail, which had a positive impact on accessibility for ethnic minority beneficiaries living in remote villages. Throughout project implementation the project’s compliance with OP/BP 4.10 was rated Satisfactory. b. Fiduciary Compliance Financial Management: At project appraisal, financial management risk was rated High. According to the ICR (p. 32), in order to mitigate this risk, the project implemented several actions including: i) adopting a Joint Ministry Circular (MOLISA, MOET, and MOF) to define the project’s financial management framework; ii) developing a Project Implementation Manual defining financial management guidelines and procedures; iii) developing and implementing a MIS in all project entities to collect information on beneficiaries, provide data for generating the payment instructions for VNPost, and generate the required financing reports; iv) separating functions of targeting beneficiaries, payments, and verification; v) implementing actions to ensure transparency throughout the implementation of the Opportunity Program; and vi) setting up an internal audit function. According to the ICR (p. 32), MOLISA complied with the financial management-related covenants of the financing agreement and submitted quarterly unaudited interim financial reports, which were accepted by the Bank, as well as annual audited financing statements covering the project. The ICR (p. 32) stated that the Bank provided financial management training to implementing agencies at all levels, resulting in increased financial management capacity at MOLISA and other relevant departments throughout implementation. Throughout project implementation financial management was rated Moderately Satisfactory. The ICR (p. 33) stated that at project closure, the project’s financial management rating was Satisfactory. Procurement: The ICR (p. 33) stated that due to MOLISA’s lack of experience in implementing Bank projects, the project procurement risk was rated Substantial at appraisal. The Bank team conducted procurement trainings for the CPMU and other implementing agencies. The project’s procurement activities were carried out in accordance with the Bank’s anti-corruption guidelines. The project experienced some procurement-related delays at the beginning of project implementation due to the project having to follow Bank and government Page 14 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) regulations, rules, and procedures. However, overall procurement performance was rated Satisfactory throughout project implementation. c. Unintended impacts (Positive or Negative) In March 2020, the government announced an income support package to support the poor, near-poor, SA beneficiaries, and other defined groups during the COVID-19 outbreak. According to the ICR (p. 28), the national beneficiary registry and MIS platform developed under the project were used to support the COVID-19 safety net package implementation. Some provinces were able to update poverty data into the system to scale up support. Also, the system was able to facilitate business administration and monitoring. d. Other --- 11. Ratings Reason for Ratings ICR IEG Disagreements/Comment Moderately Outcome Moderately Satisfactory Satisfactory The project's objective lacked clarity and was insufficiently specific. Also, the Bank team did not anticipate adequately the challenges that would arise from Bank Performance Satisfactory Moderately Satisfactory consolidation of programs. The project was not restructured when it became apparent that program consolidation would not be possible. Quality of M&E Substantial Substantial Quality of ICR --- Substantial 12. Lessons The ICR (p. 36-38) included useful lessons learned that were adapted by IEG:  Consulting and collaborating with various stakeholders involved in project implementation is critical to ensure buy-in, especially when it comes to introducing major changes. In this project, the project team worked with stakeholders at the provincial Page 15 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) and central levels to introduce a new payment modality, which replaced payment by local staff with an independent payment agency (VNPost). However, the project team did not sufficiently address the opposing views by local staff when it came to supporting out-of- school children, resulting in implementation challenges, and it did not anticipate institutional and procedural challenges involved in program consolidation.  Without sufficient attention to building capacity at the local and central levels during project preparation and developing a plan to maintain and reward trained and skilled staff, project implementation and the sustainability of project outcomes are likely to suffer. In this project, government staff did not have sufficient capacity to operationalize the new MIS during the first year of project implementation, resulting in implementation delays.  When government programs are being reformed, proactive adjustments in resource allocation and staffing will typically be necessary to address temporary disruptions. This project experienced a high staff turnover and reduction at the local level because of government administration reform resulting in local staff having to take on additional tasks and carrying out project activities such as learning to use the new system and updating data into the system. This resulted in irregular updates of the database. 13. Assessment Recommended? No 14. Comments on Quality of ICR The ICR provided an adequate overview of project preparation and implementation. Also, the ICR was internally consistent, appropriately critical, and sufficiently outcome driven. Furthermore, the identified lessons learned were useful and can be applied to similar Bank projects. However, the ICR did not provide a traditional economic analysis. It also did not note that OP/BP 4.10 was triggered and whether the project’s compliance was satisfactory. Overall, the quality of the ICR is rated Substantial. a. Quality of ICR Rating Substantial Page 16 of 17 Independent Evaluation Group (IEG) Implementation Completion Report (ICR) Review VN-Social Assistance System Strengthenin (P123960) Page 17 of 17