Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00005422 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IDA-56910) ON A CREDIT IN THE AMOUNT OF SDR 55.4 MILLION (US$ 77 MILLION EQUIVALENT) TO THE SOCIALIST REPUBLIC OF VIETNAM FOR THE RENOVATION OF GENERAL EDUCATION PROJECT August 26, 2021 Education Global Practice East Asia And Pacific Region CURRENCY EQUIVALENTS (Exchange Rate Effective August 26, 2021) Vietnamese Dong Currency Unit = (VND) 22,787 VND = US$1 US$1.42 = SDR 1 FISCAL YEAR July 1 - June 30 Regional Vice President: Victoria Kwakwa Country Director: Carolyn Turk Regional Director: Daniel Dulitzky Practice Manager: Tobias Linden Task Team Leader(s): An Thi My Tran ICR Main Contributor: Akiko Sawamoto ABBREVIATIONS AND ACRONYMS CBA Cost-Benefit Analysis CBE Competency-Based Education CPD Continuous Professional Development CPF Country Partnership Framework CPS Country Partnership Strategy CTC Core Technical Committee DPO Detailed Project Outline ESMP Environmental and Social Management Plan ETEP Enhancing Teacher Education Project FCRE Fundamental and Comprehensive Renovation of Education GC General Curriculum GEC General Education Curriculum GOV Government of Vietnam HCMC Ho Chi Minh City ICR Implementation Completion and Results Report IDA International Development Association IRI Intermediate Results Indicator IRR Internal Rate of Return ISR Implementation Status and Results Report LMS Learning Management System LTTU Lead Teacher Training University M&E Monitoring and Evaluation MOET Ministry of Education and Training MOF Ministry of Finance MPI Ministry of Planning and Investment MTR Mid-Term Review NCFLT National Center for Foreign Language Testing NCSDGEQ National Centre for Sustainable Development of General Education NLSA National Large-Scale Assessment NLSAS National Large-Scale Assessment System ODA Overseas Development Assistance OECD Organisation for Economic Co-operation and Development PAD Project Appraisal Document PDO Project Development Objective PISA Programme for International Student Assessment PMU Project Management Unit READ Russia Education Aid for Development RGEP Renovation of General Education Project SABER System Assessment and Benchmarking for Education Results SDR Special Drawing Right SEDP Socio-Economic Development Plan SSTC Subject-Specific Technical Committee TOC Theory of Change TTL Task Team Leader UOM University of Melbourne, Australia VEPH Vietnam Education Publishing House VNEN Vietnam Escuela Nueva TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) .....................................10 II. OUTCOME .................................................................................................................... 13 A. RELEVANCE OF PDOs ............................................................................................................13 B. ACHIEVEMENT OF PDOs (EFFICACY) ......................................................................................15 C. EFFICIENCY ...........................................................................................................................22 D. JUSTIFICATION OF OVERALL OUTCOME RATING ....................................................................24 E. OTHER OUTCOMES AND IMPACTS (IF ANY)............................................................................24 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 26 A. KEY FACTORS DURING PREPARATION ...................................................................................26 B. KEY FACTORS DURING IMPLEMENTATION .............................................................................28 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 30 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................30 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE .....................................................33 C. BANK PERFORMANCE ...........................................................................................................34 D. RISK TO DEVELOPMENT OUTCOME .......................................................................................37 V. LESSONS AND RECOMMENDATIONS ............................................................................. 37 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 40 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 49 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 51 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 52 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 57 ANNEX 6. SUPPORTING DOCUMENTS (IF ANY) ..................................................................... 60 The World Bank Renovation of General Education Project (P150058) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P150058 Renovation of General Education Project Country Financing Instrument Vietnam Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency SOCIALIST REPUBLIC OF VIETNAM Ministry of Education and Training Project Development Objective (PDO) Original PDO The Project Development Objective is to raise student learning outcomes by: (i) revising and implementing the curriculum following a competency-based approach; and (ii) improving the effectiveness of instruction by creating and disseminating textbooks aligned with the revised curriculum. Page 1 of 64 The World Bank Renovation of General Education Project (P150058) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 77,000,000 14,186,011 16,544,124 IDA-56910 Total 77,000,000 14,186,011 16,544,124 Non-World Bank Financing 0 0 0 Borrower/Recipient 0 0 0 Total 0 0 0 Total Project Cost 77,000,000 14,186,011 16,544,124 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 30-Jun-2015 08-Aug-2016 27-Nov-2018 31-Dec-2020 31-Dec-2020 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 23-Dec-2020 15.73 Cancellation of Financing Reallocation between Disbursement Categories KEY RATINGS Outcome Bank Performance M&E Quality Unsatisfactory Moderately Unsatisfactory Modest RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 09-Dec-2015 Satisfactory Satisfactory 0 02 29-Jun-2016 Satisfactory Satisfactory 0 Page 2 of 64 The World Bank Renovation of General Education Project (P150058) 03 22-Dec-2016 Satisfactory Satisfactory .22 04 06-Jun-2017 Satisfactory Satisfactory 1.72 05 14-Feb-2018 Moderately Satisfactory Moderately Unsatisfactory 2.43 Moderately 06 01-Oct-2018 Moderately Unsatisfactory 5.43 Unsatisfactory Moderately 07 18-May-2019 Moderately Unsatisfactory 5.43 Unsatisfactory Moderately 08 18-Jan-2020 Moderately Satisfactory 10.03 Unsatisfactory Moderately 09 26-Dec-2020 Moderately Unsatisfactory 15.73 Unsatisfactory SECTORS AND THEMES Sectors Major Sector/Sector (%) Education 100 Other Education 100 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Human Development and Gender 100 Education 100 Access to Education 36 Education Financing 25 Science and Technology 13 Teachers 13 Standards, Curriculum and Textbooks 13 ADM STAFF Role At Approval At ICR Regional Vice President: Axel van Trotsenburg Victoria Kwakwa Page 3 of 64 The World Bank Renovation of General Education Project (P150058) Country Director: Victoria Kwakwa Carolyn Turk Director: Claudia Maria Costin Daniel Dulitzky Practice Manager: Harry Anthony Patrinos Tobias Linden Task Team Leader(s): Michael F. Crawford An Thi My Tran ICR Contributing Author: Akiko Sawamoto Page 4 of 64 The World Bank Renovation of General Education Project (P150058) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1 1. Education has been a cornerstone of Vietnam’s aspirations to achieve upper middle-income status or higher for the next generation of its citizens. Economic and political reforms of 1986 (known as Đổi Mới or renovation), with its transition from a centrally planned to a market economy, spurred rapid economic growth and poverty reduction. Vietnam’s per capita income grew from below US$100 to over US$1,500 by 2012, transforming the country from one of the world’s poorest nations into a lower middle-income country. The 2011–2015 Socio-Economic Development Plan (SEDP) of the Government of Vietnam (GOV) exemplified Vietnam’s continued political commitment to education, which envisioned that fast development of high-quality human resources would be vital for increasing the country’s competitiveness and productivity, as well as transitioning to a knowledge-based economy. The GOV’s desire to invest in education to improve its quality and outcomes was also underpinned by devoting nearly 20 percent of public expenditures to education beginning in 2010, which is significantly above the Organisation for Economic Co-operation and Development (OECD) countries’ average of 12.9 percent. 2 2. Intensive investment in education backed by political commitment yielded positive results in terms of both education access and outcomes. Vietnam achieved almost universal access to primary education by 2013 3 with a 98 percent net enrollment rate, and about 7.5 million children enrolled in primary school in 2014, including a proportionate number of ethnic minority children (1.3 million). Moreover, in its first-time participation in the 2012 Programme for International Student Assessment (PISA), Vietnam surpassed the OECD countries’ average, outperforming many developed economies. 4 3. Despite these achievements, the Vietnamese education system faced persistent challenges. First and foremost, issues remained in education quality and school enrolment. For the eight-year-old category, one out of four was unable to write a basic sentence without errors. 5 Previous large-scale grade 5 reading assessments also confirmed the abilities of the poorest students were inadequate to write at the level expected for their age. 6 Second, ethnic minority children were still less likely to attend upper secondary or tertiary education and more likely to underperform, compared to their Kinh 7 counterparts. Third, the skills in high demand by employers were in short supply. While Vietnamese workers had relatively strong levels of literacy and numeracy, employers reported difficulties in hiring new workers equipped with technical, higher-order cognitive, and behavioral skills. 8 Furthermore, the underlying challenges facing the education system were partly attributed to its “focus on content 1 This section is based on information obtained from the Project Appraisal Document (PAD) for the Renovation of General Education Project (RGEP) dated June 9, 2015. 2 OECD. 2011. Education at a Glance. 3 UNESCO Institute for Statistics. Vietnam: Education and Literacy. http://uis.unesco.org/en/country/vn. 4 Vietnam scored 511 (17th place out of 65 tested countries) on math, 508 on reading (19th place), and 528 on science (8th place). 5 World Bank. 2015. PAD for the RGEP. 6 ibid. 7 The Kinh people are the most predominant ethnic group, accounting for about 85 percent of the population in Vietnam. 8 Such high-order cognitive and behavioral skills include the ability to think critically, solve problems, and communicate convincingly verbally and in writing with clients and colleagues (World Bank. 2013. Skilling Up Vietnam: Preparing the Workforce for the Modern Market Economy). Page 5 of 64 The World Bank Renovation of General Education Project (P150058) learning at the expense of inculcating competencies and soft skills such as critical analysis, problem solving, and communication skills” (World Bank, p. 53). 9 4. To improve the quality of education and make it more relevant to market needs, the country embarked on education reform efforts in 2013, with the issuance of Resolution No. 29–NQ/TW of Party Plenum XI (2013), which, among other initiatives, proposed the Fundamental and Comprehensive Renovation of Education (FCRE). The FCRE— which was ratified on November 3, 2013 in the eighth Session of the Communist Party Central Steering Committee— envisioned that the renovation of general education10 would drive the country toward a “learning society” by (i) orienting the education system away from being content-based and toward being more competency- and quality- based; and (ii) providing Vietnamese children with new cognitive and behavioral skills. To achieve the educational vision set out in Resolution No. 29, the GOV/Prime Minister proposed the “Reform of General Education Curriculum and Textbooks after 2015” to the National Assembly, which was subsequently approved by the National Assembly in November 2014. By adopting a more coherent, high-quality new general education curriculum 11 promoting key qualities, competencies, and skills for all graduates as 21st century citizens, Vietnam aimed to strategically cultivate human resources with the capacity to apply learning and bring innovation to real-world challenges, which would be vital for accelerating the country’s economic productivity and growth. 5. The Renovation of General Education Project (RGEP) was designed to contribute to the country’s higher-level objectives. Together with three other World Bank 12-financed education projects active in the country portfolio at the time, 13 the RGEP was to support the country in realizing its FCRE vision. Moreover, the RGEP was not only well aligned with the Education Development Strategic Plan (2003–2020), whose objectives included renovating the education system and moving closer to regional and international quality standards, 14 but its activities were also specifically tied to the areas listed in the GOV’s 2011–2015 SEDP as key to renovating the Vietnamese education sector, including teaching, learning methods, and learning outcome assessments for all educational levels; the application of Information, Communication, and Technology in teaching and learning; the revision of the general education curriculum and textbooks; and the promotion of students’ creativity. 9 World Bank. 2019. Project Performance Assessment Report (PPAR), Vietnam: Education Projects—School Readiness and Escuela Nueva (Report No. 135418). 10 According to Article 28 of the 2019 Education Law, which took effect from July 1, 2020, general education consists of two stages: (i) general education comprised of primary and lower-secondary education; and (ii) vocationally oriented education (upper-secondary education). 11 In the RGEP that supported the revision of the curriculum, the “curriculum” was defined as “the General Curriculum (GC) and Subject-Specific Curricula. The GC comprises the following components: (i) the features of the ideal upper secondary school graduate; (ii) the subjects to be taught at each grade and their classification as either compulsory or elective; (iii) time allocations for each subject by grade level (i.e., the number of instructional periods devoted to each subject per academic year); and (iv) definitions of core competencies and qualities, as well as their proficiency levels at each grade band endpoint” (World Bank. 2015. PAD for the RGEP, p. 32). 12 Hereafter referred to simply as the “Bank.” 13 These education projects were as follows: (i) Vietnam School Readiness Promotion Project (P117393) (approved in February 2013 and closed in June 2017), which sought to improve early childhood education by promoting school readiness for disadvantaged children through Vietnam’s existing Early Childhood Education Program; (ii) School Education Quality Assurance Program (P091747) (approved in June 2009 and closed in December 2016), which aimed to improve educational outcomes and completion rates, especially for disadvantaged primary students, through supporting the GOV’s full-day schooling reform program; and (iii) Vietnam: Global Partnership for Education—Escuela Nueva Project (P120867) (approved in November 2012 and closed in May 2016), which introduced new teaching and learning practices that promote active learning and de-emphasize rote memorization in classrooms targeting the most disadvantaged groups of primary students. 14 World Bank. 2019. PPAR, Vietnam: Education Projects—School Readiness and Escuela Nueva (Report No. 135418). Page 6 of 64 The World Bank Renovation of General Education Project (P150058) 6. Furthermore, the RGEP was directly aligned with two of the three pillars of the strategic framework of the Country Partnership Strategy (CPS) FY2012–FY2016 for Vietnam at the time of project appraisal (i.e., Pillars 1. Competitiveness and 3. Opportunity) and their designed outcomes. Specifically, as part of Outcome 1.3 (Increased Capacity for Innovation and Value Addition) under Pillar 1, the CPS sought to foster innovation by strengthening the education system’s capacity to inculcate students with the skills and competences that would be critical for productive employment and economic growth. Also, Outcome 3.2 (Improved Basic Public Service Delivery and Access) under Pillar 3 was set to improve access to basic education services (particularly for ethnic minority children and those in remote areas) and learning outcomes, while simultaneously emphasizing the importance of adopting student assessments to measure the quality of education service delivery and learning outcomes. The close alignment of the Bank’s CPS program with national priorities, coupled with the Bank’s strong presence in all aspects of education coverage and quality as a longstanding key partner for the GOV, made the Bank well positioned to support the country’s education reform efforts. Theory of Change (Results Chain) 7. The RGEP sought to attain its ultimate objective (i.e., to improve the learning outcomes of Vietnamese students enrolled in basic education 15) by implementing its two primary interventions: (i) revision and implementation of a new competency-based curriculum 16 and (ii) improvement of the effectiveness of instruction through the creation and dissemination of textbooks aligned with the revised curriculum. 8. The project was built on the assumption that the desired outcome would be facilitated by establishing the following conditions: (i) increased readiness of teachers to implement the new competency-based curriculum (PDO Indicator 2); and (ii) equitable access of disadvantaged students to textbooks aligned with the new curriculum and other learning materials (PDO Indicator 3). To create the first condition, the project was expected to support the following activities: (i) establishing a qualified Core Technical Committee (CTC) and Subject-Specific Technical Committees (SSTCs) to lead the development and piloting of the competency-based curriculum from grades 1–12; and (ii) delivering face-to-face and online training for teachers and school managers, including offering online professional development modules aligned with the new curriculum and training for Curriculum Champions to serve as peer coaches. To create the second condition, the project intended to carry out the following: (i) producing a set of textbooks aligned with the new curriculum to be approved by the Ministry of Education and Training (MOET), along with guidance materials for textbook publishers to be developed by the CTC and SSTCs; (ii) disseminating these textbooks to schools with a high number of economically disadvantaged students; and (iii) producing a selection of bilingual textbooks for ethnic minority students in consultation with ethnic minority communities and leaders. 9. Furthermore, in conjunction with the curriculum reform, the project envisaged enhancing learning assessments and analysis for the continuous improvement of curriculum and policy by expanding the National Large-Scale Assessment System (NLSAS) and establishing two national centers to support research and capacity building on curriculum development and assessment. 15 Though the PDO does not restrict its scope to basic education, this is a natural reading within the context of the CPF at the time of appraisal and of project preparation and implementation. 16 According to the PAD for the RGEP (World Bank 2015), core competencies include “problem solving, communications, critical thinking, self-management, and self-learning” (para. 72). Page 7 of 64 The World Bank Renovation of General Education Project (P150058) 10. Ultimately, the project’s achievements were expected to contribute to fostering high quality human capital equipped with 21st century skills, and to increase the employability and productivity of the future labor force. The results chain shown in Figure 1 illustrates the project’s theory of change (ToC). 17 Figure 1. Project Theory of Change (Results Chain): RGEP Project Development Objectives (PDOs) 11. The PDO of the RGEP is “to raise student learning outcomes by: (i) revising and implementing the curriculum following a competency-based approach; and (ii) improving the effectiveness of instruction by creating and disseminating textbooks aligned with the revised curriculum.” 18 In line with the Bank Guidance on Implementation Completion and Results Report (ICR) for Investment Project Financing Operations (2020), this ICR considers that the overarching objective of the RGEP was “to raise student learning outcomes” while the statements that follow this objective were key interventions/outputs designed to contribute to achieving the objective. Key Expected Outcomes and Outcome Indicators 12. The expected outcome of the RGEP was to raise student learning outcomes. 17 The ToC presented in Figure 1 was reconstructed by the ICR author for this Implementation Completion and Results Report (ICR) based on the information provided in the PAD; there was no ToC in the PAD. 18 Financing Agreement, April 8, 2016. Page 8 of 64 The World Bank Renovation of General Education Project (P150058) 13. The following PDO-level indicators were developed to measure the expected outcome 19: • PDO Indicator 1: Student learning gains (as measured by “an increase 0.12 standard deviations in math and Vietnamese test scores of students under the new curriculum compared to the baseline cohort”); • PDO Indicator 2: Teacher preparedness for new curriculum (as measured by “25 percent of teachers implementing the new curriculum will be certified as prepared for curriculum implementation”); and • PDO Indicator 3: Equitable access to learning materials (as measured by “In the bottom 10 percent of schools - based on income distribution - textbooks will be distributed to raise the ratio of student to textbook ratio to 1.1 times its baseline rate”) Components 14. At the time of project appraisal, the RGEP consisted of the following four components, with an International Development Association (IDA) Credit of Special Drawing Rights (SDR) 55.4 million (US$77 million equivalent). 20 Component 1: Curriculum Development (Estimated: US$17.5 million IDA, including US$1 million contingencies; Actual: US$10.46 million IDA). Support the development and implementation of a revised competency-based curriculum for all school subjects from grades 1 to 12 through two sub-components: Subcomponent 1.1. Curriculum Development: Provide support for the establishment of, and implementation of activities by, a CTC and up to 11 SSTCs for curriculum revision. Subcomponent 1.2. Curriculum Implementation: Support the implementation of the revised curriculum through: (i) curriculum piloting; (ii) the delivery of face-to-face and online training to teachers and Curriculum Champions who can act as peer coaches for fellow teachers as they seek to understand and implement the new curriculum; and (iii) the development of guidance materials for publishers of textbooks and other pedagogical materials aligned with the Learning Outcome Standards for the new curriculum. Component 2: Development and Dissemination of Aligned Textbooks (Estimated: US$21.6 million IDA, including US$1 million contingencies; Actual: US$0.48 million IDA). Support the development of textbooks and pedagogical materials aligned with the new curriculum and their provision to schools with high numbers of economically disadvantaged students through two sub-components: Subcomponent 2.1. Development of Textbooks Aligned with the New Curriculum: Support the development and appraisal of textbooks in alignment with the new learning outcome standards upon completion and official adoption of the new curriculum by MOET. The CTC and its members appraise the textbooks and their alignment with the new curriculum while the SSTCs add editors and textbook writers to the subject-specific expertise embodied in the SSTC membership. Following the CTC’s recommendations on textbook alignment, the Steering Committee makes the final decisions on the authorization of textbooks. In consultation with ethnic minority leaders and communities, a selection of textbooks is translated into ethnic minority languages. 19 World Bank. 2015. PAD for the RGEP. 20Following the restructuring on December 23, 2020, which entailed the cancellation of unused funds, the total IDA credit of the project was brought down to US$16.54 million at project closing. Page 9 of 64 The World Bank Renovation of General Education Project (P150058) Subcomponent 2.2. Provision of Textbooks to Disadvantaged Schools: Provide sets of textbooks to schools with high numbers of economically-disadvantaged students based on the results of a needs assessment survey. Component 3: Learning Assessment and Analysis for Continuous Improvement of Curriculum and Policy (Estimated: US$35.4 million IDA, including US$1 million contingencies; Actual: US$5.48 million IDA). Improve and expand the NLSAS and research capacity to inform policies for General Education through four subcomponents: Subcomponent 3.1. Establishment of the NCSDGEQ and the NCFLT: Support the establishment of up to a total of six sites for the National Centre for Sustainable Development of General Education (NCSDGEQ) and the National Center for Foreign Language Testing (NCFLT). The NCSDGEQ is designed to advance research in areas related to curriculum, assessment, learning, and the educational policies that affect these while the NCFLT is expected to become a leading center for research and training aimed at improving the teaching and learning of foreign languages. Subcomponent 3.2. Research and Capacity-Building on Curriculum Development and Assessment: Sponsor research programs carried out by the NCSDGEQ in areas related to the new curriculum and the learning outcome standards to ensure continuous review and improvement of the curriculum in light of emerging research; and provide capacity- building and technical assistance on the development and administration of assessment instruments, and analysis of results for the NCSDGEQ staff and relevant officials to improve the coverage, technical quality, and alignment of the national assessment system with the curriculum. Subcomponent 3.3. Expansion of the National Large-scale Assessment System: Improve the coverage, technical quality, alignment, validity, and reliability of the national assessment system through: (i) integration of curricula and assessment; (ii) closer alignment of assessment with learning goals; (iii) training for educators and administrators at general education levels (primary, lower secondary, and upper secondary) to design and implement assessments, and analyze and report the results; and (iv) creation of specialized expertise within MOET to allow for continuous improvement of the National Large-Scale Assessment (NLSA). Subcomponent 3.4. Consultation and Outreach Campaign: Develop and implement consultation and outreach campaigns to promote the public awareness of the reform and to instill public trust in the transparency and meritocracy of the reform process. Component 4: Project Management (Estimated: US$2.5 million IDA; Actual: US$0.12 million IDA). Support Project Management and Monitoring and Evaluation (M&E) activities through two subcomponents: Subcomponent 4.1. Project Management: Finance the incremental costs associated with project management to ensure efficient program management and early identification of corrective measures to solve any implementation challenges. Subcomponent 4.2. Monitoring and Evaluation: Support the development and implementation of Project M&E to ascertain that the determined indicators are achieved and to continuously identify areas for improvement. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION (IF APPLICABLE) Revised PDOs and Outcome Targets Not applicable. Page 10 of 64 The World Bank Renovation of General Education Project (P150058) Revised PDO Indicators Not applicable. Revised Components Not applicable. Other Changes, Rationale for Changes, and Their Implication on the Original Theory of Change Other Changes and Their Rationale 15. The RGEP underwent a restructuring at the end of the project’s life. On December 23, 2020, upon receipt of a formal request from the Ministry of Finance (MOF), SDR 40,016,637 (US$60.46 million equivalent) of IDA financing was cancelled from the original committed credit amount of SDR 55.4 million (US$77 million equivalent). 21 The cancelled funds were comprised of two primary sources of unused funds: (i) about US$21 million allocated to textbook development (under Component 2), which was dropped due to the government’s decision not to use IDA funds for developing textbooks or online learning management system (LMS) platforms, 22 and the MOET’s decision to halt production of bilingual textbooks for ethnic minority students and textbook distribution to disadvantaged schools; and (ii) close to US$30 million originally planned to support key activities under Component 3, including the construction of the NCSDGEQ and NCFLT (which constituted the majority of project financing) and their associated research and capacity building activities. 23 16. The need for the project’s restructuring resulted from the cumulative effects of significant delays and changes in implementation arrangements, including: (i) project effectiveness reached 13.5 months after project approval due to the delay in countersigning; 24 (ii) the National Assembly’s announcement in September 2017 (Resolution 51/2017/QH14) to postpone the rollout of the new curriculum to September 2020 (due to a combination of delays in project effectiveness and the curriculum development), 25 with its resulting change to the schedule of the national assessment from 2019 to 2020; (iii) the cancellation of textbook development by the MOET under Component 2 as a result of the National Assembly’s policy reversal to refrain from using public funds to finance this activity, with issuance of Resolution No. 122/2020/QH14 dated June 19, 2020; and (iv) the delay and eventual withdrawal of the MOET from the construction of the NCSDGEQ and NCFLT under Component 3. In addition, the weak capacity of the Project Management Unit (PMU) made it a challenge to carry out the agreed-upon actions on time. 26 Strict social 21 In the Restructuring Paper on a Proposed Project Restructuring of RGEP (World Bank 2020), it was stated that US$57.7 million (the equivalent of SDR 40.0 million) of IDA financing would be cancelled from the original committed credit amount. However, the actual cancelled funds amounted to US$60.46 million because of exchange rate shifts in the period between the preparation of the restructuring paper and the date of cancellation. 22 Upon agreement by the management board of the RGEP and the Enhancing Teacher Education Project (ETEP), managing the development of the LMS and Teacher Education Management Information System later became part of the ETEP (MOET. 2021. RGEP Project Completion Report). 23 For more details of activities planned under Component 3, see the forthcoming section, “Components.” 24 The project was approved on June 30, 2015 and became effective on August 8, 2016 after nearly 10 months of prolonged internal government’s discussion as to whether the government would use Overseas Development Assistance (ODA) funds to support the curriculum reform and textbook revision. Moreover, additional one-year delay in the curriculum development, coupled with the setbacks in textbook development, precluded the initially planned rollout of the new curriculum in grade 1 in September 2018. As result, the MOET requested the National Assembly to issue the resolution to postpone the rollout to September 2020. 25 To date, the new curriculum has been rolled out in 2020 at grade 1 only; the rollout for grades 2–12 is expected to take place from 2021 to 2024 (See Annex 6: Table 6.1 New Curriculum Rollout Plans). 26 More details on the issue of the PMU’s capacity will be provided in Section III.B. Page 11 of 64 The World Bank Renovation of General Education Project (P150058) distancing measures adopted between February and May 2020 in response to the outbreak of the COVID-19 pandemic further exacerbated the implementation delays. 17. Some revisions were also made to the implementation arrangements and schedules. As referenced earlier, the National Assembly’s decision not to use state budget for textbook development and instead to allow for the privatization of textbook development led to key changes to the implementation arrangements. It shifted the MOET’s role from managing “textbook development” to “textbook appraisal” to ensure the quality of textbooks developed by publishing houses and their alignment with the new curriculum. As a result, instead of developing a single set of textbooks for each grade from grade 1 to grade 12 under the purview of the MOET, as originally planned at the project appraisal, multiple sets of textbooks have been produced by publishing houses in accordance with the textbook development guidelines prepared by the MOET. 18. In the end, despite efforts to formally restructure the project after the November 2018 mid-term review (MTR) and subsequent discussions to reflect the aforementioned changes as part of the restructuring, the GOV and the Bank could not agree on a revised implementation plan that would ensure the completion of the remaining activities within a reasonable time period. Thus, the project funds for unfinished activities were ultimately cancelled through the aforementioned restructuring. Implication of Changes on the Original Theory of Change 19. The cancellation and modification of certain activities had important implications for the ToC conceived at the project appraisal. Mobilizing private sector financing for textbook development in exchange for cancelling IDA financing, in the end, boded well for the country’s new multiple-textbook policy. The change of implementation arrangements not only facilitated the capacity building of the MOET as a “textbook appraiser” and of publishers as “textbook developers” as previously mentioned; it also allowed the country to follow in the footsteps of many high- performing Asian education systems in which the adoption of multiple textbooks have helped break the link between the textbook and the exam, i.e., by encouraging schools and teachers to pay more attention to the curriculum rather than merely to rely on textbooks by exercising their authority to select their preferred textbooks.27 With these positive outcomes of the new developments, the project remained relevant to its primary intervention (i.e., developing and implementing the competency-based curriculum) as illustrated in the ToC. 20. Nonetheless, the significant delays in the implementation schedules, especially the postponement of the rollout of the new curriculum and the subsequent administration of NLSA, hindered the project from measuring the progress of its originally desired outcome (i.e., improved student learning outcomes). Moreover, the discontinuation of translating textbooks and disseminating the textbooks aligned with the new curriculum to disadvantaged schools 28 resulted in the project failing to meet its originally expected target of PDO Indicator 3 (equitable access to learning materials), thereby undermining the project’s achievement of the desired outcome as envisioned in the ToC. 21. Overall, as a result of the cancellation of financing for a substantial portion of originally planned activities under Components 2 and 3, the level of outputs and outcomes became significantly lower than expected ex ante. Disruptions caused by the ramifications of political decisions on the curriculum reform, which adversely affected the project outcome, also undermined the initially held underlying assumptions. 27 Smart and Jagannathan. 2018. Textbook Policies in Asia: Development, Publishing, Printing, Distribution, and Future Implications. 28 For more details about the implications of these revisions for the project’s impact on poverty reduction, see Section II.B. Page 12 of 64 The World Bank Renovation of General Education Project (P150058) 22. Reasoning for No Split Rating: It was determined that a split rating does not apply to this project for the following reasons: (i) the PDO remained unchanged throughout the project’s implementation period; and (ii) the restructuring, which took place in December 2020 at the project’s closing, cancelled undisbursed funds but did not entail any formal revisions to objectives or outcome targets, or reduction in grades or subjects targeted by the project, that would have narrowed the overall ambition or scope of the project during implementation. Completing civil works on time has proven difficult in Vietnam, as the experience of other World Bank-financed projects in the education and other sectors attests but eliminating these activities from the project did not significantly reduce the difficulty of improving student learning outcomes. The set of activities under the project was reduced to focus on laying the necessary foundation for the future progress of the general education system in Vietnam as a whole through three main activities: (i) the development and implementation of the new competency-based curriculum, (ii) the design and delivery of teacher training aligned with the new curriculum, and (iii) the development and piloting of the NLSA in alignment with the new curriculum and learning outcome standards. The scaled-down project was also commensurate with the corresponding reduction in funding. Indeed, while the decrease in project commitments at closing largely stemmed from the cancelation of the originally planned civil works and production/dissemination of textbooks, most of the remaining funds were used to cover costs associated with the above-mentioned activities. 29 II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 23. The relevance of the PDO was and continued to be well aligned with the World Bank’s country strategies throughout the life of the project. 30 The Country Partnership Framework (CPF) FY18–FY22 31 shifted its primary focus from basic to tertiary education to strengthen the relevance, quality, and responsiveness of tertiary education institutions. With Vietnam’s graduation from the IDA in July 2017, this new strategy is intended to assist the country’s efforts to succeed as a middle-income country by developing a stronger and more competitive labor force with relevant and in-demand skillsets. Nonetheless, as part of its focus on investing in people and knowledge (Focus Area 2), the current CPF continues to underscore the need for making targeted efforts to further improve the equity and 29 In this light, the revision to the project scope and the resulting reduction in commitments apply to one of the particular instances cited in Draft Guidelines for Reviewing World Bank ICR: A Manual for IEG ICR Reviewers (2017) formulated by the Independent Evaluation Group (IEG), wherein changes to the project scope at restructuring may not require a split rating. These draft guidelines also state that “if the scope of the project shrank, project commitments decreased (for example through cancellation of funds), and a good case is made that the lowering of project scope was commensurate with the lower commitment size, then this would support a decision not to apply a split rating and to assess the project based on the revised scope (and objectives)” (pp. 46–47). 30 At the appraisal stage, the RGEP’s PDO was also in line with the priorities of CPS FY12–FY16, which, among other things, underscored the Bank’s support for (i) strengthening Vietnam’s competitiveness in the regional and global economy by fostering innovation capacity and the development of skills relevant to the labor market; and (ii) broadening access to opportunities “to ensure that more citizens can benefit from the country’s development as a middle-income country” (World Bank. 2011, p. 17) by improving basic public service delivery and access. The RGEP was designed to address these priorities by (i) aiding the GOV in successfully designing and implementing the new competency-based curriculum; and (ii) enhancing the effectiveness of teaching and ensuring the availability of, as well as access to, the new curriculum-aligned textbooks for all children enrolled in basic education, including those in ethnic minority and economically disadvantaged communities. 31 World Bank. 2017. Country Partnership Framework for the Socialist Republic of Vietnam for the Period FY18–FY22. Page 13 of 64 The World Bank Renovation of General Education Project (P150058) quality of general education, based on the notion that without such changes Vietnam would not be able to truly capitalize on its well-performing basic education foundation as students progress through the system. 24. Moreover, the objectives of the RGEP remained critical to achieving the GOV’s education sector priorities upon which CPF FY18–FY22 was built. The 2016–2020 SEDP identifies improving the quality of human resources as one of its primary objectives, emphasizing the need to continue to renovate the universal education program and its learning assessment system. 32 The Vietnam 2035 report (whose findings and identified development constraints also formed the basis of the current CPF) calls for ensuring high-quality and relevant basic education for all, which is essential for building workforces with high levels of advanced skills gained through education and, ultimately, to Vietnam’s overall economic success. 33 25. The high relevance of the RGEP’s objectives to and their alignment with the strategic needs of the education sector in Vietnam at project closing were also evident in the 2019 Education Law, which took effect on July 1, 2020. The provisions of Article 31 (General Education Curriculum) of this law specify that “qualities” and “competencies” are required for students in general education, whereas those of Article 29 (General Education Curriculum) of the previous 2005 Education Law and its amendments 34 had only required “knowledge” and “skills” standards. This legislative change in 2019–2020, which institutionalized the country’s strategic shift in priorities from content-based to competency- and quality-based education, demonstrates that the RGEP’s continued support for the GOV’s FCRE by focusing its efforts on improving student learning through the introduction of the new competency-based curriculum remained relevant at project closing. Relevance Rating: High 32 Socialist Republic of Vietnam. 2016. The Five-Year Socio-Economic Development Plan 2016–2020. 33 The World Bank and Ministry of Planning and Investment of Vietnam. 2016. Vietnam 2035: Toward Prosperity, Creativity, Equity, and Democracy. Washington, DC: World Bank. 34 The 2005 Education Law and its subsequent amendments became effective in January 2006 and July 2010, respectively. Page 14 of 64 The World Bank Renovation of General Education Project (P150058) B. ACHIEVEMENT OF PDOs (EFFICACY) 26. In what follows, the ICR will evaluate the project’s achievement of its objective based on the indicator results and other sources of information. In doing so, it will assess the project’s efficacy by examining (i) the extent to which its desired outcome was achieved and (ii) how the key interventions/outputs defined as part of the PDO statement contributed to the RGEP’s objective and would support Vietnam’s continuing efforts to improve student learning outcomes as it rolls out the new curriculum for grades 2–12 in the coming years. The achievement of the PDO-level indicators and Intermediate Results Indicators (IRIs) are presented in Annex 6: Table 6.2 Achievement of PDO-level and Intermediate Results. As summarized in Table 6.2, the RGEP had three PDO-level indicators and 11 IRIs, of which six IRIs fully met their targets by the project’s completion. As will be highlighted below, although the RGEP could not achieve its desired outcome within the project implementation period, it succeeded in laying the building blocks that are essential for improving student learning outcomes in the years to come. Objective: To raise student learning outcomes 27. PDO Indicator 1 (Student learning gains) was unmeasurable because of the delay in developing the new General Education Curriculum (GEC)—which was not completed and adopted by the MOET until December 2018—and the subsequent postponement of administering the National Large-Scale Assessments (NLSA) aligned with the new GEC. At project appraisal, the project set its end-target that students taught under the new curriculum would demonstrate an increase of at least 0.12 standard deviations in mathematics and Vietnamese test scores compared to the baseline cohort. However, the planned assessments could not be administered within the project timeframe. At project closing, while the new curriculum had been applied to the first cohort of grade 1 students nationwide during the 2020–2021 school year, no baseline or control group data was available to measure learning gains. The first cohort of grade 3 students whose NLSA results were to be employed for estimating the results of PDO Indicator 1 in the revised implementation plan would not be introduced to the new curriculum until the 2022–2023 school year. It should also be noted that it was planned that learning gains would have been measured in more than one grade. 35 28. Nevertheless, the ex-post field surveys joined by the Bank team in Lam Dong and Tuyen Quang provinces 36 provided some indicative evidence of improved outcomes for grade 1 students, especially in Vietnamese. According to teachers’ evaluations of student reading comprehension, by the 19th–23rd week of the 2020–2021 school year after the new curriculum was rolled out in September 2020, the majority of students who had finished learning syllables could pronounce them clearly and read well, and by the end of the first semester in January 2021, many children could read books and comics (i.e., reference and additional learning materials, excluding textbooks) other than the first- grade Vietnamese books that were part of the curriculum. 37 In mathematics, teachers commented that students who had been engaging in various creative and experiential activities under the new curriculum exhibited confidence and speed in the area of communication skills. 29. Despite its failure to attain the overarching desired outcome within the project implementation period, the project managed to roll out the new curriculum for grade 1 and create an environment conducive to renovating the general 35 According to the results framework in the PAD, the original plan was to measure learning gains on the basis of the results of the NLSA administered in grades 5, 9, and 11 in the final year of project implementation (Year 5) and those of the baseline partially derived from 2011 and 2014 exams and the first iteration of the NLSA in Year 2. 36 Of the six provinces in which the ex-post field surveys were conducted, two were joined by the Bank team. The evidence however is only suggestive, since the surveys relied on anecdotal observations by teachers, rather than direct evidence of student achievement, included no comparison with the pre-project situation, and were not conducted on a representative sample of teachers. 37 MOET. 2021. RGEP Project Completion Report. Page 15 of 64 The World Bank Renovation of General Education Project (P150058) education system in Vietnam. By partially completing its key interventions/outputs, which were designed as essential constituents to achieve the outcome (as illustrated in the ToC), the project succeeded in establishing two major building blocks for improving student learning outcomes. 30. Building block one: revising and implementing the curriculum following a competency-based approach. The project successfully developed, piloted, and began the implementation of a new, revised competency-based curriculum under Component 1, and, by undertaking the first round of the NLSA, it also built the foundation for the continuous improvement of the curriculum under Component 3. 31. Under Component 1 (Curriculum Development), the RGEP’s success in the curriculum renovation was comprised of a comprehensive set of activities and outputs as follows: 32. The CTC and 17 SSTCs 38 were established and carried out a series of activities, ranging from the revision of the curriculum to overseeing the development of interactive online training tools for teachers to build their capacity to implement the new curriculum. The project overachieved its IRI 1 (Core Technical Committee Qualifications) end-target (15 years 39) by having CTC members on board with an average of 35.17 years of relevant professional experience. 33. In December 2018, the MOET officially endorsed and launched 29 new competency-based school curricula consisting of a general curriculum and 20 subject-specific curricula. The CTC and SSTCs drafted guidance materials for textbook publishers on how to develop textbooks and other pedagogical materials aligned with the learning outcome standards and content of the new curriculum. 34. Following the completion and endorsement of the curriculum, teacher training was delivered using professional development modules aligned with the new curriculum to improve teachers’ readiness to teach the curriculum. Four new teacher professional development modules on competency-based teaching and assessment practices were developed to prepare teachers for implementing the curriculum by fostering their understanding of the curriculum and its implications for classroom practices and learning assessments. 40 Both face-to-face and interactive online training were offered to teachers to prepare them to teach the new curriculum and for Curriculum Champions to work effectively as peer coaches for their fellow teachers. 35. During the development of the new curriculum and professional development modules, the project was facilitated by technical assistance support from the University of Melbourne (UOM), Australia, which had previously conducted empirical research on key learning competencies and skills for the 21st century, as well as by other experts in competency-based learning and assessments. This arrangement allowed the project to ensure the high quality of the design and delivery of training modules and of curriculum development by achieving the following: (i) adapting some of the key competencies highlighted in the UOM’s research to the Vietnamese context in the new competency-based curriculum; (ii) having about 200 master trainers trained in the new curriculum participate in writing the training modules for teachers under the guidance of a group of experts from the UOM, the University of Cambridge, and 38 The 17 SSTCs consisted of three education level coordinators, two Social Sciences Coordinators, three Natural Sciences Coordinators, and nine other subject chief editors. 39 The end-target of IRI 1 was originally set at 12 years. However, it was revised to 15 years in December 2017, when the initial group of CTC members came onboard with an average of 15 years of experience; the target was not formally revised. 40 These four modules were as follows: Module 1 (Introduction to the new competency-based education [CBE] curriculum); Module 2 (Application of CBE teaching methods); Module 3 (CBE student learning assessment); and Module 4 (School-based curriculum development). However, only the first two training modules were conducted in 2019 and 2020; the remaining modules will be covered under the ETEP (MOET 2021). Page 16 of 64 The World Bank Renovation of General Education Project (P150058) Vietnam National University; (iii) adopting a new blended learning approach to the online training modules; 41 and (iv) having the online modules assessed by the experts to guarantee their quality. 36. While no formal evaluation of teacher training or its impact on improving the effectiveness of instruction was conducted under the RGEP because the national rollout of training modules occurred just a few months prior to project closing, an online teacher satisfaction survey was administered to all the teachers and school managers who had completed modules 1 and 2 in accordance with the 2020 Continuous Professional Development (CPD) Program. The survey results showed a high level of satisfaction with the modules among the respondents, including in areas such as the usefulness of the modules in classroom applications and their contribution to improving student learning. 42 For example, of the 433,325 teachers who participated in the online satisfaction survey, 418,600 (96.6 percent) responded favorably to the survey questions regarding the usefulness of the modules, with an average score of 3.26 (on a 5-point scale, where 5 is the maximum and 1 is the minimum), and 417,814 (96.4 percent) provided positive responses to statements regarding the contribution of the modules to enhancing student learning experiences and results, also with an average score of 3.26. The same trend was observed across the board for both teachers and school managers, including teachers from different groups (e.g., based on gender, ethnic background, educational level, and those teaching in disadvantaged versus advantaged areas), although the survey response rate of school managers was higher at 70.3 percent than that of teachers at 56.5 percent. 37. In terms of project indicators linked to teacher training, the RGEP underachieved PDO Indicator 2 and IRIs 2 and 9, but overachieved IRI 3. Of the 151,930 teachers who were deemed prepared for the new curriculum implementation, 43 only 4,564 teachers were supported by the RGEP, which accounted for 1.21 percent of teachers. 44 As a result, the project did not meet either its 25 percent end target for PDO Indicator 2 (Teacher preparedness for the new curriculum) or that of 150,000 teachers for IRI 9 (Number of Additional Qualified Primary Teachers Resulting from Project Interventions). IRI 2 (Online Training System Readiness) was also underachieved, with only 2.65 percent of teachers having finished at least one online module and face-to-face training by the project’s completion. 45 Nonetheless, the project considerably exceeded the target of IRI 3 (for 8,000 Curriculum Champions), with 21,320 41 The online training modules were comprised of five days of self-paced learning on LMS platforms, three days of face-to-face training, and seven days of completing assignments via LMS. This new blended training model is expected to set an example for future professional development training for teachers. It has also proven efficient during the pandemic crisis. 42 In relation to usefulness, the respondents were asked to indicate their level of agreement with the following statements on a scale from 1 (strongly disagree) to 5 (strongly agree): (i) The content of the online CPD modules is relevant to the professional standards required at the school where I teach; (ii) The assessment of my CPD at the school has focused on developing my professional competencies; and (iii) I am confident that I can meet the requirements for teaching the new competency-based general education curriculum effectively once I complete these online modules. In terms of contribution to improved student learning, the respondents were asked to provide similar ratings in relation to the following: (i) The content of the online CPD modules has enhanced the quality and delivery of my lessons; (ii) The module content has improved the academic performance of my students; and (iii) I am confident that I can apply what I have learned to improve the academic performance of my students. 43 In principle, the new curriculum requires teachers to complete at least the first three modules to be certified to teach the new curriculum. However, because of the delay in finalizing module 3, which was not made available until after the new curriculum rolled out in grade 1, the MOET made an adjustment to the required qualification condition so that teachers who completed the first two modules were also deemed to be qualified to teach the new curriculum. 44 Of the 151,930 qualified teachers who have completed modules 1 and 2 developed by the RGEP (both face-to-face and online training offered via LMS), 4,564 were funded by the RGEP, 13,922 by the ETEP, and 133,444 by provincial funds. The 150,930 teacher deemed qualified represented 40.21 percent of primary teachers. 45 As cited in Table 6.2 in Annex 6, an additional 94.78 percent of teachers were granted an LMS account to access online modules and completed online module 1 and face-to-face training, with funding support from the ETEP and localities. Page 17 of 64 The World Bank Renovation of General Education Project (P150058) Curriculum Champions who were supporting their peers to help them understand and implement the new curriculum by project end. 38. Following the launch of the new curriculum and the rollout of teacher training, the curriculum was piloted in 2018 for a total of 9,732 experimental class hours for 15 subjects in 1,165 classrooms at 48 schools at three grade levels (grades 3, 8, and 12). As a result, IRI 5 (Curriculum Piloting) significantly overachieved its original target for pilot/treatment classrooms (30 classrooms per grade). However, its target for control classrooms (30 classrooms control monitored) was not met because of the changes made to the assessment of treatment vis-à-vis control classrooms. 46 Hence, there was no monitoring of control classrooms as originally planned at project approval. 39. The new curriculum was rolled out to the first cohort of grade 1 students across the country at the beginning (September) of the 2020–2021 school year. Because of the delay in the initial rollout, the entire process of the new curriculum reform at all grade levels is expected to be completed in the 2024–2025 school year (see Table 6.1 New Curriculum Rollout Plans in Annex 6). 40. Under Component 3 (Learning Assessment and Analysis for Continuous Improvement of Curriculum and Policy), the RGEP managed to make some achievements in supporting the GOV’s efforts in enhancing the NLSAS and the capacity to help inform policies for general education, primarily through the preparation and administration of the first round of the NLSA. Thus, the project fostered an environment that will help the new competency-based curriculum achieve its full potential when it is finished being rolled out at all general education grade levels in the coming years. 41. The NLSA for primary, lower, and upper secondary education was developed, piloted, and finalized in alignment with the new curriculum and its learning outcome standards. Despite the challenges presented by the COVID-19 pandemic, the first round of the NLSA was administered to 12 nationally representative samples of grades 5, 9, and 12 47 students on 14 subject exams from June to July 2020. With this outcome, the project achieved IRI 6 (Expanded NLSAS) whose target was to administer the NLSA exams to one in each of the three educational levels (primary, lower, and upper secondary). As evidenced by the completion of the NLSA in math and Vietnamese for grade 5 in July 2020, the project also met its target of IRI 10 by having a system for learning assessment at the primary level in place. The project also achieved IRI 11 (Utility of the Learning Assessment System) target of three learning assessments by conducting formative, summative, and national (i.e., NLSA) assessments by project end. 42. While conducting only the first round of the NLSA hampered the RGEP from comparing the baseline and endline results, as well as assessing the learning gains, preparing and administering the NLSA helped strengthen the national capacity to conduct NLSAs periodically: (i) test-item banks were developed in alignment with the new competency- based curriculum; (ii) 100 researchers and teachers of 19 subjects were trained “in the structure and technical matrix 46 In lieu of monitoring control classrooms, a comparison was made between the control and treatment classrooms by administering survey questionnaires to the teachers and school managers of the schools participating in the curriculum piloting to discern their perceptions of treatment vis-à-vis control (regular) classrooms. The teachers who participated in the piloting had been trained in the new learning material, teaching methods, and assessments before teaching pilot classes. While a summary report produced under the direction of the PMU largely focused on the piloting exercise, it did not provide comparative analysis of the treatment and control classrooms. 47 In the original plan, the NLSA was planned to be administered to grade 11 at the upper secondary level. National assessments had never been conducted for grade 12 students to minimize disruptions to those preparing for high-stakes exams. However, the MOET decided to change the plan in 2018 to administer the NLSA to grade 12 so that the NLSA results of grade 12 could be used to provide insights into education policymaking. Page 18 of 64 The World Bank Renovation of General Education Project (P150058) of the 2019 exam”; 48 (iii) 62 experts and researchers from the Department of Quality Management, Vietnam Institute of Educational Sciences, were fully trained in organizing NLSAs; (iv) 430 researchers and teachers received extensive training in exam question preparation techniques from international experts; (v) a total of 1,733 general teachers, experts, and administrators from upper secondary schools, the Department of Quality Management at the Vietnam Institute of Educational Sciences, and Universities of Education also received training from international experts on each step of administering the NLSA; and (vi) approximately 20 teachers and experts provided training to their peers on assessment techniques. 49 43. The RGEP inability to complete other activities under Component 3 inhibited it from maximizing its impact by enhancing national capacity for conducting policy-informed research on learning assessment and analysis. Notably, the establishment of the NCSDGEQ and the NCFLT 50 did not advance beyond the initial planning and design stages, 51 due to substantial delays in the civil works procurement process. The derailment of the establishment of the NCSDGEQ and the NCFLT, in turn, hampered the project from materializing research and capacity-building activities related to curriculum development and assessment, which were planned to be undertaken by these centers. To date, the refurbishment and provision of equipment and furniture was completed for the NCSDGEQ’s three satellite centers located at Thai Nguyen University, Danang University of Education, and Ho Chi Minh City University of Education. 52 44. Building block two: appraising textbooks developed by publishers to ensure quality assurance. While various revisions were made to the implementation arrangements for textbook creation and dissemination under Component 2 (Development and Dissemination of Aligned Textbooks), the project succeeded in having the MOET appraise textbooks developed by publishing houses to ensure their quality and alignment with the new curriculum. 45. The RGEP’s engagement in the textbook development process diverged from its original vision to address the procurement challenge and to better align the originally planned activities with evolving shifts in relevant policies. First, the MOET’s failed attempt to procure a large group of individual textbook writers and editors 53 led to the GOV issuing a report to the National Assembly Standing Committee, 54 which articulated a policy of increasing parental 48 MOET. 2021. RGEP Project Completion Report. 49 ibid. 50 While the establishment of the NCFLT was dropped during the project implementation, the MOET instead founded the National Testing Center under the Department of Quality Control. (MOET 2021). 51 The MOET approved the Feasibility Study Report in July 2020 to build two national centers in Hanoi. However, it was too late for the RGEP to complete the construction, along with other unfinished activities under Component 3, by project closing. Based on the decision not to restructure the project, this activity was discontinued. 52 MOET. 2021. RGEP Project Completion Report. 53 A total of 54 consulting bidding packages for the recruitment of 37 chief authors, 87 authors, and 51 editors for textbook compilation for grades 1, 2, and 6 were advertised in February 2020. Although 175 candidates were selected and contacted to negotiate their contracts, the negotiations were unsuccessful as most of the selected candidates were already under contract with publishers for textbook compilation (RGEP Aide-Memoire, June 2020). 54 Report No. 209/BC-CP submitted by the GOV to the National Assembly Standing Committee regarding the implementation of the National Assembly’s Resolution No. 88/2014 on general education curriculum and textbook renovation. The Vietnamese policy uses the word ‘socialization’ (xã hội hóa) which generally refers to privatization and contributions by various entities in society, or “a system of formal and informal co-payments to finance education” (London 2021, p. 34). In this light, socialization of textbooks usually connotes parental contributions for purchasing textbooks while that of textbook development means financing by publishers with permits for publishing textbooks. The socialization of textbooks was first pronounced in Resolution No. 88/2014/QH13 dated November 28, 2014 issued by the National Assembly on the innovation of the curriculum and textbooks for general education: “Implement socialization in compiling textbooks; there are a number of textbooks for each subject. The [MOET] disseminates evaluation criteria for textbooks allowed to be used on the basis of the evaluation results of the National Textbook Evaluation Council; guide the selection of textbooks in general education establishments. Encouraging organizations and individuals to compile textbooks based on the general education curriculum. To proactively implement the Page 19 of 64 The World Bank Renovation of General Education Project (P150058) contributions to purchasing textbooks and expanding financing by commercial publishers for textbook development, through which the MOET would not compile textbooks using public funds on the condition that at least one set of suitable textbooks was approved by the MOET. Second, based on this outcome, the National Assembly released Resolution No.122/2020/QH14 (dated June 19, 2020), which requires the MOET to facilitate textbook production without employing public funds and also to limit its role to appraising and approving textbooks developed by publishers. Third, the new GOV’s restrictions on the use of Overseas Development Assistance (ODA) funds for “recurrent expenditures,” which initially came into effect in March 2016 after project approval in June 2015 and was subsequently updated in May 2020, 55 also inhibited the mobilization of IDA funds for procuring technical assistance. 46. Following the GOV’s decision, the MOET withdrew from its previously envisaged role to lead the development and dissemination of textbooks and instead prioritized the use of project funds for textbook appraisal for quality assurance. The RGEP supported the MOET in outlining measures to monitor the quality of textbooks and to increase transparency in textbook selection, especially since Vietnam announced its new multiple-textbook policy in 2015,56 under which schools and localities would be responsible for selecting their materials from a list of approved textbooks. 57 47. The RGEP also provided technical support to assist the textbook development and selection process by achieving the following: (i) developing training materials for textbook appraisers and offering training sessions for the members of the National Textbook Appraisal Committee for grade 1 textbooks; (ii) preparing textbook development guidelines to ensure the textbooks produced by publishers would be well aligned with the new curriculum; 58(iii) developing detailed guidance for textbook appraisal, which included quality standards, criteria, and indicators for textbooks; and (iv) preparing and disseminating guidance on textbook selection criteria to the staff of the Bureau of Education and Training and the Department of Education and Training, as well as school managers. 48. By project end in December 2020, as evidenced by the result of IRI 4 (Textbook Production), a total of 110 new textbooks were produced59 for nine subjects at the primary level (for grades 1 and 2) and for 12 subjects at the secondary level (for grade 6). Of the 110 textbooks produced, five sets of new grade 1 textbooks (a total of 46 new general education curriculum, the [MOET] organized the compilation of a set of textbooks. This textbook set is assessed and approved in a fair manner with textbooks compiled by organizations and individuals.” (MOET. 2021. RGEP Project Completion Report, p. 25) 55 Decree No. 16/2016/ND-CP on the management and use of ODA capital and preferent loans of foreign sponsors stipulated that ODA financing (including IDA) can be leveraged only for procuring goods and civil works but not for recurrent expenditures, such as technical assistance, training, workshops, and operating costs. The decree was updated in May 2020 by Decree No. 56/2020/ND-CP, which continued to exclude recurrent expenditures from ODA funding. 56 The new policy “one curriculum, many textbooks” was announced in the 2015 Decision Note, with an initially planned rollout in 2018–2019, following the piloting for the three preceding years (Smart and Jagannathan, 2018). 57 To date, the MOET has issued two regulations on textbook selection. While the 2020 regulation stipulated that school councils would be responsible for selecting textbooks, the 2021 regulation specifies that provincial committees have the authority to select their preferred textbooks. Despite this change, teachers are still expected to work with textbooks more effectively as they implement the new curriculum. 58 Once the guidelines for the compilation and appraisal of textbooks are finalized, they will be made available for free online. They will be comprised of the following: (i) introduction of the general curriculum and subject-specific curricula; (ii) guidance on the compilation of competence-based textbooks; (iii) curricular 33 and criteria for textbook evaluation; (iv) spelling rules in textbooks; (v) guidance on the implementation of the non-discrimination framework in textbooks; and (vi) guidance on the integration of financial literacy and environmental protection into textbooks. 59 By May 2021 (i.e., after project closing), a total of 118 new textbooks have been produced, appraised, and approved for nine subjects for grades 1 and 2, and 12 subjects for grade 6. As mentioned above, the production of these textbooks was not financed by the project but by publishing houses. Page 20 of 64 The World Bank Renovation of General Education Project (P150058) textbooks) have been appraised and approved by the MOET. 60 As a result, the target of IRI 4 (target: 72 [= 1 per grade per academic subject] new textbooks approved) was not met because the textbook approval of all subjects for all twelve grade levels of general education was not achieved. Although the number of approved textbooks per subject exceeded the set target (i.e., one textbook per subject) since at least 2 to 5 textbooks per subject were produced for grades 1, 2, and 6 at each level, grade 2 and 6 textbooks were still at the appraisal stage at the time of the project’s completion; after the MOET’s approval, 32 textbooks (for grade 2) and 40 textbooks (for grade 6) became available in February 2021. 49. Despite these achievements in textbook development and appraisal, the project did not manage to implement the dissemination of textbooks under Component 2, which was expected to serve as yet another building block for improving student learning outcomes. In effect, the MOET called off the provision of textbooks to disadvantaged schools for students and instead proposed that they borrow from school libraries. This decision was made because the GOV repronounced that the State shall extend its financial support to ensure that students in underserved communes, including ethnic minority students and those with difficult circumstances, have access to textbooks, in accordance with the extant programs and regulations 61 as well as with the Government Decree No. 86/2015/ND-CP stipulating that all students be provided with books, notebooks, and school supplies by parents, schools, or local authorities. Owing to these developments, the project failed to achieve its targets for PDO Indicator 3 (Equitable Access to Learning Materials, target: 1.1 times baseline rate). No data was available to estimate PDO Indicator 3 because no official surveys were undertaken to assess students’ access to learning materials (including textbooks) and teachers’ access to teaching materials, which made it impossible to establish a baseline or to measure progress. 50. The issue with the framing of the objective statement: Another issue that emerged in the assessment of the project outcome relates to the framing of the objective statement, especially the second statement that follows the overarching objective (i.e., “improving the effectiveness of instruction”). The aforementioned online survey results revealed that the majority of teachers and school managers who had completed the required introductory training modules on the new competency-based curriculum and teaching methods believed these modules helped them improve their teaching and student learning. Nonetheless, more substantive evidence (e.g., classroom observations showing evidence of teachers’ increased application of teaching methods in alignment with the new curriculum and textbooks) to corroborate how the means (“creating and disseminating textbooks”) contributed to the ends (“improving the effectiveness of instruction”) would have allowed a meaningful evaluation of the causal relationship between the means and the ends. 60World Bank. 2020. Restructuring Paper on a Proposed Project Restructuring of RGEP. 61These programs and regulations include the following: (i) Program 135 for Socio-economic Development for especially disadvantaged ethnic minority, mountainous, border, remote, and isolated communities under which the government is expected to provide textbooks free of charge (normally through library loans) for these disadvantaged groups of students by having “localities buy textbooks from provincial books and educational equipment companies, to supply to target students” (Asian Development Bank, 2018, p. 71); (ii) Decree 86/2015/ND-CP by the government stipulating that all students be provided with books, notebooks, and school supplies by parents, schools, or local authorities; and (iii) Joint Circular No. 109/2009/TTLT/BTC-BGDĐT by the MOF and MOET ordering the provision of textbooks to ethnic minority and disadvantaged students. Page 21 of 64 The World Bank Renovation of General Education Project (P150058) Justification of Overall Efficacy Rating 51. The overall efficacy is rated as Modest. The project was unable to show that it had raised student learning outcomes, even in grade 1 where the majority of activities ultimately were focused, 62 and hence could not achieve the overarching goal of raising student learning outcomes. Nevertheless, it successfully established the building blocks that would allow the country, in conjunction with other activities, to increase the quality of student learning and ultimately to improving learning outcomes going forward. Specifically, the project contributed to creating an environment conducive to renovating the general education system in Vietnam by developing, piloting, and partially implementing the new competency-based curriculum. This achievement was evidenced by (i) successfully having the new curriculum completed under the purview of qualified CTC and SSTCs and subsequently endorsed by the government; (ii) developing a robust and scalable system for increasing teachers’ readiness to implement the curriculum through face-to-face and online professional development training dedicated to facilitating their adaptation to the new curriculum; (iii) rolling out the new curriculum nationwide for grade 1 in the final year of project implementation, with clear plans for subsequent grades to follow; and (iv) administering the NLSA exams to a nationally representative samples of students in each of the three educational levels (primary, lower, and upper secondary) and having developed the capacity for subsequent administrations of the NLSA. The professional development system for teachers established under the REGP has already been used successfully to train large numbers of teachers with funding from other sources, notably from local government budget funds which are a predictable source of funds. Furthermore, teachers and school managers indicated a high level of satisfaction with the new professional development training modules aligned with the competency-based curriculum, including how they helped teachers improve teaching and student learning. This suggests that the project has and will likely continue to contribute to increasing the effectiveness of instruction, which was set out as one means of achieving improved learning outcomes, as defined in the statement of the PDO. 52. While the project failed to achieve all three PDO Indicators and four IRIs, it met or surpassed the end targets of some indicators linked to the first building block (including six fully-achieved IRIs and one partially-achieved IRI). As part of the curriculum reform, the project supported the textbook development process by having the MOET serve as a textbook appraiser to ensure quality assurance of the newly developed textbooks by private publishing houses and their alignment with the new curriculum. 53. Nonetheless, as a result of the project’s withdrawal of textbook distribution to disadvantaged schools and its discontinuation of translating a selection of aligned textbooks for ethnic minority students, the RGEP was unable to execute the dissemination of textbooks, which would have served as another building block for raising student learning outcomes. This outcome was also reflected in the project’s failure to fulfill the end target of both PDO Indicator 3 and the IRI aligned with the aforementioned objective. Ultimately, the project did not reach the originally envisioned goal for the revised curriculum and textbooks to be fully rolled out for all grades, followed by periodic assessments of student learning gains through test scores in math and Vietnamese. C. EFFICIENCY Assessment of Efficiency and Rating 54. Based on the ex-post cost-benefit analysis (CBA), despite the significant delays in implementation and the low disbursement, the RGEP interventions resulted in marginally positive returns. The benefits include returns in the form 62It should be noted that the PDO as written covered the whole basic education sector and its scope was not narrowed during implementation. Page 22 of 64 The World Bank Renovation of General Education Project (P150058) of incremental productivity of beneficiary students over the course of their working lives through improved learning outcomes, as well as fostering of cognitive and socio-emotional skills following the adoption of the RGEP-supported competency-based education (CBE) curriculum. Other outcomes that are expected from the RGEP interventions include higher completion and transition rates across grades and educational levels. 63 There is a lag of several years between the (i) project closure and (ii) full rollout of the CBE curriculum in all grades and the collection of data on realized benefits. Hence, the economic benefits of the RGEP interventions are estimated for 11 cohorts of students who are slated to benefit from the CBE curriculum during the project’s lifespan (i.e., 2015–2020) and until the start of the next curriculum reform in the future. The benefit streams are estimated based primarily on the following information: the rollout schedule of the new curriculum developed under the RGEP; the schedule of a future curriculum reform; education indicators on enrollment, dropout, and transition; and labor market information related to returns to education and skills, wages, employment, and economic growth. The present value of gross benefits in the base case is estimated to be at US$731.098 million. The costs include (i) project costs and depreciation costs of direct investment in education non-infrastructure items (curriculum design, capacity building for the education workforce, and large-scale assessments), equipment, and research materials; and (ii) indirect/opportunity costs associated with undisbursed/cancelled project funds. The total economic costs of RGEP in the base case scenario (current value in 2021) were equal to about US$649.59 million, which mainly comprised RGEP’s direct financial costs (US$18.13 million) and the opportunity cost of cancelled funding (US$630.76 million). Moreover, as described in the forthcoming section on fiduciary compliance, the annual external financial audit reports were of unqualified (clean) opinions, suggesting that the disbursed funds were used judiciously and for their intended purposes. Overall, RGEP investments generate a net present value (NPV) of about US$81.50 million, which is equal to about 15.7 percent of the NPV benefits of the US$519 million expected ex ante, with an internal rate of return (IRR) of 1.24 percent in the base case scenario compared with the IRR of 27 percent predicted ex ante. 64 55. In terms of implementation efficiency, the RGEP encountered a series of challenges. The amount of time elapsed between project approval and effectiveness, which amounted to 13.5 months, was substantially longer than anticipated at appraisal 65 and, as a result, shortened the originally planned implementation period for the project. The ambitious timeframe 66 of the project that entailed comprehensive and highly technical activities—including competency-based curriculum development for all subjects for grades 1–12, textbook development, large-scale teacher training activities to support the rollout of the new curriculum and textbooks countrywide, national assessments aligned with the new curriculum, and consultations with a variety of stakeholders (given the public nature of the reform effort)—did not bode well when decisions on investments in ODA-funded projects and internal processing often involve lengthy and cumbersome approval procedures. The delay in the MOET’s approval of the new curriculum (which did not take place until December 2018), the National Assembly’s decisions in keeping with the MOET’s and GOV’s proposals on the postponement of the curriculum rollout and cancellation of the use of IDA funds for textbook development, and the MOET’s extensive internal review process caused significant delays in implementation, thereby contributing to the project’s reduced efficiency. As a result, despite the MOET’s and the 63 The spillover effects of education, including positive externalities on socio-economic indicators such as health, productivity of non-beneficiaries, more shared prosperity (equity), and increased future revenues for firms, which are likely positive in this case are not reflected in the analysis. 64 The details of the economic analysis are included in Annex 4. 65 The project was expected to meet its effectiveness condition (i.e., the completion of the Project Operations Manual acceptable to the Bank) shortly after countersigning the Financing Agreement, which was originally scheduled to take place by the end of January 2016. However, it was not countersigned until April 8, 2016, followed by project effectiveness on August 8, 2016. 66 For instance, the majority of the work led by the CTC on the curriculum revision under Component 1 was planned to be completed during the first 12 months, followed by three months of consultations and public review. Page 23 of 64 The World Bank Renovation of General Education Project (P150058) Bank’s continued efforts to get key activities back on track, the project suffered from low disbursement rates. The project was able to disburse only about 21.5 percent (US$16.54 million), leading to the cancellation of US$60.46 million from the original committed credit amount of US$77 million. The systems established under the project and applied for the implementation of the new curriculum in grade 1, are available for use for other grades beyond the project period. However, not all these building blocks were completed, RGEP itself only funded the training of a very small number of teachers, and more than 20 percent of project funds were used for one grade out of the 12 in basic education. 56. The results of the economic analysis and the assessment of project implementation efficiency suggest that the use of the RGEP funds was low and not as efficient as what would have been expected in the operation’s sector. Despite the marginally positive results in the ex-post CBA, the significant delays in countersigning and the implementation of key activities resulted in reducing the project’s overall efficiency. Based on this, this ICR rates the RGEP’s efficiency as Negligible. D. JUSTIFICATION OF OVERALL OUTCOME RATING 57. The ICR rates the overall outcome Unsatisfactory, considering that the PDO’s relevance was rated as High and its efficacy and efficiency were rated as Modest and Negligible, respectively. The project was ambitious and highly relevant; however, implementation delays, policy changes and a failure to restructure the project early enough to reduce the project scope appropriately indicated that the project had major shortcomings in achieving its objective efficaciously and efficiently. Nonetheless, while the project could not achieve its ultimate objective within the project timeframe, the critical foundations that it established are expected to help the country attain improved student learning outcomes in subsequent years. E. OTHER OUTCOMES AND IMPACTS (IF ANY) Institutional Strengthening 58. The collaboration between the RGEP and the Enhancing Teacher Education Project (ETEP) 67 on teacher training allowed the projects to strengthen the institutional arrangements to ensure effective execution of the educational reforms, especially in the area of teacher training, build a stronger synergy, and streamline operational efficiency. The MOET and the Bank agreed that the two projects would work together to establish the online teacher training system via an LMS portal as part of their joint efforts to improve continuous professional development for teachers and school masters in a way that aligns well with the new curriculum. While the ETEP adopted a total of 54 modules for teachers and school managers to support their continuous professional development 68 in accordance with the newly developed professional standards for teachers and school managers aligned with the new competency- based curriculum, as well as with professional development needs identified by individual teachers and school managers, the RGEP provided support by engaging international and national curriculum experts contracted under the project in the training module development processes to provide quality assurance. 67 The ETEP (P150060) aims to strengthen teacher training institutions to enhance teacher and principal effectiveness through improved continuous professional development. The project was approved in June 2016 and became effective in June 2017. It is still ongoing and being implemented by the MOET. 68 A total of 54 modules have been adopted for teachers and school managers at each educational level (i.e., nine modules each for primary, lower secondary, and upper secondary teachers and school managers). The RGEP supported the development of three of the nine modules for primary teachers. Page 24 of 64 The World Bank Renovation of General Education Project (P150058) 59. Furthermore, to amplify the institutional benefits of collaboration, not only was a joint team of the two PMUs formed to work collaboratively in accordance with an integrated work plan, but based on the Bank’s recommendation, a new organizational arrangement was also established to accelerate the synergy and avoid duplication of efforts. 69 The new arrangement prompted the following key changes: (i) one project director overseeing both the RGEP and ETEP to enhance coordination between the projects; and, (ii) relevant and qualified technical leaders of functional departments of the MOET and Lead Teacher Training Universities (LTTUs) assigned to manage core tasks under the projects as part of their regular duties, while remaining PMU staff were to focus on their primary role of providing logistical support for the technical leaders. Ultimately, these structural changes helped strengthen the MOET’s institutional capacity by allowing its line staff and LTTUs to engage more directly in the reform process and provide technical input. Additionally, positioning key line staff and LTTU leaders allowed for improving sustainability of the reform process through long-term execution and monitoring. Mobilizing Private Sector Financing 60. While the project did not formally mobilize private sector financing, the challenge originating from the change in the arrangement of textbook financing presented an opportunity for the MOET to engage the private sector in textbook development and publishing under Component 2. As Vietnam transitioned from its previous single- textbook system to the new multiple-textbook one, the RGEP was initially intended to provide financing support for textbook production. However, the GOV’s decision to cancel the use of IDA funds for textbook development activities and the MOET’s subsequent withdrawal of assigning the Vietnam Education Publishing House (VEPH)—a 100 percent state-owned enterprise under the direct ownership and supervision of the MOET—to manage textbook development opened the door for the full-scale entry of private publishing houses into the textbook production market. It remains to be seen how this market will evolve or how such public–private competition will become viable, while ensuring the availability of better-quality textbooks at affordable prices. 70 Nonetheless, improvement in textbook market entry for private publishers, combined with increased capacity-building training for textbook writers, editors, and appraisers under the RGEP, has created great potential for accelerating the liberalization and maturity of the market for textbook development in Vietnam, thereby allowing commercial publishers to play a greater part in supporting the renovation of the education sector. This move will also increase households’ contributions to the education of their children, though with, as yet, no adequate protections for those unable to pay. Poverty Reduction and Shared Prosperity 61. While poverty reduction was not an explicit objective of the RGEP, it sought to address the issues of equity in accessing textbooks and a quality education among ethnic minority children and other disadvantaged learners. Specifically, under Component 2, the MOET planned to (i) translate a selection of textbooks and their samples (including five sets of approved textbook samples) into eight ethnic minority languages; 71 (ii) convert grade 1 textbooks selected by schools in which visually impaired students are enrolled to Braille textbooks; and (iii) provide sets of textbooks to schools with a high number of enrolled students from economically disadvantaged families. Moreover, PDO Indicator 3 was specifically designed to monitor the project’s progress in improving equity in access to learning materials. However, as described in Section II.B., the project failed to make progress on this front 69 This new arrangement was formalized by the MOET’s issuance of Official Plan No. 791 specifying the roles and responsibilities of each concerned party of the RGEP and ETEP. 70 Among the 46 MOET-approved textbooks for grade 1 students, the VEPH still had the highest number of books, with 34 approved titles comprising three complete sets, while Hanoi Pedagogical University, Ho Chi Minh City (HCMC) University of Education, and Vietnam National University—HCMC had 12 textbooks approved but did not have a complete set of textbooks. 71 Plan No. 1160/KH-BGDDT issued on October 30, 2019. Page 25 of 64 The World Bank Renovation of General Education Project (P150058) because the textbook provision activity was cancelled. In effect, without official textbook access surveys, neither baseline nor end-line data was available. 62. Nevertheless, given the existing challenges in access to textbooks among underserved groups of students in some areas, despite extant programs and regulations (as referenced in footnote 63), a more rigorous mechanism needs to be in place to continue to monitor and ensure equitable access to and affordability of textbooks and other learning materials in all areas. Otherwise, there could be a potential adverse effect on students whose parents cannot afford to buy new textbooks, as well as those in disadvantaged schools should school localities not procure enough books for each student to borrow. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 63. Groundwork laid to facilitate the curriculum reform: Vietnam’s participation in Russia Education Aid for Development (READ) Trust Fund-financed activities aimed at strengthening and improving the effectiveness of the country’s assessment system had laid the groundwork for the new curriculum reform during the project preparation. With READ funds, part of which were used to explore Vietnam’s participation in the 2012 PISA, the MOET benefited from technical interactions with and advice from leading global experts and practitioners on how best to build the capacity of a broad range of stakeholders at both national and local levels 72 to facilitate successful curriculum and assessment reforms. By offering the MOET hands-on training opportunities to consider how to build competency-based standards and an assessment framework into the new curriculum, the READ activities helped improve the project’s readiness to embark on the curriculum reform. 64. Technical design built on lessons learned: To address the potential high level of risk associated with the technical complexity of the reform, the project incorporated lessons learned from previous curriculum reform development processes into the design. 73 One of the major shortcomings identified in the previous curriculum reform was a lack of coherency and clarity in the use of endorsed principles (which focused on student-centered, active, and collaborative learning). This issue largely stemmed from the following: (i) the previous curriculum reform was segmented by school subsystem (i.e., primary, lower-secondary, and upper-secondary education); and (ii) a large number of curriculum developers (who are often also textbook developers) worked independently of each other, which resulted in incoherence and duplication of content in the curriculum. Drawing on these lessons, the project sought to mitigate this risk by designing and implementing new structural arrangements under the CTC. This involved empaneling a small group of top national experts to conduct the main curriculum revision and serve as the chairpersons of each SSTC. Such arrangements allowed curriculum developers to establish a common vision for the new curriculum and coordinate their efforts in creating more coherent and rigorous curriculum content across subjects and years of general education. 65. Inadequate political climate considerations and risk-mitigation measures: While the project identified (i) the political system not adequately equipped to facilitate decision making and communication across functions, as well as (ii) limited institutional capacity of the implementing agency, among the potential risks in the appraisal phase, rigorous risk-mitigation measures were not proposed. Rather, the PAD suggested (i) the team’s continued, 72 The broad range of stakeholders included teachers, school leaders, provincial departments, MOET staff, teacher training colleges, and universities. 73 The previous reform was initiated under the guidance of National Assembly Resolution No. 40/2000/QH 10 and rolled out sequentially, beginning in 1997. Page 26 of 64 The World Bank Renovation of General Education Project (P150058) close monitoring of the political climate to ensure achievement of the PDO, despite a potential change in leadership, and (ii) that the PMU should be staffed by qualified MOET employees and consultants with adequate skills to coordinate and manage the implementation involving a wide range of stakeholders and agencies. Given that this was the first Bank-financed project to work extensively on a curriculum reform in Vietnam and that many MOET staff members who were expected to be part of the project had no previous experience in engaging in Bank- financed operations, more suitable measures should have been implemented to alleviate any potential adverse outcomes identified. 66. What seems to underlie the inadequacy of the risk-mitigation measures was the project’s inability to sufficiently take the political climate into consideration in steering the reform process. There was already a sign of challenges during the project preparation in navigating the possible ramifications of political decisions on the curriculum reform. The MOET expressed its reservations about pushing through the project approval without the National Assembly’s official endorsement of the curriculum reform. 74 Even after the approval by the Board in June 2015, the MOF and Vietnamese president also voiced their dissent in relation to the use of IDA funds for the RGEP on the grounds that ODA resources should be utilized only for interventions for which national funds or expertise were inadequate or unavailable. To garner their buy-in, the Bank’s country director for Vietnam at that time held high-level meetings with the GOV to explain the importance of mobilizing technical assistance to successfully facilitate the curriculum reform under the RGEP. In doing so, an overture to conduct a restructuring soon after the signing was also offered so that the project cost could be reduced from US$77 million to US$19.4 million by strictly limiting the use of IDA funds to support the key areas wherein leveraging technical assistance would be crucial. 75 The Bank’s persistent efforts to gain support for the RGEP ultimately paid off by securing the signing of the Financing Agreement ten months after project approval. Nevertheless, the government’s adoption of the new decree restricting the use of ODA funds for recurrent expenditures (including technical assistance), as well as its subsequent policy reversals, including the discontinuation of IDA financing for the textbook development and dissemination and the timing of the curriculum rollout, suggest that more careful measures to cope with decisions on policy changes informed by an analysis of the underlying driver behind the policy decisions and divergent views among key government stakeholders would have minimized the impact of unforeseen policy outcomes. 67. Highly ambitious objective and timeframes: The RGEP’s project design was highly ambitious in light of its timing, the sequence of activities, and its objective setting. The original plan was for the project to improve student learning outcomes by completing the key activities in the following sequence: (i) finalizing the curriculum revision within 15 months—three-month validation process by the CTC for the general curriculum, followed by the drafting 74 During the Bank’s preparation mission from September 23–October 24, 2014, the MOET expressed hesitation with moving forward through the Detailed Project Outline (DPO) to the Ministry of Planning and Investment (MPI) for circulating to other government agencies (including the MOF and the Office of the Government) for their review and approval until the National Assembly officially endorsed the curriculum reform. The mission noted its concerns that, while the National Assembly’s approval of the new resolution on curriculum and textbook reform during its plenary from October–November 2014, as well as the MPI’s ability to garner approval from the Prime Minister for the DPO and for putting the project on the ODA list in a timely fashion, would ultimately determine the project’s readiness for the Board Meeting and securing IDA funds without delay, the extensive and cumbersome internal review process typical of Vietnamese governmental institutions could inhibit timely progress toward project approval. 75 The key project activities proposed to be retained included the following: (i) developing and implementing the new competency-based curriculum, for which technical assistance was essential; (ii) establishing an online teacher training system aligned with the new curriculum; and (iii) providing training to national experts to ensure that the national large-scale assessment system would meet international standards. Ultimately these were the activities which remained once the restructuring was completed, though not until right before the project's closing. Page 27 of 64 The World Bank Renovation of General Education Project (P150058) of the Subject-Specific Curricula for nine months, and the consultation, appraisal, and adoption of the curriculum in its entirety by the MOET within the following three months; (ii) developing an exam to set the baseline for PDO Indicator 1 (student learning gains) in Year 1 and administering the exam in Year 2; (iii) rolling out the new curriculum for the first cohort of grade 1 students by Year 3; (iv) piloting the first round of the NLSA for two of the three levels of basic education in Years 2, 3, and 4; and (v) administering the NLSA to nationally representative samples in each of the three levels in Year 5. Considering the highly technical project activities and such pre-existing constraints on the institutional capacity of the MOET, as well as the onerous political and legislative processes involved in the curriculum reform, the project design was overly ambitious. In this light, even had the project gone smoothly without the initial delays in countersigning, given the need for adjustments and adaptation by school stakeholders (especially teachers and students) to the new curriculum in classrooms during the curriculum transition, the project would likely have faced challenges in achieving the outcome in just two years after the rollout of the new curriculum or having only a quarter of teachers equipped with the necessary skills to teach the new curriculum. 68. An insufficiently articulated theory of change. To be achieved, the PDO required a comprehensive set of activities. 76 While the project identified necessary building blocks, it did not include a fully sufficient set of interventions (some of which, such as comprehensive and ongoing teacher training, were picked up by ETEP which was approved a year after RGEP; others, such as early childhood education, were not). Moreover, the level of ambition of some indicators was insufficient to achieve the PDO: for example, if the end-of-project target of 25 percent of teachers are equipped to teach the new curriculum, it is unlikely that the average student performance for the country as a whole will increase; and it is not clear how improving the textbook ratio by 10 percent for ethnic minority students is likely to improve the effectiveness of instruction for all minority students, let alone how it could improve the general effectiveness of instruction in the sector. These challenges of the theory of change were compounded by the lack of ways to measure two of the PDO indicators. B. KEY FACTORS DURING IMPLEMENTATION 69. The following were identified as key factors that contributed to the overall delays in the implementation and underachievement of the RGEP compared with its expected outcomes. 70. Inadequate coordination among different stakeholders to facilitate the curriculum reform process: One of the key bottlenecks that constrained the timely and effective implementation of the RGEP was inadequate coordination among stakeholders engaging in the curriculum reform process. While the MOET played a key role as the implementing agency in leading the reform, a series of adjustments and revisions made to project activities illustrated the challenges the project faced in facilitating dialogue among key stakeholders to allow for collective and coherent decision-making. This made it extremely challenging also to agree on a major restructuring of the project in a timely fashion. Some of the instances that reflected these challenges include the following: 76For example, the World Bank Education Global Practice now advocates five pillars for raising learning outcomes in basic education: (1) Learners that are prepared and motivated to learn; (2) Teachers that are effective and valued; (3) Classrooms that are effective learning spaces; (4) Schools that are safe and inclusive; and (5) Education systems that are well managed (https://www.worldbank.org/en/topic/education/overview#2, accessed June 5, 2021). As early as in its Education Sector Strategy 1999, the World Bank articulated broad-based education reform as necessary for improving learning. Page 28 of 64 The World Bank Renovation of General Education Project (P150058) (i) Conflicts between the originally planned project activities and state policies and regulations: Some conflicts arose between the initially planned activities and extant state policies and regulations, which became apparent during the project implementation and required course corrective actions. For example, one of the key objectives and components of the RGEP was to improve equity in access to textbooks and other learning materials by disseminating textbooks aligned with the new curriculum. However, the project ultimately cancelled the planned activity due to the existence of an MOF–MOET Joint Circular (issued in 2009) on the provision of textbooks to ethnic minority and disadvantaged students and a GOV Decree (issued in 2015) stipulating the provision of school materials (including textbooks) to all students by parents, schools, or local authorities. Given the pre-existing gap between the de jure and de facto practice, setting up measures to address the underlying factor(s) contributing to inequitable access, despite the extant regulations (rather than administering surveys to assess the availability of learning materials), might have helped create a more sustainable mechanism to ensure more equitable access beyond the project period. Similarly, the emerging conflict over the use of public and ODA funds for textbook development and publication, as well as the National Assembly’s ultimate decision to discontinue these activities with IDA financing, indicates that more thorough and careful discussions should have been undertaken among stakeholders prior to project approval about the implications of the multiple-textbook policy adopted in 2015 on the roles of and areas for capacity building for the MOET and commercial publishers in undertaking the curriculum reform. (ii) Continuing lack of consensus over the deployment of IDA funds for technical assistance and capacity building activities: The lack of consensus over the use of IDA funds, which had inhibited the timely conclusion of the project signing and the materialization of a formal restructuring, was another instance of how inadequate coordination among the stakeholders continued to undermine the curriculum reform process during project implementation. As described in Section III. A, the project began with an extensive delay in the GOV countersigning of the Financing Agreement because of the divergent views among the government stakeholders as to whether ODA funds should be deployed for project activities wherein technical assistance and capacity building would be essential to ensure quality curriculum reform. This initial obstacle not only resulted in shortening the project implementation period by almost one year but also caused delays in implementation, leading to the National Assembly’s announcement to postpone the new curriculum rollout. As a result, the duration of the new curriculum implementation under the RGEP’s support was further reduced. Moreover, the continuing challenge in resolving the conflicting views concerning the handling of recurrent expenditures as evidenced in the GOV’s issuance of Decrees No. 16 and 56 (see footnote 55), ultimately precluded the project’s restructuring after the MTR. In response to the MOET’s formal proposal in August 2020 for the RGEP’s restructuring and extension, the MOF held a meeting with the Bank and representatives from the MOET to discuss the proposal. The Bank stated that the restructuring would be conditional upon the eligibility of IDA financing to support capacity building and other non-infrastructure activities so that the RGEP could help the country achieve educationally impactful results through the curriculum reform. However, as appropriate action was not taken by the established deadline (November 15, 2020), the restructuring did not materialize. 71. Weak communication strategies: Weak and insufficient communication strategies also contributed to the inadequate coordination among stakeholders. Despite the Bank team’s repeated advice to the PMU, the RGEP suffered from not having systematic communication plans in place well ahead of the new curriculum rollout. Even at the time of the MTR mission (which was undertaken in November 2018), the PMU had still not been able to recruit a qualified communication consultant who could lead the development and implementation of Page 29 of 64 The World Bank Renovation of General Education Project (P150058) communication strategies or sign a contract with a communication consulting firm. Rather, communication activities remained limited and ad hoc, relying primarily on personal-level initiatives by the then chief technical officer. The curriculum renovation, coupled with the introduction of a new multiple-textbook policy, required an understanding and buy-in from all stakeholders (including policymakers and the general public) of their rationale and objectives. However, the project did not manage to outline a set of concrete, well-defined communication strategies, which should have included well-thought-out plans to address “communication crises,” in particular those stemming from possible divergent/conflicting interests and incentives facing different stakeholders that might generate unforeseen policy outcomes that could hinder the reform. Toward the end of the project, communication activities picked up pace, actively engaging in media outreach campaigns in coordination with the MOET’s Communication Department to promote the transparency of the curriculum renovation and address public concerns about the introduction of the new curriculum. The project also amplified its web presence by making regular updates on the project’s website and fan page. Nonetheless, implementing effective communication strategies at an earlier stage could have facilitated better coordination among stakeholders and minimized disruptions due to frequent policy changes. 72. Limited capacity of the PMU: The project also encountered challenges in mobilizing a team of PMU personnel with high-level administrative and coordination skills to navigate the curriculum reform process, which entailed comprehensive and highly technical activities and required a broad range of stakeholders. The limited management capacity of the PMU resulted in constant delays in implementation, including procurement activities, and ineffective utilization of technical experts tasked with supporting the development of the new curriculum, teacher training modules, and NLSAs. The PMU also experienced high staff turnover, including multiple new project directors and chief technical officers. 77 While forming a joint PMU with the ETEP helped the two projects strengthen the institutional arrangements and streamline the management of teacher training activities, the continued limited capacity of the PMU and frequent leadership changes contributed to undermining implementation progress. 73. Continuation of professional development training despite disruptions caused by COVID-19: The outbreak of the COVID-19 pandemic caused disruptions in the final year of RGEP implementation. Due to the difficulty of organizing intensive training sessions in such unprecedented times, some of the sessions needed to be rescheduled or switched to an online format. Nonetheless, the training model adopted under the RGEP and ETEP, which combined both face-to-face and online training sessions for teachers and school managers, minimized the adverse impact of the pandemic on the implementation of training activities, as well as assisted teachers and school managers in improving their readiness to teach in accordance with the new curriculum. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 74. The project’s M&E design and ToC had some important shortcomings. 77 The RGEP had three project directors (appointed in 2015, 2019, and 2020, respectively) and two chief technical officers. Page 30 of 64 The World Bank Renovation of General Education Project (P150058) (i) The framing of the compound PDO statement: As described earlier in Section II.B., while the outcome-based overarching objective (“to raise student learning outcomes”) was clearly articulated in the PDO statement, the link between the means (“creating and disseminating textbooks”) and the ends (“improving the effectiveness of instruction”) was unclear, thereby undermining the logic of the ToC. Having no indicator developed to assess the effectiveness of instruction (as described below in (ii)) made it even more difficult to monitor and evaluate how the textbook creation and dissemination would contribute to greater effectiveness of instruction and, eventually, to improved learning outcomes. (ii) Alignment of PDO-level indicators with key interventions as defined in the PDO: While PDO Indicator 1 (student learning gains) was clearly formulated and directly linked to the RGEP’s overarching objective (“to raise student learning outcomes”), neither PDO Indicator 2 (teacher preparedness for the new curriculum) nor PDO Indicator 3 (equitable access to learning materials) was well formulated to monitor and measure the progress of the two key interventions through which the project was designed to achieve improved learning outcomes. 78 Although five IRIs (1, 2, 3, 5, and 9) linked to Component 1 helped capture progress on the revision and implementation of the new curriculum, no PDO-level or intermediate results indicator was identified to determine how or to what extent a competency-based approach was built into the curriculum. Similarly, without any indicators to track progress toward achieving “improved effectiveness of instruction,” it was difficult to assess how either of the main components of the project activities (i.e., the curriculum revision, and the textbook creation and dissemination) would lead to the expected outcome. The meaning of “effectiveness” was also not clearly defined. While the project’s results framework included multiple indicators (PDO Indicator 2, IRI 2, and IRI 9) to monitor the progress of teachers’ “readiness” for instruction under the new curriculum, it would have been more useful to include an outcome-based indicator to determine how or to what extent the teachers certified to teach the new curriculum, as well as those who were yet to be certified but had completed some portion of the training, were applying improved teaching and learning practices in their classrooms. (iii) Specifications of project indicators and methodologies for measuring indicator results: Some project indicators, as well as the methodologies for data collection and analysis, were not clearly defined in designing the results framework during project preparation. The remaining ambiguity in the definitions and methodologies to measure key indicators, including PDO Indicators 1 and 3—both of which were expected to entail the complex and arduous process of data collection and analysis to set the baselines through administering exams and surveys—precluded the timely establishment of M&E arrangements and subsequent progress monitoring. M&E Implementation 75. The project’s M&E was implemented primarily through the following activities: (i) data collection and analysis by the PMU on project-financed inputs, outputs, and outcomes to assess achievement toward project objectives as per the results framework; and (iii) preparing implementation progress reports biannually in conjunction with the Bank- led supervision missions. 76. In addition, the project conducted ex-post field surveys in six provinces to examine how the new curriculum has been implemented in grade 1 and so that any challenges faced by stakeholders (including localities, school managers, teachers, students, and their parents) could be addressed as the new curriculum is sequentially rolled out to other 78These two interventions were (i) revising and implementing the curriculum following a competency-based approach and (ii) improving the effectiveness of instruction by creating and disseminating textbooks aligned with the revised curriculum. Page 31 of 64 The World Bank Renovation of General Education Project (P150058) grades in the coming years. As part of the curriculum piloting exercise, survey questionnaires were also administered to the teachers and school managers of the participating schools to investigate their perceptions of experimental vis- à-vis non-experimental (regular) classrooms as well as to identify lessons learned from the piloting to be incorporated into the full rollout of the new curriculum, though the data from these surveys was not systematically analyzed to gain insight into the impact of the changes at the classroom level. 77. The aforementioned ambiguity in the definitions of some project indicators and methodologies for measuring them prompted the PMU to update the results framework during project implementation by finalizing and making adjustments to the specifications of those indicators and their measurement and by reflecting the revisions to the implementation schedules and arrangements. 79 For example, PDO Indicator 1 was initially designed to be measured by comparing the mathematics and Vietnamese test scores of the grade 3 cohort at the baseline and the endline (i.e., before and after the rollout of the new curriculum). However, because of the overall delay in the rollout (with only grade 1 rolled out in the final year of project implementation), the estimation method was modified to compare the same test score results for the treatment and control groups of grades 3 and 8 through an impact evaluation study (for the complete list of descriptions of the PDO-level and intermediate results indicators, see Table 6.3 in Annex 6). Nonetheless, the plan did not materialize by project end due to the delay in finalizing the study design and the project failing to get extended to complete the study. M&E Utilization 78. The use of M&E data was appropriate throughout the life of the project. Using the M&E report templates prepared by the PMU based on the results framework, the M&E information was closely monitored and updated by the PMU, which allowed the Bank team to review whether the project was progressing as planned and to provide recommendations on how project activities could be adapted to ensure achievement of the PDO. 79. As referenced in the previous section on M&E implementation, an adjustment was made to the data collection methodology of multiple project indicators to allow the PMU and Bank teams to monitor and assess how the project was progressing, in light of the changes to the implementation arrangements and schedules. Following the MTR mission in November 2018, structural changes to the project implementation arrangements were proposed to speed up the implementation, especially in the area of teacher training, and to improve coordination and synergies between the RGEP and the ETEP. 80. Despite these efforts, the project did not succeed in turning things around across the board. The delays in adjusting the definitions of some indicators to improve their alignment with actual implementations and in finalizing methodologies to set the baselines for the indicators requiring extensive M&E arrangements (including the design and development of survey instruments and protocols) hindered the project from employing those indicator results as evidence of achievement/non-achievement of the project outcome to inform subsequent interventions or make timely course corrections. Moreover, as a result of not having been able to complete the impact evaluation study to measure student learning gains based on the test score results of the grade 3 and 8 treatment and control groups on time, the project could not utilize that data to determine the impact of the new curriculum on student learning outcomes. 80 Multiple options were discussed for amendments to the project design and scope (including the results 79 RGEP Aide-Memoire, June 2018. 80 By its completion, the project managed to accomplish the following activities related to the impact evaluation study: (i) developing the technical design of the randomization; (ii) designing math and Vietnamese exams; (iii) preparing test blueprints; (iv) finalizing the specification of the exams; and (v) developing operational forms for the first round of tests. While this Page 32 of 64 The World Bank Renovation of General Education Project (P150058) framework), as well as the implementation period, to account for the government’s decision not to use project funds for some project activities. Nevertheless, the restructuring plan ultimately fell through because appropriate action was not taken by the authorities by the agreed-upon date to meet the condition for restructuring. 81 Justification of Overall Rating of Quality of M&E 81. The ICR rates the overall quality of the M&E as Modest. There were some important shortcomings in the project’s M&E design, especially with regard to the framing of the PDO statement, the design of PDO-level indicators, and the delays in finalizing the definitions of project indicators and measurements, all of which precluded a meaningful and timely evaluation of project outcomes within the framework of the ToC. The PMU and Bank teams made collective efforts to revise the data collection methodology of some project indicators to improve their alignment with the new developments in relation to the curriculum rollout, as well as textbook development and distribution. Nevertheless, the overall implementation delays, combined with the project’s inability to restructure and to collect the baseline and endline data for measuring student learning gains, did not allow for timely course corrections or full utilization of M&E data to provide evidence of achievement of the project’s ultimate objective of improving learning outcomes. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 82. Environmental compliance: Throughout its life, the project’s performance in environmental safeguards was rated Satisfactory. This Category B project complied with the OP 4.01 Environmental Assessment to mitigate any potential adverse environmental and social impacts owing to the planned construction of the NCSDGEQ and NCFLT research centers under Component 3. An Environmental and Social Management Plan (ESMP) was developed by the MOET under the Bank’s guidance to ensure the project’s compliance with the Bank’s safeguard policies and national regulations. Upon the Bank’s approval, the ESMP was disclosed to the public on the MOET’s website. However, in the end, the construction was cancelled due to the severe implementation delays, as well as the Bank’s and the MOET’s mutual decision not to restructure the project. 83. Social compliance: The RGEP, which triggered OP 4.10 Indigenous Peoples, 82 conducted a Social Assessment to identify potential barriers related to the curriculum and textbooks for ethnic minority students. Based on the review of and findings from policy documents, the existing curriculum and textbooks, and consultations with representatives of ethnic minority groups, the project developed a bias avoidance framework, 83 which was integrated into the curriculum and textbook development process. It also served as an Ethnic Minority Development Plan to ensure that the new curriculum and textbooks were bias-free and culturally relevant to ethnic minority students. The production of bilingual textbooks was an important vehicle to help ethnic minority students access the curriculum, but it was approach might have been successful to measure project outcomes, it was a different approach from that envisaged during project preparation in which measurement of learning improvement under the project was to be based on the national system of learning measurement. 81 Details on how the restructuring overtures and negotiations played out between the parties are provided in Section III.B. Key Factors during Implementation. 82 The project did not involve any displacement of or land acquisition from ethnic minority populations. 83 The recommendations captured in the bias-avoidance framework were included in Circular No. 33/2017/TT-BGDĐT (dated December 22, 2017) issued by the MOET as standard: the form and content of textbooks should be without prejudice related to “race, religion, profession, gender, age group and social status” (MOET. 2021. RGEP Project Completion Report, p. 31). The framework has also been concretized into indicators according to textbook standards as specified in Circular No. 33/2017/TT- BGDĐT (ibid.). Page 33 of 64 The World Bank Renovation of General Education Project (P150058) discontinued under the RGEP. However, curricula (but not the accompanying textbooks or learning materials) for eight ethnic minority languages were developed in consultation with experts from related fields. 84 84. Fiduciary compliance: (i) Financial management performance: The project was compliant with all the fiduciary requirements during project implementation, and its financial management arrangements were found to be acceptable to the Bank based on the following: (a) timely submission of the interim financial reports in a format and with content satisfactory to the Bank; (b) internal and external audits carried out as per the Bank’s recommendation; and (c) unqualified (clean) opinions received for the project’s audited financial statements, with no major control or accountability issues raised during annual external audits. (ii) Procurement performance: The project continued to remain compliant with the Bank’s procurement guidelines and policies throughout the implementation cycle. However, constant delays in the procurement of goods and technical assistance services and contracts implementation, combined with the cancellation of (a) as many as 59 IC packages under the component of textbook development following the National Assembly’s decision not to use IDA financing for the activity and (b) the construction of the NCSDGEQ and NCFLT, contributed to slow disbursement progress during implementation and resulted in a low disbursement rate at project completion. 85 C. BANK PERFORMANCE Quality at Entry 85. The Bank performance in ensuring quality at entry had both strengths and limitations. The design of the curriculum development, the major component of the RGEP, was technically sound. It not only incorporated the lessons learned from the previous curriculum reform to facilitate a more rigorous and cohesive general education curriculum by mobilizing the CTC and SSTCs; it also supported the GOV’s key education sector priorities and the country’s critical needs to foster 21st-century learners equipped with high-quality cognitive and market-relevant competencies and skills as driving forces for promoting the country’s productive and equitable growth. 86. The rationale of the RGEP and its ultimate goal of improving learning outcomes as a result of the effective implementation of the new competency-based curriculum was supported by the following: (i) a body of extant research evidence demonstrating the empirical relationship between better-designed curricula and student learning 86 and (ii) the findings of the READ Trust Fund-financed self-diagnosis of the national assessment system in Vietnam and the System Assessment and Benchmarking for Education Results (SABER) report, both highlighting the inadequacy of the country’s NLSAS because of its lack of technical capacity and its irregular and inconsistent implementation. 87 87. During project preparation, the RGEP benefited from interactions and consultations with relevant stakeholders, including (i) technical experts on competency-based assessments, pedagogy, and curriculum development under the 84 RGEP Aide-Memoire, June 2020. 85 The project’s disbursement rate remained at only seven percent at the time of the MTR in November 2018 and 22 percent even at project end in December 2020. 86 World Bank. 2015. PAD for the RGEP. 87 For example, one of the key findings of the SABER report was the irregular funding of the NLSA program, which hinged on donors. This issue prevented the establishment of a coherent learning assessment system that allows for continuous monitoring and improvement of the effectiveness of curriculum and teaching. Page 34 of 64 The World Bank Renovation of General Education Project (P150058) purview of the READ Trust Fund; (ii) the MOET, the MOF, and other policy leaders at the nascent phase of the project to ensure the alignment of its objectives with sector priorities and the consensus on the planned activities; and (iii) ethnic minority representatives to address any potential barriers related to curriculum and textbooks for ethnic minority communities. 88. Nevertheless, some aspects could have been improved. First, as described in Section III. A, setting out better measures to mitigate or cope with the risks of unforeseen policy outcomes by taking into account political economy more adequately in navigating the education reform during the preparation phase might have helped the project avoid or alleviate the adverse impacts of the multiple policy changes to the curriculum reform process on the implementation progress. Second, the project’s implementation arrangements were inadequate, especially in terms of the role and structure of the steering committee for the curriculum reform. The committee, which was chaired by the minister and the vice minister of the MOET, was mainly responsible for directing the overall development of the new curriculum and ensuring the quality assurance of textbooks. However, without representatives from a broad range of stakeholders, including the National Assembly and media, the committee could not serve as a vehicle for facilitating dialogue or deeply engaging the key stakeholders outside the MOET in the curriculum reform process. Third, the framing of the project’s PDO, the weaknesses in the theory of change, and the design of the results framework had significant limitations, which presented a challenge for the effective monitoring of project implementation. Fourth, there was room for the RGEP to better integrate the experiences and lessons learned from the Vietnam Escuela Nueva (VNEN) project (November 2012–May 2016). Such a project, which used a student- centered model of teaching and learning, also involved pedagogical renovation, as well as the production and distribution of learning materials (including textbooks) for a selection of schools with the most disadvantaged groups of primary students. While the VNEN project was implemented on a pilot basis with grants from the Global Partnership for Education, and its textbooks were based on the previous curriculum (because it predated the new curriculum reform), the RGEP could have benefited from the VNEN’s experience in light of students’ and teachers’ adaptation to learning with a focus on 21st-century skills development and ensuring equitable access to learning materials for ethnic minority and other disadvantaged students. 88 Quality of Supervision 89. Throughout the implementation, the Bank team, which consisted of a good mix of technical, operational, and administrative experts, provided continuous implementation support by undertaking biannual supervision missions, including an MTR halfway through the project implementation. The project benefited from the continuity in the team by having members who retain the institutional memory of the project (including the task team leader [TTL] at project end, who was part of the team from the very beginning and assumed the TTL role in 2018). The availability of all key staff in Vietnam also allowed deeper and more timely engagement with the MOET and facilitated the intensive supervision support needed. The team prepared detailed aides-memoire in which the project’s progress, technical and operational areas of concern, any identified risks, and solutions offered during the mission were well documented, along with a list of time-bound follow-up actions agreed upon with counterparts. 90. To remediate the significant delays caused by the initial holdup in countersigning, as well as the policy changes related to the use of project funds for the curriculum development and new curriculum rollout plans, the Bank 88In effect, the RGEP’s ex-post field survey provided anecdotal evidence that VNEN schools and provinces demonstrated a higher level of preparedness for and conformity with the new curriculum implementation. This is presumably because in such schools and provinces, teachers and school managers learned to work with a multiple set of textbooks (both VNEN and traditional textbooks) and apply new assessment practices aligned with the VNEN model. Page 35 of 64 The World Bank Renovation of General Education Project (P150058) engaged in extensive discussions with the MOET and other government counterparts. The team carried out an MTR mission in November 2018 to resolve significant implementation delays and slow disbursement by (i) revisiting the project scope and components and the results framework; (ii) discussing restructuring options; and (iii) agreeing on actions to be taken in order to improve project performance and enhance project sustainability after project closing. 91. After the MTR, multiple restructuring options were considered, all of which were not only to scale back some of the project activities as a result of the policy ramifications but also to allow for a partial or full assessment of student learning gains (i.e., the project outcome) by extending the implementation period. As described in Sections III. A and B, the Bank consistently emphasized the importance of using IDA resources to support technical assistance and capacity building activities under the new curriculum reform, while either reducing or cancelling funds through a restructuring of other activities for which national resources could be mobilized. The Bank’s approach to restructuring was justified and built on lessons learned from the previous curriculum reform in Vietnam, which suffered from low levels of curriculum coherence and lack of alignment with assessment standards because of inadequate technical assistance to support the capacity building of stakeholders to lead the development and implementation of that new curriculum. 92. While the restructuring did not materialize under the RGEP—largely due to the remaining lack of consensus on the handling of recurrent expenditures in IDA financing—key structural adjustments made to the PMUs and organizational structures of the RGEP and ETEP not only strengthened synergies and complementarities (without duplication) between the two projects but also helped ensure the operational sustainability of activities implemented under the RGEP. 93. Nevertheless, some shortcomings in the Bank supervision could have been addressed to support more effective implementation. First, there was some inconsistency in the submission of Implementation Status and Results Reports (ISRs), especially in the early phase of project implementation, which inhibited the timely reporting of project progress and results. 89 This limitation also prevented the team and management from communicating on the identified bottlenecks and the possible course corrections to be made in a timely manner. Second, there remained some scope for the team to apply lessons learned from preceding interventions with similar components (e.g., the VNEN project) in navigating and coordinating the reform effort more successfully; given its public nature, the reform process inevitably requires effective engagement and consultations with a broad range of stakeholders. Therefore, the project could have benefited from more extensive technical support by the Bank team to the PMU in defining a process for facilitating a dialogue with stakeholders and devising strong communication strategies early on to bolster public and political support for the curriculum reform and promote collective actions. 94. Furthermore, despite its persistent efforts to help turn the project around, there appeared to be some room left for the Bank to play a more active role in bringing together policymakers involved in the reform process, including the National Assembly, GOV, MOF, MPI, and MOET. Given the political system ill-equipped to coordinate among the key stakeholders and across functions as observed in the curriculum reform process from the beginning, it might have helped for the Bank to go one step further by discerning the implications of the restructuring for policymakers with varied interests and incentives, as well as by maximizing its role in facilitating the collective buy-in and new pathways 89 Multiple instances during the first two years of implementation showed that ISRs were submitted to management prior to the planned missions with an indication that the ISRs, including performance ratings, would be updated at the conclusion of the mission. However, no updates or revisions were provided afterward. Page 36 of 64 The World Bank Renovation of General Education Project (P150058) to achieve the expected outcome, which would ultimately have contributed to improving the country’s productivity and growth. Justification of Overall Rating of Bank Performance 95. Based on the strengths and limitations of Bank performance as highlighted above and considering the overall project’s outcome rating Unsatisfactory, the overall Bank performance is rated as Moderately Unsatisfactory. D. RISK TO DEVELOPMENT OUTCOME 96. Despite the numerous challenges it faced in attaining the expected outcomes, the RGEP succeeded in creating an environment aligned with the country’s efforts to continue the renovation of the general education system in Vietnam. With the GOV’s commitment to complete the full rollout of the new competency-based curriculum in all grades (grades 1–12) sequentially by the 2024–2025 school year, as shown in Table 6.1 of Annex 6, various institutional mechanisms and the technical capacities developed under the RGEP will serve as a foundation for the country as it continues on its path to improving the quality of student learning and ultimately improving learning outcomes. The guidance materials produced by the CTC and SSTCs for the development and evaluation of textbooks and other pedagogical materials will ensure that the new textbooks to be rolled out for different grades in the coming years will be created and appraised properly in alignment with the learning outcome standards and content of the new curriculum. The shift to private provision of textbook development and production is new but highly promising. 97. The institutional sustainability of the professional development activities for teachers and school managers will be ensured by retaining some of the PMU members working as part of the joint RGEP-ETEP team under the ETEP until its closing in June 2022; together with the ETEP team members, they will continue developing the remainder of the online training modules and monitoring their quality under the leadership of the project director who oversaw both the RGEP and ETEP.90 Moreover, several MOET officials who have been closely engaged in or consulted regarding the implementation of the RGEP in activities related to teacher training, curriculum development, and learning assessments are expected to integrate their institutional memories and insights gained from the RGEP into their work on curriculum review and appraisal upon their return to their own MOET functional departments. V. LESSONS AND RECOMMENDATIONS 98. Strategic structural rearrangement can improve operational synergies and sustainability. With an eye to increasing synergies with the ETEP in the area of professional development for teachers and school managers and improving the project’s sustainability after completion, the RGEP worked closely with the ETEP (which became effective 10 months after the RGEP did) to restructure organizational arrangement by: (i) forming a joint team of the two PMUs (RGEP and ETEP), headed by one project director; and (ii) assembling personnel from relevant functional departments of the MOET and LTTUs. This strategic rearrangement not only helped to streamline the efficiency of teacher training activities under the two projects; it has also contributed to strengthening the continuity of the curriculum reform and teacher training efforts initiated under the project operations beyond the project’s life by allowing key line staff and experts to retain and integrate their institutional memories and experiences into their regular work. 90 A total of nine modules are planned to be completed by the closing of the ETEP (MOET, 2021). Page 37 of 64 The World Bank Renovation of General Education Project (P150058) 99. To mitigate the risk of the project falling behind schedule, especially in a context wherein the dynamics of politics can strongly affect the outcome of reforms, fortifying the Bank’s role in addressing any emerging conflicting perceptions and interests among key stakeholders, as well as in undertaking early initiation of dialogue with counterparts to pursue possible paths for restructuring, is critical for enabling timely course corrections. In countries such as Vietnam, where the political system ill-equipped for cross-functional coordination and the government’s cumbersome internal decision-making processes could impede timely and collective actions to achieve the expected outcome under the country-led reform, the Bank needs to become even more proactive in initiating dialogue with its counterparts on restructuring early enough to allow timely course corrections within the project’s original timeframe or within a reasonable time period with an extension. It also needs to strengthen its role as a facilitator to foster a shared vision and support collective reform actions. Otherwise, there is a good chance that adjustments through restructuring would no longer be relevant or achievable. 100. The absence of well-defined communication strategies or risk mitigation measures to cope with the political system not adequately equipped for cross-functional collaboration ahead of the curriculum reform process can undermine collective decision-making and coordination among key stakeholders during implementation. The RGEP did not manage to establish communication strategies that clearly defined how to address potential unforeseen policy outcomes or interferences stemming from divergent views, interests, or concerns held by different stakeholders before the curriculum reform process started in full swing. This limitation contributed to the project’s vulnerability to disruptions due to the ramifications of political decisions on the curriculum reform. Considering that reform processes inherently require extensive coordination and engagement of a broad range of stakeholders as well as their collective commitment and decision-making, setting out well-crafted measures and communication strategies to facilitate dialogue and buy-in from relevant stakeholders needs to be prioritized as much as the planning and rollout of the new curriculum itself. 101. To avoid emerging conflicts between the planned project activities and the existing state policies during project implementation, it is essential to embed an institutional mechanism that promotes better coordination among stakeholders in navigating the reform process. Putting such a mechanism in place could help ensure the alignment of project activities with relevant existing policies. It could also provide an opportunity for the project to investigate the underlying factors that contribute to the poor functioning of any of such policies during the project preparation and consider how the project can address the gaps that exist between de jure and de facto practices in the application of those policies on the ground. Under the RGEP, textbook distribution to disadvantaged schools was eventually cancelled due to the overlap of its objective with extant relevant state policies. The implications of the government’s adoption of the multiple-textbook policy in 2015 for the potentially increased roles and responsibilities of commercial publishers, textbook authors, and editors in textbook development and publishing under the new curriculum reform were not fully taken into account at the early stage of the project. This partly contributed to the reassignment of textbook production and appraisal at a later stage. In light of this, setting up a mechanism that allows for collective and coherent decision-making (e.g., by establishing a steering committee comprising representatives from relevant institutions) could help lower the risk of an adverse impact of policy inconsistencies or unexpected policy disruptions on project outcome. 102. To increase the chances of the project’s success in major reforms, such as a curriculum reform implemented under the RGEP, it is crucial to design the project at the right level of ambition, factoring in such constraints as the capacity of the implementing agency and other stakeholders, the political environment, and timeframes. The Page 38 of 64 The World Bank Renovation of General Education Project (P150058) RGEP’s project design was characterized by (i) its highly ambitious objective that was planned to be achieved within just four and a half years after approval and only two years after the implementation of the new curriculum, and (ii) a set of activities requiring intensive and extensive preparation and coordination among a wide group of stakeholders. The onerous political and legislative processes involved in the curriculum reform, coupled with the results framework that includes some indicators needing additional explanation and development of methodologies for data collection and analysis (to first establish baselines and, subsequently, to monitor progress and measure results), imposed additional burdens for the PMU, which already had limited capacities. Thus, to ensure the successful and timely implementation of major reforms in a context wherein rigorous technical and handholding support is necessary, designing projects with an adequate level of complexity and realistic objectives is key. . Page 39 of 64 The World Bank Renovation of General Education Project (P150058) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: To raise student learning outcomes Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion PDO1. Student learning gains Text Partially derived from A gain of at least 0.12 0.00 (Text) 2011 and 2014 Exams standard deviation in and 1st iteration of math and Vietnamese the Project’s NLSA. test scores 30-Jun-2016 31-Dec-2020 31-Dec-2020 Comments (achievements against targets): The planned assessments to measure student learning gains could not be undertaken within the project timeframe as the new curriculum was implemented only in grade 1 at project closing. Objective/Outcome: Revising and implementing the curriculum following a competency-based approach Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion PDO2. Teacher preparedness Percentage 0.00 25.00 1.21 Page 40 of 64 The World Bank Renovation of General Education Project (P150058) for new curriculum 10-Nov-2015 31-Dec-2020 31-Dec-2020 (Percentage) Comments (achievements against targets): 1.21 percent of primary teachers supported by the RGEP were deemed to be qualified as prepared for the new curriculum implementation by completing training modules 1 and 2. An additional 39 percent of teachers also became qualified, with support from the ETEP (3.68 percent) and localities (35.32 percent). Objective/Outcome: Improving the effectiveness of instruction Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion PDO3. Equitable access to Text Ratio of available 1.1 times baseline rate 0.00 learning materials (Text) textbooks in math and reading to enrolled students in primary schools in disadvantaged areas. 10-Nov-2015 31-Dec-2020 31-Dec-2020 Comments (achievements against targets): The indicator could not be measured as the project did not provide textbooks as planned, for the Government Decree 86/2015/ND-CP stipulates that all students should be provided with books, notebooks, and school supplies by their parents, schools, or local authorities. Page 41 of 64 The World Bank Renovation of General Education Project (P150058) A.2 Intermediate Results Indicators Component: Component 1: Curriculum Development Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IR1. Core Technical Text Committee seated 12 years 35.17 years Committee Qualifications 10-Nov-2015 31-Dec-2018 31-Dec-2020 Comments (achievements against targets): The end-target of IRI 1 was originally set at 12 years. However, it was revised to 15 years in December 2017, when the initial group of CTC members came onboard with an average of 15 years of experience. However, the target was not formally revised. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IR5. Curriculum Piloting Text No pilot available until 30 classrooms per Experimental 9,732 the revisions are made grade + 30 classroom class hours / 15 control monitored subjects / 1,165 classrooms / 48 schools of 3 grade levels 15-Dec-2015 31-Dec-2020 31-Dec-2020 Comments (achievements against targets): Page 42 of 64 The World Bank Renovation of General Education Project (P150058) The curriculum was piloted in 2018 for a total of 9,732 experimental class hours for 15 subjects in 1,165 classrooms at 48 schools at three grade levels (grades 3, 8, and 12). However, because of the changes made to the assessment of treatment vis-à-vis control classrooms, there was no monitoring of control classrooms as originally planned at project approval. As a result, while IRI 5 significantly overachieved its original target for pilot/treatment classrooms (30 classrooms per grade), its target for control classrooms (30 classrooms control monitored) was not met. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IR2. Online Training System Text Temporary system by 50 percent of teachers 2.65 percent of Readiness Viettel is being have had at least teachers supported by explored for teacher some training the RGEP were training granted an LMS account to access online modules and completed online module 1 and face-to- face training. 31-Dec-2019 31-Dec-2020 31-Dec-2020 Comments (achievements against targets): An additional 94.78 percent of teachers also met the required online readiness condition, with support from the ETEP (3.98 percent) and localities (90.8 percent). Indicator Name Unit of Measure Baseline Original Target Formally Revised Actual Achieved at Page 43 of 64 The World Bank Renovation of General Education Project (P150058) Target Completion IR3. Curriculum Champions Number 0.00 8000.00 21,320.00 10-Nov-2015 31-Dec-2020 31-Dec-2020 Comments (achievements against targets): A total of 21,320 teachers were trained under the RGEP to serve as Curriculum Champions for grades 1, 5, and 9. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IR9. Number of qualified Number 0.00 150000.00 4,564.00 primary teachers resulting from the project 31-Dec-2019 31-Dec-2020 31-Dec-2020 interventions Number of teachers trained Number 0.00 150000.00 447,189.00 31-Dec-2019 31-Dec-2020 31-Dec-2020 Comments (achievements against targets): A total of 4,564 primary teachers supported by the RGEP were deemed to be qualified for teaching the new curriculum by completing modules 1 and 2 developed by the RGEP. An additional 147,366 primary teachers became qualified, with support from the ETEP (13,922 teachers) and provincial funds (133,444 teachers). "Number of teachers trained" was the only corporate results indicator that had been added to the results framework before project closing. A total of 447,189 teachers received training by project end. Page 44 of 64 The World Bank Renovation of General Education Project (P150058) Component: Component 2: Development and Dissemination of Aligned Textbooks Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IR4. Textbook Production Text No new textbooks 72 = 1 per grade per Not achieved have been approved academic subject 10-Nov-2015 31-Dec-2020 31-Dec-2020 Comments (achievements against targets): 110 textbooks were produced for nine subjects at primary level (for grades 1 and 2) and for 12 subjects at the secondary level (for grade 6). However, these textbooks were not financed by the project. Component: Component 3: Learning Assessment and Analysis for Continuous Improvement of Curriculum and Policy Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IR6. Expanded National Text Current system Exams administered at A total of 14 exams Large-Scale Assessment measures learning of 5th, 9th, and 11th (round 1) were System high school leavers grades for nationally administered to the only representative sample nationally representative sample of grades 5, 9, and 12* Page 45 of 64 The World Bank Renovation of General Education Project (P150058) from June to July 2020. 10-Nov-2015 31-Dec-2020 31-Dec-2020 Comments (achievements against targets): *While the NLSA was initially planned to be administered to grade 11 at the upper secondary level, it was revised to grade 12 in 2018 based on the MOET’s decision to employ the NLSA results to provide insights into education policymaking. In the past, national assessments had never been conducted for grade 12 students to minimize disruptions to those preparing for high-stakes exams. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IR10. System for learning Yes/No No Y Yes assessment at the primary level 31-Dec-2019 31-Dec-2020 31-Dec-2020 Comments (achievements against targets): Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IR11. Utility of the learning Number 1.00 3.00 3.00 assessment system 31-Dec-2019 31-Dec-2020 31-Dec-2020 Comments (achievements against targets): Page 46 of 64 The World Bank Renovation of General Education Project (P150058) Formative, summative, and national (i.e., NLSA) learning assessments were conducted to provide insights into education policymaking. Component: Component 4: Project Management Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion IR7. Direct project Number 0.00 5000000.00 3,953,221.00 beneficiaries 10-Nov-2015 31-Dec-2020 31-Dec-2020 IR8. Female beneficiaries Percentage 0.00 50.00 50.50 Comments (achievements against targets): Page 47 of 64 The World Bank Renovation of General Education Project (P150058) B. KEY OUTPUTS BY COMPONENT Objective/Outcome 1: To raise student learning outcomes PDO Indicator 1: Student learning gains Outcome Indicators Not applicable Intermediate Results Indicators PDO Indicator 1, the only indicator linked to the ultimate objective (“to raise student learning outcomes”), was unmeasurable because of the delay in developing the new competency-based curriculum— which was not completed and adopted by the MOET until December 2018—and the subsequent postponement of administering the NLSA aligned with the new curriculum. The first cohort of grade 3 students whose NLSA results were to be employed for estimating the results of PDO Indicator 1 would not be introduced to the new curriculum until the 2022–2023 school year. Nonetheless, the project succeeded in creating an environment Key Outputs by Component conducive to improving student learning outcomes by: (i) rolling out (linked to the achievement of the Objective/Outcome 1) the new competency-based curriculum for the first cohort of grade 1 students; (ii) administering the first round of the NLSA to 12 nationally representative samples of grades 5, 9, and 12 students on 14 subject exams; (iii) training 21,320 teachers to serve as Curriculum Champions for grades 1, 5, and 9; and (iv) having a total of 4,564 primary teachers who became qualified to teach the new curriculum by completing the first two training modules. In addition, a total of 110 new textbooks were produced by publishing houses for nine subjects at the primary level (for grades 1 and 2) and for 12 subjects at the secondary level (for grade 6); of the 110 textbooks, five sets of new grade 1 textbooks have been appraised and approved by the MOET. Page 48 of 64 The World Bank Renovation of General Education Project (P150058) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Michael F. Crawford Task Team Leader(s) Hoang Xuan Nguyen Procurement Specialist(s) Cung Van Pham Financial Management Specialist Nghi Quy Nguyen Social Specialist Thuy Cam Duong Social Specialist Khang Van Pham Social Specialist Supervision/ICR An Thi My Tran Task Team Leader(s) Hoang Xuan Nguyen Procurement Specialist(s) Cung Van Pham Financial Management Specialist Mary A. Dowling Team Member Dilip Parajuli Team Member Dung Thi Thuy Dao Team Member Viet Quoc Trieu Team Member Nga Thuy Thi Nguyen Procurement Team Son Van Nguyen Environmental Specialist Dung Kieu Vo Team Member Thong Trung Le Social Specialist Huyen Thi Thanh Le Procurement Team B. STAFF TIME AND COST Page 49 of 64 The World Bank Renovation of General Education Project (P150058) Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY14 3.035 9,066.35 FY15 30.294 213,621.09 FY16 47.892 224,807.03 FY17 9.810 50,787.04 FY18 0 0.00 FY19 0 9,076.27 FY20 0 24,353.88 Total 91.03 531,711.66 Supervision/ICR FY16 .900 113,566.23 FY17 6.960 197,970.84 FY18 20.763 180,062.99 FY19 31.798 244,876.01 FY20 26.733 188,550.04 Total 87.15 925,026.11 Page 50 of 64 The World Bank Renovation of General Education Project (P150058) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Curriculum Development 17.50 10.46 59.8% Development and Dissemination of Aligned 21.60 0.48 2.2% Textbooks Learning Assessment and Analysis for Continuous 35.40 5.48 15.5% Improvement of Curriculum and Policy Project Management 2.50 0.12 4.8% Total 77.00 16.54 21.5% Page 51 of 64 The World Bank Renovation of General Education Project (P150058) ANNEX 4. EFFICIENCY ANALYSIS Economic Benefits By the project closure in 2020, new CBE curricula have been fully developed for grade 1 to grade 12 and rolling out has started at grade 1 in 2020. During the project life cycle, only one cohort entering grade 1 in 2020 was taught in the RGEP curriculum, a number of cohorts of students would benefit from this curriculum after the project closure. Rolling out of the curriculum to grade 2 to grade 12 is scheduled from 2021 to 2024 with full rolling out at all grades to be achieved in 2024. Details of this schedule is provided in Panel A, Table 1. Within the education sector, adoption of this new curriculum is expected to raise learning outcomes, cognitive skills, including core competencies such as problem solving, communications, critical thinking, self-management, and self-learning, and socio-emotional/behavioral skills. Principal outcomes expected from the RGEP interventions also include higher completion rates and higher transition rates across grades and levels of education. According to the results chain, these rates respond to (i) development and rolling out of new sets of textbooks and assessment approaches which are CBE and better support students’ skill development; (ii) improvements in enhanced teacher quality due to pedagogical training; (iii) better school management due to CPD training provided to school managers. On the labor market, it is expected that students benefitted from CBE curriculum as the RGEP-supported curriculum would be better equipped with relevant competencies and skills and compensated with higher wages. As indicated by various Vietnamese employer surveys and the recent Higher Education report, despite Vietnamese workers’ relatively high levels of education attainment, a significant proportion of firms in Vietnam report difficulties in recruiting employees with advanced cognitive skills, socio-emotional skills, and job-specific technical skills, indicating large skills gaps. For an accurate benefit stream, the economic analysis would look at realized data on benefits for different interventions. However, such data is not available systematically particularly since there is a lag of several years after the project closure to have the curriculum fully rolled out at all grades and to have the data on realized benefits available. Economic benefits of RGEP is estimated for all the cohorts that would benefit from the RGEP curriculum during the project lifecycle from 2015-2020 and after the project closure until the next comprehensive curriculum reform. Core modeling assumptions of the analysis are made about how benefits would be accrued and were based on rolling out schedule of the RGEP curriculum, enrollment, drop-out rates, wages, and schedule of future comprehensive reforms and summarized below. 1. Wage premia resulting from improved cognitive and social skills of students are about 2.5 percent for primary school, 1.5 percent for secondary school, 1.0 percent for upper secondary school, and 1.0 percent for tertiary education. These rates are adjusted to be only 50 percent of the rates originally expected from RGEP interventions (RGEP PAD) as to reflect the delay in rolling out of the RGEP curriculum, changed/cancelled project activities, non-verification of impact of the curriculum on students’ competency, and the changing conditions of the labor market. It is further assumed that, (i) only graduates experiencing the curriculum at all grades of that level would benefit from it and dropouts before completion of all grades at that level would not benefit from the RGEP curriculum, (ii) only graduates experiencing the curriculum at all levels of education – Page 52 of 64 The World Bank Renovation of General Education Project (P150058) primary to upper secondary school – would fully benefit at the specified rates of returns of that level, otherwise the rates would be reduced by 50 percent. 2. It is assumed that, from 2029, a new comprehensive curriculum reform replacing the RGEP- supported curriculum will be rolled out with a roll-out schedule similar to RGEP curriculum. Alongside of the implementation of the new curriculum, benefits of the RGEP curriculum will phase out gradually and be expected to be insignificant as the new one fully rolled out to primary and secondary school. The rolling out schedule of the new curriculum is summarized in Panel B, Table 1. 3. The dropout rates at each grade and across levels of education are expected to reduce by 1 percent per year purely as results of RGEP curriculum-induced improvement in education quality and relevance. The positive impact of the RGEP curriculum on dropout rates ceases to null when the new curriculum replacing the RGEP will be rolled out. 4. Projection of average wages, growth rates of real wage, employment rates for graduates at different levels of education are based on income targets from the Vietnam Socio-economic Development Strategy and historical data calculated using the Vietnam Household Living Survey (VHLSS) and forecast of the IMF and WB. For primary and lower secondary graduates, economic benefits are calculated after students enter the labor market at the official age of 15. Given the assumed rolling out schedules, in the base case, we consider: (i) 11 cohorts who entered grade 1 during 2013-2024 will benefit from the new curriculum supported by RGEP at different points of their education, (ii) RGEP-supported curriculum would be fully rolled out at all grades and levels during 2024- 2028, (iii) two cohorts who started grade 1 in 2020 and 2021 would be fully taught with the RGEP curriculum; (iv) economic benefits of the RGEP curriculum become insignificant starting from 2033, i.e., for the cohort that enters Grade 1 in 2023. Note that the assumptions listed are highly conservative and would likely result in lower bounded estimation of the RGEP economic benefits. The accrued economic benefits for each beneficiary cohort and graduates across levels of education are presented in Table 2. The present value of gross benefits, given the key assumptions described, is estimated at USD $731.098 million. Tertiary graduates, who benefit from the RGEP curriculum, have the highest economic benefits despite of the fact that they come from only 5 cohorts, entering grade 1 from 2013 to 2017. This is mainly driven by the high employment rates, high wage premium to tertiary education, and improvement in quality of pipeline students from high schools presumably induced by the RGEP curriculum. Cohorts which enter grade 1 from 2017 onwards would mostly go to the labor market after 2033, when the new curriculum will have been rolled out and wage premia of the RGEP curriculum will be gradually diminished, and therefore, accrued benefits would decrease over time. The cohort that enters grade 1 in 2023 will no longer benefit from the RGEP regardless of their education attainment. Table 1 Rolling-out schedule Rolling-out schedule of the of the RGEP- Lower secondary Upper secondary new curriculum replacing Primary graduates supported graduates graduates the RGEP curriculum curriculum 2020 G1 2029 2021 G1, G2 G6 2030 2022 G1, G2, G3 G6, G7 G10 2031 Page 53 of 64 The World Bank Renovation of General Education Project (P150058) 2023 G1, G2, G3, G4 G6, G7, G8 G10, G11 2032 2024 G1, G2, G3, G4, G5 G6, G7, G8, G9 G10, G11, G12 2033 Table 2: RGEP economic benefits (present value, 2021) Lower Upper Cohorts Primary Tertiary Benefits secondary secondary (enter Grade 1 in year) graduates graduates by cohort graduates graduates 2013 26,062,665 88,969,659 115,032,324 2014 22,789,095 75,204,629 97,993,724 2015 20,868,415 58,794,936 79,663,351 2016 25,320,938 45,194,917 71,288,583 141,804,438 2017 22,288,585 36,810,705 37,409,148 96,508,438 2018 18,555,918 29,044,714 - 47,600,632 2019 15,529,310 20,342,330 - 35,871,640 2020 15,242,888 32,591,948 18,442,780 - 66,277,616 2021 10,414,131 22,726,733 - - 33,140,864 2022 5,354,596 11,851,337 - 17,205,933 2023 - - - - Benefits by 31,011,615 148,864,769 149,835,446 331,666,955 731,098,960 education attainment Economic Costs There are three categories of direct costs during the project life cycle. The first category includes financial costs of the RGEP which includes project management spending and recurrent expenditures (net of the human capital investment amount) funded by the Bank. The second cost is associated with investment in human capital through capacity building and technical assistance. It should be noted that funding for textbook and NLSA development and teacher/school management/MOET staff training is considered investment in human capital instead of recurrent expenditure as per Vietnam regulations and subject to investment depreciation. The analysis uses five percent of annual depreciation for human capital investment. The economic cost of the human capital investment would then equal to its depreciated value during the medium lag of five years before economic benefits realized. The analysis also considers opportunity costs associated with the undisbursed/cancelled fund given a significant proportion of the originally committed funds were cancelled at the request of the Government. Specifically, at the end of project closure in 2020, a total of US$60.46 million IDA Credit, 74.93 percent of the original credit, was cancelled due to delayed disbursement and shifting project demand. The analysis views there is an opportunity cost associated with the cancellation given that the funding might have been invested in 2015 and generated positive social benefits elsewhere. For a highly conservative estimation of opportunity costs, the analysis uses average social benefits generated through hypothetical alternative investments in infrastructure in the East Asia Pacific region (EAP). Research literature finds that estimated rates of return to investments in infrastructure in EAP ranging between 31 to 106 percent 91. The base- case scenario takes the mid-range value of 68.5 percent, a lag of one before returns are realized 92, and a 91 Canning, David, and Esra Bennathan. "The social rate of return on infrastructure investments." World Bank Policy Research Working Paper 2390 (2000). 92 The discounting of future flows means that returns in the early years would dominate the calculation of rate of return. Accordingly, if it takes several years instead of one year for the alternative investment to reach its full potential, the rate of return would be overestimated. However, a similar consideration applies to the estimates of the rate of return to RGEP investment, so that when we compare alternative rates of return to those found on RGEP investment, it is expected to be overestimated in Page 54 of 64 The World Bank Renovation of General Education Project (P150058) deprecation rate of 5 percent to calculate the opportunity costs. Details on the project costs are presented in Table 2. Total economic costs of RGEP in the base-case scenario (current value in 2021) equal to about USD $649.59 million, consisting of direct financial costs of USD $18.13 million, depreciation costs of human capital investments of USD $0.70 million, and opportunity costs of RGEP cancelled funds of about USD $630.76 million 93. Table 3: RGEP economic costs (current value, 2021) 2015 2016 2017 2018 2019 2020 Costs by category & total Panel A. RGEP financial costs 18,133,351 (i) Curriculum - - 2,593,286 2,081,238 2,457,791 4,517,406 11,649,721 development (ii) Textbook - - - - 469,658 57,745 527,403 development (iii) Supports for - - 56,025 119,408 1,032,474 4,615,389 5,823,296 sustainable development of general education and learning assessment (iv) Project - - 24,215 30,032 42,749 35,933 132,930 management Panel B. Economic costs due to depreciation of investment in human capital 696,419 (i) Curriculum - - 100,332 80,521 95,090 174,774 450,717 development (ii) Textbook - - - - 18,171 2,234 20,405 development (iii) Supports for - - 2,168 4,620 39,945 178,565 225,298 sustainable development of general education and learning assessment Panel C. Opportunity costs of the project cancelled fund 630,763,866 Economic benefits of 630,763,866 630,763,866 alternative investment Costs by year & total - - 2,776,026 2,315,819 4,155,878 640,345,912 649,593,635 Cost-Benefits Analysis Table 4 provides results of the cost-benefit analysis. Overall, despite of significant delay and low disbursement by project closure, RGEP investments generate positive NPV benefits of about US$81.50 million, which equal to about 15.7 percent of the NPV benefits of USD $519 million expected ex ante, and IRR of 1.24 percent compared with the IRR of 27 percent predicted ex ante. Sensitivity Analysis The analysis looks at two additional scenarios. To assess risk and the expected NPV of the project, it is similar proportions. Although both are probably overestimated, the issue may be worse for infrastructure investment where there is considerable evidence that construction may lead demand for infrastructure services (World Bank (1994), Canning and Esra, 2000), and thus, making the estimated opportunity costs highly conservative. 93 It should be noted that RGEP opportunity cost is calculated until full depreciation of the alternative investment and its total value over time is discounted back to 2020 when the fund is cancelled. Page 55 of 64 The World Bank Renovation of General Education Project (P150058) assumed that (i) the social rates of returns of restructured funds in alternative projects and (ii) wage premium of the RGEP curriculum are subject to change. Case-1 scenario uses the base-case assumptions exception that the depreciation rate of alternative investments is assumed to increase to 6 percent, that is the alternative investments would depreciate at a higher rate than the RGEP investments. In this scenario, NPV would increase to US$128.96 million and IRR to 2.05 percent. Case-2 scenario uses Case-1 assumptions and, in addition, accounts for the impact of Covid-19 on the labor market and economic growth. It is assumed that the impact is mainly reflected in slower real wage growth rates during 2021-2024, being about only 60-80 percent of the pre-Covid-19 growth rates. In Case- 2, RGEP investments result in NPV of US$85.84 million and IRR of 1.403 percent. Table 4: Net benefits and internal rate of returns Costs and Benefits Base-case scenario Case 1 Case 2 (PV, US$ million) Gross economic benefits of RGEP HC investments 731,098,960 731,098,960 687,989,944 Economic costs 649,593,635 602,143,147 602,143,147 - of HC investments 696,419 696,419 696,419 - of project financial costs 18,133,351 18,133,351 18,133,351 -of cancelled funds 630,763,866 583,313,377 583,313,377 Net Benefits 81,505,325 128,955,813 85,846,798 IRR 1.243% 2.051% 1.403% Page 56 of 64 The World Bank Renovation of General Education Project (P150058) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS Comments on the Bank’s ICR from MOET MINISTRY OF EDUCATION AND TRAINING RGEP PROJECT MANAGEMENT UNIT SOCIALIST REPUBLIC OF VIETNAM Independence – Freedom – Happiness Ref: 56/CV-RGEP Re: RGEP PMU comments on “RGEP Implementation Completion and Results Report” Hanoi 20 August 2021 To: The World Bank in Vietnam On 12 August /2021, RGEP Project Management Unit (PMU) received the letter from the World Bank (WB) regarding “Implementation Completion and Results Report” (for the Renovation of General Education Project / RGEP). After reviewing the report, RGEP PMU would like to have some comments on the “Implementation Completion and Results Report” as follows: 1. In principle, RGEP PMU agree with the judgments/assessments about the success as well as constraints in the design and implementation of RGEP stated in the “Implementation Completion and Results Report” of the World Bank (WB). Although the expected outcomes are not achieved within the project implementation period, the project achievements related to: i) development and implementation of competency and quality based General Education Curriculum (GEC) 2018; ii) national large scale assessment of the competency-based students’ learning outcomes; iii) socialisation of the compilation and appraisal of textbooks are considered as “the root” for fundamental renovation of general education and resolutions for the implementation of the GEC 2018. 2. More specifically, we would like the authors of the report to review and revise the following points: - Correct the number of textbooks in paragraph 48, specifically: The number of textbooks for grades 1, 2 and 6 compiled, appraised and approved is 118 textbooks (the number was already reported to the WB on 12 May 2021, whereas it is stated as 110 textbooks in the report). At the end of the project, only 46 textbooks for grade 1 were approved; whereas the textbooks for grade 2 (32 textbooks) and grade 6 (40 textbooks) were approved and made available in February 2021 (it is currently stated in the report that: grade 2 and 6 textbooks “would not become available until September 2021 after the MOET’s approval”). - Correct one sentence in paragraph 55: “Following the cancellation of US$60.46 million from the original committed credit amount of US$77 million, the project’s disbursement rate remained at about 21.5 percent (US$16.54 million) at closing” should be corrected to: The project can disburse about 21.5% (equivalent to USD 16.54 million) only, leading to the cancellation of USD 60.46 million from the original committed credit amount of USD 77 million. - Revise some judgements about the political system – one of the factors affecting the project implementation and outcomes; avoid using serious generalised terms for the political system, which are inappropriate for this report. The recommended revisions are presented in details in the attached Annex. - Please reconsider the following statement in paragraph 57: “…. the project’s achievements were low Page 57 of 64 The World Bank Renovation of General Education Project (P150058) and came at a higher than expected cost.” 3. Correct the names of units/sections in the Vietnamese version: - Paragraph 8: “Ủy ban Kỹ thuật Chủ chốt (CTC) và các Ủy ban Kỹ thuật theo Môn học cụ thể (SSTC)” should be corrected to: Ban phát triển Chương trình tổng thể (CTC) và Ban phát triển Chương trình môn học (SSTC). - Paragraph 71: “Vụ Truyền thông của Bộ GD&ĐT” should be corrected to: Trung tâm Truyền thông của Bộ GD&ĐT. RGEP PMU would like to report to the World Bank the above comments. Summary of Borrower’s ICR Objective and components: The RGEP sought to support the renovation of general education and the quality assessment of general education, contributing to improving the quality of general education and creating high- quality human resources to meet the requirements of industrialization and modernization of the country. More specifically, the project supported the following: (i) developing new competency-based general education curriculum and disseminating new textbooks aligned with the curriculum; (ii) renovating exams/tests, and evaluations of the quality of competency-based general education; and (iii) strengthening (a) the research capacity in administering exams/tests to assess the quality of general education and (b) the foreign language testing capacity. The project consisted of four components: (i) Support in development of new general education program; (ii) Support in development of textbooks in accordance with the new general education program; (iii) Assist in evaluating and analyzing learning results to continuously improve curriculum and policy of general education; and (iv) Project management. All four components were designed to contribute to changing the levels to the overall objectives of the project to create a modified curriculum, textbooks, and specific assessment tools aligned with the learning objective for the school system in Vietnam. Project Implementation Results: The RGEP had three Project Development Objective (PDO) Indicators and 11 Intermediate Results (IR) Indicators. While these indicators remained unchanged throughout the project implementation, amendments were made to the descriptions of the indicators to help stakeholders agree on the correct understanding and collection of measurement information. By the end of the project, out of the three PDO indicators, only one indicator (PDO2) met the target; the two remaining indicators (PDO1 and PDO3) were not measurable and unachieved. Out of 11 IR indicators, only seven indicators met and exceeded their targets, and the four remaining indicators failed (IR4, IR6, IR7 and IR9). Financial Performance: As of December 31, 2020, the project disbursed USD 18,442 million (23%), of which loan Page 58 of 64 The World Bank Renovation of General Education Project (P150058) capital was disbursed USD 16,544 million (22%) and counterpart disbursement was USD 1,898 million (63%). The disbursement rate compared to the design was low due to the slow implementation of the project. Recommendations: Based on the implementation results, the following recommendations were provided to sustain the activities implemented under Components 1, 2, and 3. Component 1 • Continue to follow and assess the 2018 general education curriculum; improve and update the curriculum; prepare detailed output standards; and develop and issue curriculum assessment standards and more detailed instructions on curriculum implementation • Continue to foster qualified teachers to help them implement the 2018 general education curriculum; transfer modules 3 and 4 to the ETEP to deploy training for professional leaders; supplement and edit the content of modules 2, 3, and 4 after receiving feedback from lecturers and professional leaders and invest adequately in organizing teacher training from all resources; and assist schools in building school programs Component 2 • Complete the process of compiling textbooks, strengthen the capacity and specialization of textbook authors and appraisal team • Provide resources to prepare textbooks for students with special needs, including those with disabilities and ethnic minority students Component 3 • Professionalize the curriculum development team and carry out educational assessments to continue curriculum innovations; strengthen the capacity of the NCSDGEQ, the National Testing Center, and Satellite Centers to be able to conduct research, deploy and advise the MOET on curriculum development and evaluation • Develop indicators for evaluating core competencies: the capacity scale of the 2018 general education curriculum • Build an item bank for wide-ranging assessments and the high school final exam Page 59 of 64 The World Bank Renovation of General Education Project (P150058) ANNEX 6. SUPPORTING DOCUMENTS References: 1. Independent Evaluation Group, The World Bank. 2019. Project Performance Assessment Report, Vietnam: Education Projects—School Readiness and Escuela Nueva (Report No. 135418). Washington, DC: World Bank. 2. London, Jonathan D. 2021. Outlier Vietnam and the Problem of Embeddedness: Contributions to the Political Economy of Learning. RISE Working Paper Series. 21/062. https://doi.org/10.35489/BSG-RISEWP_2021/062. 3. OECD. 2011. Education at a Glance 2011: OECD Indicators, OECD Publishing. http://dx.doi.org/10.1787/eag-2011-en. 4. Smart, Andy and Shanti Jagannathan. 2018. Textbook Policies in Asia: Development, Publishing, Printing, Distribution, and Future Implications. Manila: Asian Development Bank. http://dx.doi.org/10.22617/TCS189651-2. 5. The Ministry of Education and Training of Vietnam. 2021. RGEP Project Completion Report. 6. The Ministry of Education and Training of Vietnam, RGEP Project Management Unit. 2015–2020. RGEP Project Implementation Progress Reports. 7. The World Bank. Project files for the Renovation of General Education Project: Aide Memoires from Bank Preparation and Supervision Missions (2014–2020) Environmental and Social Management Framework (2015) Project Appraisal Document (June 2015) Project Financing Agreement (Aril 2016) Project Operations Manual (July 2016) Project Restructuring Paper (December 2020) 8. ———. Project Implementation Status and Results Reports (Sequence 1–9, 2015–2020) 9. ———. 2011. Country Partnership Strategy for The Socialist Republic of Vietnam for the Period FY12–FY16. 10. ———. 2013. Skilling Up Vietnam: Preparing the Workforce for the Modern Market Economy. 11. ———. 2017. Country Partnership Framework for The Socialist Republic of Vietnam for the Period FY18–FY22. 12. The World Bank and Ministry of Planning and Investment of Vietnam. 2016. Vietnam 2035: Toward Prosperity, Creativity, Equity, and Democracy. Washington, DC: World Bank. http://hdl.handle.net/10986/23724. Page 60 of 64 The World Bank Renovation of General Education Project (P150058) Table 6.1 New Curriculum Rollout Plans Grade Timeline Grade 1 2020–2021 school year Grades 2 and 6 2021–2022 school year Grades 3, 7, and 10 2022–2023 school year Grades 4, 8, and 11 2023–2024 school year Grades 5, 9, and 12 2024–2025 school year Table 6.2 Achievement of PDO-level and Intermediate Results Indicators PDO-level and Intermediate Unit of End Target End Result Results Indicators Measure PDO 1: Student learning gains Text A gain of at least 0.12 standard The planned assessments to measure student deviation in math and Vietnamese test learning gains could not be undertaken within the scores. project time frame as the new curriculum was only applied to grade 1 at the time of the project’s closure. PDO 2: Teacher preparedness for % 25 percent RGEP supported 1.21 percent of primary teachers new curriculum to be qualified as prepared for the new curriculum implementation by completing training modules 1 and 2. An additional 39 percent of teachers also became qualified, with support from the ETEP (3.68 percent) and localities (35.32 percent). PDO 3: Equitable access to Text 1.1 times baseline rate The indicator could not be measured as the project learning materials did not provide textbooks as planned, for the Government Decree 86/2015/ND-CP stipulates that all students should be provided with books, notebooks, and school supplies by their parents, schools, or local authorities. Component 1: Curriculum Development IRI1: Core Technical Committee Text 1215 years* 35.17 years Qualifications (Component 1.1) *The end-target of IRI 1 was originally set at 12 years. However, it was revised to 15 years in December 2017, when the initial group of CTC members came onboard with an average of 15 years of experience. However, the target was not formally revised. IRI 2: Online Training System Text 50 percent of teachers have had at RGEP support 2.65 percent of teachers to be Readiness (Component 1.2) least some training granted an LMS account to access online modules Page 61 of 64 The World Bank Renovation of General Education Project (P150058) PDO-level and Intermediate Unit of End Target End Result Results Indicators Measure and completed online module 1 and face-to-face training. An additional 94.78 percent of teachers also met the required online readiness condition, with support from the ETEP (3.98 percent) and localities (90.8 percent). IRI 3: Curriculum Champions Number 8,000 A total of 21,320 teachers were trained under the (Component 1.2) RGEP to serve as Curriculum Champions for grades 1, 5, and 9. IRI 5: Curriculum Piloting Text 30 classrooms per grade + 30 classroom The curriculum was piloted in 2018 for a total of (Component 1.2) control monitored 9,732 experimental class hours for 15 subjects in 1,165 classrooms at 48 schools at three grade levels (grades 3, 8, and 12). However, because of the changes made to the assessment of treatment vis-à-vis control classrooms, there was no monitoring of control classrooms as originally planned at project approval. As a result, while IRI 5 significantly overachieved its original target for pilot/treatment classrooms (30 classrooms per grade), its target for control classrooms (30 classrooms control monitored) was not met. IRI 9: Number of additional Number 150,000 A total of 4,564 primary teachers supported by the qualified primary teachers RGEP were deemed to be qualified for teaching the resulting from project new curriculum by completing modules 1 and 2 interventions developed by the RGEP. (Component 1.2) An additional 147,366 primary teachers became qualified, with support from the ETEP (13,922 teachers) and provincial funds (133,444 teachers). Component 2: Development and Dissemination of Aligned Textbooks IRI 4: Textbook production Text 72 = 1 per grade per academic subject 110 textbooks were produced for nine subjects at (Component 2.1) the primary level (for grades 1 and 2) and for 12 subjects at the secondary level (for grade 6). However, these textbooks were not financed by the project. Component 3: Learning Assessment and Analysis for Continuous Improvement of Curriculum and Policy IRI 6: Expanded National Large- Text Exams administered at 5th, 9th, and A total of 14 exams (round 1) were administered to Scale Assessment System 11th grades for nationally the nationally representative sample of grades 5, 9, (Component 3.3) representative sample and 12* from June to July 2020. *While the NLSA was initially planned to be administered to grade 11 at the upper secondary level, it was revised to grade 12 in 2018 based on Page 62 of 64 The World Bank Renovation of General Education Project (P150058) PDO-level and Intermediate Unit of End Target End Result Results Indicators Measure the MOET’s decision to employ the NLSA results to provide insights into education policymaking. In the past, national assessments had never been conducted for grade 12 students to minimize disruptions to those preparing for high-stakes exams. IRI 10: System for learning Yes/No Yes Yes assessment at the primary level (Component 3.3) IRI 11: Utility of the learning Number 3 3 assessment system (Component 3.3) Component 4: Project Management IRI 7: Direct project beneficiaries Number 5,000,000 3,953,221 (Component 4.2) IRI 8: Female beneficiaries % 50 50.5 percent (Component 4.2) Table 6.3 Descriptions of PDO-level and intermediate results indicators Indicator End Target Original Description 94 Revised Description (where applicable) PDO Indicator 1: Student A gain of at Students taught under the new curriculum The treatment groups of grade 3 and 8 learning gains least 0.12 will show an increase of at least 0.12 students taught under the new curriculum standard standard deviation in math and will show an increase of at least 0.12 deviation in Vietnamese test scores compared to the standard deviation in math and Vietnamese Math and baseline cohort. test scores compared to the control groups. Vietnamese test scores PDO Indicator 2: Teacher 25 percent 25 percent of teachers implementing the 25 percent of primary teachers preparedness for new new curriculum will be certified as prepared implementing the new curriculum will be curriculum for curriculum implementation. qualified as prepared for curriculum implementation by completing training modules 1 and 2.* *While teachers who completed online training modules 1, 2, and 3 were technically certified to teach the new curriculum, those who completed modules 1 and 2 were still deemed to be qualified by the MOET because module 3 was not completed until after the new curriculum rolled out in grade 1. PDO Indicator 3: Equitable 1.1 times In the bottom 10 percent of schools (based access to learning materials baseline rate on income distribution), textbooks will be 94 World Bank. 2015. PAD for the RGEP. Page 63 of 64 The World Bank Renovation of General Education Project (P150058) Indicator End Target Original Description 94 Revised Description (where applicable) distributed to raise the ratio of student to textbook ratio to 1.1 times its baseline rate. IRI 1: Core Technical Average years of professional experience per Committee Qualifications member (Component 1.1) IRI 2: Online Training System Milestones for systems development and The percentage of teachers who were Readiness (Component 1.2) implementation as per the Results Indicators granted an LMS account to access online table modules and who completed online module 1 and face-to-face training IRI 3: Curriculum Champions Reported Annual Average Person-hours of The number of general teachers trained (Component 1.2) training provided under the RGEP to serve as Curriculum Champions IRI 4: Textbook production Approved textbooks (Component 2.1) IRI 5: Curriculum Piloting Schools/Classes (in grade 3) Testing the New Schools/Classes that took part in the piloting (Component 1.2) Curriculum of the new curriculum IRI 6: Expanded National Large scale exams successfully implemented Large-Scale Assessment System (Component 3.3) IRI 7: Direct project Students, teachers, and education official The number of people or groups (including beneficiaries who master the principles of the revision students, teachers, school managers, and (Component 4.2) and/or initiate implementation of the new lecturers at teacher training universities) curriculum who directly benefited from interventions under the RGEP, as well as those who participated in training organized jointly by the RGEP and the ETEP IRI 8: Female beneficiaries 50 percent of overall beneficiaries (Component 4.2) IRI 9: Number of additional Primary education teachers comprise The number of primary teachers qualified to qualified primary teachers roughly 40 percent of all teachers. This teach the new curriculum under the RGEP resulting from project indicator will be expected to be 0.4 times interventions the overall number of teachers certified as (Component 1.2) trained in the new curriculum. IRI 10: System for learning This is a binary (“Yes” or “no”) measurement assessment at the primary indicating the existence of the system. level (Component 3.3) IRI 11: Utility of the learning Tiered system scale as per World Bank Core assessment system Indicator guidance (Component 3.3) Page 64 of 64