Document of The World Bank FOR OFFICIAL USE ONLY Report No: PAD1081 INTERNATIONAL DEVELOPMENT ASSOCIATION PROJECT APPRAISAL DOCUMENT ON A PROPOSED CREDIT IN THE AMOUNT OF SDR 55.40 MILLION (US$77.00 MILLION EQUIVALENT) TO THE SOCIALIST REPUBLIC OF VIETNAM FOR A RENOVATION OF GENERAL EDUCATION PROJECT JUNE 9, 2015 Education Global Practice East Asia and Pacific This document has a restricted distribution and may be used by recipients only in the performance of their official duties. Its contents may not otherwise be disclosed without World Bank authorization. CURRENCY EQUIVALENTS (Exchange Rate Effective May 31, 2015) Currency Unit = Vietnamese Dong 21,365 Dong = US$1 US$0.47 = 10,000 Dong US$1 = SDR 0.719 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS AWP Annual Work Plan OECD Organization for Economic Co-operation and Development BOET Bureaus of Education and PCRA Procurement Capacity and Training Risk Assessment CTC Core Technical Committee PAD Project Appraisal Document CPS Country Partnership PDO Project Development Strategy Objective DOET Departments of Education PISA Program for International and Training Student Assessment ECE Early Childhood Education POM Project Operation Manual ECOP Environment Codes of READ Russia Education Aid for Practice Development EMP Environmental SA Social Assessment Management Plan ESMF Environmental and Social SABER System Assessment and Framework Benchmarking for Education Results FM Financial Management SEDP Socio-Economic Development Plan GC General Curriculum SEQAP School Education Quality Assurance Program GRS Grievance Redress Service SRPP School Readiness Promotion Project GOV Government of Vietnam SSC Subject-Specific Curricula IFRs Interim Financial Reports SSTC Subject Specific Technical Committees IRR Internal Rate of Return STEP Strategic Teaching & Evaluation of Progress M&E Monitoring and Evaluation TEC Training, Events, and Conferences MOET Ministry of Education and TIMSS Trends in International Training Mathematics and Science Study NCFLT National Center for Foreign TT Task Team Language Testing NCSDGEQ National Centre for TOR Terms of Reference Sustainable Development of General Education Quality NLSA National Large-Scale VNIES Vietnam Institute of Assessment Educational Science NLSAS National Large-Scale WB World Bank Assessment System Regional Vice President: Axel van Trotsenburg Country Director: Victoria Kwakwa Senior Global Practice Director: Claudia Maria Costin Practice Manager / Manager: Harry Patrinos Task Team Leader: Michael F. Crawford VIETNAM Renovation of General Education Project (P150058) TABLE OF CONTENTS Page I.  STRATEGIC CONTEXT .................................................................................................1  A.  Country Context ............................................................................................................ 1  B.  Sectoral and Institutional Context................................................................................. 2  C.  Higher Level Objectives to which the Project Contributes .......................................... 6  II.  PROJECT DEVELOPMENT OBJECTIVES ................................................................7  A.  PDO............................................................................................................................... 7  B.  Project Beneficiaries ..................................................................................................... 7  C.  PDO Level Results Indicators ....................................................................................... 8  III.  PROJECT DESCRIPTION ..............................................................................................9  A.  Project Components ...................................................................................................... 9  B.  Project Financing ........................................................................................................ 12  C.  Lessons Learned and Reflected in the Project Design ................................................ 13  IV.  IMPLEMENTATION .....................................................................................................14  A.  Institutional and Implementation Arrangements ........................................................ 14  B.  Results Monitoring and Evaluation ............................................................................ 15  C.  Sustainability............................................................................................................... 16  V.  KEY RISKS AND MITIGATION MEASURES ..........................................................16  A.  Risk Ratings Summary Table ..................................................................................... 16  B.  Overall Risk Rating Explanation ................................................................................ 16  VI.  APPRAISAL SUMMARY ..............................................................................................17  A.  Economic and Financial (if applicable) Analysis ....................................................... 17  B.  Technical ..................................................................................................................... 20  C.  Financial Management ................................................................................................ 21  D.  Procurement ................................................................................................................ 22  E.  Social (including Safeguards) ..................................................................................... 22  F.  Environment (including Safeguards) .......................................................................... 23  G.  World Bank Grievance Redress .................................................................................. 24  Annex 1: Results Framework and Monitoring .........................................................................25  Annex 2: Detailed Project Description .......................................................................................32  Annex 3: Implementation Arrangements ..................................................................................44  Annex 4: Implementation Support Plan ....................................................................................57  Annex 5: Risk Diagnosis ..............................................................................................................59  . PAD DATA SHEET Vietnam Renovation of General Education Project (P150058) PROJECT APPRAISAL DOCUMENT . EAST ASIA AND PACIFIC GEDDR Report No.: PAD1081 . Basic Information Project ID EA Category Team Leader(s) P150058 B - Partial Assessment Michael F. Crawford Lending Instrument Fragile and/or Capacity Constraints [ ] Investment Project Financing Financial Intermediaries [ ] Series of Projects [ ] Project Implementation Start Date Project Implementation End Date 01-Jul-2015 31-Dec-2020 Expected Effectiveness Date Expected Closing Date 01-Oct-2015 31-Dec-2020 Joint IFC No Practice Senior Global Practice Country Director Regional Vice President Manager/Manager Director Harry Anthony Patrinos Claudia Maria Costin Victoria Kwakwa Axel van Trotsenburg . Borrower: SOCIALIST REPUBLIC OF VIETNAM Responsible Agency: Ministry of Education and Training Contact: Doan Van Ninh Title: Chair of Standing Unit Telephone No.: 01184985853386 Email: dvninh@moet.edu.vn . Project Financing Data(in USD Million) [ ] Loan [ ] IDA Grant [ ] Guarantee [X] Credit [ ] Grant [ ] Other Total Project Cost: 77.00 Total Bank Financing: 77.00 Financing Gap: 0.00 . i Financing Source Amount BORROWER/RECIPIENT 0.00 International Development Association (IDA) 77.00 Total 77.00 . Expected Disbursements (in USD Million) Fiscal 2016 2017 2018 2019 2020 0000 0000 0000 0000 0000 Year Annual 7.00 10.00 20.00 25.00 15.00 0.00 0.00 0.00 0.00 0.00 Cumulati 7.00 17.00 37.00 62.00 77.00 0.00 0.00 0.00 0.00 0.00 ve . Institutional Data Practice Area (Lead) Education Contributing Practice Areas Cross Cutting Topics [ ] Climate Change [ ] Fragile, Conflict & Violence [X] Gender [ ] Jobs [ ] Public Private Partnership Sectors / Climate Change Sector (Maximum 5 and total % must equal 100) Major Sector Sector % Adaptation Mitigation Co-benefits % Co-benefits % Education General education sector 100 Total 100 I certify that there is no Adaptation and Mitigation Climate Change Co-benefits information applicable to this project. . Themes Theme (Maximum 5 and total % must equal 100) Major theme Theme % Human development Education for the knowledge economy 50 Human development Education for all 50 Total 100 . ii Proposed Development Objective(s) The Project Development Objective is to raise student learning outcomes by: (i) revising and implementing the curriculum following a competency-based approach; and (ii) improving the effectiveness of instruction by creating and disseminating textbooks aligned with the revised curriculum. . Components Component Name Cost (USD Millions) Curriculum Development 17.50 Development and Dissemination of Aligned Textbooks 21.60 Learning Assessment and Analysis for Continuous 35.40 Improvement of Curriculum and Policy Project Management 2.50 . Systematic Operations Risk- Rating Tool (SORT) Risk Category Rating 1. Political and Governance Substantial 2. Macroeconomic Moderate 3. Sector Strategies and Policies Low 4. Technical Design of Project or Program High 5. Institutional Capacity for Implementation and Sustainability Substantial 6. Fiduciary Substantial 7. Environment and Social Low 8. Stakeholders Substantial 9. Other Low OVERALL Substantial . Compliance Policy Does the project depart from the CAS in content or in other significant Yes [ ] No [ X ] respects? . Does the project require any waivers of Bank policies? Yes [ ] No [ X ] Have these been approved by Bank management? Yes [ ] No [ X ] Is approval for any policy waiver sought from the Board? Yes [ ] No [ X ] Does the project meet the Regional criteria for readiness for implementation? Yes [ X ] No [ ] . Safeguard Policies Triggered by the Project Yes No Environmental Assessment OP/BP 4.01 X Natural Habitats OP/BP 4.04 X iii Forests OP/BP 4.36 X Pest Management OP 4.09 X Physical Cultural Resources OP/BP 4.11 X Indigenous Peoples OP/BP 4.10 X Involuntary Resettlement OP/BP 4.12 X Safety of Dams OP/BP 4.37 X Projects on International Waterways OP/BP 7.50 X Projects in Disputed Areas OP/BP 7.60 X . Legal Covenants Name Recurrent Due Date Frequency Schedule 2, Section I.A.1. X CONTINUOUS Description of Covenant The Recipient, through the MOET, shall take all necessary measures to ensure that appropriate project management arrangements are maintained throughout project implementation Name Recurrent Due Date Frequency Schedule 2, Section I.E.5. X Semi-Annual Description of Covenant The Recipient shall take all measures necessary on its part to regularly collect, compile, and submit to the Association, as part of the Project Reports, information on the status of compliance with the safeguards instruments. Name Recurrent Due Date Frequency Schedule 2, Section I.D.1 X Yearly Description of Covenant The Recipient shall furnish to the Association by October 31 of each year, Annual Work Plans containing eligible project activities and eligible expenditures proposed to be included in the project in the following calendar year for the Association’s no-objection. . Conditions Source Of Fund Name Type IDA Project Operations Manual Effectiveness Description of Condition The Recipient, through the MOET, shall have adopted the Project Operations Manual, in form and substance satisfactory to the Association Source Of Fund Name Type IDA Retroactive Financing Disbursement Description of Condition iv Withdrawals up to an aggregate amount not to exceed SDR 1,800,000 may be made for payments made prior to the signing of the financing agreement but on or after September 30, 2015, for eligible expenditures. Team Composition Bank Staff Name Role Title Unit Michael F. Crawford Team Leader (ADM Lead Education GEDDR Responsible) Specialist Hoang Xuan Nguyen Procurement Specialist Procurement Specialist GGODR Cung Van Pham Financial Management Sr Financial GGODR Specialist Management Specialist An Thi My Tran Team Member Education Specialist GEDDR Anna Coronado Team Member Program Assistant GEDDR Christopher Robert Team Member Sr Financial GGODR Fabling Management Specialist Huong Thi Mai Nong Counsel Associate Counsel LEGES Junxue Chu Team Member Senior Finance Officer WFALN Khang Van Pham Environmental Specialist Consultant GENDR Michel J. Welmond Team Member Program Leader EACVF Nghi Quy Nguyen Social Development Social Development GSURR Specialist Specialist Nguyet Minh Nguyen Team Member Program Assistant EACVF Nina Masako Eejima Counsel Senior Counsel LEGES Ria Nuri Dharmawan Counsel Associate Counsel LEGES Extended Team Name Title Office Phone Location Dan Thanh Nguyen . Locations Country First Location Planned Actual Comments Administrative Division Socialist Country-wide Republic of Vietnam . Consultants (Will be disclosed in the Monthly Operational Summary) Consultants Required ? Consultants will be required v I. STRATEGIC CONTEXT A. Country Context 1. Progress in education is an important part of Vietnam’s long term aspirations to achieve upper- middle income status or better within the next generation. Vietnam, like the East Asia region as a whole, has recovered from the global economic downturn in a reasonably strong position but with substantial risks. The region is projected to account for two-fifths of global growth and one-third of global trade in the short-term future. In Vietnam, diminishing inflation, strong export growth, and expanding foreign exchange reserves are all contributing to a reasonably stable macroeconomic environment. Growth from 2008 to 2014 was 5.8 percent (annual average). Overall export growth from 2011-2013 was above 22.8 percent, with high-tech exports growing by a robust 69.4 percent.1 These positive developments are offset by data from three major indices2 showing that Vietnam’s competitiveness is declining with respect to that of its neighbors and other comparison countries. 2. Vietnam’s structural reform agenda takes time to implement, and exports currently remain dominated by “less sophisticated products with low value added and embodying modest technology.”3 Emphasis on improved competitiveness and greater value added are essential to continuing the country’s economic and social progress. The share of labor in agriculture continues to decline, and more workers are moving “one chair over” to a more productive segment of the labor market: from farming to informal, from informal to low-productivity formal, and from lower- productivity formal to high-productivity formal. This transition raises skill requirement at every level of employment. 3. Data on poverty and consumption provide encouraging news: recent consumption growth has favored the poor. The national poverty rate fell from 20.7 percent in 2010 to 17.2 percent in 2013. Although poverty rates for ethnic minorities and those living in rural areas remain high, household expenditures continue to grow overall. Moreover, expenditures in rural areas have risen for households at every level of income along the economic spectrum.4 The overarching structural transformation of the economy described above, from rural, small-holder agriculture to urban employment, tracks the main pathway out of poverty. Investment in the education of Vietnam’s children is the cornerstone of this pathway; for this reason, the general populace’s demand for education and expectations for improvements in education quality are both rising steadily. 4. The recent 2011- 2015 Socio-Economic Development Plan (SEDP) of the Government of Vietnam (GOV) recognizes the potential impacts, opportunities, and challenges of globalization. In particular, it acknowledges the importance of countries’ mutual dependence, integration, competiveness, and cooperation, all of which are driven by populations’ levels of knowledge and education. One of the three strategic breakthrough measures for socioeconomic development 1 Taking Stock, World Bank. December 2013. Page 32 (table 2.2). Note: figure estimated by taking annual averages from 2011, 2012, and the first 10 months of 2013. 2 Indices: (i) the World Economic Forum Global Competitiveness Index; (ii) the World Bank’s Doing Business rankings; and (iii) the IDA Country Policy and Institutional Assessment. 3 Taking Stock, World Bank. December 2013. Page 8. 4 Taking Stock, World Bank. December 2013. Page 48. 1 proposed in the SEDP is “quick development of high quality human resources” for the country’s industrialization, modernization, and development of a knowledge-based economy. Improving education quality and outcomes is a key aspect of the SEDP measure and reflects the GOV’s desire to invest in education. Indeed, this policy goal prompted the GOV to allocate nearly 20 percent of public expenditures to education starting in 2010. This percentage is significantly above the Organization for Economic Co-operation and Development (OECD) average of 12.9 percent.5 B. Sectoral and Institutional Context 5. The growth in social demand for education and training reflects both traditional values and the concern in Vietnamese households that children acquire the new skills needed to succeed in a knowledge-based economy. The government has made sustained efforts to meet these demands and to continuously improve education access and quality. Vietnam has achieved universal primary and lower secondary education. In 2012, the net primary education enrollment rate was 98 percent, with 88.5 percent of enrolled children completing primary education. It is now aiming for universal preschool education for 5-year-old children by 2015 and universal upper secondary education enrollment by 2020. Vietnam is expanding education access (particularly for disadvantaged and ethnic minority groups) and ensuring gender equality in education and training.6 6. Vietnam’s K-12 education system is large and centrally-managed. In 2013, it consisted of 14.8 million students in primary and secondary schools (7.2 million in primary, 4.9 million in lower secondary, and 2.7 million in upper secondary) and over 850,000 direct teachers (381,000 in primary, 315,000 in lower secondary, and 151,000 in upper secondary). The responsibility for the schools is ultimately in the hands of the national-level Ministry of Education and Training (MOET). However, provincial-level Departments of Education and Training (DOETs) directly oversee upper secondary schools, and district-level DOETs directly oversee primary and lower secondary schools. 7. The MOET issues learning standards for all school subjects and for all grade levels. MOET also approves the textbooks, pedagogical materials and teachers’ guides used by teachers and education administrators to select what to teach and how. These two taken together constitute “the curriculum.” The current curriculum was developed under the guidance of National Assembly Resolution No. 40/2000/QH 10. As per the Resolution, the full school curriculum was rolled- out sequentially so that the 1997 cohort was the first to receive the new curriculum for each grade level as it progressed through 12th grade. After twelve years, all twelve grades were operating under a “new” curriculum—although by this point the grade 1 “new” curriculum was twelve years old. 8. Vietnam participated in the Program for International Student Assessment (PISA) for the first time in 2012 and its results were very encouraging.7 The country scored higher than the OECD country average and outperformed many developed economies. Furthermore, results of the recent Strategic Teaching & Evaluation of Progress (STEP) literacy survey showed that the literacy levels 5 Education at a Glance, OECD. 2011. Page 248. 6 Resolution 29 (part A, item 1, paragraph 3). 7 Vietnam scored 511 (17th place out of 65 tested countries) on math, 508 on reading (19th place), and 528 on science (8th place). 2 of younger adults from urban areas in Vietnam are comparable to the literacy levels of their peers in many OECD countries (including Austria, Germany, Denmark, Canada, Norway and France).8 9. Although Vietnam has succeeded in promoting increased access to education, significant challenges of quality remain. One out of four eight-year-olds is unable to write a basic sentence without errors.9 Previous large-scale Grade 5 reading assessments confirmed inadequate abilities of the poorest students to write at the level expected for their age. Enrollment rates may be high, but it is clear that the quality of primary education can be further improved. 10. Most importantly, demand for skills is running far ahead of supply. Despite Vietnamese workers’ relatively strong levels of literacy and numeracy, employers report difficulties in hiring new workers with adequate skills. Employers emphasize their need for employees who are equipped not only with technical skills, but also with higher-order cognitive and behavioral skills, such as the ability to think critically, solve problems, and communicate verbally and in writing in a convincing manner to clients and colleagues.10 A coherent, high-quality curriculum that promotes advanced skills for all graduates is a critical part of Vietnam’s strategy to overcome this scarcity of skills in the labor market. Failure to address this problem will significantly hamper economic productivity and growth. 11. Criticisms of the previous curriculum reform focus on two issues. The strongest criticism is that the principles of the reform were insufficiently clear and inconsistent. Observers have cited vague and contradictory statements from MOET in its guidance to teachers, sometimes in the same official publication. The goals of the reform were to promote variously “active learning” or “student –centered learning” or “discovery learning” or “collaborative learning” or “autonomous learning methods.” MOET was criticized for not making the meaning of these terms sufficiently clear, and for not stating clearly and concretely what teachers should do differently in the classrooms under the new curriculum.11 12. The second criticism of the previous curriculum reform is that it was segmented by school sub- system. Those who revised the primary school curriculum were only tangentially involved, if at all, with the revision of the junior secondary school curriculum; idem from junior secondary to senior secondary. No coordinated sense of the direction of the curriculum as a whole could emerge given the changing composition of technical experts and stakeholders throughout the process. 13. The SEDP of 2011-2015 lists the following as key to the development of the Vietnamese education sector: renovation of teaching, learning methods, learning outcome assessments for all educational levels, the application of Information, Communication and Technology (ICT) in teaching and learning, the revision of general education curriculum and textbooks, and the promotion of students’ creativity. 8 Vietnam estimates from World Bank staff analysis using STEP household level data; note that while young Vietnamese adults perform relatively well, Vietnamese urban workers perform generally below their peers. 9 Young Lives, 2013. 10 Skilling Up Vietnam: Preparing the Workforce for the Modern Market Economy, World Bank. November 2013. 11 Duong Thi Hong Hieu, “Changing Pedagogies: The Vietnamese Case from the International Perspective” Journal of the University of Education of Ho Chi Minh City, vol. 54, 2013. 3 14. Resolution 29 –NQ/TW of Party Plenum XI (2013) requires the national education and training system to improve the knowledge, human resource training, and foster the talent of the Vietnamese people. It orients the education system away from being content-based, and toward being more competency- and quality-based. It requires general education to focus on developing intelligence, physical well-being, forming proper citizenship qualities and competencies, promoting talents, and providing career orientation for students. It also stipulates that graduates should have fundamental general knowledge that prepares them for either post-secondary education or immediate entry into the job market. 15. In response to Resolution 29, MOET submitted a proposal to the National Assembly entitled “Reform of General Education Curriculum and Textbooks after 2015.” The proposal would achieve the aims of Resolution 29 and result in a new general curriculum and new or substantially revised subject-specific curricula (SSC), textbooks, teacher guides, and other pedagogical materials. Materials for grades 1, 6, and 9 would be piloted in the 2016-2017 academic year, and materials for other grades in primary, lower secondary, and upper secondary schools would be developed and piloted annually for the four subsequent academic years. 16. The new general education curriculum is expected to develop the key qualities and competencies of 21st century citizens. As any good curriculum does, it starts from a conception of the “ideal graduate” who has harmoniously developed ethically, intellectually, and physically, and who can apply learning to real world challenges. 17. The Bank has been a key partner for the GOV. The Government-Bank collaboration has addressed virtually all aspects of education coverage and quality, from pre-primary education to the promotion of Science, Technology, and Innovation capacity, and includes the graduate education and research system. The current portfolio of education projects focuses on: (a) Readiness to Learn for Young Children: The School Readiness Promotion Project (SRPP - P17393) seeks to improve early childhood education by promoting school readiness for disadvantaged children through Vietnam’s existing Early Childhood Education (ECE) program. In addition to strengthening the ECE program’s capacity, the SRPP expands full-day preschool enrollment, improves preschool quality assurance mechanisms, and develops preschool teachers’ and education managers’ professional expertise. (b) School Coverage and Instructional Time: The School Education Quality Assurance Program (SEQAP - P091747), which supports the GOV’s full-day schooling reform program by upgrading educational infrastructure and facilities, developing human resources, and reinforcing policy frameworks, aims to improve educational outcomes and completion rates for disadvantaged primary education students. (c) Improved Pedagogy: The Vietnam Global Partnership for Education - Escuela Nueva Project (VNEN - P120867) introduces new teaching and learning practices that promote active learning and de-emphasize rote memorization. The VNEN is especially notable for inaugurating and advancing this new pedagogical approach. 4 18. The advances under the VNEN are particularly relevant to this proposed Project. They have sought to replace traditional memorization-based learning with more active learning methods, more student-centered teaching, and greater emphasis on the ability to apply learning outside the classroom. The revised curriculum is expected to further these advances by providing a more coherent, focused, and rigorous learning program that promotes both the knowledge and the competencies needed for deeper understanding of curriculum content and greater ability to apply knowledge. 19. The need for high-quality education does not end with secondary school graduation. The government and the Bank also promote the autonomy of universities along with a host of other policy improvements at the tertiary level such as the aforementioned advancement of science, technology, and innovation capacity. 20. The Fundamental and Comprehensive Renovation of Education (proposed in Resolution 29 NQ/TW, which was ratified on November 4, 2013 in the 8th Session of the Communist Party Central Steering Committee) aims to produce a “learning society” necessary for Vietnam’s continuing industrialization, modernization and international integration. The renovated general education will provide Vietnamese children with the new cognitive and behavioral skills necessary for this vision. A modernized post-secondary education will build on these skills for the new world of work and research. The World Bank will strategically support the Education Renovation through a programmatic approach. 21. The World Bank is discussing with the Government additional areas of intervention to achieve a more comprehensive package of support to the Fundamental and Comprehensive Renovation of Education Program. One area under discussion is the Government’s plans to transform Vietnamese teachers into professionals capable of providing new content and new ways of instruction. To affect this change, teachers will need more customized support and the education system will need to develop the requisite feedback mechanisms. Different approaches, including lending operations that may potentially transform schools of education from training providers into coordinators of teacher support services, are being considered. The package of support also envisages support to policy shifts to bring training and other services closer to the teacher either through more classroom and school level interventions. 22. A second area is expected to be the modernization of higher education. The Vietnamese government’s vision is for more accessible higher education that meets a wider diversity of learning needs, is better linked to the needs of the economy and society, has a more diverse set of providers, and can produce world-class research. Recent policy changes allow higher education institutions greater autonomy and promote the development of quality assurance mechanisms in order to create the potential to meet these ends. 23. Additional work in these two areas, in addition to this project, will strategically support key and catalytic elements of the Education Renovation in a phased manner. This interdependent set of interventions represents a system-wide perspective and a programmatic approach to supporting Vietnam’s Education Renovation effort. This package of support is also fully consistent with the World Bank’s overall strategy for education reform, especially as manifest in the Systems Approach for Better Education Results (SABER). 5 C. Higher Level Objectives to which the Project Contributes 24. The proposed Project supports the World Bank’s overarching goals of reducing poverty and promoting shared prosperity. The new, improved curriculum developed by the proposed Project will enable Vietnamese schools to produce better-educated graduates who have the ability to meet the rising skill demands of 21st century employment. These higher standards present a special challenge and opportunity for students from families in the bottom 40 percent of the socio- economic spectrum: raising learning outcomes among this group will take greater effort but will also contribute in important ways to poverty reduction, social mobility, and equity. Special efforts will be made to track the progress of the most disadvantaged. This will help promote the Twin Goals of ending poverty and promoting shared prosperity. At the same, the revised curriculum will provide an opportunity to ensure appropriate inclusiveness and relevance for disadvantaged groups and with respect to gender. 25. More specifically, the proposed project is directly aligned with the 2012-2016 Country Partnership Strategy (CPS) for Vietnam. The CPS’ strategic framework stands upon three “pillars” - competitiveness, sustainability, and opportunity - and the proposed project relates to both competitiveness outcome 1.3 and opportunity outcome 3.2. 26. Outcome 1.3 seeks to increase Vietnam’s “capacity for innovation and value addition.” It notes that successfully fostering innovation will depend on strengthening the Vietnamese education system’s ability to teach students skills relevant in increasingly sophisticated labor markets. The CPS indicates that future Bank education projects will focus on the education system’s capacity to deliver competencies that will be critical for employment and economic growth. By supporting the design and creation of a competency-based curriculum that incorporates 21st century skills into Vietnamese classrooms, the proposed project will contribute significantly to this outcome. Furthermore, the proposed project’s commitment to developing high-quality assessments calibrated for use with the new curriculum aligns closely with this CPS’s emphasis on measuring student learning. 27. The proposed operation will also contribute directly to CPS outcome 3.2, which seeks to improve “basic infrastructure and public service delivery and access” by fostering better learning outcomes through increased educational efficiency, more effective teaching, and higher school completion rates. Like outcome 1.3, outcome 3.2 emphasizes the importance of measuring the quality of service delivery. The support for curriculum and assessment enhancements and research included in the proposed project will enable the GOV to carry out such student outcome measurements more easily and more precisely. The use of improved data through feedback mechanisms will inform more effective targeting to further improve education quality and learning outcomes. 28. Finally, the proposed project advances the CPS’ goal of maximizing the benefits of the World Bank’s “Knowledge Bank” role by stressing the importance of technical assistance in the curriculum design process. This process began under the auspices of the Vietnam Russia Education Aid for Development (READ) Trust Fund, which enabled the Bank to assemble a team of external experts who provided the highest levels of technical advice drawing on international best practices in successful curriculum and assessment Reform. This technical assistance has supported the MOET during the READ implementation period. Project Preparation has built on the results of 6 29. READ activities and promoted intensive technical interactions with leading experts and practitioners to take advantage of the growing knowledge base for assessment and curriculum improvement. II. PROJECT DEVELOPMENT OBJECTIVES A. PDO 30. The Project Development Objective (PDO) is to raise student learning outcomes by: (i) revising and implementing the curriculum following a competency-based approach; and (ii) improving the effectiveness of instruction by creating and disseminating textbooks aligned with the revised curriculum. B. Project Beneficiaries 31. The beneficiaries of the Project include the following groups: (a) Vietnamese primary, lower secondary, and upper secondary students: With a new curriculum, pedagogical materials, and assessment system that improve upon previous efforts to increase student learning outcomes in the context of Vietnam’s rapidly growing economy, all Grades 1-12 students will be better equipped to pursue a wide range of opportunities after they complete secondary school. More interesting and engaging classes are likely to modestly attenuate dropout rates. It should be noted that the Project is expected to benefit students of both genders equally. Indeed, specific efforts will be made to ensure that the revised curricula and pedagogical materials avoid biases related to gender, ethnic origin or other non-majority statuses. (b) Vietnamese teachers and school administrators: Vietnamese teachers and school administrators will benefit from the coherence of the new and improved education system. Not only will they have new pedagogical materials that better equip them to teach students, but they will also be able to plan lessons based on an improved curriculum that allows them to accomplish more successfully their goal of improving student learning. (c) Vietnamese policymakers: Policymakers will have increased capacity to develop and implement policy interventions by utilizing data collected and analyzed through the assessment and research systems supported by Component 3.Vietnamese and foreign employers: Employers will benefit from graduates who are better prepared to face the challenges of the 21st-century workplace, including the ever-increasing importance of improved problem-solving and critical thinking abilities. (d) Vietnamese tertiary education institutions: Students entering Vietnam’s tertiary education system will be better equipped to handle rigorous coursework, allowing these institutions to challenge their student bodies and produce highly-qualified graduates. 7 (e) Vietnamese society as a whole: Society as a whole will benefit from the development of a more educated population and stronger economy. C. PDO Level Results Indicators 32. The Projects main outputs will be: (a) A revised and improved curriculum. The new curriculum will improve on the existing one by: (i) Appropriately balancing the number of topics covered with the depth of treatment per topic in each subject at each grade level; (ii) Progressing coherently from grade to grade and being consistent across subjects within each grade-level; (iii) Being “competency-based” and promoting the creation of graduates with strong competencies in: (1) problem-solving and critical thinking; (2) communicating; (3) working in teams; (4) managing their own learning; and (5) managing their conduct, inter alia. (iv) Promoting pedagogies that support the ability to apply knowledge and minimize rote learning and memorization; and (v) Recognizing that the curriculum does not only equal what is taught, but also includes how it is taught. (b) Adequate understanding of the new curriculum and its implications for classroom practices among Vietnam’s teachers. This understanding will be fomented both by: (i) a comprehensive on-line interactive training system and by the use of highly-trained “curriculum champions” in schools and at the district or provincial DOETs; and (ii) the drafting, printing, and distribution of textbooks, teachers’ guides, and other materials aligned with the new curriculum to at-risk schools; (c) A comprehensive set of analyses of student learning outcomes under the new curricula, with special attention to the progress of poor and disadvantaged students; and (d) An expanded National Large-Scale Assessment System (NLSAS) that will: (i) be aligned with the new curriculum; (ii) adhere to the new MOET policy of sample-based testing of students at each sub-level of education (5th grade for primary, 9th grade for junior secondary, and 11th grade for senior secondary); and (iii) promote the consequential analysis and use of assessment data from all levels for the improvement of learning. 33. Although the Project will have the above-mentioned major outputs, it will have three key PDO Indicators. 8 (a) Indicator 1: Student learning gains (as measured by “an increase0.12 standard deviations in Math and Vietnamese test scores of students under the new curriculum compared to the baseline cohort”). Data on both school and non-school factors will be collected and the revised curriculum will be piloted using both treatment and control classrooms to permit attribution of learning gains. (b) Indicator 2: Teacher preparedness for new curriculum (as measured by “25 percent of teachers implementing the new curriculum will be certified as prepared for curriculum implementation”); and (c) Indicator 3: Equitable access to learning materials (as measured by “In the bottom 10 percent of schools - based on income distribution - textbooks will be distributed to raise the ratio of student to textbook ratio to 1.1 times its baseline rate”). 34. Several technical issues related to Project indicators will be addressed during the construction of indicator baselines. The baseline for indicator 1 will be constructed by analyzing learning outcomes from existing standardized test microdata at the 5th grade and 12th grade levels. Anchor items from these baselines will be reused in subsequent testing to discern changes to learning outcomes. These anchor items will be incorporated into the expanded NLSA system (output (iv) above) to do this. The gains themselves will rely on output numbers (i), (ii) and (iii) above. 35. Indicator 2 will measure the impact of Output # (b) above – the extent of understanding among teachers of the new curriculum and its implications for classroom practice. The online teacher training system will be used to determine if the minimum levels of acceptable performance have been attained. Teachers who attain these minimum levels or above will be certified. As the certification is new, the baseline for this indicator is the null situation of no teachers certified for the new curriculum. 36. Indicator 3 requires technically sound surveys of teaching and learning materials, both for the construction of baselines and for the measurement of progress under the Project. Surveys will be conducted immediately following appraisal. III. PROJECT DESCRIPTION A. Project Components 37. Component 1: Curriculum Development (US$17.5 million, including US$1 million contingencies). This component will support the development and implementation of a revised curriculum that is competency-based for all school subjects from grades 1 - 12. More specifically, this component seeks to: (i) develop and pilot a revised curriculum that aims to promote students’ acquisition of both competencies and content knowledge; (ii) deliver online and face-to-face training to educators and administrators on implementing the new curriculum; and (iii) draft materials to guide publishers on developing appropriately aligned textbooks and other pedagogical materials. This work will primarily be undertaken by the Core Technical Committee (CTC) and 9 the Subject-Specific Technical Committees (SSTCs), who will ensure that the new curriculum exhibits coherence, rigor, and focus. 38. Subcomponent 1.1: Curriculum Development. This subcomponent will support the creation and work of a Core Technical Committee (CTC) for curriculum revision plus up to eleven Subject Specific Technical Committees (SSTCs). The CTC will be composed of the chairpersons of each SSTC. The CTC will carry out the main curriculum revision, with the technical “legwork” conducted in each of the SSTCs. The work of the CTC will ensure that the curriculum is coherent and coordinated across subjects and years. Outputs of the work of the CTC and the SSTCs will be, inter alia: (i) comprehensive micro-level reviews of Vietnam’s existing curriculum; (ii) partial and selective reviews of other countries’ successful “general curricula” (including their definitions of key terms, desired characteristics of graduates, selection of specific subjects, and basic first-order time allocations); (iii) draft SSC; (iv) indicative menus of pedagogical approaches and learning resources aligned with the new curriculum; and (v) guidance for the development and evaluation of textbooks and other pedagogical materials. 39. Subcomponent 1.2: Curriculum Implementation. Once the curriculum has been developed, this Project will support several ancillary activities to ensure that the curriculum is implemented appropriately. Such activities include: (i) curriculum piloting; (ii) delivery of face-to-face and online training to teachers and Curriculum Champions who can act as peer coaches for fellow teachers as they seek to understand and implement the new curriculum; and (iii) development of guidance materials for publishers of textbooks and other pedagogical materials that are aligned with the Learning Outcome Standards for the new curriculum. 40. Component 2: Development and Dissemination of Aligned Textbooks (US$21.6 million, including US$1 million contingencies). This component will support the development a set of textbooks aligned with the new curriculum and the provision of aligned textbooks to schools with high numbers of economically-disadvantaged students. 41. Subcomponent 2.1: Development of Textbooks Aligned with the New Curriculum. After the new curriculum has been completed and officially adopted by MOET, the CTC and the SSTCs will be “converted” into mechanisms to develop a set of textbooks aligned with the new Learning outcome standards. The CTC and its members will migrate to the role of “appraisers” of textbooks while the SSTCs will add editors and textbook writers to the subject-specific expertise embodied in the SSTC membership. The enhanced SSTC committee will work under a management structure of subject specific editors and a general editor. The CTC will review and appraise the extent to which texts align with the new curriculum and are therefore suitable for classroom use and official adoption by MOET. The CTC’s recommendations on textbook alignment will be transmitted to the Steering Committee, who will make final decisions on the authorization of textbooks. This structure will preserve the institutional memory and knowledge of the nature of the new curriculum, and guide the development of aligned texts. 42. This sub-component will also seek to translate a selection of textbooks into ethnic minority bilingual textbooks. The textbooks will be printed and disseminated to ethnic minority learners who are also economically disadvantaged. The subjects and languages chosen for translation will be selected under the Project in close consultation with ethnic minority leaders and communities. 10 43. Subcomponent 2.2: Provision of Textbooks to Disadvantaged Schools. Under this subcomponent, the Project will provide sets of textbooks to schools whose students are most economically-disadvantaged. A survey will be conducted to assess the level of need in different geographic areas and among schools. Based on survey results, the books will be distributed to libraries of schools at the bottom 10 percent based on household income. The books will be on loan for students for the duration of the academic year. 44. Component 3: Learning Assessment and Analysis for Continuous Improvement of Curriculum and Policy (US$35.4 million, including US$1 million contingencies). In order to ensure that learning outcomes are measured and analyzed for continuous improvement of curriculum and policy, this component will support: (i) the establishment of the National Centre for Sustainable Development of General Education (NCSDGEQ) and the National Center for Foreign Language Testing (NCFLT); (ii) research and capacity-building in curriculum development and assessment for the Centers’ staff and relevant officials; (iii) expansion of the National Large-Scale Assessment system (NLSAS); and (iv) consultations/outreach campaigns. A significant focus of this component will be to understand better and narrow the learning achievement gaps that exist within the Vietnamese public education system. 45. Subcomponent 3.1: Establishment of the NCSDGEQ and the NCFLT. To facilitate the research and measurement activities, this Project will support the construction of up to a total of six sites for the NCSDGEQ and the NCFLT. The NCSDGEQ will be staffed by full-time staff and individuals seconded from agencies such as the Vietnam Institute of Educational Science (VNIES) and General Department of Education Testing and Accreditation (GDETA), inter alia. It will also house “visiting researchers” from the university sector. The purpose of the NCSDGEQ will be to advance research in several areas related to curriculum, assessment, learning, and the educational policies that affect these. Moreover, as improved foreign language skills have been identified as a national priority, the NCFLT will be become a leading center for research and training aimed at improving the teaching and learning of foreign languages. 46. Subcomponent 3.2: Research and Capacity-Building on Curriculum Development and Assessment. The educational elements in a curriculum require continuous review and, in many cases, adjustment. The process of continuous improvement enhances the effectiveness of the curriculum and promotes more and better learning. Goals for reading and literacy skills, and learning outcomes for early readers, for instance, should be reviewed and perfected on an on-going basis. Research advances various domains related to education need to be understood, and, where relevant, appropriately incorporated into the pedagogy or learning outcomes standards. While formal revision of the curriculum and its Learning Outcome Standards usually takes place about once every ten years, review of the curriculum in light of emerging research is an on-going process. This subcomponent will sponsor research programs in several areas related to the new curriculum and the Learning Outcome Standards. This research will be carried out by a National Centre for Sustainable Development of General Education Quality, which is expected to continue to function after the Project ends. 47. Subcomponent 3.3: Expansion of the National Large-scale Assessment System. This subcomponent will improve the coverage, technical quality, alignment, validity, and reliability of 11 the national assessment system. More specifically, it will support: (i) integration of curricula and assessment; (ii) closer alignment of assessment with learning goals; (iii) training for educators and administrators at various levels of the system to design and implement assessments and analyze and report the results; and (iv) creation of specialized expertise within MOET about continuously improving large-scale assessment capacity. In addition, this sub-component will finance the implementation of two rounds of learning assessment in each of the education levels (Primary, Lower, and Upper Secondary) that are of the desired level of quality and incorporate the assessment of core competencies. 48. Subcomponent 3.4: Consultation and Outreach Campaign. This subcomponent will develop and implement a Consultation and Outreach campaign to promote the public’s awareness of the reform and to instill public trust in the transparency and meritocracy of the reform process. The key messages of the reform will be delivered through several forms of media, such as print, television, word-of-mouth and social media. The primary source of information, however, will be the MOET website devoted to explaining the reform to the public. 49. Component 4: Project Management (US$2.5 million). This component combines Project Management with Monitoring and Evaluations (M&E). 50. Subcomponent 4.1: Project Management. This subcomponent will finance the incremental costs12 associated with project management, including activities designed to ensure efficient program management and early identification of corrective measures to solve any implementation challenges. 51. Subcomponent 4.2: Monitoring and Evaluation. This subcomponent supports the development and implementation of Project M&E in order to ensure that the determined indicators are achieved and to continuously identify areas of the Project that can be improved. Key Project Indicators combine both information derived from the expanded and aligned NLSAS along with Project- specific indicators such as the number of teachers reaching certification in mastery of the concepts underlying the revisions to the curriculum. B. Project Financing 52. The lending instrument will be Investment Project Financing and will be financed by an IDA Credit in the amount of US$77 million. The total cost of the project will finance goods, works, non-consulting services, consultants’ services, training and workshops, and incremental operating costs for the project. . It is expected to be implemented over a five-year period. Retro-active financing: Withdrawals up to an aggregate amount not to exceed SDR 1.8 million (US$2.5 million equivalent) may be made for payments made prior to the signing of the financing agreement but on or after September 30, 2015, for eligible expenditures. 12 “Incremental Operating Costs” refers to reasonable costs of incremental eligible expenditures incurred by the MOET in the implementation of the project, based on annual work plans and budgets approved by the Bank, including the costs of communication costs; information and communication campaigns; monitoring and evaluation costs; and wages of contracted staff; but in all cases excluding salaries, salary allowances, and salary supplements of the Recipient’s civil servants (“công chức”, “viên chức”). 12 Project Cost and Financing IDA Project cost % Project Components Financing (US$ M) Financing (US$ M) 1. Curriculum Revision 16.5 16.5 100 2. Development and Dissemination of Aligned 20.6 20.6 100 Textbooks 3. Learning Assessment and Analysis for 34.4 34.4 100 Continuous Improvement of Curriculum and Policy 4. Project Management 2.5 2.5 100 Contingency 3.00 3.00 100 Total Costs 77.00 77.00 100 Total Project Costs Total Financing Required 77.00 77.00 C. Lessons Learned and Reflected in the Project Design 53. Coherence, focus, and rigor underpin successful curricula. The nature of curriculum-setting as a political process creates a bias toward accumulation of topics beyond optimal learning points. School hours and especially instructional time remain fixed while the number of potentially worthwhile topics has no hard limit. This tension is often resolved by overloading curricula with a variety of topics from different proponents. Recent experience has shown that improved mastery and better learning outcomes are associated with a smaller number of topics in the curriculum to allow appropriate depth-of-treatment and mastery of each.13 Explicit commitment is needed to create a curriculum that avoids this pitfall. Focus in a curriculum is achieved when the number of topics included matches the available time to treat each with sufficient depth to promote the highest average mastery. Coherence describes the degree to which the topics are clearly presented and the subsequent builds on the previous. Rigor is achieved when the cognitive complexity of the topics chosen is appropriately challenging for the developmental stage and abilities of the target students. 54. A small number of principles promote effective curriculum change. Coherence, focus, and rigor are goals of a good curriculum. They are achieved through adherence to a small number of principles in addition to those stated in the paragraph above. Most important among these are: (i) clear and logical grade-to-grade progress of learning topics; (ii) consistency of treatment and 13 See Schmidt et al. “Why Schools Matter” Oxford University Press, 2012. P.360 13 presentation across subjects at a given grade; (iii) consistency of treatment but with appropriately increasing complexity in new iterations from grade-to-grade; (iv) attention to how content is taught and not just what content is taught; and (v) above requires schools to promote the acquisition of generic competencies, such as problem-solving, teamwork, self-management of learning, communication skills, and self-management abilities along with content knowledge. 55. Empirical analysis complements judgment. Expert judgment and the political expression of preference have traditionally played a central role in the definition of curricula. The advent of iterative internationally-comparable tests such as PISA and especially Trends in International Mathematics and Science Study (TIMSS) has allowed these techniques to be supplemented with empirical research on the connections between curricula, time-on-task, and learning outcomes. The trend will be toward curricula and policies being informed more by empirical analysis of data about learning. 56. Clear and simple definitions with concrete implications for classroom practice are required. Many reforms have espoused vague and theoretically interesting reform topics but have not made clear to teachers, parents, and students what these terms mean and what concretely should change in classroom practices. Successful reforms clearly define key terms, provide simple and concrete examples of what these mean, and demonstrate indicative and desirable changes to classroom practices. 57. Long-term commitment is required to change classroom practices. Experience from the “Escuela Nueva” schools both in Colombia and Vietnam affirms that changing teachers’ classroom practices requires sustained commitment. Significant effort is needed to ensure understanding and ownership of curricular and pedagogical models and standards. 58. Assess and Adjust at Every Sub-level of Education. Both classroom and systematic, large-scale assessment provide the information needed to adjust and optimize teaching. Policy-makers require large amounts of technically-sound assessment data to understand the factors that promote learning on average and among sub-groups and individual learners. Sample-based periodic assessments of the progress of cohorts provide an indispensable measure of aggregate learning from year to year. Continuous diagnosis of progress is the foundation of sound analysis of educational performance. IV. IMPLEMENTATION A. Institutional and Implementation Arrangements 59. The agency responsible for implementing the Project is the MOET. The MOET will ensure that appropriate project management arrangements are established, including a management entity headed by a director that will have primary responsibility for Project implementation (including, inter alia, procurement and financial management). Technical issues related to curriculum and textbook revision will be under the purview of the Chief Technical Officer, who will chair the CTC. 60. The MOET will ensure that curriculum and textbook-related project deliverables are consistent with Government/Ministry policies and the goals of the Project. When this is not the 14 case, the Minister will return them to the CTC for further revision. When deliverables are of the desired quality, the Minister will authorize their official adoption. 61. The MOET will ensure close coordination and collaboration among Project implementers and relevant GOV agencies (including, inter alia, the Ministry of Planning and Investment and the Ministry of Finance), and other people and entities involved with the Project.. B. Results Monitoring and Evaluation 62. The three Key Project Indicators will measure changes to student learning, readiness of teachers to implement the new curriculum, and access of disadvantaged student to new, aligned textbooks and pedagogical materials. New standardized exams will be developed that use items from previous standardized exams and will therefore be scalable with these exams. They will allow student learning outcomes under the new curriculum to be compares with the status quo ante. Initial measure of teacher preparedness will be constructed using the key features of the curricular revisions as these are incorporated into revised learning standards. 63. The intermediate indicators will focus on: (a) The composition and qualifications of the CTC for Curriculum and Assessment Reform; (b) The progress, technical quality and effective use of the online training system; (c) The effectiveness of curriculum champions; (d) The progress and coverage of the production of aligned textbooks; (e) The effectiveness of curriculum piloting efforts; and (f) The coverage and quality of the expanded NLSAS. 64. Technical issues related to monitoring and evaluation pertain to assessing learning outcomes in such a way that baselines scores-obtained under the old curriculum equate to and are comparable with new scores from students taught under the new curriculum. Under Project preparation, technical work was carried out to: (i) identify existing standardized tests and corollary data from which anchor test items14 might be drawn; and (ii) analyze the technical quality of the potential anchor items. Additional activities were conducted to identify gaps in measurement of student characteristics to allow for the improvement of questionnaires. Technical challenges also exist to measure progress or quality under the areas of the progress where success depends on sound professional judgment. The MOET and Project implementers will ensure that capacity built through Project activities is incorporated into their efforts to produce the highest-quality M&E data and the soundest possible measurement of Project indicators. 14 An “anchor item” is a standardized test question, which is used in multiple tests and in multiple time periods to allow the performance of different cohorts of test takers to be scored on a single, comparable scale. 15 C. Sustainability 65. As indicated in subsection A of this section, the MOET is committed to implementing this Project successfully. The MOET has successfully partnered with the Bank on a number of projects. Ongoing projects include the SRPP, the SEQAP, and the VNEN. Projects that closed in the past seven years include the Primary Education for Disadvantaged Children Project (P044803), the Primary Teacher Development Project (P051838), and the Support for National Education-for-All Plan (P085260). Through these interactions, the MOET has become familiar with Bank procedures and practices. However, the Bank has yet to work extensively with the MOET on curriculum reform, and many MOET officials involved in this project have never worked directly with the Bank. In addition, revising an entire national curriculum requires substantial high-quality human capital, and the appropriate management of such human capital is crucial to success. V. KEY RISKS AND MITIGATION MEASURES A. Risk Ratings Summary Table Risk Categories Rating (H, S, M, or L) 1. Political and governance Substantial 2. Macroeconomic Moderate 3. Sector strategies and policies Low 4. Technical design of project or program High 5. Institutional capacity for implementation and Substantial sustainability 6. Fiduciary Substantial 7. Environment and social Low 8. Stakeholders Substantial 9. Other Low Overall Substantial B. Overall Risk Rating Explanation 66. The Project supports a reform that was initiated by the GOV through Resolution 29-NQ/TW. Therefore, the level of alignment between the interests, objectives, and motivations of the GOV and those of the World Bank is high. There remains, however, a number of key risks. 67. The highest risk to the achievement of the PDO relates to the technical design. The reform of the curriculum and textbooks involves high technical complexity. It requires a significant level of coordination of technical experts who have a shared understanding of the principles of the reform – coherence, rigor, and focus. Based on previous reforms, however, MOET is accustomed to having a large number of curriculum and textbook developers working independently of each other. This modus operandi led to incoherence and duplication of content in the curriculum. The likelihood that MOET will apply a similar modus operandi in this round of reform poses high risk to the achievement of the PDO. This Project seeks to mitigate this risk by empaneling a CTC 16 comprising a small group of top national experts who will be involved in all key activities under the Project. 68. Political and Governance, Institutional Capacity, Fiduciary, and Stakeholders, are assessed to be substantial. The next election of the National Assembly will take place in early 2016, followed by the selection of the new Government. The transition is expected to be relatively smooth and will not cause much disruption. The Bank team will monitor the political dialogue closely to ensure that the PDO is maintained despite the potential change in leadership. In regards to institutional and fiduciary capacity, MOET has had limited experience with the Bank guidelines, and implementation involves several key groups and agencies (CTC, SSTC, textbook developers, Center Staff). Financial and procurement management systems exist, but may be inconsistent with Bank guidelines in certain areas. Where incompatibilities are present, MOET sometimes follow national procurement procedures in parallel with Bank procedures. This leads to procurement delays, and noncompliance. Moreover, low financial management capacity can lead to weak internal control of a project and low-quality financial reporting. However, fiduciary mitigation measures have been identified and are being put in place (see Sections C and D, and Annex 3 for more details). In regards to stakeholders, the involvement of the general public and politicians in addition to high-level MOET officials increases the potential for divergent views and heightens the risk that the best technical pathways for reform will not be selected. VI. APPRAISAL SUMMARY A. Economic and Financial (if applicable) Analysis 69. The Project aims to raise learning outcomes by revising and enhancing the relevancy of the national curriculum and textbooks for grades 1-12 following a competency-based approach. The revision, coupled with improved capacity in assessment and research, is expected to lead to higher quality education. Research supports this presupposition. 70. The central basis of Project benefits will be the ability of employees to be more effective and efficient over the course of their working lives due to the larger and more coherent set of knowledge and skills they possess. Especially relevant are competencies that allow them to know how to succeed in solving novel and unstructured problems. The use of the key principles of good curriculum reform will make new curriculum sufficiently different from its predecessor to promote these different work-place behaviors and with them increased labor productivity. Evidence for the effectiveness of better-designed curricula is growing in the education effectiveness literature. For example, a recent study carried out by the Institute of Education Sciences within the United States Department of Education examines the relative effectiveness of different math curricula on student achievement in early elementary schools through experimental design. Schools were randomly assigned to one of four selected math curricula that are widely-used and representative of different instructional approaches. The study finds that student math achievement under two of the four curricula were significantly higher than that of the two less effective curricula. The scores for the top two were .24 to .3 standard deviations higher than the bottom two, translating to 9-12 points higher at the end of a school year if the more effective curricula had been taught.15 These results 15 “Achievement Effects of Four Early Elementary School Math Curricula.” Institute of Education Sciences (2009). http://ies.ed.gov/ncee/pubs/20094052/summ_2.asp 17 provide additional evidence that curriculum does indeed have an empirical relationship with learning. 71. Literature on theoretical growth suggests that learning, in turn, yields economic growth. There are at least three mechanisms through which this can occur - improved human capital and productivity, increased innovative capacity, and the facilitation of the diffusion and transmission of knowledge to understand and apply new technology.16 A study by Hanushek and Kimko (2000) using international assessment results from 1960 – 1990 shows that a standard deviation higher in test performance leads to one percentage point increase in annual growth.17 A series of studies by Barro (2001), Woessmann (2002, 2003), Bosworth and Collins (2003), Coulombe and Tremblay (2006), and Jamison et al. (2007) have shown similar results, suggesting that the quality, more so than quantity of education, is an important factor in economic development. 72. Based on these research findings, the Project is expected to result in positive outcomes. The anticipated benefits of the Project include improvement in student cognitive ability, particularly as they relate to the core competencies – problem solving, communications, critical thinking, self- management, and self-learning. As indicated by various Vietnamese employer surveys, despite Vietnamese workers’ relatively strong levels of literacy and numeracy, employers still report difficulties in hiring new workers equipped with not only with technical skills, but also with higher- order cognitive and behavioral skills. Therefore, it is expected that workers who have these competencies would be compensated with higher wages. An economic analysis was undertaken to quantify these benefits. Core modeling assumptions were made about how project benefits would be accrued and were based on data for enrollment, drop-out rates, wages, and the State Bank of Vietnam’s refinancing rate. 73. As described in previous sections of the Project Appraisal Document (PAD), the revised curriculum and textbooks will be rolled out simultaneously in one year. For every subsequent year, it is expected that the cohort of students would receive an incremental one-twelfth of the benefits. For example, the first cohort to enter the labor market under this Project would have received only one year of studies under the revised curriculum. This group could then expect to receive a twelfth of the benefits, or a twelfth of the expected increase in wages. The second cohort would receive two-twelfths of the full curriculum and an equivalent proportion in benefits. Benefits for University graduates are calculated using the same logic. However, their benefits are delayed by four years to account for time spent at the university. The analysis also assumes that a certain percentage of the students who drop out of primary and lower secondary schools participate in the labor market. It is expected that 2 percent of primary school students18 and 12 percent of lower secondary school students19 participate in paid work when they drop out of school. Multiplying the enrollment data by dropout rates, labor market participation, and expected wage increase yields the total economic benefits produced by these various groups of students. 74. The curriculum is expected to be continually adjusted throughout the implementation process. However, the curriculum will eventually have to undergo another comprehensive round of 16 Hanushek & Woessmann. International Encyclopedia of Education (2010), vol. 2, pp. 245-252 17 Ibid. 18 Multiple Indicators Cluster Survey. UNICEF, 2006 19 Le, Duc; Tran, Tam. “Why Children Drop Out of School.” Young Lives, 2013. 18 revisions after three cohorts have been through the curriculum for the entire series of grades 1 – 12. The analysis accounts for this depreciation of the curriculum’s relevance, and begins to phase out benefits at the end of the 14th year after the roll-out. Once the phase-out begins, each subsequent cohort receives one-twelfth less of the benefits than the previous, and the change in wages is one percent less than before the phase-out. 75. The analysis takes a very conservative approach in estimating the change in wages before and after curriculum revision, with the highest being 3 percent for primary school, 2 percent for lower secondary and 1 percent for upper secondary and university. The rationale is that an additional year of schooling in primary school, when students are acquiring the foundational tools for learning, would bring about a higher change in wages than an additional year in upper grades when the subject-matters are more specialized. The analysis also employs a conservative estimate for the Refinancing Rate, or the rate that the Central Bank lends to banks, of 10 percent. The average rate between 2000 and 2014 was 9.05 percent, but has risen to as high as 15 percent to curb inflation between 2008 and 2012. 76. Based on these assumptions, the Project is expected to produce positive returns at an increasing rate up through the 15th year after the curriculum has been rolled out, with benefits optimized around 2031. For university graduates, because they do not enter the labor market until four years after reaping the benefits of the revised curriculum, their benefits optimize around 2035. As shown in Figure 1 below, high-school graduates produce the greatest benefits because: (i) they would have undergone the full curriculum from grades 1 – 12; and (ii) they have high absolute and relative labor force participation rate. The group with the second highest level of benefits is lower secondary school drop-outs. This can be attributed to the high drop-out rate and extended time spent under the revised curriculum (as compared to primary school students). 77. Table 1 below provides a comparison of the Internal Rate of Return (IRR), Cumulative Discounted Net Benefits, and the Benefit to Cost Ratio under different scenarios for wage differentials between before and after the curriculum revision. Under all scenarios, the IRR is greater than the 15-year average Refinancing rate of 9 percent, Cumulative Discounted Net Benefit is positive, and Benefit Cost Ratio is greater than 1. In the worst case scenario, in which the revised curriculum would only lead to a 2 percent wage increase for students who drop out of primary school, 1 percent for lower secondary, and 1 percent for upper secondary and university students, the benefit to cost ratio is still US$2.67 in benefits for every US$1 in investment. 78. There are several limitations to the approach taken in this analysis due to the limited data available. However, it is expected that the limitations lead to an underestimation of project benefits. For example, this analysis only takes into account private benefits and not public benefits, such as positive effects on health, crimes, etc. As such, the benefits in this analysis are likely to be lower than total societal benefits. The second limitation is that the analysis only takes into account the discounted stream of earnings through the phase-out of the project. The benefits of the project, however, are expected to be sustained throughout the students’ lifetime and beyond the duration of the Project. 19 Table 1: Comparison of Project Profitability under Different Wage Differentials Scenario Change in wages IRR Cumulative Benefit Cost (Prim/ Lower Sec/ Upper Sec/Uni) discounted net Ratio (US$) benefits (US$) High 5pct/3pct/2pct/2pct 27.20 519,817,550.09 6.84 Low 2pct/1pct/1pct/1pct 19.86 203,059,482.43 2.67 Figure 1: Discounted Project Benefits by Year and Education Level B. Technical 79. The key technical issues have revolved around designing a way for the new curriculum to better prepare students for the challenges they will face in their educational careers beyond secondary school (for those who continue on) and, most importantly, in their work lives and as life-long learners. The goal is to change education in a way that allows students to apply and use what they have learned in school in real-word situations and on novel problems. 80. The inclusion of competencies along with content has been seen as a main vehicle for promoting this ability to apply knowledge. The key competencies to be promoted are: (i) critical thinking/problem solving; (ii) communications; (iii) self-management of learning; (iv) self- management (conduct); and (v) teamwork. These competencies are generically applicable across main learning domains and for the application of knowledge generally, but the mean different things in different learning and application contexts. Communication for Mathematics or Science is not quite the same as communication in Vietnamese Language and Literature or Geography, although some characteristics are shared. Clarifying both which competencies are key and how to develop them across disciplines and grades in a consistent and holistic way has been central to the technical aspects of Project design. 81. Similarly, identifying the key principles for revision of the curriculum has been centrally important. Improving the curriculum will require the incorporation of the three main organizing 20 principles of clarity, focus and rigor. These in turn rely on appropriate breadth/depth trade-offs, and on matching the cognitive complexity of a subject to students’ current abilities while also building sequentially on previous learning. Further attention is needed to consistent treatment of topics and themes across grades and appropriate grade-to-grade progressions. 82. Achieving these outcomes requires the “back-mapping” of learning programs. Back-mapping begins with a consideration of what a senior secondary school graduate ought to know and be able to do. It then organizes the twelve-year educational experience to reach this point—using the principles above. 83. Producing successful curricula relies on use of these principles but also on sound professional judgment. The Project has sought to incorporate a structure and mechanisms which best promote the application of both these principles and the professional judgment that necessarily accompanies their application. C. Financial Management 84. A Financial Management (FM) Assessment for the proposed Project has been carried out. The assessment has concluded that the project meets the minimum Bank financial management requirements, as stipulated in BP/OP 10.00. The following key risks have been identified: (i) as the management entity will be newly established, its staff may not have relevant experiences with the Bank financed projects; and (ii) as there are a large number of (consulting) packages, and training and workshop events, internal controls may not be effectively designed and implemented to ensure eligible uses of funds. 85. The main actions required to mitigate the identified risks are: (a) Appointment of adequate FM staff: (Full time) Chief Accountant with appropriate accounting background and relevant experiences shall be appointed as part of the project management arrangements and found satisfactory to the Bank; (b) FM section of the Project Operation Manual (POM)20 and training: FM section (as a chapter of the POM) shall be developed as an integral part of the POM and to be found acceptable by the Bank. Training on financial management aspects will also be carried out. (c) Financial Management Information System: a Financial Management Information System which could keep track of payments by (consulting) package, beneficiaries of payment, sources of fund, currency, category of expenditure and component of the project should be developed. An acceptable version shall be developed within 6 months of signing of Financing Agreement. 20 A detailed draft of the POM has been developed and is being finalized 21 D. Procurement 86. Procurement for the proposed project will be carried out in accordance with the Bank's "Guidelines: Procurement of Goods, Works and Non-consulting Services under IBRD Loans and IDA Credits and Grants by World Bank Borrowers" dated January 2011, revised July 2014 ("Procurement Guidelines"), and "Guidelines: Selection and Employment of Consultants under IBRD Loans and IDA Credits and Grants by World Bank Borrowers" dated January 2011, revised July 2014 ("Consultant Guidelines"). 87. The project implementing agency will be the MOET. MOET will establish appropriate project management arrangements, including a management entity, and be responsible for all procurement activities financed by IDA funds. 88. A procurement capacity and risk assessment (PCRA) of MOET was conducted by the Bank team in October 2014 and updated in March 2015. The assessment identified that procurement under the project may risk delays and noncompliance which could arise from: (i) MOET bureaucracy in internal approval/appraisal procedures that would delay the procurement clearance and decision making processes; (ii) management entity's expected weak procurement planning and monitoring, and its tendency to follow the national public procurement procedures rather than the Bank’s Guidelines when there is conflict between them; (iii) lack of clarity on who is responsible for the decisions on selection of individual consultants (ICs) for developing curriculums and textbooks, and how to effectively manage a huge number of the hired ICs; (iv) assigned procurement staff's lack of adequate familiarity with Bank procurement procedures and policies; and (v) possible governance and corruption issues. Given MOET's accumulated experience in the Bank's procurement practices and considering that most of the envisaged international and national competitive bidding packages under the project will be neither very high in value nor very complex in nature, the procurement risk for the proposed project is rated as "Substantial". 89. To mitigate the risks and build up capacity, MOET has agreed to take a number of measures. Details of mitigation measures to address the identified risks, as well as the procurement arrangements for the project are presented in Annex 3. 90. The Bank will provide timely and specific guidance with respect to all procedures. The provision of specific guidance will raise the efficiency and effectiveness of the acquisition of the relevant procurement and administrative capacities of staff working on this Project. E. Social (including Safeguards) 91. As the Bank’s safeguard policy on Indigenous People (OP 4.10) was triggered for this Project, MOET has conducted a Social Assessment (SA) that identifies potential barriers related to curriculum and textbooks for ethnic minority communities. Ethnic minority children are a disproportionate share of both poor children overall and disadvantaged learners. This is true in spite of encouraging recent data showing trends for improvement.21 Additionally, a review of policy documents, existing curriculum and textbooks, and consultations with representatives of 21 See Patrinos and Hall (Editors), Indigenous Peoples, Poverty, and Development, Cambridge University Press: New York, 2012. 22 Ethnic Minority groups, the assessment found that: (i) the frequency of mentions of ethnic minorities in textbooks is low; (ii) the curriculum does not explicitly encourage the introduction of culturally and locally relevant materials; and (iii) the primary language of instruction for ethnic minority students is Vietnamese, while little support is offered in ethnic minority students' mother tongue. 92. The SA also examines issues related to gender equality and fair treatment of students with disabilities in the existing curriculum and textbooks. The assessment found that there is a disproportionately low percentage of content about women, girls, and children with special educational needs in textbooks. Moreover, the study reports that out of 763 textbook authors in the previous curriculum reform, only 171 were women. 93. In order to mitigate such bias, the SA proposes recommendations for policy makers, curriculum and textbook developers, education managers on the development and implementation of bias-free curriculum and textbooks. The key recommendations include: (i) ensuring that bias-avoidance is a criteria for Project deliverables; (ii) building capacity for writers and a selection of teachers and administrators on multicultural and inclusive education; (iii) defining a process for broad community consultation and support; (iv) building staffing models that result in highly-competent and diverse teams; and (v) ensuring that there are equal opportunities to introduce culturally and locally relevant materials into the curriculum. 94. These recommendations have been considered in the design of the Project and have been translated into a bias-avoidance framework that includes an Ethnic Minority Development Plan (EMDP). The Project seeks to build capacity for ethnic minority teachers on implementing the new curriculum, translate a number of selected textbooks into several ethnic minority languages, and develop an on-going research agenda on ethnic minorities' access to and equity in education. Furthermore, as indicated in the POM, bias-avoidance will be a criterion for development and evaluation of the revised curriculum, textbooks, and other pedagogical materials. In addition, recruitment and selection of curriculum and textbooks developers will be a fair and transparent process that prioritizes technical expertise but also promotes group diversity (e.g., with respect to gender, ethnic background, etc.) to ensure that the revised curriculum and textbooks are well- balanced and appropriate for a wide range of learners. The SA, which includes the EMDP, has been disclosed locally in Vietnamese on November 14, 2014 and at the World Bank’s Infoshop in English on January 13, 2015 for public access.   F. Environment (including Safeguards) 95. Component 3 of the Project supports the Development of a NCSDGEQ and a NCFLT. It is expected that the Centers will be constructed and renovated within the current premises of MOET and MOET’s universities. Their exact locations, however, will be finalized during implementation. The Project has been classified as Category B, and the OP 4.01 is triggered as the project may cause adverse environmental and social impacts. During Project preparation, an Environmental and Social Framework (ESMF) was developed by MOET to ensure that activities during implementation are in compliance with the World Bank safeguard policies and national regulations. 23 96. As the locations of the centers have not yet been identified, the site-specific impacts for subprojects will not be known until implementation phase. Nonetheless, according to the ESMF, the key negative impacts -- noise, dust, solid waste, wastewater, traffic and labor safety, and unexploded ordnances -- are likely to be localized, short-term, and moderate. Moreover, such effects can be expected to be managed and mitigated by good design and compliance with construction practices as described in the Environment Codes of Practice (ECOPs), which will be included in the bidding documents and relevant contract documents. Once the sites have been identified and prior to the implementation, the management entity shall ensure the appropriate review of the design of the construction/rehabilitation work. Additional measures for mitigation of adverse impact are described in the ESMF. One such measure is the development of an Environmental Management Plan (EMP) model that is specifically for the Project. Moreover, according to the ESMF, Public Consultation was conducted during the project preparation and will continue to be undertaken during Project implementation as necessary. The ESMF, which includes the ECOP and EMP templates, has been disclosed locally in Vietnamese on October 31, 2014 and at Infoshop in English on November 14, 2014 for public access. G. World Bank Grievance Redress 97. Communities and individuals who believe that they are adversely affected by a World Bank (WB) supported project may submit complaints to existing project-level grievance redress mechanisms or the WB’s Grievance Redress Service (GRS). The GRS ensures that complaints received are promptly reviewed in order to address project-related concerns. Project affected communities and individuals may submit their complaint to the WB’s independent Inspection Panel which determines whether harm occurred, or could occur, as a result of WB non-compliance with its policies and procedures. Complaints may be submitted at any time after concerns have been brought directly to the World Bank's attention, and Bank Management has been given an opportunity to respond. For information on how to submit complaints to the World Bank’s corporate Grievance Redress Service (GRS), please visit http://www.worldbank.org/GRS. For information on how to submit complaints to the World Bank Inspection Panel, please visit www.inspectionpanel.org. 24 Annex 1: Results Framework and Monitoring . Country: Vietnam Project Name: Renovation of General Education Project (P150058) . Results Framework . Project Development Objectives . PDO Statement The Project Development Objective is to raise student learning outcomes by: (i) revising and implementing the curriculum following a competency- based approach; and (ii) improving the effectiveness of instruction by creating and disseminating textbooks aligned with the revised curriculum. These results are at Project Level . Project Development Objective Indicators Cumulative Target Values End Indicator Name Baseline YR1 YR2 YR3 YR4 YR5 YR6 YR7 YR8 YR9 Target Partially A gain of A gain of derived at least at least from 2011 0.12 0.12 and 2014 Baseline standard standard Student learning Baseline Initial test 2nd round Exams test deviation deviation gains (Text) test results test and 1st administer in Math in Math developed analyzed prepared iteration ed and and of the Vietnames Vietnames Project’s e test e test NLSA. scores. scores. Teacher New preparedness for 0 0 3.8 10.9 25 25 curriculu new curriculum 25 (Percentage) m not yet available Equitable access 1.1 times 1.1 times 1.1 times to learning baseline baseline baseline materials (Text) rate rate rate . Intermediate Results Indicators Cumulative Target Values End Indicator Name Baseline YR1 YR2 YR3 YR4 YR5 YR6 YR7 YR8 YR9 Target Core Technical Committe Committee e not yet 12.00 12.00 12.00 12.00 Qualifications seated (Text) System fully System 17.2% of 50% of 50% of No functionin Online Training pilot 1 teachers teachers teacher g System system year after have had have had have had 24 months Readiness currently curriculu at least at least at least after (Text) exists m some some some curriculu approved training training training m approval Curriculum Champions 0.00 2500 6250 8000 8000 8000 8000 (Number) No new 72 = 1 per Textbook textbooks grade per Production 0 0 24 72 have been 48 academic (Text) approved subject 26 30 30 No pilot classroom classroom available s per Curriculum s per grade + 30 Piloting until the 0 0 TBD TBD grade + 30 classroom (Text) revisions classroom control are made control monintore monitored d Exams administer ed (1 in 2 exams 2 exams 2 exams each of Exams Current piloted piloted piloted the three administer Expanded system (for 2 of 3 (for 2 of 3 (for 2 of 3 education ed at 5th, National Large- measures education education education levels – 9th, and Scale learning levels – levels – levels – primary, 11th Assessment of high primary, primary, primary, lower, and grades for System school lower, and lower, and lower, and upper nationally- (Text) leavers upper upper upper secondary representa only secondary secondary secondary ) for tive ) ) ) nationally- sample representa tive sample Direct project beneficiaries 1600000.0 5000000.0 5000000.0 0.00 0.00 0.00 0.00 (Number) - 0 0 0 (Core) Female beneficiaries (Percentage - 0.00 0.00 0.00 0.00 50.00 50.00 50.00 Sub-Type: Supplemental) - (Core) 27 Number of additional qualified primary teachers resulting from 0.00 0.00 0.00 32500.00 92500.00 150,000 150,000 project interventions. (Number) - (Core) System for learning assessment at the primary No No No No Yes Yes Yes level (Yes/No) - (Core) Utility of the learning assessment system 1.00 1.00 1.00 1.00 2.00 3.00 3.00 (Number - Sub- Type: Supplemental) - (Core) . 28 Indicator Description . Project Development Objective Indicators Responsibility for Data Indicator Name Description (indicator definition etc.) Frequency Data Source / Methodology Collection Student learning gains Students taught under the new curriculum Periodic Exams management entity/MOET (Text) will show an increase of at least 0.12 exams standard deviation in Math and starting one Vietnamese test scores compared to the year after baseline cohort. new curriculum implemented Teacher preparedness for 25 percent of teachers implementing the Annual once Online training management entity/MOET new curriculum new curriculum will be certified as the training (Percentage) prepared for curriculum implementation. system ready Equitable access to learning In the bottom 10 percent of schools (based Annual once School survey management entity/MOET materials (Text) on income distribution), textbooks will be textbooks are distributed to raise the ratio of student to ready textbook ratio to 1.1 times its baseline rate. . Intermediate Results Indicators Responsibility for Data Indicator Name Description (indicator definition etc.) Frequency Data Source / Methodology Collection Core Technical Committee Average Years of professional Experience Yearly CVs of CTC management entity/MOET Qualifications Per member Members Online Training System Milestones for systems development and Yearly Review of contractor management entity/MOET Readiness implementation as per the Results progress and online system Indicators table above use statistics Curriculum Champions Reported Annual Average Person-hours of Yearly Management instruments & management entity/MOET training provided surveys 29 Textbook Production Approved textbooks Yearly after Reports from the Textbook management entity/MOET year two of appraisal committee implementati on. Curriculum Piloting Schools/Classes Testing the New Intensive Trial management entity/MOET Curriculum piloting for 12 months Expanded National Large- Large scale exams successfully 2nd Iteration Exams NCSDGEQ/MOET Scale Assessment System implemented of Exams for all three grade levels Direct project beneficiaries Students, teachers, and education official Yearly Various sources management entity/MOET who master the principles of the revision and/or initiate implementation of the new curriculum Female beneficiaries 50% of overall beneficiaries Yearly Various sources management entity/MOET Number of additional Primary education teachers comprise Yearly once Online system use statistics management entity/MOET qualified primary teachers roughly 40% of all teachers. This new resulting from project indicator will be expected to be 0.4 times curriculum interventions. the overall number of teachers certified as and trained in the new curriculum. certification system are available System for learning This is a binary (“Yes” or “no”) Yearly Exam data management assessment at the primary measurement indicating the existence of entity/MOET level the system. Utility of the learning Tiered system scale as per World Bank Yearly Assessment of exam management assessment system Core Indicator guidance when utility entity/MOET system in place 30 98. Several technical issues related to Project indicators will be addressed during the construction of indicator baselines. The baseline for indicator 1 will be constructed by analyzing learning outcomes from existing standardized test microdata at the 5th grade and 12th grade levels. Anchor items from these baselines will be reused in subsequent testing to discern changes to learning outcomes. These anchor items will be incorporated in the expanded NLSAS (output # 2) to do this. The gains themselves will rely on outputs numbers 1, 3, and 4 above. The analysis itself comes under output # 5, so this indicator will therefore encompass or contain information related to all aspects of the PDO. 99. Indicator 2 will measure the impact of output # 2 above – the extent of understanding among teachers of the new curriculum and its implications for classroom practice. The online teacher training system will be used to determine if the minimum levels of acceptable performance have been attained. Teachers who attain these minimum levels or above will be certified. As the certification is new, the baseline for this indicator is the null situation of no teachers certified for the new curriculum. 100. Indicator 3 requires technically sound surveys of teaching and learning materials, both for the construction of baselines and for the measurement of progress under the Project. Surveys will be conducted immediately following appraisal. 31 Annex 2: Detailed Project Description VIETNAM: Renovation of General Education Project 101. The Project Development Objective (PDO) is to raise student learning outcomes by (i) revising and implementing the curriculum following a competency-based approach; and (ii) improving the effectiveness of instruction by creating and disseminating textbooks aligned with the revised curriculum. To achieve these aims, the Project will have four components. Component 1 will advance the revision of the curriculum and build capacity for teachers and Curriculum Champions to implement the revised curriculum. Component 2 will support the development of appropriately aligned pedagogical materials, including the conversion of selected textbooks into bilingual textbooks for several ethnic minority languages. Component 3 will improve and expand the National Large-Scale Assessment System (NLSAS) and research capacity in order to inform policies for General Education. Component 4 supports Project Management and Monitoring and Evaluation. Component 1: Curriculum Development (US$17.5 million, including US$1 million contingencies). 102. The objectives for Component 1 are to: (i) develop a revised curriculum; (ii) deliver online and face-to face training so that teachers are willing and able to teach the revised curriculum; and (iii) write guidance for publishers to create textbooks and other pedagogical materials that are aligned with the curriculum. To achieve these objectives, the project will establish a Core Technical Committee (CTC) and Subject-Specific Technical Committees (SSTCs) to carry out the revision of the curriculum. The CTC and SSTCs will then oversee the development of an interactive online training tool to build teacher capacity on implementing the new curriculum. The CTC and SSTCs will also draft the guidance for textbook publishers on how to develop materials that are aligned with the direction and content of the new curriculum. Subcomponent 1.1: Curriculum Development 103. The Project will support the revision of the curriculum. The “curriculum,” for the purposes of the Project, is defined as the General Curriculum (GC) and Subject-Specific Curricula (SSC). The GC comprises the following components: (i) the features of the ideal upper secondary school graduate; (ii) the subjects to be taught at each grade and their classification as either compulsory or elective; (iii) time allocations for each subject by grade level (i.e., the number of instructional periods devoted to each subject per academic year); and (iv) definitions of core competencies and qualities, as well as their proficiency levels at each grade band endpoint. 104. The SSC comprises the subject-specific learning outcome standards that are accompanied by examples of learning opportunities or experiences for teachers to consider employing. The learning standards should be designed in such a way that ensures teachers’ and parents’ understanding of the learning goals and provides the right level of guidance for teaching and assessment. This sub-component will be implemented through the following activities. 32 Core Activities for Curriculum Revision 105. The curriculum revision process will take up to 15 months from the start of the validation process for the GC to the adoption of the curriculum in its entirety by the MOET. Within two weeks after the Project becomes effective, the CTC will begin the validation process for the GC. The validation process will be completed within 3 months, and the drafting of the SSC within 9 months. The remaining 3 months will be spent on consultation, appraisal, and adoption. Throughout the revision process, there will be continuous review and adjustment of both the GC and SSC to ensure that the two are appropriately aligned. The core activities of the curriculum revision process are: (i) Validation, Revision, and Writing. The majority of the work on curriculum revision will take place during the first 12 months of the Project. CTC members will first review the draft of the GC and decide whether final adjustments are needed. In validating the draft, the CTC may suggest minor changes. However, in the absence of any technical reasons to introduce major changes, the CTC is not expected to do so. The CTC will also lead the development of the (SSC) to ensure that both will be completed within one year, after which there will be three months of consultations and public review. Once a reasonable level of agreement has been reached in regards to the ideal graduate profile, the schedule of compulsory and elective subjects, the allocation of time per subject, and definitions and proficiency levels of competencies, CTC members will begin to work with their respective SSTC members to draft the SSC. (ii) A key task for the SSTCs to accomplish before developing the learning standards will be to conduct a thorough review of the existing curricula. The purpose of the review is to identify elements that should be preserved and elements that should be revised. There are two approaches to this task. The first is to benchmark Vietnam’s curriculum against that of top performing countries and to create curriculum comparison maps. The second approach is “back-mapping.” Back-mapping first requires the identification of the minimum knowledge, skills, and competencies that a student graduating from upper secondary school must possess. The SSTC will then back-map to determine what students must know and be able to do at the end of each grade band endpoint. Doing so will allow SSTCs to analyze where the curriculum falls short of the learning goals or is excessively demanding. SSTCs will also consider the extent of transferability and repetition of competencies across subjects and grades so as to prevent redundancy and overloading of the curriculum. (iii) Consultation. Given the public nature of the reform effort, there is a clear need for wide-ranging and extensive consultations with a variety of stakeholders. Domestic and international experts on the development of curricula, pedagogical materials, and assessment will be involved throughout the curriculum revision process. Moreover, parents, teachers, students, civil society organizations, government officials, and citizens from all parts of the country and all socioeconomic backgrounds may be consulted in order to ensure that the curriculum reflects the viewpoints and values of Vietnamese society. Such consultations will be accomplished through a series of reviews (internal, independent, public, and fidelity reviews) held in provinces across the country. The events may be conducted both in-person and/or virtually and may 33 leverage a wide variety of methods and media, including, inter alia, surveys, interviews, focus groups, and large meetings. The CTC and SSTCs will, as appropriate, incorporate findings from these consultations into the curriculum. (iv) Appraisal and adoption. The curriculum will be appraised by a committee that is independent of standard-development responsibilities. After the CTC and SSTCs have incorporated comments from the various consultations and public reviews, the CTC will submit the revised curriculum to the SC. Once the SC believes that the curriculum is of sufficient quality to be appraised, the SC will send the curriculum to the Appraisal Committee for review and appraisal. In the case that the Appraisal Committee recommends revisions for the curriculum, the validation report must clearly indicate which criteria have not been met, and send the recommendations directly to the SC. The SC may or may not accept the Appraisal Committee’s recommendations. In the case that the SC agrees with the Appraisal Committee’s recommendations for revisions, the SC will send the Appraisal Committee’s comments to the CTC for the latter to revise and resubmit. When the SC is satisfied that the curriculum meets the required levels of quality and is consistent with Government/Ministry policies and the goals of the Project, they will authorize its official adoption. Ancillary Activities for Curriculum Revision 106. The core work of the CTC and the SSTCs will be supported, as needed, by activities in capacity building and research that will be undertaken throughout the revision process. 107. Capacity building. The CTC will lead all technical aspects of the curriculum revision and ensure that subsequent pedagogical resources are appropriately aligned with the founding documents. In preparation for such work, CTC members will participate in intensive capacity and consensus building sessions in Vietnam and abroad. The program will begin with a two to three- week retreat abroad and followed by a series of week-long sessions in Vietnam. Such sessions will provide an opportunity for CTC members to interact with international advisors and with their CTC colleagues. 108. Research. Curriculum revision will draw on available research to the extent feasible. Before the curriculum revision process begins, the CTC will work with their respective Subject- Specific Curriculum Subcommittee to refine the research agenda proposed in Component 3 of the Project’s Implementation plan. Core questions include, inter alia, learning progression, curriculum development and implementation, pedagogical materials, professional development, and assessment at the levels of classrooms, schools, and the education system as a whole. Findings from the research projects will provide evidence-based guidance for improving learning outcomes. An elaboration of the research topics is provided in the Research section of Component 3. 109. Subcomponent 1.2: Curriculum Implementation. In order to ensure that the revised curriculum is implemented appropriately in the classroom, the Project seeks to develop the capacity of teachers to translate the written curriculum into classroom practices for effective learning. This will require teachers to have a deep understanding of the rationale and motivation behind the reform, the new concepts and terminologies, the elements and approaches that are different from those of the existing curriculum, and teachers’ critical role in transmitting the new 34 curriculum to students. This sub-component supports the training for teachers and education managers to develop a thorough understanding of these key aspects of the reform. 110. There are two main modes of delivery for the training. The first is an interactive, online training system that includes a series of self-paced training modules. At least a third of teachers and education managers are required to complete the training modules and obtain a certificate as proof of completion and proficiency with the new curriculum. The second mode of training is an in-person element that is offered to a select number of Master Teachers and Curriculum Champions. 111. 1.2.1 Online training system. The online training system may include the following elements: (i) A diagnostic test of teachers’ current knowledge of the new curriculum; (ii) Self- paced training modules on rationale and motivation for the reform, definitions of key terminologies for the reform, with emphasis on competencies and the desired proficiency levels for each subject and grade band endpoint, and classroom practices and pedagogical techniques that promote competencies; and (iii) forums to exchange ideas with other teachers on what they have learned and how they might apply the knowledge to their own classrooms. 112. The system will be able to track individual teacher progress, support individual teacher profiles, and collect data on teacher credentials and performance. As such, the system requires each teacher to recreate a login. Data will be made available for research on teacher effectiveness, with confidentiality of individual teachers’ information strictly enforced. 113. Teachers who complete all the modules will receive a certificate as verification of their completion of the training and proficient understanding of the new curriculum. 114. 1.2.2 Face-to-Face training. A select group of Master Teachers and Curriculum Champions (about 1 per school) receive face-to-face. Master Teachers are top performing teachers who are given additional training to support their colleagues in applying the competency-based approach to specific subjects. Curriculum Champions are given more comprehensive, general training on how the revised curriculum will affect the school experience. Approximately 85 percent of all Curriculum Champions are selected from the pool of top performing teachers, with the remaining 15 percent comprising exceptional administrators, Bureaus of Education and Training (BOET), and DOET officials who are nominated by their superiors for having great potential to be ambassadors of the reform. They work closely with school administrators to implement the new curriculum, including supporting the development of local content (20 percent of curriculum). 115. Master Teachers and Curriculum Specialists also collaborate in training the remaining teachers on how to access and navigate the online training system. The level of distinction between Master Teachers and Curriculum Champions varies by education level, as each level requires different types of support. In primary schools, for example, the roles of Master Teachers and Curriculum Champions might be less differentiated because the curriculum at that level is more general in nature and thus requires less support from subject-specific specialists. 35 116. 1.2.3: Guidance for Developers of Textbooks and Other Pedagogical Materials. In order to ensure that textbooks and other pedagogical materials abide by the principles of the reform and aligned with the learning outcome standards, CTC and SSTC members will write guidelines for selecting, writing, and evaluating quality textbooks and other pedagogical materials. The CTC and SSTC for each subject are responsible for producing one set of guidelines for the respective subject that runs from grades 1 – 12. The criteria articulated in the guidelines will concentrate on the most significant elements of Vietnam’s new learning outcome standards. They are not meant to dictate classroom practices, but rather to help teachers receive the appropriate tools by guiding publishers to be purposeful and strategic in both what to include and what to exclude in the instructional materials. 117. The guidelines will cover a range of topics, including, inter alia, content, learning and teaching, structure and organization, language, and layout. The section on Content, for example, might recommend that the objectives align with those stated in the General and SSC, information is up-to-date, concepts are accurate and precise, treatment of the subject balances between depth and breadth, multiple perspectives are offered on issues, and the level of difficulty is consistent with the cognitive level of students. Whenever appropriate, the guidelines will emphasize the approaches undertaken in the reform. For example, for the section on Learning and Teaching, the guidelines will discuss the need for balanced coverage of core competencies, such as Problem- Solving, Creativity, Collaboration, Self-Management, and Self-Study. These guidelines should be of comparable quality, breadth, and depth to those designed by top performing countries. Component 2: Development and Dissemination of Aligned Textbooks (US$21.6 million, including US$1 million contingencies) 118. Under the current education system, there exist approximately 128 textbooks - at least one textbook per subject per grade. Each textbook is also accompanied by a workbook and a teacher’s guide. A workbook is essentially a book of practice exercises for students. The teacher’s guide provides general guidelines for pedagogical techniques to be employed as well as specific lesson plans for each lesson. In this round of Reform, MOET is considering eliminating the development of workbooks, which it may encourage independent publishers to develop. 119. This component aims to support: (i) the development of a set of textbooks by MOET and accompanying pedagogical materials; (ii) the conversion of a select number of primary school textbooks into bilingual textbooks for several ethnic minority languages that have written form; and (iii) the production and dissemination of textbooks and other materials to disadvantaged schools. Subcomponent 2.1: Development of Textbooks Aligned with the New Curriculum 120. Textbook Development. This Project supports the development of one set of textbooks from grades 1 – 12 for all subjects in Vietnam’s national curriculum. For each subject, there is a Chief Grade Band Editor who oversees the development of the textbooks for each of the four grade bands (Grades 1-3, 4-5, 6-9, 10-12). Such structure ensures that the content of the textbooks align well with the curriculum and that there is appropriate grade-to-grade progression. The Chief Grade 36 Band Editor manages a team of writers and works in close coordination with other Chief Grade Band Editors to ensure coherence across grade bands and subjects matters. 121. The number of writers within each team depends on the subject and grade band. For example, the number of writers for Science in lower grade bands, when Science is treated as an integrated subject, would be fewer than that in Upper Secondary School, when Science is taught as distinct subjects – Chemistry, Physics, and Biology. Similarly, while subjects such as Mathematics and Vietnamese Language might require extensive textbook development, Physical Education might not require any written texts. As such, the number of writers for each subject and grade band will vary with need. The Chief Editor, in close consultation with the Chief Technical Officer and the management entity director, will determine the number of writers required for his or her respective subject based on the level of complexity of the content and anticipated length of the textbook. 122. It is expected that the majority of the writers will be former SSTC members who transition into the new role as textbook authors. Encouraging SSTC members to remain with the project through textbook development will raise the level of coherence and consistency between the Subject-Specific Curriculum and textbooks. 123. In order to meet the stated timeline, the committees of textbook developers will begin their work at least six months prior to the anticipated adoption date of the curriculum. Activities during this preliminary phase will include: (i) a careful review the existing textbooks to determine areas that require revision; (ii) development of an initial plan for revising textbook content, chapter organization, and design in cases where content does not align with learning outcome standards; and (iii) creation of a project schedule and revision of the content development plan to reflect input from authors and educators. 124. Once the curriculum has been officially adopted, textbook developers will: (i) develop the detailed outlines and make writing assignments; (ii) create page and cover design for textbooks and all ancillary materials; and (iii) plan teacher editions and ancillary materials. Textbook authors will ensure to check facts from at least two independent sources and review for accuracy and non- bias. There will be a well-defined process for quality review, verification, proofreading, and revision. The process of textbook development will take no longer than one year from appraisal of the curriculum to being ready for printing. 125. Before production, textbooks will be appraised by a group of independent appraisers whose responsibilities are independent from drafting. To ensure that textbooks align well with the curriculum, the Chief Technical Officer, in close collaboration with the management entity director, will empanel the Textbook Appraisal Committee to include experts on the subject matter, pedagogy, assessment, and textbooks. Some of the members of the Committee may be CTC members. 126. Supplementary Materials. This subcomponent will also finance the development of supplementary materials, which for the purposes of the Project are defined as written texts designed to promote learning of the curriculum. Such materials may include storybooks for early readers, workbooks, teacher’s guides, and/or lists of suggested resources. Provision of storybooks 37 to schools for early readers is not a practice in Vietnam. Therefore, the Chief Technical Officer and the management entity director will have to deliberate and decide on the extent to which these will be developed and provided for under the Project. 127. In regards to teacher’s guides, the existing materials offer very explicit instructions on how teachers should teach the content. As stated in Component 1, STC and SSTC members will ensure that the new pedagogical materials developed under this Project will provide a sufficient level of guidance to teachers while at the same time giving them the flexibility to adapt the material to the specific teaching and learning conditions. For example, the new teacher’s guides might include suggestions for additional resources for teaching, typical stumbling blocks for students, common questions and sample answers, and guidance on designing assessments. These materials will undergo appraisal and adoption processes similar to those of textbooks. 128. MOET will have copyright ownership of the textbooks and other materials that are developed by MOET under the Project. 129. Production and dissemination of Textbooks and Related Materials for Ethnic Minority Learners. Vietnam’s population comprises 54 ethnicities, with the Kinh ethnicity being the majority and accounting for 85 percent of the population. While ethnic minorities only account for 15 percent of the population, they make up 50 percent of the nation’s poor. In regards to education, ethnic minority students score much lower than do their Kinh counterparts in mathematics and reading tests. Language seems to be a significant contributing factor to this discrepancy, as ethnic minority children whose mother tongue was Vietnamese scored higher than those whose mother tongue was not Vietnamese. In order to ensure that all children in Vietnam receive equal access to the new textbooks, this sub-component seeks to convert a selection of textbooks into bilingual textbooks for several ethnic minority languages for which there is written form. Moreover, the textbooks will be printed and disseminated to ethnic minority learners who are also economically disadvantaged. The subjects and languages chosen for translation will be decided on under the Project in closer consultation with the ethnic minority leaders and communities. Subcomponent 2.2 Provision of Textbooks to Disadvantaged Schools 130. Currently in Vietnam, families are responsible for purchasing textbooks for their children. For families from low socio-economic backgrounds, the cost of textbooks could be a high percentage of total family income. Under this sub-component, the Project will print and disseminate textbooks to students in targeted provinces with high concentration of disadvantaged students, including ethnic minority students, students with disabilities, and those from families of low socio-economic backgrounds. The books will be distributed to school libraries, where they will be on loan for students for the duration of the academic year. The books are expected to have a life-span three years, so redistribution might be necessary during the project. 38 Component 3: Learning Assessment and Analysis for Continuous Improvement of Curriculum and Policy (US$35.4 million, including US$1 million contingencies) 131. In order to promote the accumulation of capacity to measure learning outcomes are measured and analyze these for continuous improvement of curriculum and policy, this component will support: (i) the establishment of the National Centre for Sustainable Development of General Education Quality (NCSDGEQ) and the National Center for Foreign Language Testing (NCFLT); (ii) capacity-building in curriculum development and assessment for the Centers’ staff and relevant officials; (iii) expansion of the NLSAS; and (iv) consultations/outreach campaigns. A significant focus of this component will be to understand better and narrow the learning achievement gaps that exist within the Vietnamese public education system. 132. Subcomponent 3.1: Establishment of NCSDGEQ and NCFLT Centers. To facilitate the research and measurement activities discussed above, the Project will support the development of a NCSDGEQ. The purpose of the center is to: (i) create a community of expert education researchers who will conduct short-term research projects and other analyses that support the revision of curriculum and textbooks or related issues and serve as a think-tank that promotes a robust education research agenda in Vietnam. 133. National Center for Foreign Language Testing - NCFLT. The purpose of the Center is to promote capacity in foreign languages for students by the end of high-school to prepare them for tertiary education and/or the workforce. The Center will conduct research on effective foreign language instructions, compile teaching and learning resources, and investigate the demand for foreign languages in different academic and labor market settings in order to tailor teaching programs. The Center will also be responsible for designing and administering assessment of proficiency in foreign languages for students at the upper secondary level. These exams may be used for selection into academic institutions or for employment. 134. Construction of the NCSDGEQ and NCFLT. To facilitate the work of the research Centers, the Project will support the construction of six buildings on five plots of MOET-owned land. In Hanoi, two buildings – one for each main center (NCSDGEQ, NCFLT) will share one plot of land. In each of the remaining locations, the satellite offices will share a building. All constructions will be permanent structures, with a minimum of expected lifespan of at least twenty years and meeting the requirements of the Project’s Environmental Social Management Framework (ESMF). For each center, the Project will finance the construction or rehabilitation, and procurement of furniture and supplies so that the center may operate. If the buildings are to be constructed instead of rehabilitated, all buildings will share the same design. 135. For the first three years while the Centers are being constructed or rehabilitated, the MOET will rent a location to house both the management entity and the research centers. 136. Subcomponent 3.2: Research and Capacity-Building on Curriculum Development and Assessment. The NCSDGEQ will be committed to undertaking and disseminating unbiased education research among educators, policymakers, business professionals, and the academic community. The Center’s greatest asset will be its reputation for scholarly integrity and technical 39 quality. Publications by the NCSDGEQ must meet standards for international peer-reviewed research journals. 137. The Center’s research agenda is expected to be well-defined and focused on key research programs that correspond loosely to traditional fields of study within the field of education, such as Curriculum, Pedagogy, Assessment, Equity and Access, and Economics of Education. Each research program is organized into four to five working groups, each led by a Research Coordinator. Researchers might be affiliated with one or more research programs. Overlapping membership on research programs will ensure that topics are addressed from multiple perspectives and based on a diversity of expertise. The researchers in most programs meet at least once a month at program meetings and participate in the Center’s annual summit. Research findings by NCSDGEQ researchers will be used to inform both broad policy reforms and more locally-targeted interventions. 138. Staffing and Coordination. The Chief Technical Officer, in close consultation with the management entity Director, will identify an individual who can be assigned to serve as the Center Director and identify other individuals who will be associated with the center and its research activities. 139. In order to achieve coordination, alignment, and cost-effectiveness, the Center will find synergies and work in tandem with existing research and assessment institutions, including, inter alia, the Vietnamese National Institute of Education Sciences, Hanoi National University of Education, and General Department of Education Testing and Assessment. Because these institutions are already addressing some of these issues relevant to the Center’s mission, it would be prudent for the center’s staff to be seconded from these institutions at the nascent phase of the Project. 140. Visiting Researchers. To support the Project’s aim to significantly raise the quality of the Center’s research capacity, the Center seeks to host Visiting Researchers who are internationally recognized as leaders in their The Visiting Researcher may be also asked to lead training workshops on international practices and various research methodologies. 141. Capacity building on assessment. To support the administration of the national large-scale assessment (NLSA) (subcomponent 3.3), the project will finance consultants and workshops to provide technical assistance on development and administration of assessment instruments and analysis of results. Such technical assistance aims to improve the coverage, technical quality, and alignment of the national assessment system with the curriculum. As such, the primary activities of this sub-component will include assistance in: (i) institutional building, such as setting of goals, planning, staffing requirements, timelines, development of examination blueprints, and development of required architecture for all examinations; (ii) technical issues related to Item Response Theory (IRT), equating designs, vertical scaling planning, and other psychometric issues; and (iii) logistics and other administration procedures, such as field testing, security procedures, test reporting and score use. 142. Training will be delivered to relevant MOET, DOET, and BOET officials as well as to content experts and psychometric staff, particularly staff of the Curriculum Development and 40 Assessment Center. This training will build on the capacity building that has taken place under the Russian Education Aid for Development (READ) Trust Fund. 143. Subcomponent 3.3: Expansion of the National Large-Scale Assessment System (NLSAS). This subcomponent will improve the coverage, technical quality, alignment, validity, and reliability of the national assessment system. More specifically, it will support: (i) integration of curricula and assessment; (ii) closer alignment of assessment with learning goals; (iii) training for educators and administrators at various levels of the system to design and implement assessments and analyze and report the results; and (iv) creation of specialized expertise within MOET about continuously improving large-scale assessment capacity. In addition, this sub-component will finance the implementation of two rounds of learning assessment in each of the education levels (Primary, Lower, and Upper Secondary) that are of the desired level of quality and incorporate the assessment of core competencies. 144. In 2009, under the auspices of READ Trust Fund, Vietnam conducted a self-diagnosis of the assessment system. That diagnosis revealed that Vietnam’s NLSAS is still “emerging.” Even though MOET’s Circular No 51/2011/TT-BGDĐT (dated December 2011) authorized NLSAS’ every three to five years in Math and Vietnamese language/literature for students in the 5th, 9th, and 11th grades, MOET is inadequately staffed to carry out NLSAs on a regular basis. Instead, it relies on staff from the Vietnam National Institute of Education Sciences (VNIES) and universities. Furthermore, the SABER report revealed that the NLSA program has irregular funding that is only provided by donors. 145. Measurement of Learning Outcomes. This Project seeks to expand the NLSAS in order to measure the effectiveness of the Project and the Reform on improving student learning outcome throughout the duration of the Project and beyond. Prior to the implementation of the revised curriculum, there will be several sources for baseline data. One such source will be the results of high-school graduation exams in the past three years. As the reliability and validity of these exams have not been analyzed, the project will finance the item-level analysis of these exams. Low- quality items will be eliminated from the dataset. The remaining items, particularly those related to the core competencies, will be used to indicate the level of student performance prior to the revision of the curriculum and textbooks. Another source of baseline data may include the first round of sample-based NLSAs at each education level (Primary, Lower, and Upper Secondary) in Math and Vietnamese administered during the first two years of Project Implementation. During Year 5 of the Project, a round of end-line data will be collected, which will include the high-school graduation exam as well as the sample-based NLSAs at the three education levels. 146. Prior to the implementation of the NLSAs, NCSDGEQ staff will design a work plan that indicates the policy questions to be addressed, specify the target population, identify the curriculum areas or constructs to be assessed, describe the data collection instruments, and assign responsibility for developing the instruments, and delegate staff for analyzing the data and producing reports. 147. Administration of NLSA. As mentioned, this sub-component will finance the development and administration of two rounds of NLSAs for each of the education levels (Primary, Lower, and 41 Upper Secondary). Doing so will ensure that the NLSAs are implemented in a timely fashion, are of the desired level of quality, and incorporate the assessment of core competencies. 148. To administer the NLSA, NCSDGEQ Director will assign responsibility for preparing the tests, supplementary questionnaires for teachers and parent, and administration manuals. The tests and supporting documents will be pilot-tested by the NCSDGEQ, and subsequently reviewed by relevant MOET officials and other competent bodies. The NCSDGEQ will also coordinate the logistics of the NLSA administration, including sampling, printing of materials, training test administrators, and ensuring tight security. 149. After the two rounds of NLSAs, the NCSDGEQ will be responsible for collecting survey instruments, scoring, cleaning and preparing data for analysis, and producing a report. The report will: (i) describe the national levels of learning achievement; (ii) compare achievement levels across geographic locations, over time, and among key subgroups, such as boys and girls, ethnic minority groups, urban and rural students, and public and private school students); (iii) provide insights on how assessment results relate to curriculum, textbook, and teacher training issues; and (iv) relate achievement to factors such as household characteristics, levels of teacher training, teachers’ attitudes toward curriculum areas, teacher knowledge, and availability of teaching and learning materials. After the NCSDGEQ submits the draft report to the MOET for approval, the report can be used by educators to inform instruction and by policymakers to shape policies on accountability, school improvements, and individual student and/or group performance. 150. Subcomponent 3.4: Consultation and Outreach Campaign. This subcomponent will develop and implement a Consultation and Outreach campaign to promote the public’s awareness of the reform and to instill public trust in the transparency and meritocracy of the reform process. The key messages of the reform will be delivered through several forms of media, such as print, television, word-of-mouth and social media. The primary source of information, however, will be the MOET website devoted to explaining the reform to the public. 151. The management entity, which is responsible for operationalizing the above objectives, will solicit the cooperation of teachers, school administrators, parents, and students to serve as spokespeople for the reform. The messages of the reform will be delivered through several forms of media, such as print, television, word-of-mouth and social media. The primary source of information, however, will be the MOET website devoted to explaining the reform to the public. The site might include major project documents, frequently asked questions, common myths about the reform, and videos of educators and parents speaking frankly about the strengths and shortcomings about this round of reform. The management entity will also identify esteemed individuals within the education sector to serve as ambassadors for the reform. Component 4: Project Management (US$2.5 million) 152. This component combines Project Management with Monitoring and Evaluation (M&E). 153. Subcomponent 4.1: Project Management. This subcomponent will finance the incremental costs associated with project management, including activities designed to ensure efficient 42 program management and early identification of corrective measures to solve any implementation challenges. 154. Subcomponent 4.2: Monitoring and Evaluation. This subcomponent supports the development and implementation of Project M&E in order to ensure that the determined indicators are achieved and to continuously identify areas of the Project that can be improved. 43 Annex 3: Implementation Arrangements VIETNAM: Renovation of General Education Project Project Institutional and Implementation Arrangements 155. In order to achieve coherence, focus, rigor, and alignment in the revised curriculum and textbooks, the Project seeks to promote effective and strategic collaboration and coordination among and within working groups. The management entity headed by a Director, will oversee all Project activities, including financial and procurement management. The Chief Technical Officer will head all technical matters. The Core Technical Committee (CTC), Subject-Specific Technical Committee (SSTC), Curriculum Appraisers and Textbook Appraisers, and the staff of the National Centre for Sustainable Development of General Education Quality (NCSDGEQ) will execute the day-to-day work on the revision of the curriculum and textbooks. International experts will be sought to provide technical assistance in areas wherein MOET or national experts lack capacity. Horizontal and vertical membership overlaps are encouraged, except where there might be conflict of interest, in order to strengthen coherence and alignment among various Project activities. 156. Authorization. As the Project is a part of a larger education reform in Vietnam, it is under the purview of the MOET, which will ensure that curriculum and textbook-related project deliverables are consistent with Government/Ministry policies and the goals of the Project. When deliverables are of the desired quality, they will be transmitted to the Minister of Education for consideration and official adoption. 157. Project Management. The government will ensure that appropriate project management arrangements are in place and that they are satisfactory to the Bank. Such management arrangements may include an entity staffed by MOET employees and consultants to coordinate, manage, and implement project activities (administration, financial and procurement management, accounting, and interpretation/translation, etc.). Once the curriculum, textbooks, and other pedagogical materials have been developed, the management entity will coordinate their printing and dissemination to disadvantaged students, including ethnic minority students, students with disabilities, and those from families of low socio-economic backgrounds. In regards to the establishment of the NCSDGEQ, the management entity will be responsible for the construction/rehabilitation of the physical infrastructure. Lastly, the management entity will arrange all logistical matters related to the Public Media Campaign. 158. Curriculum Revision. All technical matters will fall under the purview of the technical committees. The CTC is responsible for the validation of the GC and for chairing the respective SSTC. The work of the CTC and the SSTCs will ensure that the curriculum is coherent and coordinated across subjects and years. Once the curriculum has been revised, the CTC and SSTCs will oversee the development of an interactive online training tool to build teacher capacity in implementing the new curriculum. Also, the CTC and SSTCs will draft the guidance for textbook publishers on how to develop materials that are aligned with the direction and content of the new curriculum. Once the curriculum has been adopted, the roles of the CTC and SSTCs will transform into textbook appraisers and textbook editors, respectively. Such a structure would allow SSTCs 44 to translate the learning standards into textbooks, and for the CTC to ensure that the textbooks are well aligned with the General and SSC. 159. Training of Teachers and Curriculum Champions. CTC and SSTC members will work with the selected consultant to develop the online training system to ensure teachers are well prepared to implement the new curriculum. Teachers who complete all the online training modules will receive a certificate as verification of their completion of the training and proficient understanding of the new curriculum. Approximately 85 percent of all Curriculum Champions will be selected from the pool of top performing teachers, with the remaining 15 percent comprising exceptional administrators, BOET, and DOET officials who are nominated by their superiors for having great potential to be ambassadors of the reform. 160. Appraisal. A committee that is independent of standard-development responsibilities will appraise the curriculum. CTCs will appraise the textbooks. 161. Textbook Development. For each subject, there will be a Chief Grade Band Editor who will oversee the development of the textbooks for each of the four grade bands (Grades 1-3, 4-5, 6-9, 10-12) to ensure that the content of the textbooks align well with the curriculum and that there is appropriate grade-to-grade progression. In doing so, he or she will manage a team of writers within the respective grade band. The Chief Grade Band Editor will also work in close coordination with other Chief Grade Band Editors to ensure coherence across grade bands and subject matters. It is expected that the majority of the writers will be former SSTC members who transition into the new role as textbook authors. Encouraging SSTC members to remain with the project through textbook development will raise the level of coherence and consistency between the Subject- Specific Curriculum and textbooks. 162. Pilot and rollout. During the pilot phase, CTC and SSTC will conduct field tests of certain sections of the curriculum and their corresponding textbook materials as they become available within live classrooms. Field testing will involve a selection of teachers to teach the chosen materials to their students for a certain period of time as designated by the CTC and SSTC. Staff of the NCSDGEQ will support the field testing by creating the sampling plan as well as developing questionnaires for administrators, teachers, parents, and students to collect both quantitative and qualitative data on student learning and quality of the lesson as perceived by survey respondents. The NCSDGEQ staff will also be responsible for analyzing the results and providing recommendations for revising the tested materials. Management entity staff will assist in coordinating with school districts, schools, and teachers to arrange for the field tests. They will also help with preparing and printing materials as requested by the NCSDGEQ staff. 163. NLSA. NCSDGEQ staff will design a work plan that indicates the policy questions to be addressed, specify the target population, identify the curriculum areas or constructs to be assessed, describe the data collection instruments, and assign responsibility for developing the instruments. The NCSDGEQ will also coordinate the logistics of the NLSA administration, including sampling, printing of materials, training test administrators, and ensuring tight security. After the two rounds of NLSAs at each education level (Primary, Lower, and Upper Secondary), the NCSDGEQ will be responsible for collecting survey instruments, scoring, cleaning and preparing data for analysis, and producing a report. 45 164. To support the administration of NLSA, the project will finance consultants to provide technical assistance on development and administration of assessment instruments and analysis of results. Such technical assistance aims to improve the coverage, technical quality, and alignment of the national assessment system with the curriculum. Training will be delivered to relevant MOET, DOET, and BOET officials as well as to content experts and psychometric staff, particularly staff of the NCSDGEQ. This training will build on the capacity building that has taken place under the READ Trust Fund. 165. Technical Assistance. The primary source of technical assistance will be provided by international and national advisors who will be available for the duration of the reform to facilitate core training activities, provide evidence-based high-quality technical input relevant to the revision process, and share international best practices on curriculum reform. 166. Research. The NCSDGEQ implements six research programs, each organized into four to five working groups and led by a Research Coordinator. Researchers might be affiliated with one or more research programs. Overlapping membership on research programs will ensure that topics are addressed from multiple perspectives and based on a diversity of expertise. The researchers in most programs meet at least once a month at program meetings and participate in the Center’s annual summit. In order to achieve the desired level of quality and innovative research, the Center will also seek the participation of Visiting Researchers who are internationally recognized as leaders in their respective fields. 167. Staffing of Centers. In order to achieve coordination, alignment, and cost-effectiveness, the Center will find synergies and work in tandem with existing research and assessment institutions, including, inter alia, the Vietnamese National Institute of Education Sciences (VNIES), Hanoi National University of Education, and General Department of Education Testing and Assessment (GDETA). Because these institutions are already addressing some of these issues relevant to the Center’s mission, it would be prudent for the center’s staff to be seconded from these institutions at the nascent phase of the Project. 168. Training, Events, and Conferences (TEC). TEC are activities that are categorized as such in the Annual Work Plan (AWP) and agreed on by the Task Team (TT) and the management entity. These activities refer to a variety of modalities, including but not limited to workshops, training, consultations, writing retreats, capacity-building, conferences, symposia, and/or seminars that are held scheduled and carried out routinely or ad hoc, domestic or abroad. For these Conferences and Events, IDA will be used to finance reasonable costs of expenditures incurred for honoraria for trainers and presenters (including civil servants, but excluding staff of the management entity); materials; equipment; venue rental; accommodation, meals, transportation, and/or per diems for those attending TEC. The Per Diem amount will not duplicate payments for other expenses that are already accounted for in the Per Diem payment. If costs related to the participation in a TEC activity have already been paid by the management entity directly to the service provider, the Per Diem amount will be zero. IDA does not finance salary, salary supplements, and/or allowance for attendance. 169. AWP and Budgets. The management entity will transmit an AWP by October 31st of each year for the duration of the Project. The AWP will state all activities proposed to be undertaken 46 during the year, including the detailed description of the purpose, scope, beneficiaries, timeline, and detailed budget for each activity. All TEC activities in the AWP must be assigned an Event Number in accordance with the FM section of the POM. This Event Number will be used as a reference number for all expenses incurred during a particular TEC activity in the Event Expenditure Log in accordance with the POM, and with Bank and GOV regulation. Once annually, at least four months before the start of the Borrower’s fiscal year, and after the draft AWP has been submitted to the Bank and any relevant government authorities for review, a meeting will be held among the TT, the management entity Director, relevant management entity staff, and key authorizing officials. The purpose of the meeting is to discuss the AWP’s feasibility and effectiveness in achieving the PDO. The AWP can only be implemented once agreed on by both the World Bank and the management entity. Minutes of the meeting will be taken and transmitted to both the TT and management entity for approval. Financial Management, Disbursements and Procurement Financial Management 170. A “Moderate” FM risk rating was assigned to the project. The following key risks have been identified: (i) as the management arrangement will be established, the envisaged staff may not have relevant experiences with the Bank financed projects; and (ii) as there are a large number of (consulting) packages, and training and workshop events, internal controls may not be effectively designed and implemented to ensure eligible uses of funds. 171. The main actions required are: Action Responsibility To Be Completed By As part of project implementation Appointment of adequate FM staff: (Full time) arrangements Chief Accountant with appropriate accounting acceptable to the MOET background and relevant experiences shall be Bank. appointed. As an integral part of FM section and training: FM section (as a chapter the POM and to be of the Project Operation Manual) shall be MOET developed and related Training shall be delivered. found acceptable by the Bank. 172. The management entity will be responsible for overall project coordination and financial management of the project. In particular, the management entity will be responsible for designing and maintaining satisfactory Project FM. 173. The Project Financial Statements will be audited by independent auditors acceptable to the Bank in accordance with Terms of Reference (TORs) acceptable to the Bank and will need to be submitted within six months after the end of the period covered by the Financial Statements. The 47 cost of the audit will be funded by the Project. The Audited Financial Statements shall be made publicly available. 174. The semi-annual Interim Financial Reports (IFRs) will be prepared by management entity consisting of the followings: (a) IFR1: Summary of Funds and Uses of Funds (b) GSDG 2.3: Report of Disbursement of ODA (c) GSDG 2.4: Report of Disbursement of Counterpart Fund (d) IFR3: Statement of Designated Account 175. Templates of GSDG 2.3 and GSDG 2.4 are regulated under the Circular 01/2014 issued by the Ministry of Planning and Investment. Templates of IFR1 and IFR3 will be documented in the financial management section of the POM. The Project IFRs shall be made publicly available. The quarterly “requests for reimbursement” must be sent to the Bank within 45 days of the reporting period. 176. The management entity will appoint an acceptable Internal Auditor (IA) for the project with clearly defined TORs and the required experience and qualification as indicated in the financial management section of the POM. The IA audit report will need to be sent to the Bank on a semi-annual basis within three months after the end of the reporting period. 48 Figure 3: Fund flow to the beneficiaries Step 1- Step 4: fund flow from MOF WB management entity to Contractors/ 6 Beneficiaries  Step 1: Contractors/ 7 Beneficiaries send payment requests to 5 management entity. State 2 Mgm 3b Commercial  Step 2: management entity sends Treasury t Bank entity payment requests and supporting documentation to ST for verification. 3a 1 4  Step 3a: State Treasury (ST) makes payment to Contractors/ Contractors/ Beneficiaries Beneficiaries (counterpart fund).  Step 3b: management entity requests commercial bank to make payments to Contractors/ Beneficiaries (ODA funds).  Step 4: Commercial Bank makes Documentation payments to Contractors/ Beneficiaries (ODA funds). Cash Flow Step 5- Step 7: Fund flow for replenishment from the World Bank to the Project  Step 5: management entity prepares Withdrawing Application (WA) and sends to MOF for co-signing.  Step 6: WA then is sent to WB.  Step 7: WB disburses funds to the Designated Account (DA) of the Project maintained at Commercial Bank. 177. Eligible expenditure means the reasonable cost of goods, works, non-consulting services, consultants’ services, training and workshops, and incremental operating costs for the project, all in accordance with legal agreement. 49 Disbursements 178. Disbursement of the Project will be made via the Advance and Reimbursement disbursement method supported by the Statement of Expenditure and Lists of Payment against contracts as stated in the Disbursement Letter (DL). The ceiling of the Designated Account (DA) will be US$5 million. Percentage of Expenditures to Category Amount of the be Financed Credit Allocated (inclusive of Taxes) (US$) Goods, Works, Non-consulting 77,000,000 100 Services, Consultants’ Services, Training and Workshops, and Incremental Operating Costs for the Project Total Amount 77,000,000 179. The Project will have a Disbursement Deadline Date of four months after the Closing Date. This is the final date on which World Bank will accept applications for withdrawal from the Recipient or documentation on the use of Credit proceeds already advanced by the World Bank. This "Grace Period" is granted in order to permit orderly project completion and closure of the Credit account via the submission of applications and supporting documentation for expenditures incurred on or before the Closing Date. Expenditures incurred between the Closing Date and the Disbursement Deadline Date are not eligible for disbursement, except as otherwise agreed with the World Bank. Procurement 180. Institutional Arrangement. The project implementing agency will be the MOET. The management entity to be established under MOET will be responsible for all procurement activities financed by IDA. 181. Procurement Capacity and Risk Assessment (PCRA). As the management arrangements had not been formally established by March 2015, the Bank team had to rely the PCRA on the previous assessment of capacities of MOET completed on October 2014. The assessment revealed that the project management arrangements would include MOET staff, who have gained necessary procurement experiences through implementation of several completed projects financed by the Bank and other international donors as well. Nevertheless, the assessment also identified that procurement under the project may risk delays and noncompliance which could arise from: (a) MOET bureaucracy in internal approval/appraisal procedures that would delay the procurement clearance and decision making processes; (b) management entity’s potential weak procurement planning and monitoring, and its potential tendency to follow the national public procurement procedures rather than the Bank's Guidelines when there is conflict between them; 50 (c) Lack of clarity on who is responsible for the decisions on selection of individual consultants (ICs) for developing curriculums and textbooks, and how to effectively manage a huge number of the hired ICs; (d) Assigned procurement staff's lack of adequate familiarity with Bank procurement procedures and policies; and (e) Possible governance and corruption issues. 182. Based on the recent public procurement assessment, the risks inherent in the country’s public procurement system are high. However, given MOET's accumulated experience in the Bank's procurement practices and considering that most of the envisaged international and national competitive bidding packages under the project will be neither very high in value nor very complex in nature, the procurement risk for the proposed project is rated as "Substantial". 183. To mitigate the above risks, the following actions were agreed between the Bank and the project management entity during project preparation. Actions Responsibility Time frame 1 Assign at least two procurement MOET As part of project officer with adequate qualifications implementation and procurement experience with arrangements World Bank-funded projects acceptable to the Bank. 2 Prepare and endorse a hands-on MOET Prior to procurement section (covering clear effectiveness rules, procedures and division of responsibilities, sample documents and evaluation report, etc.) as part of the POM 3 Provide regular and ad-hoc trainings management Continuous, on the Bank’s procurement entity & WB following signing policies/procedures to project staff of management entity, as well as other relevant MOET staff in various departments, who will be involved in assisting MOET’s appraisal/approval of/for procurement decisions made by the management entity. 4 Recruit Procurement Consultants as management Early stage and needed entity throughout the project implementation 5 Mobilize adequate technical expertise management Throughout the (in house and/or external consultants) entity project to prepare technical specifications for implementation IT system, equipment, civil works packages as needed 51 184. Applicable Guidelines. For contracts financed in whole or in part by the IDA Credit under the proposed project, procurement shall be carried out in accordance with the Bank’s “Guidelines: Procurement of Goods, Works and Non-consulting Services under IBRD Loans and IDA Credits and Grants by World Bank Borrowers” dated January 2011, revised July 2014 (“Procurement Guidelines”), and “Guidelines: Selection and Employment of Consultants under IBRD Loans and IDA Credits and Grants by World Bank Borrowers” dated January 2011, revised July 2014 (“Consultant Guidelines”), as well as the relevant provisions in the Financing Agreement. 185. Procurement thresholds. The specific procurement methods, their application thresholds, and the thresholds for the Bank’s Prior Review to be applied for each of the contract under the proposed project are indicated in the Table below. During the project implementation, these thresholds may be subject to change upon the Bank’s official notice in sufficient time in advance to ensure smooth implementation without any disruption. Furthermore, the use of particular procurement and selection methods such as Force Account, Framework Agreements, Limited International Bidding, Procurement Agents and Construction Managers and/or Selection of particular types of consultants shall be subject to the Bank’s prior approval, and will be specified in the Procurement Plan agreed with the Bank. Table 2: Summary of Procurement Arrangements Expenditure Contract Procurement Method Bank Prior Review (*) Category Value (US$) >=$3,000,000 ICB All the ICB contracts <$3,000,000 NCB (**) First NCB contract for each Goods implementing agency <$100,000 Shopping N/A NA DC All DC contracts >=$20,000,00 ICB All the ICB contracts <$20,000,000 NCB First NCB contract for each implementing agency and all NCB contracts exceeding US$15 million Works (according the most updated prior review thresholds) <$200,000 Shopping N/A NA DC All DC contracts >=$300,000 QCBS, QBS, FBS, LCS Firms: All contracts >= $ 500,000 for firms; first contract for each method regardless of value; SSS Consultant contract >=$50,000 Services Individuals: only in exceptional (***) <$300,000 CQS cases for competitive selection; NA SSS SSS contract >= $50,000 NA IC Audit contracts (****) 52 Notes ICB – International Competitive LCS – Least Cost Selection : Bidding CQS – Selection Based on Consultants’ NCB – National Competitive Bidding Qualification DC – Direct Contracting SSS – Single (or Sole) Source Selection QCBS – Quality and Cost Based IC – Individual Consultant selection procedure Selection NA – Not Applicable QBS – Quality Based Selection FBS – Fixed Budget Selection * Contracts below these Prior Review thresholds shall be subject to Post Review on an annual basis. The rate of post review will be initially 20 percent. This rate may be adjusted during project implementation based on the procurement performance. **Where goods are not normally available within Vietnam, the method of procurement will be ICB even if the contract value is less than US$3 million/contract. **The NCB procedures shall be those set forth in Vietnam’s procurement laws and regulations, but subject to modifications, waivers, and exceptions as set forth in the “NCB Annex” to the Financing Agreement. *** Shortlists for contracts below US$0.5 million/contract may comprise entirely national consultants. **** For Audit contracts below the prior review threshold, the TTL/FMS may require to prior review TOR, shortlist, TER, draft contract, etc. from a technical perspective. 186. Procurement of Goods and Works under the NCB procedures shall comply with provisions stipulated in the “NCB Annex for Vietnam” – an Annex to Schedule 2 (Procurement) of the project Financing Agreement. 187. Engagement of Government owned universities professors, school teachers, scientists from research institutes. It is expected that a large number of ICs (tentatively estimated at around 750 at the time of project preparation) who are professors, teachers, scientists, and staff of Government owned research institutes, universities, schools will need to be involved in the project activities such as the development and review of general curriculum, SSC, pedagogical materials, etc. These are the only resources in Vietnam that would have the functions and capabilities to deliver the services required under the project; hence, they may be considered as uniquely qualified for the proposed assignments and their participation would be critical and essential to the project implementation. These persons would carry out tasks that are incremental to their statutory activities. The Bank may therefore agree on the hiring of these individuals only if the management entity can demonstrate that (i) such hiring of any individual does not conflict with any employment or other laws or regulations or policies of the GOV; (ii) every individual would meet the eligibility criteria set forth in paragraphs 1.13 (c) and (d) of the Consultants Guidelines; and that (iii) such hiring of any individual would not create a conflict of interest. 53 188. Procurement Plan. For each contract under the proposed project to be financed by the IDA Credit, the procurement methods, estimated costs, prior review requirements, and time frame shall be reflected in the Procurement Plan. The draft Procurement Plan for the first 18 months implementing the proposed project has been developed by MOET and been agreed to by the Bank team during negotiations. 189. The above mentioned Procurement Plan shall be updated throughout the duration of the project as needed or at least annually. The updated Procurement Plan should retain all contracts under management in the “contracts under management” and previously awarded contracts in the “completed contracts” sections of the Procurement Plan to present a full picture on procurement under the project. All procurement plans, their updates or modifications shall be subject to Bank’s prior review and no-objection. The Bank shall arrange after project negotiations the publication on the Bank’s external website of the agreed initial procurement plan and all subsequent updates. The management entity shall arrange the publication of all the procurement plans on the Vietnam Pubic Procurement Review (both paper and online version). Environmental and Social (including safeguards) 190. Component 3 of the Project supports the Development of a NCSDGEQ and a NCFLT. It is expected that the Centers’ newly constructed and renovated buildings will be set within the current premises of MOET and MOET's universities. Their exact locations, however, will be finalized during implementation. The Project has been classified as Category B, and the OP 4.01 is triggered as the project may cause adverse environmental and social impacts. During Project preparation, an Environmental and Social Framework (ESMF) was developed by MOET to ensure that activities during implementation are in compliance with the World Bank safeguard policies and national regulations. 191. As the locations of the centers have not yet been identified, the site-specific impacts for subprojects will not be known until implementation phase. Nonetheless, according to the ESMF, the key negative impacts -- noise, dust, solid waste, wastewater, traffic and labor safety, and unexploded ordnances -- are likely to be localized, short-term, and moderate. Moreover, such effects can be expected to be managed and mitigated by good design and compliance with construction practices as described in the Environment Codes of Practice (ECOPs), which is part of the ESMF and will be included in the bidding documents and relevant contract documents. Once the sites have been identified and prior to the implementation, the management entity shall ensure the appropriate review of the design of the construction/rehabilitation work. 192. Additional measures for mitigation of adverse impact are described in the ESMF. One such measure is the development of a template for an Environmental Management Plan (EMP), which consists of the set of mitigation, monitoring and institutional measures to be taken during Project implementation phase to eliminate, offset, or reduce adverse impacts. Moreover, according to the ESMF, Public Consultation was conducted and will continue to be undertaken during Project implementation as necessary. 193. The project will be implemented nationwide, including locations where there is intensive presence of indigenous people (known as ethnic minority people in Vietnam), such as Northern 54 Mountain, Central Highland, Mekong Delta regions. Although it is expected that the project has no adverse impact on ethnic minority communities, it is important that the new curriculum and pedagogical materials will portray ethnic minority peoples and their heritage in a respectful and positive manner. The proposed Project constitutes a significant opportunity to ensure that Vietnamese public schools treat ethnic minority people's curriculum and assessment issues fairly. 194. As the Bank's safeguard policy on Indigenous People (OP 4.10) was triggered for this Project, MOET has conducted a social assessment that identifies potential barriers related to curriculum and textbooks for ethnic minority communities. Based on a review of policy documents, existing curriculum and textbooks, and consultations with representatives of Ethnic Minority groups, the assessment finds that: (i) the frequency of mentions of ethnic minorities in textbooks is low; (ii) the curriculum does not explicitly encourage the introduction of culturally and locally relevant materials; and (iii) the primary language of instruction for ethnic minority students is Vietnamese while little support is offered in ethnic minority students' mother tongue. 195. The social assessment also examines issues related to gender equality and fair treatment of students with disabilities in the existing curriculum and textbooks. The assessment finds that there is a disproportionately low percentage of content about women, girls, and children with special educational needs in textbooks. Moreover, the study reports that out of 763 textbook authors in the previous curriculum reform, only 171 were women. 196. In order to mitigate such bias, the social assessment proposes recommendations for policy makers, curriculum and textbook developers, education managers on the development and implementation of bias-free curriculum and textbooks. The key recommendations include: (i) ensuring that bias-avoidance is a criteria for Project deliverables; (ii) building capacity for writers and a selection of teachers and administrators on multicultural and inclusive education; (iii) defining a process for broad community consultation and support; (iv) building staffing models that result in highly-competent and diverse teams; and (v) ensuring that there are equal opportunities to introduce culturally and locally relevant materials into the curriculum. 197. These recommendations have been considered in the design of the Project. For example, the Project seeks to build capacity for ethnic minority teachers on implementing the new curriculum, translate a number of selected textbooks into several ethnic minority languages, and develop an on-going research agenda on ethnic minorities’ access to and equity in education. Furthermore, as indicated in the POM, bias-avoidance will be a criterion for development and evaluation of the revised curriculum, textbooks, and other pedagogical materials. In addition, recruitment and selection of curriculum and textbooks developers will be a fair and transparent process that prioritizes technical expertise but also promotes group diversity (e.g., with respect to gender, ethnic background, etc.) to ensure that the revised curriculum and textbooks are well- balanced and appropriate for a wide range of learners. 55 Monitoring & Evaluation 198. The three Key Project Indicators will measure changes to student learning, readiness of teachers to implement the new curriculum, and access of disadvantaged student to new, aligned textbooks and pedagogical materials. 199. The intermediate indicators will focus on: (a) The composition and qualifications of the CTC for Curriculum and Assessment Reform (CTC); (b) The progress, technical quality and effective use of the online training system; (c) The effectiveness of curriculum champions; (d) The progress and coverage of the production of aligned textbooks; (e) The effectiveness of curriculum piloting efforts; and (f) The coverage and quality of the expanded NLSAS. 200. Technical issues related to monitoring and evaluation pertain to assessing learning outcomes in such a way that baselines scores-obtained under the old curriculum equate to and are comparable with new scores from students taught under the new curriculum. Technical challenges also exist to measure progress or quality under the areas of the progress where success depends on sound professional judgment. To mitigate the risks from these challenges, measures were taken during Project preparation to: (i) identify existing standardized tests and corollary data from which anchor test items might be drawn; and (ii) analyze the technical quality of the potential anchor items. Additional activities were conducted to identify gaps in measurement of student characteristics to allow for the improvement of questionnaires. 56 Annex 4: Implementation Support Plan VIETNAM: Renovation of General Education Project (P150058) Strategy and Approach for Implementation Support 201. The implementation support strategy for this Project will comprise regular dialogue with MOET and the management entity and intensive interaction to support the technical quality of implementation. It will also include semi-annual joint reviews of Project implementation as part of formal Supervision, and regular oversight of fiduciary activities. Regular dialogue and early support will facilitate early identification of problems and obstacles and will enable timely provision of technical advice to address any obstacles. The approach to effective Implementation Support has begun with intensive efforts to ensure the comprehensiveness and quality of the POM prior to Negotiations. It is expected that the POM will be the first and often best source of detailed information on the specifics of implementation. While not all technical challenges can be foreseen, the strategy depends on a strong and actively-used POM to minimize areas of ambiguity for Implementation. 202. As part of Supervision, lessons learned from other education projects and from implementation experience will be incorporated into the dialogue to allow for continuing improvement of implementation. 203. With regard to the supervision of the fiduciary aspects of the operation, the World Bank will conduct financial management and procurement supervision at appropriate intervals. Emphasis from the beginning will be placed on creating shared understanding of the nature of good compliance and its value as a tool for management and effectiveness of the Project. All efforts will be made to minimize of eliminate “busywork” or any type of reporting that does not serve a clear and important purpose. Project managers will be expected to review and analyze fiduciary reports and opine on the quality of fiduciary performance as it affects the Project’s ability to meet its objectives – and not only its ability to comply in a narrow sense. In addition, the Bank team will provide support for continued improvements to borrower capacity for fiduciary matters. 57 Implementation Support Plan 204. The table below presents the implementation support plan for the Project Time Focus Skills Needed Resource Partner Role Estimate First twelve Assuring quality of Curriculum revision US$120,000 p.a. months technical committees experience; assessment and project baselines data analysis Assistance to survey Survey expertise US$150,000 p.a. research on textbook availability and use 12-48 months Promoting high Core World Bank US$120,000 p.a. quality alignment of education project teaching and supervision skills; learning with the online training systems revised curriculum; production skills assuring full and effective dissemination of new curriculum Other Safeguards Curriculum bias US$25,000 p.a. Compliance expertise and textbook production expertise Training Online systems design US$15,000 p.a. Effectiveness and maintenance Research Program Research Management US$10,000 p.a. Management Expertise Skills Mix Required Skills Needed Number of Staff Weeks Number of Trips Comments Task Team Leader 15/yr 3/yr Washington-based Education Specialist 18/yr n/a Hanoi-based Curriculum Design 8/yr for first two years 3 International Consultants Assessment and Test 8/yr 3 International Design Consultants Program Assistant 10/yr n/a Hanoi & Washington Based Legal Analysis 2 Hanoi & Bangkok Offices Financial Management 3 Hanoi Office Analysis and Capacity Building Procurement Analysis 3 Hanoi office and Capacity Bldg. Disbursement Analysis 1 0 To be done remotely/ virtually 58 Annex 5: Risk Diagnosis VIETNAM: Renovation of General Education Project Risk Categories Rating (H, S, M, or L) 1. Political and governance Substantial 2. Macroeconomic Moderate 3. Sector strategies and policies Low 4. Technical design of project or program High 5. Institutional capacity for implementation and Substantial sustainability 6. Fiduciary Substantial 7. Environment and social Low 8. Stakeholders Substantial 9. Other Low Overall Substantial General Commentary about Overall Risks (Substantial) 205. The Project supports a reform that was initiated by the GOV through Resolution 29- NQ/TW. As such, there exists a high level of alignment between the interests, objectives, and motivations of the GOV and those of the World Bank. The reforms supported by the proposed operation will benefit greatly from efforts to develop a shared understanding of educational concepts, project objectives, and modus operandi among stakeholders during the project preparation phase. Achieving this level of technical coordination is not easy and will require significant efforts to: (i) identify, recruit, and train large numbers of qualified people with specialized skills; (ii) share information, resources, and responsibilities across a large number of MOET departments; and (iii) maintain high levels of commitment and performance from MOET staff with key technical responsibilities. All of this requires steadfast focus on the technical quality of the reform, shared understandings of project goals, and transparent talent management and procurement processes that allow for the selection of highly-qualified academic and fiduciary staff. Setting up the right conditions under the preparation phase has allowed for effective implementation, including getting buy-in from teachers and officials at the provincial and local levels. Even with the successful preparation phase, implementation will still pose its own challenges. A number of key risks will remain, including maintaining consistency and coherence during the nationwide curricular roll-out, building MOET capacity for certain less familiar activities (especially assessment), and evaluating the efficacy of the new curriculum’s content. Furthermore, the GOV’s emphasis on delivering a portion of the new curriculum and textbooks by 2018 puts great time pressure on the MOET. The Bank will support the MOET to ensure that Project implementation balances timeliness and quality. 59 Political and Governance Risks (Substantial)   206. Preparations for the next Party Congress in 2016 have started. Key outcomes will be the adoption of the socio-economic orientations for 2016 – 2020 and the selection of a new leadership for the Party. The next election of the National Assembly will take pace in early 2016, followed by the selection of the new Government. The transition is expected to be relatively smooth and will not cause much disruption. The space for dialogue and public debate has continued to expand. In particular, there is now much more open discussion of corruption and of Government effectiveness. The Government has increased efforts to promote transparency and citizen engagement to improve government accountability, as illustrated by revisions of the Anti- Corruption Law, the Land Law and the Constitution. While the changes did not address all of the critical issues, the consultation process for the revisions of provided good opportunities and space for public debate. The landmark revision to the Constitution introduced a clearer definition of State responsibilities in exercising human rights, which paved way for the preparation of three important draft laws on Access to Information, on Associations and on Referendum. Technical Design of Project Risks (High) 207. This Project involves a large number of stakeholders, and the working mechanisms and interdependencies among them are complex and remain untested. There exists a high level of risk that aspects of project implementation will prove difficult, especially given the vast scope of the Project (which includes all primary and secondary education in the country). In order for the reform to succeed, policymakers and practitioners will have to establish a common vision for the new curriculum. However, it is unclear whether agreements made in principle will translate into an actual shared vision. The involvement of the general public and politicians in addition to high- level MOET officials increases both the potential for divergent views and interests and the risk that the best technical pathways for reform will not be selected. There is also a risk that decision- making and coordination mechanisms designed for the implementation of the proposed Project’s vision will be inadequate for such a complex, large-scale undertaking, particularly because the proposed Project’s activities are new territory for so many of the stakeholders involved. Finally, there is a significant risk that opacity within decision-making structures and communication channels will impede successful reform. 208. The Bank will ensure that the MOET have access to field experts and practitioners who can comment on effective means of structuring an effort as large as this one. Indeed, this has already begun with the provision of technical assistance from a number of research experts and practitioners experienced in curriculum reform of high-performing countries. International advisors and national experts will provide ongoing technical support to the MOET’s development of curriculum content and their organizational structure preparations. There has been frequent consultation with MOET and policy leaders at the nascent phase of the Project to ensure alignment of objectives and consensus on activities to be included in this Project. Furthermore, the Bank continues to work with the MOET to ensure proper documentation and the institutionalization of policies guiding coordination, communication, and decision-making. 60 Institutional Capacity for Implementation Risks (Substantial) 209. The availability of competent staff with the skills, organizational knowledge, and financial resources to implement the proposed Project is still unknown, and the process for procuring that staff is not yet transparent. While the MOET has some staff with experience from other Bank projects and the ability to provide staff with training in a variety of skills, it has not undertaken a project of this nature before. For this reason, MOET has started to prepare a comprehensive set of written processes and systems in place to support the implementation of the proposed Project. The ability of project managers to monitor implementation milestones and continue monitoring reform effects after the Project ends is unclear given the novelty of this project and potential concerns regarding suboptimal transparency for these types of activities. In addition, this project requires a large number of areas to be monitored. The impact of the new curriculum and textbooks on learning outcomes is contingent on teachers’ willingness and ability to implement and maintain new teaching methods. As such, project managers must not only monitor the development of the curriculum and textbooks, but also coordinate with pre-service and in-service teacher training institutions to ensure that teachers are well-supported. 210. Both formative and summative evaluation will be used to monitor implementation and Project results. Training is included in the project preparation phase to ensure that there are adequate skills, experience, processes, and policies to support monitoring and evaluation. The Bank will also emphasize the importance of teacher effectiveness, as teachers will play a key role in ensuring a smooth transition to the new curriculum. The Bank will utilize prior review to guarantee, inter alia, the competitive selection of candidates for key financial and procurement positions. The MOET has continued to collaborate closely with the Bank to facilitate MOET’s production of a high-quality, clear, and concise POM that establishes processes and systems to support Project implementation and ensure comprehension of and adherence to relevant rules and regulations. An advanced draft of the POM has been developed and is being finalized. Fiduciary Risks (Substantial) 211. The legal framework for public procurement, which has been improved through recent revisions to procurement law, continues to have key shortcomings in comparison to international best practices. Even though procurement during implementation will be highly decentralized, Vietnam’s lack of qualified procurement staff and high procurement staff turnover rates are causes for concern. In short, many implementing agencies do not have the necessary capacities for procurement and contract management. One particular risk for Bank-financed projects is the practice of following national procurement procedures in parallel with Bank procedures. This leads to procurement delays, and noncompliance. Procurement complaints about Bank-financed projects in Vietnam are relatively common. Low financial management capacity, particularly at sub- national levels, can lead to weak internal control of a project and low-quality financial reporting. Disbursement is impacted by counterpart funding and approval delays, and financial reporting and internal and external oversight mechanisms are below international standards for both the public and corporate sectors. Although a relatively strong anti-corruption legal framework exists on paper—e.g., the Law on Anti-Corruption, the National Strategy on Anti-Corruption until 2020, and the Law on Procurement—implementation is lagging. Conflicts of interest are widespread, and oversight institutions are not fully independent from the national executive. 61 212. The Bank continues to support the government in improving the performance of its public procurement system through “Institutional Development Funds” for strengthening procurement audit capacity, updates to the assessment of the procurement legal framework, assistance in further developing the e-GP system, and measures to strengthen existing procurement legislation. Bank- financed projects in Vietnam also routinely include support for contract management and intensive procurement training and capacity-building. In addition, a national strategy for public procurement training and capacity-building has been developed, and its implementation should facilitate the development of a procurement profession, build long-term training resources and capacity, and enhance the public procurement system so that funds are used more economically, efficiently, and transparently. The POM will clarify procurement responsibilities. To the extent possible, internal audit capacities will be used in the context of the Project. However, external consultants for procurement and contract management assistance will be engaged as needed. The country office’s strong support for the GOV’s implementation of its Anti-Corruption Strategy, enhanced technical assistance, and dialogues about transparency will help to mitigate the risk of such problems. Stakeholder Risks (Substantial) 213. There is clear alignment of borrower and Bank interests, objectives, and motivations for undertaking this project (see Resolution 29 of Session XI of National Assembly VIII). Nonetheless, there is a substantial probability for disrupted borrower relations should the MOET’s and/or other GOV entities’ views on the reform diverge significantly and publicly from the technical advice coming from the Bank. The involvement of high-level political actors from the National Assembly heightens this risk. 214. However, trust has been built through past Bank projects and the ongoing READ Trust Fund. Moreover, the latter has enabled the Bank to gain invaluable experience providing the GOV with relevant technical assistance related to curriculum and assessment. There is an ongoing consultation process with development partners and beneficiaries to ensure that the views of internal and external stakeholders are considered in the project design and implementation processes. In addition, the Bank has emphasized (and will continue to emphasize) that curricular decisions are the purview of the GOV and that the Bank’s role is to provide technical assistance without making decisions for the GOV. This emphasis is important given that controlling curriculum is a significant aspect of national sovereignty. Other Risks (Low) 215. As advised by the SORT Interim Guidance Notes, the most relevant risks (rated as high and substantial) have been addressed above. Macroeconomic, Sector Strategies and Policies, and Environment and Social risks are considered to be low or moderate, and therefore do not require further discussion here. 62