Document of The World Bank FOR OFFICIAL USE ONLY Report No: ICR00003549 IMPLEMENTATION COMPLETION AND RESULTS REPORT (Loan 8110 – SV) ON A LOAN IN THE AMOUNT OF US$60 MILLION TO THE REPUBLIC OF EL SALVADOR FOR THE EDUCATION QUALITY IMPROVEMENT PROJECT June 28, 2019 Education Global Practice Latin America And Caribbean Region CURRENCY EQUIVALENTS Currency Unit = U.S. dollar FISCAL YEAR July 1 - June 30 Regional Vice President: Axel van Trotsenburg Country Director: Yaye Seynabou Sakho Senior Global Practice Director: Jaime Saavedra Chanduvi Practice Manager: Emanuela Di Gropello Task Team Leader(s): Robert J. Hawkins ICR Main Contributor: Silvia Guallar Artal ABBREVIATIONS AND ACRONYMS CPF Country Partnership Framework CPS Country Partnership Strategy EDUCO Community-Managed Schools Program (Educación con Participación de la Comunidad) EMIS Education Management and Information System FM Financial Management FMLN Farabundo Martí National Liberation Front (Frente Farabundo Martí para la Liberación Nacional) GDP Gross Domestic Product ICR Implementation Completion and Results Report IFTS Inclusive Full-Time School INFOD National Institute for Teacher Training (Instituto Nacional de Formación Docente) IPP Indigenous Peoples Plan IRR Internal Rate of Return IS Integrated System M&E Monitoring and Evaluation MCC Millennium Challenge Corporation MINED Ministry of Education (Ministerio de Educación, Ciencia y Tecnología, also known as MINEDUCYT) NPV Net Present Value PDO Project Development Objective SFLAC Spanish Fund for Latin America and the Caribbean SNEe National System for Education Evaluation (Sistema Nacional de Evaluación Educativa) TFSCB Trust Fund for Statistical Capacity Building TABLE OF CONTENTS DATA SHEET .......................................................................................................................... 1 I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES ....................................................... 5 A. CONTEXT AT APPRAISAL .........................................................................................................5 B. SIGNIFICANT CHANGES DURING IMPLEMENTATION ................................................................9 II. OUTCOME .................................................................................................................... 12 A. RELEVANCE OF PDOs ............................................................................................................ 12 B. ACHIEVEMENT OF PDOs (EFFICACY) ...................................................................................... 12 C. EFFICIENCY ........................................................................................................................... 15 D. JUSTIFICATION OF OVERALL OUTCOME RATING .................................................................... 16 E. OTHER OUTCOMES AND IMPACTS (IF ANY) ............................................................................ 16 III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME ................................ 18 A. KEY FACTORS DURING PREPARATION ................................................................................... 18 B. KEY FACTORS DURING IMPLEMENTATION ............................................................................. 19 IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME .. 19 A. QUALITY OF MONITORING AND EVALUATION (M&E) ............................................................ 19 B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE ..................................................... 21 C. BANK PERFORMANCE ........................................................................................................... 22 D. RISK TO DEVELOPMENT OUTCOME ....................................................................................... 22 V. LESSONS AND RECOMMENDATIONS ............................................................................. 23 ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS ........................................................... 24 ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION ......................... 39 ANNEX 3. PROJECT COST BY COMPONENT ........................................................................... 41 ANNEX 4. EFFICIENCY ANALYSIS ........................................................................................... 42 ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS ... 48 ANNEX 6. REVISION HISTORY OF PROJECT INDICATORS........................................................ 60 ANNEX 7. THEORY OF CHANGE WITH SUPPORTING LITERATURE .......................................... 76 ANNEX 8. SUPPORTING DOCUMENTS .................................................................................. 79 The World Bank Education Quality Improvement Project (P126364) DATA SHEET BASIC INFORMATION Product Information Project ID Project Name P126364 Education Quality Improvement Project Country Financing Instrument El Salvador Investment Project Financing Original EA Category Revised EA Category Partial Assessment (B) Partial Assessment (B) Organizations Borrower Implementing Agency Republic of El Salvador Ministry of Education Project Development Objective (PDO) Original PDO The objective of the Project is to improve access, retention and graduation rates for students in the Lower Secondary Education and the Upper Secondary Education of the Borrower's public schools adopting the Inclusive Full Time School Model (IFTS Model). Page 1 of 82 The World Bank Education Quality Improvement Project (P126364) FINANCING Original Amount (US$) Revised Amount (US$) Actual Disbursed (US$) World Bank Financing 130,000 128,817 128,817 TF-10361 : 60,000,000 60,000,000 59,587,404 IBRD-81100 : 98,680 0 0 TF-14166 : Total 60,228,680 60,128,817 59,716,221 Non-World Bank Financing 0 0 0 Borrower/Recipient 10,400,000 4,701,928 6,562,644 Total 10,400,000 4,701,928 6,562,644 Total Project Cost 70,628,680 64,830,745 66,278,865 KEY DATES Approval Effectiveness MTR Review Original Closing Actual Closing 13-Dec-2011 15-May-2012 17-Oct-2016 31-Dec-2017 31-Dec-2018 RESTRUCTURING AND/OR ADDITIONAL FINANCING Date(s) Amount Disbursed (US$M) Key Revisions 23-Jan-2013 1.17 Change in Components and Cost Change in Institutional Arrangements 13-Jul-2015 13.71 Change in Results Framework Change in Components and Cost Reallocation between Disbursement Categories Change in Implementation Schedule Other Change(s) 28-Jun-2017 33.62 Change in Results Framework Change in Loan Closing Date(s) Change in Procurement Change in Implementation Schedule 15-Mar-2018 50.76 Reallocation between Disbursement Categories Page 2 of 82 The World Bank Education Quality Improvement Project (P126364) KEY RATINGS Outcome Bank Performance M&E Quality Moderately Satisfactory Moderately Satisfactory Substantial RATINGS OF PROJECT PERFORMANCE IN ISRs Actual No. Date ISR Archived DO Rating IP Rating Disbursements (US$M) 01 29-Mar-2012 Satisfactory Satisfactory .05 02 22-Oct-2012 Satisfactory Moderately Satisfactory .05 03 06-Mar-2013 Satisfactory Moderately Satisfactory 1.30 04 27-Aug-2013 Satisfactory Moderately Satisfactory 1.30 05 13-Mar-2014 Moderately Satisfactory Moderately Satisfactory 4.29 Moderately 06 27-Oct-2014 Moderately Unsatisfactory 10.28 Unsatisfactory Moderately 07 06-Mar-2015 Moderately Unsatisfactory 12.06 Unsatisfactory Moderately 08 05-Sep-2015 Moderately Unsatisfactory 13.84 Unsatisfactory 09 21-Apr-2016 Moderately Satisfactory Moderately Unsatisfactory 17.32 10 22-Nov-2016 Moderately Satisfactory Moderately Satisfactory 23.97 11 30-Jun-2017 Moderately Satisfactory Moderately Satisfactory 35.11 12 08-Dec-2017 Moderately Satisfactory Satisfactory 46.40 13 18-Jun-2018 Moderately Satisfactory Satisfactory 54.50 14 27-Dec-2018 Moderately Satisfactory Satisfactory 59.29 Page 3 of 82 The World Bank Education Quality Improvement Project (P126364) SECTORS AND THEMES Sectors Major Sector/Sector (%) Education 100 Public Administration – Education 6 Secondary Education 47 Tertiary Education 47 Themes Major Theme/ Theme (Level 2)/ Theme (Level 3) (%) Human Development and Gender 100 Education 100 Access to Education 38 Education Financing 25 Science and Technology 11 Teachers 13 Standards, Curriculum and Textbooks 13 ADM STAFF Role At Approval At ICR Regional Vice President: Pamela Cox Axel van Trotsenburg Country Director: Carlos Felipe Jaramillo Yaye Seynabou Sakho Senior Global Practice Director: Keith E. Hansen Jaime Saavedra Chanduvi Practice Manager: Chingboon Lee Emanuela Di Gropello Task Team Leader(s): Robert J. Hawkins Robert J. Hawkins ICR Contributing Author: Silvia Guallar Artal Page 4 of 82 The World Bank Education Quality Improvement Project (P126364) I. PROJECT CONTEXT AND DEVELOPMENT OBJECTIVES A. CONTEXT AT APPRAISAL Context 1. Country context. The 2008 global economic crisis represented a significant setback for El Salvador. After several years in the early 2000s of inclusive economic growth with important gains in terms of reduction of poverty and inequality, the global crisis strongly impacted El Salvador’s economy. As a result of the sharp decline in remittances, investment and foreign demand, the country’s Gross Domestic Product (GDP) contracted 2.1 percent in 2009, which rapidly deteriorated the fiscal situation and reversed the poverty and inequality trends.1 Within this context, the Farabundo Martí National Liberation Front (Frente Farabundo Martí para la Liberación Nacional, FMLN) party came into power in June 2009 after winning the Congressional and Presidential elections, and developed an Anti-crisis Plan for 2009-11 that included actions to mitigate the immediate impacts of the crisis while addressing longer-term development challenges such as social service delivery. By Project appraisal, El Salvador was starting to recover from the crisis, with a positive GDP growth of 2.1 percent, but this recovery was still fragile. 2. To understand the country context, it is also important to mention El Salvador’s high violence and crime rates, which are structural characteristics. This problem disproportionately affects the population in the bottom 40 percent and is the result of i) the socioeconomic dynamics of inequality and limited opportunity; ii) drug trafficking and the availability of drugs, drug money and weapons in the country; and iii) ineffective police and criminal justice institutions.2 In 2010, crime overtook economic issues as the number one problem in the country according to perception surveys and in 2011, when the Project was prepared, El Salvador ranked second in the world in terms of homicide rates per 100,000 inhabitants, following Honduras.3 Consequently, preventing exposure to crime and violence was one of the main priorities of the Government. 3. Sectoral and institutional context. By 2010, El Salvador had made significant progress in expanding access to primary education, reaching a net enrollment rate of 94.5 percent. However, this coverage was mostly expanded through the creation of small, community-managed schools that were under-resourced and lacked adequate equipment and effective teachers.4 Furthermore, most schools operated in two shifts, limiting the time for learning. As a result, the quality of education was low and student performance at grade 6, as measured by standardized achievement tests, was below the regional average in every subject area (mathematics, science, reading). 4. These quality shortcomings compounded over the years, translating into high repetition and dropout rates in secondary education. Dropouts accelerated at age 13, during the transition from primary to lower secondary education, and disproportionately affected the poor. Of the students aged 13 to 17 years old at the bottom income quintile, less than 60 percent enrolled in upper secondary education and only 15 percent enrolled in tertiary education, compared to 1 From 2001 to 2007, the Poverty headcount ratio at US$5.50 a day (2011 PPP) was reduced from 46.2% to 39.2% and inequality, measured through the GINI index, dropped from 51.4 to 45.2. In 2008, both social outcomes experienced a setback, with the poverty rate rising to 43.4% and the GINI index to 46.9 (WDI data). 2 Country Partnership Strategy for 2010-2012 (Report No. 50642-SV). 3 UNODC 2011. 4 These schools were created under the Community-Managed Schools Program (Educación con Participación de la Comunidad , EDUCO). Page 5 of 82 The World Bank Education Quality Improvement Project (P126364) 82 and 39 percent, respectively, for those in the top income quintile.5 The main reason for dropping out of school among youth aged 13-18 years was the lack of interest (35 percent), followed by the need to contribute to the household either through remunerated work for men (27 percent) or through domestic work for women (30 percent).6 This data highlights the lack of relevance of the education provided and the high opportunity cost of staying in school. 5. Additionally, school absenteeism was, and still is, a risk factor for involvement in criminal activity. Gangs tend to recruit poor children, between 12 and 25 years old,7 with lower education levels, and those that are out-of-school are particularly vulnerable to recruitment.8 6. In this context, the Ministry of Education (Ministerio de Educación, MINED9) articulated its 2009-2014 Education Sector Strategy10 with the concept of inclusive education as the organizing principle for all levels of school education. This included proposals to professionalize the teacher force, improve coverage of pre-primary education, better integrate science and technology across the curriculum, and begin the reform of the curricula of technical secondary education. School governance structures were also reformulated. A key program in this strategy was the Inclusive Full- Time School (IFTS) model, which was seen as the first phase to a more comprehensive reform of the secondary school system and aimed to address the problems of low-quality education and the exclusion of economically disadvantaged students. This strategy also meant to reduce crime and violence by addressing the challenges of retention and providing additional hours of schooling. The model had two main pillars: the extension of the school day, from 25 to 40 hours a week, in lower secondary education (grades 7 to 9) and the implementation of a new governance model, where schools were going to be clustered to share resources, leading to a more efficient allocation of the public education financing. 7. This Project was designed to support MINED in the implementation of the IFTS model, leveraging the Bank experience with the extension of the school day in other countries. Its objectives were aligned with the Country Partnership Strategy (CPS) for 2010-2012 (Report No. 50642-SV), particularly with regards to strengthening the delivery of social services (specifically, improving the quality and coverage of secondary education) and enhancing employment opportunities. It was also part of a more comprehensive package of interventions on human development. 5 Data from the Project Appraisal Document. 6 World Bank (2010); “Youth Development and Economic Opportunities in El Salvador,” Technical Paper prepared for the Non-Lending Technical Assistance on El Salvador: Human Development for Poverty Reduction. 7 Cruz, 2009; Santacruz Giralt et al., 2001. 8 Knox, 2018. 9 Currently, Ministry of Education, Science and Technology ( Ministerio de Educación, Ciencia y Tecnología, MINEDUCYT) 10 Plan Social Educativo “Vamos a la Escuela” 2009 -2014. Page 6 of 82 The World Bank Education Quality Improvement Project (P126364) Theory of Change (Results Chain) 8. The results chain used to evaluate the Project is shown in Figure 1: Figure 1. Results Chain11 9. The logic of the results chain of the Project is clear and well-supported by evidence in the literature. For further details, see Annex 7. Project Development Objectives (PDOs) 10. The objective of the Project was to improve access, retention and graduation rates for students in the Lower Secondary Education and the Upper Secondary Education of the Borrower’s public schools adopting the Inclusive Full Time School Model (IFTS Model). 11. While the Project included implementation of the IFTS model as a PDO indicator, this ICR considers its implementation as an intermediate outcome to facilitate the objectives of access, retention and graduation objectives. The Project was designed to pilot this pedagogical model as an effective approach to improve the quality of education, building on international experience from Uruguay, Chile, Argentina, and Mexico. The adoption of the IFTS Model was thus a prerequisite and means to improve access, retention and graduation rates for students in lower and upper secondary education, rather than an outcome in itself. The results chain above reflects this reasoning. 11Transportation appears with an asterisk because in the July 2015 restructuring, school feeding was added to the Model, financed with counterpart funds, and it would enter in the same category as transportation in the results chain. Page 7 of 82 The World Bank Education Quality Improvement Project (P126364) Key Expected Outcomes and Outcome Indicators 12. The PDO was to be measured through the following four indicators: (a) Improvement in the Access Rate in lower secondary education 12 (b) Improvement in the Retention Rate for lower secondary education (c) Improvement in the Graduation Rate for upper secondary education (d) Number of municipalities with clusters effectively implemented13 Components 13. The Project had the following two components: Component 1: Adoption of the IFTS model (Appraisal: Total: US$65.25 million, of which Bank US$54.85 million; Actual: US$61.61 million, of which Bank US$55.05 million; 93 percent of total Project cost) 14. Component 1 aimed to support the creation of appropriate teaching-learning conditions in selected schools to enable their adoption of the IFTS model. 1.1 Professional development and training: this subcomponent sought to support the design and implementation of three types of training: (i) accreditation program on the IFTS pedagogical model for MINED’s technical staff and directors and teachers in the targeted schools offering secondary education; (ii) orientation training on the IFTS model for the directors and teachers in all the remaining schools from the 29 targeted municipalities; and (iii) specifically designed training for MINED’s technical staff. 1.2 Educational materials and equipment: this subcomponent sought to support the acquisition of educational materials (books and art, sports and recreational materials) and equipment (computers, laboratory inputs and tools and equipment for students with special needs or disabilities) in the targeted schools offering secondary education. 1.3 Improvement of learning facilities: this subcomponent sought to support the improvement of learning facilities in the targeted schools offering secondary education through the renovation and construction of educational facilities and/or the acquisition of new furniture. 1.4 Extension of the teaching schedule: this subcomponent sought to finance the extra teaching hours required to extend the school day in the targeted schools offering secondary education. 1.5 Investments and/or activities (subprojects) in the participating and eligible schools: this subcomponent sought to directly transfer funds to the targeted schools offering lower secondary so that they could cover the other incremental costs, besides the teaching cost, associated with the implementation of the IFTS model, as well as the cost of some minor maintenance works. Component 2: Improvement of MINED’s Institutional Capacity and the Schooling System’s Governance (Appraisal: Total: US$5 million, all Bank resources; Actual: US$4.45 million; 7 percent of total Project cost) 15. Component 2 aimed to strengthen MINED’s capacity and support selected governance reforms at the central, departmental and cluster level to enable the implementation of the IFTS model. 12 This Project measured access rate as the transition rate between cycles. 13 This indicator aimed at monitoring the implementation of the IFTS Model. This indicator would be considered achieved in a given municipality if at least 4 of the following 6 factors were in place for all schools within all clusters in that given municipality: (i) New pedagogical approach implemented; (ii) Longer school day implemented; (iii) School directors and teachers of all grades trained in the features of the new model; (iv) Schools earmarked for civil works fully completed, including furniture; (v) Educational equipment and material delivered, and (vi) Territorial reorganization of IFTS schools within the cluster completed. Per the explanation in paragraph 11, this indicator is considered an intermediate indicator and not an outcome indicator. Page 8 of 82 The World Bank Education Quality Improvement Project (P126364) 2.1 Strengthening MINED’s capacity: this subcomponent sought to strengthen MINED’s policy-making, planning, implementation and monitoring capacity through the provision of technical assistance to, inter alia, develop policies, regulations and laws to regulate the IFTS model, develop and publish education standards, strengthen the Education Management and Information System (EMIS), improve the learning assessments and implement a communication strategy of the IFTS model. 2.2 Enhancing local governance mechanisms to support the clusters of IFTS: this subcomponent sought to support the design, implementation and process evaluation of the new IFTS governance model, which would replace the school-based management model that was in place at that time, in seven pilot municipalities. 2.3 Monitoring and evaluation: this subcomponent sought to strengthen MINED’s capacity, both at the central and departmental level, to monitor and evaluate the Project implementation. For that, it would hire technical coordinators and administrative coordinators in each of the 14 departmental offices to supervise the implementation of the IFTS in the 29 selected municipalities and would provide training in strategic planning and Project management. 2.4 Impact evaluation: this subcomponent sought to finance the evaluation of the impact of the IFTS model on both educational and social outcomes. B. SIGNIFICANT CHANGES DURING IMPLEMENTATION Revised PDOs and Outcome Targets 16. There were no changes to the Project's development objectives. Revised PDO Indicators 17. The restructuring approved on July 13, 2015 revised the results framework to capture the changes in the targeting and nature of Project activities. The first three PDO indicators were split into two to ensure that all Project development objectives were monitored for both lower and upper secondary education, and their targets were adjusted downwards. Additionally, the definition of PDO Indicator 4 was revised to simplify the monitoring of the implementation of the IFTS model at the school level (see Annex 6 for details on all changes). Revised Components 18. Shortly after Project effectiveness, on January 23, 2013, a restructuring was approved to change the implementation arrangements as a result of an internal reorganization of MINED, creating a specific Project Implementation Unit to be financed with Project funds. This change entailed the addition of a subcomponent under Component 2. 19. On July 13, 2015, a restructuring was approved to reflect strategic changes to Component 1, moving away from having MINED technical staff implement teacher training towards a more scalable cascading model to trains selected teachers to become the trainers and redefining eligibility criteria for the training. Changes were also made to the modality for teacher compensation and provision of transportation. Additionally, school feeding was added in the IFTS schools, financed with counterpart funds, and the use of extra time payments to teachers was enabled to expand the supply of upper secondary education. 20. In addition to the adjustments previously described, the July 2015 restructuring also entailed changes to the Project’s scope, concentrating the resources for the implementation of the IFTS Model into 40 out of the 101 initially Page 9 of 82 The World Bank Education Quality Improvement Project (P126364) targeted Integrated Systems (IS).14 The schools in the remaining 61 IS benefitted from the in-service training for secondary education teachers, educational materials, small works, and capacity-building activities from Component 2, with the purpose of getting them ready to fully implement the IFTS Model at a later stage. Other Changes 21. Other Project changes during implementation included reallocation across disbursement categories and changes in the disbursement schedule (July 2015, June 2017 and March 2018), changes in the World Bank’s procurement regulations to the Systematic Tracking and Exchanges in Procurement System (June 2017), and a one-year extension (June 2017). Rationale for Changes and Their Implication on the Original Theory of Change 22. This Project was an opportunity to test the implementation of a new education approach in El Salvador: the IFTS Model. As a result, most of the activities under Component 1 experienced changes to reflect the lessons learned during implementation and overcome different operational bottlenecks; it was a process of continuous improvement. The evolution of the most important pieces of the Model is summarized below: • Extension of the school day: the original Project design anticipated a shift in the school day from 25 to 40 class hours per week. In the July 2015restructuring, however, the definition of the extended day changed to 30 or more class hours per week to align it with MINED’s regulations. Additionally, the pedagogical content offered during the extra hours evolved: at first, most teachers would teach the same content in the regular and expanded school hours, which explained the low take-up and motivation of the students. When this was discovered, a significant effort was made to change the content to arts, sports and more practical trainings such as entrepreneurship, cooking and Information and Communication Technologies, decided in agreement with students to ensure their participation. Finally, in order to find enough professionals to offer the workshops, the regulation changed to increase the pool of eligible workers from only contract teachers (original design) to non- contract teachers (July 2015) and then craft professionals without a teaching degree but with a closer link to the community (June 2017). • Infrastructure: in the July 2015 restructuring, the approach for the infrastructure interventions changed, concentrating the same number of learning spaces to be renovated into fewer schools to offer a more comprehensive set of infrastructure inputs given the poor state of many of the schools targeted for this activity, which required more extensive works than originally envisaged. The design of the integral infrastructure that was implemented in the Project has become the infrastructure standard for MINED. • Territorial reorganization: the Project originally envisioned that the school clusters, later called IS, would work as “campuses” of a same unit, with full integration of the school resources and student mobility. However, the country’s high crime and violence and the existing legislation prevented the fluid mobility of teachers and students. As a result, although some coordination between the different schools remained (evidenced by the monthly meetings of directors and teachers from the different schools belonging to the same IS and the shared pedagogical plans elaborated at the IS level), some elements of the IFTS Model, such as school equipment, became exclusive for students on school premises. • Student transportation: the transportation strategy was revised in the July 2015 restructuring, its direct provision by local firms was replaced by a stipend to socio-economically disadvantaged students in secondary education. Because of the country’s high crime and violence, firms were not willing to offer transportation services across 14The Integrated Systems are the clusters of primary and secondary schools that were created to test the new governance approach under the IFTS Model, where schools would share resources and work in close coordination. The 29 municipalities targeted by the Project were divided into 101 Integrated Systems. Page 10 of 82 The World Bank Education Quality Improvement Project (P126364) gang lines, since it endangered their workers and the students. Then, the scope of this component was reduced, since the need for the movement of students decreased as more schools in the IS offered workshops on their premises. • School feeding: this activity was added during the July 2015 restructuring and financed with counterpart funds, acknowledging its key role in allowing the most vulnerable students to stay in school for the extended day. • Teacher training: in-service teacher training was the activity that changed the most significantly. Moving away from the original approach of having MINED technical staff implementing it, the Project tested a new strategy based on a cascade training model, in which selected teachers (specialists) were trained first to later train other teachers. The primary focus of the training was subject content divided into eight modules and was delivered mainly in person, with some online activities. An independent review of the teacher training programs in El Salvador, including the ones supported by the Project, found that, these followed good international good practices overall.15 The new in-service training approach would set the foundation for the creation of the National Institute for Teacher Training (Instituto Nacional de Formación Docente, INFOD). Additionally, given the positive reception of the training from teachers, training in art, physical education and indigenous cultures were added to the list of subject areas offered in the last year of implementation. 23. MINED’s commitment to the IFTS Model was also illustrated in its internal reorganization. The new structure elevated the Project’s coordination to a Directorate level, creating a new unit that would report directly to the Vice Minister, with four key project staff: (i) Project coordinator, to ensure the correct implementation of the Project; (ii) Territorial coordinator, to support the governance changes; (iii) Organizational coordinator, to support the school clustering; and (iv) Pedagogical Coordinator, to support the implementation of the pedagogical reforms focused on active student-centered approaches and teacher professional development. This new unit lacked resources and MINED requested that the Project support its financing. Given that these expenses were not accounted for at Project design, a new subcomponent was added to support the management, financial and technical needs of the Project. 24. All the changes described above entailed budget revisions and, therefore, reallocation of funds across disbursement categories. Additionally, the delays in project implementation, particularly in the infrastructure component, resulting from MINED’s strategic shift and other operational bottlenecks impacted implementation, requiring an extension of the Project closing date, to fully implement and evaluate elements of the IFTS Model. 25. Finally, the revision of PDO indicators in the July 2015 restructuring served several purposes (further detail is provided in Annex 6). First, the split of the first three PDO indicators into two ensured the correct monitoring of the PDO for both lower and upper secondary education. Second, as a result of MINED’s decision to concentrate project resources in 40 IS, the universe of schools where the Project was expected to impact access, retention and graduation was reduced and, therefore, the baseline figures had to be recalculated. Third, when the baseline was recalculated, the team recognized that in the recent years student access, retention, and graduation rates in lower and upper secondary education had shown a declining trend at the national level due to macro factors such as deteriorating economic conditions, migration and increased crime and violence. The team thus set targets to keep indicators at their baseline level until Project closing, which would be an improvement given national trends. Finally, the methodology to calculate the access rate changed to align it with the UNESCO definition of effective transition rate and the definition of PDO Indicator 4 was revised to simplify the monitoring of implementation of the IFTS Model at the school level. 26. Given that these restructurings did not entail any changes to the PDO, the theory of change remains unchanged. 15 Characterizing In-service Teacher Training in El Salvador (David Evans, unpublished) Page 11 of 82 The World Bank Education Quality Improvement Project (P126364) II. OUTCOME A. RELEVANCE OF PDOs Assessment of Relevance of PDOs and Rating 27. The relevance of the Project’s objectives is High. The Project’s objectives were fully aligned with the CPS at the time of Project preparation,16 particularly with its strategic objectives of strengthening the delivery of social services (specifically, improving the quality and coverage of secondary education) and enhancing employment opportunities. They also remained highly relevant to the Bank’s Country Partnership Framework (CPF) for 2016-2019 (Report No. 95185), which explicitly mentions improving secondary school attainment and enhancing youth employability and skills as two of the Bank’s engagement priorities to boost competitiveness, expand opportunities, and help break the vicious cycle of low growth and migration. 28. At the time of preparation, the Project was also aligned with El Salvador's 2010-2014 National Development Plan, which highlighted the need to improve the quality of education, reduce dropout rates and prevent violence, and the 2009- 2014 Education Sector Plan, which already introduced the IFTS Model as the new pedagogical model for the country. The Project objectives remained highly relevant to the priorities of the updated National Development Plan 2014-2019, which places investments in education among the top three national priorities, and with the Education Sector Plan, both of which point towards improvements in the quality of education by strengthening teacher training, modernizing infrastructure and equipment in schools, increasing education access for the poor and vulnerable, and ensuring safety in schools; all of which were supported by the Project. 29. Finally, relevance is also demonstrated by the expansion of the IFTS Model to 45 additional IS (349 schools) with US$100 million financing from the Millennium Challenge Corporation (MCC). B. ACHIEVEMENT OF PDOs (EFFICACY) 30. This section assesses Project performance in relation to the periods before and after the second restructuring, at which the Project scope and PDO targets were revised. Achievement of outcomes over the entire duration of the Project is evaluated separately for both original and revised targets and scope (101 IS and 40 IS, respectively) to arrive at an overall outcome rating. 31. As mentioned earlier, this Report considers that the Project aimed to improve access, retention and graduation rates for students in lower and upper secondary education of the Borrower’s targeted public schools through the adoption of the IFTS Model and, therefore, the implementation of the IFTS model is not considered a PDO indicator, but rather an intermediate outcome. 32. Finally, it is worth mentioning that the Project managed to implement a semi-experimental evaluation with administrative data to measure the impact of the bundle of activities under the IFTS Model in access, retention and graduation rates in project schools compared to a set of control schools chosen with propensity score matching. Therefore, although the efficacy assessment will look at achievement of the PDOs, the ratings will also be based on the results of the impact evaluation, since the fluctuation of the access, retention and graduation rates in the targeted IS can be due to many factors outside the Project’s control (macroeconomic shocks, spikes of violence, etc.) while the 16 Country Partnership Strategy for 2010-2012 (Report No. 50642-SV) Page 12 of 82 The World Bank Education Quality Improvement Project (P126364) impact evaluation determines causality. Assessment of Achievement of Each Objective/Outcome Objective 1: Increase access (transition rates) to lower and upper secondary education (rating: modest) Original (101 IS) Revised (40 IS) Target Achieved % Achieved Rating Target Achieved % Achieved Impact17 Rating Lower secondary 98.8% 95.1% -48% Negligible 96.6% 96.7% 100% Not significant Modest Upper secondary NA NA NA NA 78% 87% 112% Not significant 33. The post-restructuring targets for PDO Indicator 1, measuring the transition rates between cycles, were achieved in lower secondary (with an access rate of 96.7 percent) and surpassed in upper secondary (87 percent compared to the target of 78 percent). The target for the original indicator, which only measured lower secondary, was not attained. Access, defined as the transition rate between education cycles, is expected to be positively impacted by all the components of the IFTS Model through the improvement of quality, which increases the pool of students ready to transition to the following education cycle (increasing retention and graduation rates). However, the availability of infrastructure is particularly important, since it enables students to continue studying within the same premises, lowering their probability of dropping out during this transition. Given that the infrastructure works were finalized close to Project closing, the impact evaluation did not capture changes in the access rates resulting from these interventions because they had not had time to materialize and are only expected to appear in the upcoming years. Since the PDO targets were attained but no causality can be claimed for these results yet, the overall rating of this objective is modest. Objective 2: Increase retention rates in lower and upper secondary education (rating: substantial) Original (101 IS) Revised (40 IS) Retention Target Achieved % Achieved Rating Target Achieved % Achieved Impact Rating Lower secondary 98.4% 93.7% -42% Negligible 91% 93.8% 103% Positive Substantial Upper secondary NA NA NA NA 92.5% 92.8% 100% Not significant 34. The revised targets for PDO Indicator 2, measuring the retention rates within the two secondary cycles, were achieved, with retention rates of 93.8 percent in lower secondary (compared to the target of 91 percent) and 92.8 percent in upper secondary (compared to the target of 92.5). The target for the original indicator, which only measured lower secondary, was not attained. The results of the impact evaluation confirm that the achievement in lower secondary can be attributed to project interventions, finding that the Project managed to decrease the dropout rate by 3.4 percent in project schools compared to control schools, which translates into 529 additional children per year that remain in the education system. On the other hand, the evaluation did not find any significant impact for upper secondary, which might be because most of the interventions targeted only lower secondary students or because the Project did not manage to close the increasing learning gaps as students progressed to upper secondary. Nevertheless, the Project’s net impact is positive, leading to more students staying in school and increasing their educational attainment. Therefore, the overall rating of this objective is substantial. 17 Project's attributable impact according to the results from semi-experimental impact evaluation carried using administrative data. Page 13 of 82 The World Bank Education Quality Improvement Project (P126364) Objective 3: Increase graduation rates in lower and upper secondary education (rating: substantial) Original (101 IS) Revised (40 IS) Target Achieved % Achieved Rating Target Achieved % Achieved Impact Rating Lower secondary NA NA NA NA 69% 71.4% 103% Positive Substantial Upper secondary 91.7% 73.8% -7% Negligible 71.5% 75.9% 106% Negative* 35. The revised targets for PDO indicator 3, measuring student graduation in the targeted IS, were achieved, with graduation rates of 71.4 percent in lower secondary and 75.9 percent in upper secondary. The target for the original indicator, which only monitored upper secondary, was not attained. The results of the impact evaluation for this objective are also positive, finding that the Project managed to increase graduation rates in lower secondary in project schools by 4.1 percent compared to the control schools, which translates into 597 additional children per year that successfully completed their school year. On the other hand, the impact evaluation found that graduation rates for upper secondary in project schools are significantly lower than in control schools. This result is likely driven by the fact that once more students graduate from lower secondary due to Project support, some of the students that now move onto upper secondary are unable to keep up with its requirements and therefore do not graduate. Nevertheless, given that the Project’s net impact is positive and increases overall educational attainment, the rating for this objective is substantial. 36. As the results chain shows, all of the IFTS Model’s components are associated with all three Project objectives, since the main goal of the Model is to improve education quality. This translates into higher retention and graduation rates, increasing the pool of students ready to transition into the next education cycle and, consequently, increasing the number of students who access the next cycle (with the infrastructure interventions reinforcing this virtuous cycle). Therefore, all the intermediate indicators that monitor the implementation of the IFTS model are associated with all three Project objectives. Their achievements are summarized below: • Improved physical environment for learning. By Project closing, a total of 36 schools were fully renovated, with integral infrastructure; 441 schools received school furniture; and 230 benefited from minor repairs. • Increased supply of materials for learning. The Project provided 195 schools with educational materials and equipment, including computers and projectors, library books, sports equipment, and art supplies. • Additional class time. In terms of extension of the school day, the number of students from lower secondary that participated and stayed in school for at least 30 hours per week increased progressively over the Project’s life, starting with 4,140 students in 2015 and covering 9,842 students by Project closing. • Lower opportunity cost of studying. Over the Project’s life, the number of students receiving the school transportation stipend increased progressively, starting with 4,161 in 2016 and covering 9,759 by Project closing.18 • Improved teaching and school management practices. A total of 1,150 master teacher trainers, 1,619 teachers, and 436 school directors were certified. • Implementation of the IFTS Model. In addition to the achievements mentioned above, of the original 213 schools offering secondary education in the 40 targeted IS, all received training on the features of the IFTS Model for directors and secondary teachers, 195 belong to an IS with a Pedagogical Plan, 133 offer 30 hours or more per week, and 112 participated in the territorial reorganization of the IS as evidenced by student 18 Although the provision of food through the school feeding also reduces the opportunity cost of studying, the number of beneficiaries was not monitored since this activity was not part of the initial Project design and was financed exclusively with counterpart funds. Page 14 of 82 The World Bank Education Quality Improvement Project (P126364) transportation. Given that the IFTS Model is considered fully implemented when at least four of the six defined elements are met, 163 of the 213 targeted schools fully adopted the IFTS model by Project closing. Justification of Overall Efficacy Rating 37. As described above, there is substantial evidence that the Project significantly improved retention and graduation rates. Improvements in the third objective of access are expected to be reflected in the coming years, since the infrastructure interventions were finalized too close to Project closing to measure their impact. The literature on school extension supports this expectation: Padilla-Romo (2017) found that the impact of extending the school day in Mexico was strongest four years after its introduction and Bellei (2009) suggests that teachers may need time to adapt to the longer school day and that it may take schools a few years to work out implementation issues. Consequently, the efficacy of the Project is rated as Substantial. C. EFFICIENCY Assessment of Efficiency and Rating 38. The Project’s benefits are Substantial. An economic analysis of all activities related to the IFTS Model shows positive results in both the cost-benefit analysis and the Internal Rate of Return (IRR).19 The Project’s economic benefits are based on the assumption that the interventions will improve the school learning environment, which will lead to improved learning and will encourage attendance for students that might not have enrolled or might have dropped out otherwise, increasing graduation rates. This will translate into higher years of education for the Project’s beneficiaries for each year of participation in the Project, which, in turn, will lead to higher productivity and earnings in the labor market. On the other hand, the costs are composed of the investment cost of implementing the components of the IFTS and the recurrent costs to maintain the improved school infrastructure, which will continue to benefit students after Project closing. Assuming that an additional year of education in El Salvador increases annual income by 9.3 percent,20 the Project’s IRR is estimated at 8.9 percent, which is similar to the estimates at appraisal (8.4 percent) and the July 2015 restructuring (9.0 percent).21 With a 5 percent discount rate, the Net Present Value (NPV) is US$64.1 million. Even under a pessimistic scenario where the estimated returns to education are reduced by 30 percent, the Project’s NPV is positive, with a 5 percent discount rate. These estimates are considered conservative, since the analysis does not take into consideration the potential benefits that are harder to quantify in monetary terms, such as the potential impact that the extended school day programs may have in reducing teen pregnancies and juvenile crime, increasing labor participation rates for mothers of school-aged children,22 or the possibility that the Project’s benefits may persist or even grow over time. Finally, it is worth mentioning that this analysis does not estimate the cost-benefit of the activities to improve the institutional capacity and governance of El Salvador’s schooling system given the difficulty to quantify the benefits, but these may also have had a positive impact by improving the system’s efficiency. 39. In terms of efficiency of implementation, the Project experienced delays as a result of MINED’s change of strategic vision and the infrastructure challenges due to security and land titling issues, which translated into a one-year extension 19 The economic analysis did not exploit the impact evaluation results since this study only measures the impact of the bundle of the IFTS Model interventions on students that are in the beneficiary schools by Project closing and does not differentiate by type of intervention or intensity of exposure to the treatment. Therefore, a more detailed approach was followed for this analysis, basing the benefit estimates on the expected impact for each activity according to the literature. To compensate for the shortcoming of not using estimates from El Salvador, the team was conservative in its assumptions to show that even then the Project was worth the investment. 20 Montenegro & Patrinos, 2014. 21 However, the comparison of these results should be taken with caution since the approach used is different. In this case, the benefits are estimated in a more precise way, by dividing the universe of beneficiaries into non-overlapping sub-universes according to their degree of exposure to the Project. 22 Berthelon & Kruger, 2011; Contreras & Sepúlveda, 2016; and Pires & Urzua, 2015. Page 15 of 82 The World Bank Education Quality Improvement Project (P126364) of the Project’s closing date. One could argue, however, that some of the inefficiencies from implementation delays were recovered, since the restructurings led to an improvement in the design of activities, especially of teacher training and infrastructure quality. 40. Consequently, the efficiency of the Project is deemed to be Substantial. D. JUSTIFICATION OF OVERALL OUTCOME RATING 41. Calculating the weighted value of the outcome rating under both objectives yields a rounded score of 4 (see Table 2 below), which translates to a final outcome rating of Moderately Satisfactory. Table 2: Application of Split Rating for Project Outcomes Before Restructuring After Restructuring no. 2 no. 2 Relevance of Objectives High Efficacy (PDO) Negligible Substantial • PDO 1 (a). Improved access in lower secondary Negligible Modest • PDO 1 (b). Improved access in upper secondary Modest • PDO 2 (a). Improved retention in lower secondary Negligible Substantial • PDO 2 (b). Improved retention in upper secondary Substantial • PDO 3 (a). Improved graduation in lower secondary Substantial • PDO 3 (b). Improved graduation in upper secondary Negligible Substantial Efficiency Substantial Outcome ratings Unsatisfactory Satisfactory Numerical value of the outcome ratings23 (1) 2 5 Disbursement (millions) US$13.71 US$45.93 Share of disbursement (2) 23% 77% Weighted value of the outcome rating [(1)x(2)] 0.46 3.85 Moderately Satisfactory Final outcome rating (0.46 + 3.85 = 4.31, rounded to 4) E. OTHER OUTCOMES AND IMPACTS Gender 42. Gender inclusiveness was considered in the design of the infrastructure investments, building gender-separated latrines to ensure girls’ privacy and safety, an intervention that has proven to be effective in keeping the girls in school in several countries. Additionally, anecdotal evidence from MINED indicates that the extra-curricular activities contributed to gender equality by offering all student activities that were more female or male dominated according to the gender norms. As such, Project activities were in line with gender inclusiveness. Institutional Strengthening 43. The Project’s second component specifically addressed institutional strengthening, targeting MINED’s capacity for policy-making, planning, and monitoring the IFTS Model. The main activities supporting institutional strengthening Page 16 of 82 The World Bank Education Quality Improvement Project (P126364) implemented with Project financing were: • Design and implementation of a communication strategy: The Project financed a consultancy for the design of a communication strategy for the IFTS Model and then supported its implementation. Each IS would have its own communication portal in a social network to share achievements and advertise upcoming events in the schools belonging to the IS. These pages are updated by staff from the IS, usually the computer science teacher. By Project closing, 38 of the 40 targeted IS had an active communication platform. It is worth mentioning that in addition to the communication activities at the IS level, the Project also financed broader communication campaigns such as radio programs, advertising spots, and multimedia banners that disseminated the IFTS model at the national level. • Development of the Education Management Information System: when the Project was designed, MINED already had a strong monitoring system that allowed full tracking of the Project’s results through school census data that was collected annually. However, this data is collected at the school level and does not allow tracking of students over time. Additionally, MINED lacked a platform to manage its different programs. Consequently, the Project financed the design of a platform to manage MINED’s education programs (Sistema Integrado de Administración de Proyectos, SIAP II) and a strategy to upgrade the information system (Sistema de Monitoreo y Evaluación para el Sistema Integrado de Escuela Inclusiva de Tiempo Pleno, SIMES). The first was implemented and the second is being finalized with MCC support. • Evaluation of governance approach for the IFTS Model: throughout project implementation, two governance approaches were piloted and evaluated in seven targeted municipalities, extracting the lessons learned through a survey and follow-up interviews to the main stakeholders and providing recommendations moving forward. • Impact evaluation of the IFTS Model: by closing, a semi-experimental evaluation with administrative data was carried out to measure the impact of the bundle of interventions under the IFTS Model on access, retention and graduation rates in the targeted IS compared to a group of control schools selected using propensity score matching. This evaluation represents an important achievement since it is the result of a continuous dialogue with MINED to raise awareness about the importance of evaluating the Project and not only monitoring its results. By the end of the Project, MINED was empowered and managed to implement it, overcoming a change in the evaluation strategy and procurement obstacles. • Technical assistance for the implementation of the IFTS Model: 28 technical assistants were hired with Project funds in the 14 departments to guide schools in the implementation of the IFTS Model, providing technical assistance in the elaboration of the pedagogical plans and the choice of the content of the extended school hours. 44. It is worth highlighting, however, that the main channel through which the Project managed to strengthen MINED’s institutional capacity was the piloting and refining of the implementation of the IFTS Model. The implementation of this Model with MCC funds (US$100 million) in 45 additional IS takes into account all the lessons learned throughout Project implementation in terms of the IS governance, teacher training strategy, extension of the school day, environmental management of works, and infrastructure design. The Project contributed to the institutionalization of the IFTS Model. 45. Finally, in addition to the activities implemented with Project funds, the Bank team secured a Trust Fund for Statistical Capacity Building (TFSCB) to develop educational indicators and build capacities to interpret statistics related to compulsory education, which led to the creation of the National System for Education Evaluation (Sistema Nacional de Evaluación Educativa, SNEe) and helped change the culture in MINED and raise awareness of the importance of evaluating the education policies and having a good monitoring system that enables informed decision-making. Although these funds were executed by the Bank, MINED was fully involved in the process, hosting the consultant hired for this activity in their offices as an additional team member and fully managing the development of the SNEe. The Bank managed only the procurement processes to reduce the administrative burden for MINED. Page 17 of 82 The World Bank Education Quality Improvement Project (P126364) 46. The above statements suggest that the operation had a positive impact on institutional strengthening and lays the groundwork for the longer-term development of the country’s capacity. The improved functioning of MINED also contributes towards the Project outcomes. Mobilizing Private Sector Financing Not applicable Poverty Reduction and Shared Prosperity 47. The Project contributed to reducing poverty and boosting shared prosperity by targeting vulnerable municipalities with lower educational outcomes in terms of access, retention and graduation rates, and higher crime rates. Additionally, although the impact evaluation could not capture changes in crime as a result of the Project’s activities, a rigorous study of a similar after-school program in El Salvador found that the extension of the school day reduced violent and anti-social behavior of participants and had positive spillover effects on other students.24 As violence disproportionately affects the population in the bottom 40 percent, it is likely that the Project also contributed to boosting shared prosperity through that angle. Other Unintended Outcomes and Impacts 48. As a result of the social safeguards work around indigenous populations, the Project supported MINED to develop a program on Náhuat language25 and identity, an activity not originally considered in the Project’s design. This is a historic achievement because it is the first time in El Salvador that MINED offered an institutionalized teacher training on "Náhuat language and cultural identity," and this effort will help preserve a language that was disappearing. A total of 15 textbooks for 6 modules of the program were developed, including those that will be used by students. Teacher training reached 89 teachers from 61 schools from the targeted area and other regions of the country, responding to the growing demand in the capital of El Salvador. Starting in Module 5, the teacher training included the participation of Náhuat speakers in a systematic way as expert consultants. They facilitated the practice of the language and strengthened the socio-cultural aspects of the training. This milestone had a positive impact among participating teachers, but also on a wider range of stakeholders, through sensitization on indigenous issues. III. KEY FACTORS THAT AFFECTED IMPLEMENTATION AND OUTCOME A. KEY FACTORS DURING PREPARATION 49. The previous education Project in El Salvador, the Excellence and Innovation in Secondary Education (EXITO) Project (P078993), was approved by the Board on November 29, 2005, but never became effective because it was not ratified by Congress. The same happened to four other projects prepared in 2005 and 2006. To mitigate the risk of this happening again, the Project was prepared quickly, capitalizing on the strong political support from the Minister of Education at the time, who became President in 2014 and took ownership of the Project due to his strong belief in the IFTS Model. 50. Another key factor during preparation is that the timeline did not allow for a diagnostic on the ground of the infrastructure needs of schools, which led to an underestimation of the depth of the renovations required and, consequently, an underestimation of the costs for this activity. Finally, it could be argued that the Project’s scope was too 24 Dinarte & Egana del Sol, 2018 25 Náhuat is the language originally spoken by the indigenous population from El Salvador, which was disappearing with the new generations. Page 18 of 82 The World Bank Education Quality Improvement Project (P126364) ambitious in terms of the number of schools targeted, given that this Project aimed to finance a pilot. B. KEY FACTORS DURING IMPLEMENTATION 51. The loan became effective on May 15, 2012, around 5 months after Board approval. Shortly after, in January 2013, a restructuring was swiftly processed to change the implementation arrangements, which led to satisfactory execution of the Project. Execution later slowed as a result of leadership changes in MINED after the March 2014 Presidential elections, which led to a change of vision within MINED that led to an in-depth restructuring that took 9 months to be agreed and processed (more details on this restructuring can be found in the section Significant Changes During Implementation). 52. Other factors that caused implementation delays were (i) delays in the confirmation of land titles and revisions to the original specifications for infrastructure interventions due to the change towards more in-depth works in fewer schools; (ii) inefficiencies in the procurement processes due to long turnaround times for each internal procurement step; and (iii) the requirement of counterpart financing for certain activities given national fiscal constraints and bureaucracy. 53. Finally, a factor outside the control of the Government and the Bank was the spike in violence, as evidenced by the homicide rates that followed the end of the truce with the gangs in 2014. During the truce, from 2012 to 2014, homicides fell to 5 per day, compared to 12 prior to the truce,26 and then increased significantly, reaching the world’s highest rate in 2015. In subsequent years, homicides declined, but crime figures still place El Salvador at the top of the global ranking. IV. BANK PERFORMANCE, COMPLIANCE ISSUES, AND RISK TO DEVELOPMENT OUTCOME A. QUALITY OF MONITORING AND EVALUATION (M&E) M&E Design 54. M&E design is assessed as Modest. As previously mentioned, the adoption of the IFTS Model should not have been a PDO indicator but rather an intermediate outcome indicator enabling improvements in the access, retention and graduation rates in secondary education. Additionally, although the PDO explicitly separated the objectives for lower and upper secondary education, the PDO indicators only monitored progress in one of the two education levels. Finally, the PDO targets were too ambitious, aiming to achieve significant improvements when the declining trends at the national level had started before Project preparation, as described and acknowledged in the July 2015 Project restructuring. 55. The country’s capacity to implement an experimental impact evaluation was also overestimated. Although MINED was able to collect data for monitoring purposes, the institutional culture did not give importance to rigorous measurement of impact. The Project initially envisioned a randomized control trial of the teacher training strategy, relying on two complex ad-hoc data collections. This evaluation strategy was designed by the Bank team, with limited ownership from MINED. As a result, despite several trainings and intensive support for the evaluation’s design and implementation, the control group was contaminated and the baseline data could not be used. Ex-post, it seems that the resources spent for the collection of the baseline data would have been better used in the implementation of a rigorous process evaluation that followed Project implementation from the beginning, especially because this Project was piloting a new pedagogical approach and, consequently, many parts of the IFTS Model were improved over time. The semi-experimental impact evaluation that was finally carried out using existing administrative data was the appropriate approach to evaluate the 26 Rodriguez (2013) Page 19 of 82 The World Bank Education Quality Improvement Project (P126364) Project given the country context. 56. On the positive side, the intermediate indicators were clearly defined, and their measurement relied on existing data sources or administrative records that could be easily generated through Project implementation. M&E Implementation 57. M&E Implementation is assessed as Substantial. The Project team closely supervised the data reported by MINED and, if any discrepancies arose, intensive technical assistance was provided to identify the reason and ensure that the measurement methodology was understood and used consistently. The mechanisms used to collect data were appropriate and the data was, in general, found to be reliable and of good quality. M&E was rated Moderately Satisfactory or Satisfactory throughout Project implementation. 58. Most of the changes made to Project indicators during implementation were to capture the changes in the nature and scope of project activities. With the exception of PDO indicators one and four, there were no substantial revisions to the calculation methodology (see Annex 6 for more detailed information). The changes did not significantly alter the meaning of the indicators or inhibit their ability to measure the outputs or outcomes as intended. 59. A minor shortcoming in M&E implementation was that the student-level information system planned under Component 2 was not implemented by Project closing. However, MINED is aware of the importance of having a functional EMIS and is now finalizing it with support of the MCC. M&E Utilization 60. M&E Utilization is assessed as High. The Bank team proactively raised funds to complement MINED’s monitoring and inform project implementation. During preparation, the team received a grant from the Spanish Fund for Latin America and the Caribbean (SFLAC), executed by MINED, to finance the five studies that would support early stages of implementation: (i) an assessment of the reform approach to improving governance and accountability; (ii) a proposal of standards of efficiency in school management, student learning, and governance; (iii) a proposal to create a control panel for school management; (iv) a communication strategy; and (v) a study on the implementation of providing lunch to students in the IFTS Model. 61. In 2012, the Bank also secured a TFSCB in the amount of US$98,680 to develop educational indicators and build capacities to interpret statistics related to compulsory education, which, as previously mentioned, led to the creation of the SNEe and helped raise awareness about the importance of evaluating education policies and having a good monitoring system that enables informed decision-making. 62. Additionally, the Bank team carried out studies to understand the challenges in the implementation of the IFTS Model. In 2017, a phone survey to directors from the targeted schools was conducted to understand the reasons for the low student take-up of the extended school day. As a result of this study, the content of the extra-curricular hours was revised to make them more appealing to students. In 2018, the findings from this study were complemented by focus group discussions with students and parents to capture their understanding of the IFTS Model and perception of the value added of the extension of the school day. Both studies were greatly appreciated by MINED and disseminated in beneficiary schools. 63. Finally, the governance and impact evaluations generated relevant insights that will be taken into account by MINED in the subsequent phases of implementation of the IFTS Model. Page 20 of 82 The World Bank Education Quality Improvement Project (P126364) Justification of Overall Rating of Quality of M&E 64. Despite the shortcomings at design, the M&E quality at implementation is considered substantial and its utilization high. Therefore, the overall quality is rated as Substantial. B. ENVIRONMENTAL, SOCIAL, AND FIDUCIARY COMPLIANCE 65. Environmental safeguards. The Project was rated Category “B” and only triggered OP/BP 4.01 on Environmental Assessment because it involved a large number of small civil works, mostly maintenance and renovation, that were not expected to generate significant environmental impact. The shift towards more in-depth interventions in July 2015 increased the environmental impact, but did not strengthen the environmental monitoring, leading to an exceptional downgrade in the environmental rating to Moderately Satisfactory. The Bank team and MINED then agreed on the use of a monitoring tool to be completed periodically that enabled closer supervision and developed a Project Management Plan for 19 of the 36 construction works to fix the observed deficiencies. In December 2017, the environmental safeguards rating was upgraded to Satisfactory and all 36 infrastructure works were environmentally certified by Project closing. In addition to the closer monitoring, the Bank team also provided training to the staff in MINED and the construction firms, enhancing their environmental awareness. 66. Social safeguards. OP/BP 4.10 Indigenous Peoples’ was triggered under the Project. The Borrower prepared and consulted an Indigenous Peoples Plan (IPP) that included five components: (i) fostering educational exchange at the local level (clustering); (ii) sensitization; (iii) cultural and curricular research; (iv) development of educational materials; and (v) training for in-service teachers. Following the creation of the Consultative Committee in 2012 and the sensitization campaign, the Government faced critical challenges in hiring a company or local organization to implement IPP actions. Implementation of social safeguards gained momentum in March 2017, however, and the MINED has also taken ownership of the new activities for the Program on Náhuat language, which represents a historical achievement. 67. Financial Management (FM). The Project complied with FM requirements and sent most audit reports to the Bank in a timely fashion. Audit opinions were unmodified throughout project implementation, although some internal control issues were reported. Interim Financial Reports (IFRs) were generally sent without delays and were always considered acceptable. While issues around documenting transfers and the discovery of ineligible expenditures led to ratings of Moderately Satisfactory, the Bank team provided ongoing support to overcome these issues and ensure that there were no questionable funds. 68. Procurement. Project procurement arrangements and plans were adequate throughout most of the implementation period. Procurement was carried out by MINED’s Procurement Division (Unidad de Adquisiciones y Compras Institucionales, UACI). The Bank provided direct training to the analysts and MINED procurement staff and contributed to quality control through close supervision. The efficiency in the procurement processes improved significantly over time, reducing the internal turnaround times for each step and improving the communication within MINED between the technical and procurement teams to avoid multiple iterations in the specifications of each process. Project procurement was rated satisfactory for most of the implementation.27 27When procuring the Project’s evaluation, in June 2018, the procurement rating was downgraded for the first time from Satisfactory to Moderately Unsatisfactory due to a complaint received from one bidding firm regarding the evaluation of proposals. This led to the cancellation of the process and the procurement through a new one, resolved satisfactorily. In November 2018, the rating was upgraded to Moderately Satisfactory. Page 21 of 82 The World Bank Education Quality Improvement Project (P126364) C. BANK PERFORMANCE Quality at Entry 69. Quality at entry is rated as Moderately Satisfactory. The Bank leveraged the political window of opportunity, managing to prepare and appraise the Project in a short timeframe, taking less than six months from concept note to Board approval. The objectives were clear and strongly aligned with the national development and education strategies and the Bank engagement priorities for El Salvador. The Project adequately addressed economic and financial issues and provided due diligence on aspects such as poverty and shared prosperity, in line with Bank priorities. 70. The swift preparation, however, resulted in shortcomings. First, the Bank underestimated the implications that the country’s high crime and violence would have on project implementation. Although some could not have been foreseen, such as the delays in the school works resulting from gang threats due to a lack of precedent on this issue, the country reality should have been taken into account when planning for student and teacher mobility within the IS. Additionally, the SFLAC funds could not be used to inform Project design and were instead used as technical assistance in the early stages of the Project. Finally, there were shortcomings in estimating infrastructure costs and the depth of the interventions needed, since the tight Project preparation timeline did not allow for a full diagnostic of the state of the beneficiary schools. Quality of Supervision 71. The quality of supervision is rated as Satisfactory. The Bank worked proactively and closely with MINED throughout implementation. The Bank team included two infrastructure consultants, one expert on teacher training, and a local consultant to implement the diagnostic studies on the implementation of the IFTS Model. The team held regular meetings with the Project Implementation Unit and the directorates implementing the different Project activities and organized missions to provide targeted technical assistance and training as needed. The frequency and quality of performance reporting was high and candid. The team quickly responded to issues during implementation, in communication and cooperation with both the Borrower and Bank management. Progress and challenges were transparently recorded to ensure all parties were on the same page and to focus dialogue on the pertinent issues, reaching solutions to achieve development outcomes, as reflected by the various restructurings that took place during implementation. 72. Another strength in the quality of the supervision was that the team composition remained stable over time, with the same Task Team Leader throughout implementation and little rotation of team members. Finally, the Bank was commended by MINED for its flexibility to respond to the changes in the implementation of the Project to overcome challenges or respond to changes in MINED’s priorities, while maintaining its focus on achieving the development outcomes. Justification of Overall Rating of Bank Performance 73. The services provided by the Bank ensured quality of supervision. However, given the shortcomings detailed at Project entry, the Bank’s overall performance is considered Moderately Satisfactory. D. RISK TO DEVELOPMENT OUTCOME 74. The risk to the development outcome is considered Low. The expansion of the IFTS Model is part of the country’s strategic objectives, both for the five-year National Development Plan and for the education sector plan. The newly elected president, elected in early 2019, included the IFTS Model as part of his electoral program. Additionally, the MCC is already funding the subsequent phase of this Model, incorporating the lessons learned from this Project in their design. Page 22 of 82 The World Bank Education Quality Improvement Project (P126364) 75. Finally, the sustainability of the teacher training strategy is expected to be ensured through the establishment of the INFOD. This institute will carry out educational research and will develop standards for teacher training programs, building upon lessons from Project implementation. V. LESSONS AND RECOMMENDATIONS 76. There is trade-off between speed of preparation and technical soundness of project design. For this Project, a swift preparation was required to take advantage of the political window of opportunity. This short preparation timeline, however, did not allow for the use of the SFLAC funds to inform project design nor for a full diagnostic of the infrastructure needs of beneficiary schools, which led to an underestimation of the costs and the depth of the interventions needed. Therefore, when a Project needs to be prepared fast, the Project Appraisal Document should foresee some flexibility to adapt activities during implementation. 77. When piloting a new education strategy, it is crucial to start small and have the flexibility to adapt the activities to respond to different implementation challenges. This Project was an opportunity to pilot the IFTS Model in a reduced universe of schools. Consequently, most of the activities in Component 1 experienced changes to reflect the lessons learned during implementation and overcome different operational bottlenecks. Given that this Project was a pilot, it could be argued that its scope was too ambitious in terms of the number of schools targeted and, therefore, it is recommended to target a smaller universe and ensure that all the implementation lessons learned are well captured in a process evaluation. 78. Project design should carefully take into account the environment in which its activities are going to be implemented. The implementation of some activities of the IFTS Model, namely construction works and teacher and student mobility, was affected by the country’s high crime and violence rates, which was underestimated during Project design. Taking this reality into account when designing project interventions would allow more realistic planning and smoother implementation. Page 23 of 82 The World Bank Education Quality Improvement Project (P126364) ANNEX 1. RESULTS FRAMEWORK AND KEY OUTPUTS A. RESULTS INDICATORS A.1 PDO Indicators Objective/Outcome: Improve access rates for students in lower and upper secondary education Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Improvement in the Access Percentage 96.30 98.80 96.60 96.70 Rate for lower secondary education 10-Nov-2011 10-Nov-2011 13-Jul-2015 31-Dec-2018 Comments (achievements against targets): Target achieved. Original Baseline value: 96.3% Original Target: 98.8% Achieved (101 IS): 95.1% Achievement against original baseline/target: -48% Revised Baseline value: 96.6% Revised Target: 96.6% Achieved (40 IS): 96.7% Achievement against revised baseline/target: 100% In 2015, the baseline was revised to account for the changes in the universe of schools that would benefit from the full implementation of the IFTS Model and to align the calculation methodology with the UNESCO definition of effective transition rate. The targets were also revised downward to reflect the declining trend of this indicator at the national level; keeping baseline levels was interpreted as an improvement. The revised target was achieved, although it is worth mentioning that this figure corresponds to the indicator value for 2017 since the data to calculate the value for 2018 will only be available in October 2019. Page 24 of 82 The World Bank Education Quality Improvement Project (P126364) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Improvement in the Access Percentage 78.00 78.00 87.00 Rate for upper secondary education 13-Jul-2015 13-Jul-2015 31-Dec-2018 Comments (achievements against targets): Target surpassed. Original Baseline value: 78% Original Target: 78% Achievement against original baseline/target: 111% This indicator was added in 2015. The target value is the same as the baseline since, given the declining trend of this indicator at the national level, staying at baseline levels was interpreted an improvement. The target was surpassed (111% achieved), although it is worth mentioning that this figure corresponds to the indicator value for 2017 since the data to calculate the value for 2018 will only be available in October 2019. Objective/Outcome: Improve retention rates for students in upper and lower secondary education Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Improvement in the Percentage 95.10 98.40 91.00 93.80 Retention Rate for lower secondary education 10-Nov-2011 10-Nov-2011 13-Jul-2015 31-Dec-2018 Comments (achievements against targets): Target achieved. Original Baseline value: 95.1% Original Target: 98.4% Achieved (101 IS): 93.7% Achievement against original baseline/target: -42% Revised Baseline value: 91% Revised Target: 91% Achieved (40 IS): 93.8% Achievement against revised baseline/target: 103% In 2015, the baseline was revised to account for the changes in the universe of schools that would benefit from the full implementation of the Page 25 of 82 The World Bank Education Quality Improvement Project (P126364) IFTS Model. The targets were also revised downward to reflect the declining trend of this indicator at the national level; staying at baseline levels was an improvement. The revised target was 103% achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Improvement in the Percentage 92.50 92.50 92.80 Retention Rate for upper secondary education 13-Jul-2015 13-Jul-2015 31-Dec-2018 Comments (achievements against targets): Target achieved. Original Baseline value: 92.5% Original Target: 92.5% Achievement against original baseline/target: 100% This indicator was added in 2015. The target value is the same as the baseline since, given the declining trend of this indicator at the national level, staying at baseline levels was an improvement. The target was achieved. Objective/Outcome: Improve graduation rates for students in lower and upper secondary education Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Improvement in the Percentage 69.00 69.00 71.40 Graduation Rate for lower secondary education 13-Jul-2015 13-Jul-2015 31-Dec-2018 Comments (achievements against targets): Target achieved. Original Baseline value: 69.0% Original Target: 69.0% Achievement against original baseline/target: 103% Page 26 of 82 The World Bank Education Quality Improvement Project (P126364) This indicator was added in 2015. The target value is the same as the baseline since, given the declining trend of this indicator at the national level, staying at baseline levels was an improvement. The target was 103% achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Improvement in the Percentage 75.00 91.70 71.50 75.90 Graduation Rate for upper secondary education 10-Nov-2011 10-Nov-2011 13-Jul-2015 31-Dec-2018 Comments (achievements against targets): Target achieved. Original Baseline value: 75% Original Target: 91.7% Achieved (101 IS): 73.8% Achievement against original baseline/target: -7% Revised Baseline value: 71.5% Revised Target: 71.5% Achieved (40 IS): 75.9% Achievement against revised baseline/target: 106% In 2015, the baseline was revised to account for the changes in the universe of schools that would benefit from the full implementation of the IFTS Model. The targets were also reviewed downward to reflect the declining trend of this indicator at the national level; staying at baseline levels was an improvement. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of schools that fully Number 0.00 29.00 213.00 163.00 adopt the Inclusive Full-time School Model 10-Nov-2011 10-Nov-2011 13-Jul-2015 31-Dec-2018 Comments (achievements against targets): Target not achieved. Original Baseline value: 0 Original Target: 29 municipalities Achievement against original baseline/target: Not applicable Page 27 of 82 The World Bank Education Quality Improvement Project (P126364) Revised Baseline value: 0 Revised Target: 213 schools Achievement against revised baseline/target: 77% In 2015, this indicator was revised to capture the number of schools, as part of an Integrated Systems, that adopt the IFTS Model instead of the number of municipalities. Of the original 213 schools offering secondary education in the 40 targeted IS, all received training on the features of the IFTS Model for their directors and secondary teachers, 195 belong to an IS with a Pedagogical Plan, 150 received educational materials and equipment, 133 offer 30 hours or more per week, 118 have access in the IS to a school with integral infrastructure, and 112 participated in the territorial reorganization of the IS as evidenced by student transportation. Given that the IFTS Model is considered fully implemented when at least four of the six elements listed above are met, only 163 out of the 213 schools targeted managed to fully adopt the IFTS Model by Project closing and, therefore, the target for this indicator has not been achieved. A.2 Intermediate Results Indicators Component: Component 1: Adoption of the IFTS model Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of certified teachers Number 0.00 1301.00 1500.00 1619.00 from lower and upper secondary school 13-Jul-2015 13-Jul-2015 30-Jun-2017 31-Dec-2018 Comments (achievements against targets): Target achieved. Original Baseline value: 0 Original Target: 1,301 Achievement against original baseline/target: 124% Revised Baseline value: 0 Revised Target: 1,500 Achievement against revised baseline/target: 108% This indicator was added in 2015 as a revision of the indicator "Number of additional qualified secondary school teachers and number of additional qualified school directors resulting from Project intervention," to better monitor progress in the teacher training. Additionally, in 2017, the targets were revised to capture the inclusion of art, physical education, and indigenous cultures as additional subject areas for which Page 28 of 82 The World Bank Education Quality Improvement Project (P126364) teachers would be certified under the Project. The original target was surpassed (124% achieved) and the revised indicator was 108% achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of certified directors Number 0.00 285.00 436.00 13-Jul-2015 13-Jul-2015 31-Dec-2018 Comments (achievements against targets): Target surpassed. Original Baseline value: 0 Original Target: 285 Achievement against original baseline/target: 153% This indicator was added in 2015 as a revision of the indicator "Number of additional qualified secondary school teachers and number of additional qualified school directors resulting from Project intervention," to better monitor the progress in the director training. The final target was surpassed (153% achieved). Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of certified teacher Number 0.00 624.00 750.00 1150.00 trainers 13-Jul-2015 13-Jul-2015 30-Jun-2017 31-Dec-2018 Comments (achievements against targets): Target surpassed. Original Baseline value: 0 Original Target: 624 Achievement against original baseline/target: 184% Revised Baseline value: 0 Revised Target: 750 Achievement against revised baseline/target: 153% This indicator was added in 2015 as a revision of the indicator "Number of additional qualified MINED technical staff (at the central and Page 29 of 82 The World Bank Education Quality Improvement Project (P126364) departmental level) resulting from Project intervention," to capture the changes in the teacher training strategy. Additionally, in 2017, the targets were revised to capture the inclusion of art, physical education, and indigenous cultures as additional subject areas for which teacher trainers would be certified under the Project. Both the original and the revised targets were surpassed. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of IFTS that are Number 0.00 201.00 195.00 195.00 being equipped with educational material and 10-Nov-2011 11-Nov-2011 13-Jul-2015 31-Dec-2018 equipment Comments (achievements against targets): Target achieved. Original Baseline value: 0 Original Target: 201 Achievement against original baseline/target: 97% Revised Baseline value: 0 Revised Target: 195 Achievement against revised baseline/target: 100% The targets for this indicator were reduced in 2015. The revised target was achieved (100%) and the original 97% achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of newly built or Number 0.00 723.00 697.00 718.00 renovated facilities 10-Nov-2011 10-Nov-2011 30-Jun-2017 31-Dec-2018 Comments (achievements against targets): Target achieved. Original Baseline value: 0 Original Target: 723 Achievement against original baseline/target: not applicable (this Page 30 of 82 The World Bank Education Quality Improvement Project (P126364) first target measured only new classrooms) Revised Baseline value 1: 0 Revised Target 1: 882 Achievement against revised baseline/target: 81% Revised Baseline value 2: 0 Revised Target 2: 697 Achievement against revised baseline/target: 103% This indicator was revised in 2015 to better capture the works in the rehabilitated schools, since schools benefitted not only from new classrooms but also from other educational facilities such as kitchens, libraries, bathrooms, dining areas, etc. Then, in 2017, the methodology to monitor the progress in this indicator was revised and the targets were revised to reflect the downsizing of the infrastructure interventions. The revised target was 103% achieved, but the original target was not achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of IFTS being Number 0.00 201.00 36.00 36.00 renovated under the Project 10-Nov-2011 10-Nov-2011 30-Jun-2017 31-Dec-2018 Comments (achievements against targets): Target achieved. Original Baseline value: 0 Original Target: 201 Achievement against original baseline/target: not applicable (this first target measured more superficial renovations) Revised Baseline value 1: 0 Revised Target 1: 46 Achievement against revised baseline/target: 78% Revised Baseline value 2: 0 Revised Target 2: 36 Achievement against revised baseline/target: 100% The targets for this indicator were revised to reflect the changes in the infrastructure strategy agreed in the 2015 restructuring, which entailed more comprehensive and in-depth infrastructure renovations in fewer schools. Then, in June 2017, the targets were reduced from 46 to 36 as a result of the higher than budgeted infrastructure costs and the poor state of many of the schools targeted for this activity, which required more extensive works than originally planned. Page 31 of 82 The World Bank Education Quality Improvement Project (P126364) Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of IFTS that receive Number 0.00 195.00 230.00 transfers for repairs or other minor works 13-Jul-2015 13-Jul-2015 31-Dec-2018 Comments (achievements against targets): Target surpassed. Original Baseline value: 0 Original Target: 195 Achievement against original baseline/target: 118% This indicator was added in 2015 to monitor the progress in the transfers made to schools for repairs and other minor works. The target was surpassed (118% achieved). Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of students in Number 0.00 34017.00 12604.00 9842.00 grades 7 to 9 that have 30 or more weekly hours of 10-Nov-2011 10-Nov-2011 13-Jul-2015 31-Dec-2018 pedagogical activities coordinated by the school Comments (achievements against targets): Target not achieved. Original Baseline value: 0 Original Target: 34,017 Achievement against original baseline/target: 29% Revised Baseline value: 0 Revised Target: 15,801 Achievement against revised baseline/target: 78% In 2015, the targets for this indicator were reduced, capturing the changes in the scope of the Project’s interventions. Neither the original nor Page 32 of 82 The World Bank Education Quality Improvement Project (P126364) the revised target were achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of students Number 0.00 11906.00 10000.00 9759.00 transported to school in the Integrated System 10-Nov-2011 10-Nov-2011 30-Jun-2017 31-Dec-2018 Comments (achievements against targets): Target almost achieved. Original Baseline value: 0 Original Target: 11,906 Achievement against original baseline/target: 82% Revised Baseline value 1: 0 Revised Target 1: 16,000 Achievement against revised baseline/target: 61% Revised Baseline value 2: 0 Revised Target 2: 10,000 Achievement against revised baseline/target: 98% This indicator was revised during the July 2015 restructuring to capture the changes in the scope of the Project interventions and school clustering and then, in June 2017, the targets were reduced due to both budgetary constraints in MINED, challenges in monitoring the use of stipends, and reduced need for movement of students as more schools in IS offered extended hours on their premises. The final target was almost achieved (98%), but not the original targets were not. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of "Integrated Number 0.00 40.00 38.00 Systems" that implemented the communication strategy 13-Jul-2015 13-Jul-2015 31-Dec-2018 for IFTS Comments (achievements against targets): Target almost achieved. Page 33 of 82 The World Bank Education Quality Improvement Project (P126364) Original Baseline value: 0 Original Target: 38 Achievement against original baseline/target: 95% Component: Component 2: Improvement of MINED's Institutional Capacity and the Schooling System's Governance Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Number of "Integrated Number 0.00 40.00 0.00 Systems" with a monitoring system 13-Jul-2015 13-Jul-2015 06-Dec-2017 Comments (achievements against targets): Target not achieved. Original Baseline value: 0 Original Target: 40 Achievement against original baseline/target: 0% By Project closing, this indicator was still at 0 because the information system SIMES was not implemented. The development of the EMIS is now being undertaken by MINED with support of the Millennium Challenge Corporation (MCC). Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Municipalities that have Number 0.00 7.00 7.00 started a process evaluation of the governance system 13-Jul-2015 13-Jul-2015 31-Dec-2018 Comments (achievements against targets): Target surpassed. In 2015, the focus of this indicator shifted from undertaking to starting the evaluation of the governance model for the IFTS to account for delays resulting from MINED's decision to review the national level governance of schools, to which the IFTS governance plan was subject. By Page 34 of 82 The World Bank Education Quality Improvement Project (P126364) closing, the Project had a complete evaluation of the governance model in the 7 selected municipalities, which implies that the revised target was surpassed, and the original target achieved. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Baseline survey for Yes/No N Y Y evaluation has been collected 10-Nov-2011 10-Nov-2011 31-Dec-2018 Comments (achievements against targets): Target achieved. The baseline survey for the impact evaluation was collected in 2014. Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Follow-up survey for impact Yes/No N Y N evaluation has been collected 13-Jul-2015 13-Jul-2015 31-Dec-2018 Comments (achievements against targets): Target not achieved. The follow-up survey to collect the end line was not conducted because the strategy for the impact evaluation changed and this exercise was not deemed necessary. However, the broad objective monitored by this indicator was achieved, since the Project completed an impact evaluation using administrative data. Page 35 of 82 The World Bank Education Quality Improvement Project (P126364) Unlinked Indicators Formally Revised Actual Achieved at Indicator Name Unit of Measure Baseline Original Target Target Completion Direct project beneficiaries Number 0.00 43871.00 96549.00 158694.00 10-Nov-2011 10-Nov-2011 13-Jul-2015 16-Apr-2018 Female beneficiaries Percentage 49.00 50.00 49.00 51.50 10-Nov-2011 10-Nov-2011 13-Jul-2015 16-Apr-2018 Comments (achievements against targets): Target surpassed. Original Baseline value: 0 Original Target: 43,871 Achievement against original baseline/target: 362% Revised Baseline value: 0 Revised Target: 96,549 Achievement against revised baseline/target: 164% In 2015, the targets for this indicator were revised to capture the changes in the scope of the project interventions and the teacher training strategy, as well as to include parents and students from technical upper secondary education into the calculations. The final target was significantly surpassed considering both original and revised values. Page 36 of 82 The World Bank Education Quality Improvement Project (P126364) B. KEY OUTPUTS BY COMPONENT Objective: Improve access, retention and graduation rates in upper and lower secondary education 1. Improvement in the Access Rate for lower and upper secondary education 2. Improvement in the Retention Rate for lower and upper Outcome Indicators secondary education 3. Improvement in the Graduation Rate for lower and upper secondary education 1. Number of schools that fully adopt the IFTS model 2. Number of certified teachers from lower and upper secondary school 3. Number of certified directors 4. Number of certified teacher trainers 5. Number of IFTS that are being equipped with educational material and equipment 6. Number of newly built or renovated facilities 7. Number of IFTS being renovated under the Project 8. Number of IFTS that receive transfers for repairs or other minor Intermediate Results Indicators works 9. Number of students in grades 7 to 9 that have 30 or more weekly hours of pedagogical activities coordinated by the school 10. Number of students transported to school in the Integrated System 11. Number of “Integrated Systems” with a monitoring system. 12. Number of “Integrated Systems” that implemented the communication strategy for IFTS 13. Municipalities that have started a process evaluation of the governance system 14. Baseline survey for evaluation has been collected Page 37 of 82 The World Bank Education Quality Improvement Project (P126364) 15. Follow-up survey for impact evaluation has been collected 1. 163 out of the 213 schools targeted managed to fully adopt the IFTS model 2. A total of 1150 master teacher trainers, 1619 teachers and 436 school directors got certified. 3. By Project closure, a total of 36 schools were fully renovated, with integral infrastructure, with a total of 718 new or renovated facilities, 441 schools received school furniture and 230 received resources for minor repairs. 4. The project provided 195 schools with educational materials and Key Outputs by Component equipment, including computers and projectors, library books, sports equipment, and art supplies. 5. In terms of the extension of the school day, the number of students from lower secondary who participated and stayed in school for at least 30 hours per week increased progressively over the project life, starting at 4,140 students in 2015 and covering 9,842 students by project closure. 6. Over the Project life, the number of students who received the school transportation stipend increased progressively, starting at 4,161 in 2016 and covering 9,759 by project closure. Page 38 of 82 The World Bank Education Quality Improvement Project (P126364) ANNEX 2. BANK LENDING AND IMPLEMENTATION SUPPORT/SUPERVISION A. TASK TEAM MEMBERS Name Role Preparation Supervision/ICR Robert J. Hawkins Task Team Leader(s) Monica Lehnhoff Procurement Specialist(s) Sandra Lisette Flores De Mixco Financial Management Specialist Erika Piber Social Specialist Alberto Treves Environmental Specialist Jimena Garrote Counsel Nancy Rocio Banegas Raudales Team Member Maria Pia Cravero Counsel Silvia Guallar Artal Team Member Jasmine Anne Pineda Team Member Norman Russle Howard Taylor Social Specialist B. STAFF TIME AND COST Staff Time and Cost Stage of Project Cycle No. of staff weeks US$ (including travel and consultant costs) Preparation FY11 1.540 37,624.78 FY12 23.073 172,487.36 FY13 0 0.00 Total 24.61 210,112.14 Page 39 of 82 The World Bank Education Quality Improvement Project (P126364) Supervision/ICR FY12 15.906 70,344.82 FY13 39.036 180,670.81 FY14 48.083 194,574.62 FY15 60.319 259,487.49 FY16 25.713 147,368.61 FY17 46.808 290,472.98 FY18 37.353 261,731.21 FY19 23.178 130,283.68 Total 296.40 1,534,934.22 Page 40 of 82 The World Bank Education Quality Improvement Project (P126364) ANNEX 3. PROJECT COST BY COMPONENT Amount at Approval Actual at Project Percentage of Approval Components (US$M) Closing (US$M) (US$M) Adoption of the IFTS Model 54.85 55.00 100% Improvement of MINED's Institutional Capacity and the 5 4.5 90% Schooling System's Governance Front-end fee 0.15 0.15 100% Total 60.00 59.7 99% Page 41 of 82 The World Bank Education Quality Improvement Project (P126364) ANNEX 4. EFFICIENCY ANALYSIS This economic analysis focuses on the Project interventions designed to improve access, retention, and graduation rates in secondary education. These interventions include all activities under component 1, aimed at implementing the Inclusive Full-Time Schools (IFTS) model: (1) professional development and training, (2) educational materials and equipment, (3) improvement of learning facilities, (4) extension of the school day, and (5) minor investments and other activities in eligible schools, including a transportation stipend, school feeding, and the provision of materials for the extended day workshops. The Project included an additional component to improve institutional capacity and governance in El Salvador’s schooling system. However, given the difficulty of measuring the economic benefit of this intervention, it was not included in the analysis. The Project’s economic benefits are based on the assumption that the interventions will improve learning conditions in participating schools and encourage attendance for students who might otherwise have dropped out or not enrolled in the first place. This will translate into more years of education for the Project’s beneficiaries, which, in turn, will lead to higher productivity and earnings in the labor market. In their study of comparable returns to education in 139 countries, Montenegro and Patrinos (2014) estimated that an additional year of education in El Salvador increases annual income by 9.3 percent.28 The analysis measures the Project’s benefits against its costs, including the investment cost of implementing the components of the IFTS model, but excluding the cost to improve institutional capacity and governance. By the end of 2018, US$61.6 million had been spent on the IFTS model over the Project’s six-year life. Additionally, as explained below, the analysis includes estimates of recurrent costs to maintain school infrastructure that will continue to benefit students after the Project’s closure. To calculate the Project’s benefits more precisely, the analysis grouped beneficiaries into different “universes” depending on which intervention they participated in each year. Different parts of the Project were implemented at different times, with educational materials and minor investments being provided early on, but training and school renovations not taking place until the Project’s fourth year. Moreover, the level of Project intensity varied by school, with some schools receiving all components of the IFTS model, while others received one or two. The intervention universes therefore range in size, from an estimated 5,899 students who benefited from all components of the Project in 2018, to 41,068 who benefited mainly from teacher training.29 Separate universes were created for the students who participated in the Project’s early activities (31,333 in 2015), and for the 21,231 students who will continue to benefit each year from school renovations after the Project’s closure. Table 1 presents estimates of the number of beneficiaries in each universe per year, along with the estimated impact in terms of additional years of education (described in more detail below). Care was taken to make sure the universes do not overlap, so that the Project’s total benefits could be determined by adding up the benefits received in each universe. 28 The authors use the same specification and estimation procedure and similar data to derive comparable returns across countries. The most recent estimate for El Salvador is for 2009. 29 The number of students benefiting from the full IFTS model was taken directly from the Project’s monitoring data. For teacher training, the number of beneficiaries was estimated based on the number of teachers certified and an average of 34 students per teacher. For the other interventions, the number of beneficiaries was estimated based on the number of schools participating in the intervention and the average number of students per school using enrollment data from El Salvador’s school census. Page 42 of 82 The World Bank Education Quality Improvement Project (P126364) Table 1. Project Impact and Beneficiaries per Year, by Intervention Universe Beneficiaries per Year Intervention Estimated Per Year, Universe1 Impact2 2013 2014 2015 2016 2017 2018 2019-2028 Training3 0.15 0 0 0 27,814 41,705 41,068 0 School 0.1 0 0 0 1,626 8,454 11,705 21,231 renovations4 Extended day5 0.20 - 0.40 0 0 4,140 6,850 8,057 9,842 0 Full IFTS Model6 0.4 0 0 0 819 4,260 5,899 0 Minor investments7 0.05 10,667 34,125 31,333 0 0 0 0 Notes: 1 The impact of receiving educational materials and equipment was not estimated separately for the analysis. 2 Estimated impact is the additional year of education students gain from participating in an intervention. 3 Some of the students who benefited from teacher training also benefited from educational materials and minor investments. 4 This group of students benefited from improved learning facilities only. 5 Some of the students who benefited from the extended school day also benefited from teacher training, educational materials, school renovations, and minor investments. 6 This group of students participating in the extended school day also benefited from all of the other interventions. 7 Some of the students who benefited from minor investments also benefited from educational materials. After 2015, these students began to benefit from teacher training and the extended school day and for the purposes of the analysis were grouped into those interventions. Source: Author’s calculations based on Project monitoring data and enrollment data from El Salvador’s school census. Based on a literature review of similar interventions in Latin America and elsewhere, the analysis estimated how many additional years of education students received from participating in the Project. The information used to calculate this benefit for each intervention is described in detail below. 1. Professional development and training: In their meta-analysis of 26 training interventions in developing countries, Popova, Evans, and Arancibia (2016) found that training programs accompanied by reading materials can improve test scores by up to 0.16 standard deviation, Page 43 of 82 The World Bank Education Quality Improvement Project (P126364) corresponding to a 0.78 additional year of education.30 Similarly, in a review of 77 randomized experiments measuring impact on learning in developing countries, McEwan (2015) found that teacher training improves test scores by 0.12 standard deviation, or a 0.59 additional year of education. However, Evans and Popova (2016) states that training is most effective when teachers receive detailed guidance on pedagogy and the programs target teachers by skill level. It is not clear if the Project provided that level of support. Moreover, the studies caution that the impact of training may overlap with other interventions, such as providing educational materials (Evans & Popova, 2016; McEwan, 2015). While some of the beneficiaries of the Project’s training component also participated in other interventions, including educational materials and equipment, some did not. Therefore, the analysis makes a conservative estimate of a 0.15 additional year of education from this component, or a quarter of the lower range described above. The investment cost was US$7.7 million. In addition to training teachers, the Project also trained several hundred school directors in an effort to improve school management in the IFTS. There is evidence in the literature that school management plays an important role in improving student outcomes (Adelman & Lemos, n.d.; Bloom, Lemos, Sadun, & Van Reenen, 2014). However, the analysis does not consider the direct impact of improved school management, because such an impact cannot be disentangled from the impact of teacher training. Thus, the analysis probably underestimates the benefit of this intervention. 2. Educational materials and equipment: Educational materials and equipment can have a positive impact on test scores and graduation rates (Glewwe & Muralidharan, 2016; Krishnaratne, White, & Carpenter, 2013; McEwan, 2015). However, the literature makes clear that these materials are most effective when paired with other interventions, such as teacher training, and are not by themselves likely to have an impact (McEwan, 2015). Therefore, the analysis assumes that students did not benefit from receiving educational materials alone, but only when they also participated in other interventions, such as teacher training (see above) or additional investments and activities (see below). The investment cost of this component was US$3.5 million. 3. Improvement of learning facilities: In a review of 39 studies from developing countries, including 16 studies from Latin America, Cuesta, Glewwe, and Krause (2016) found that sturdy walls, floors, and roofs, and access to basic services such as water and sanitation, have a positive impact on test scores and, to a lesser extent, on enrollment and attendance. Analyzing test scores from 16 Latin American countries, Treviño et al. (2010) found that one additional element of infrastructure, such as a sports field or dining room, corresponds to between a 0.05 and 0.20 additional year of education. In Peru, Paxson and Schady (2002) found that spending on school infrastructure corresponded to a 14.5 percent increase in attendance, or a 0.145 additional year of education. Similarly, in the United States, poor ventilation in classrooms was found to reduce class attendance by 10 to 20 percent (Barrett, Treves, Shmis, Ambasz, & Ustinova, 2019). The analysis uses a conservative estimate of a 0.10 30 To convert the impact on test scores in standard deviations into equivalent years of education, the analysis benchmarks the effect size to the natural growth in academic achievement per year for a typical middle school or high school student. Since that information is not available for El Salvador, the analysis uses estimates from Bloom, Hill, Black, and Lipsey (2008), which reports the mean effect sizes for annual grade-to-grade gains for students in the United States. Given the relatively poor quality of education in El Salvador, the annual gains in academic achievement for students in that country are likely much lower than in the United States. Therefore, the actual benefit of an intervention as converted into years of education is probably greater in El Salvador than what the analysis estimates using the data from Bloom et al. (2008). Page 44 of 82 The World Bank Education Quality Improvement Project (P126364) additional year of education from this intervention, or the lower-middle range in the literature. The investment cost was US$38.6 million. Given that most of the renovations occurred in the last two years of the Project, it seems likely that students will continue to benefit from school infrastructure after the Project’s closure. New school buildings may last 20, 30, or even 60 years.31 However, in interviews conducted during the Project’s impact evaluation with teachers and directors of the renovated schools, many expressed concern about finding enough funds to properly maintain the facilities. Moreover, it may be dubious to assume that renovated schools in Central America will continue to benefit students decades after the initial investment, given that the physical condition of schools in the region is particularly poor (Duarte, Jaureguiberry, & Racimo, 2017). To be conservative, the analysis assumes that students will continue to benefit from the renovated schools up to 10 years after the Project’s closure. It assumes a total recurrent cost of US$225,000 per year, based on the estimated maintenance costs for 2018 for the 36 schools that were renovated. 4. Extension of the school day: Although results vary, extended school day programs in Brazil, Chile, Colombia, Mexico, Peru, and Uruguay have generally been found to improve test scores by 0.05 to 0.15 standard deviation per year, corresponding to a 0.24 to 0.73 additional year of education (Agüero, 2016; Bellei, 2009; Berthelon, Kruger, & Vienne, 2016; Cabrera-Hernández, 2015; Cerdan- Infantes & Vermeersch, 2007; Cruz, Loureiro, & Sa, 2017; García Marín, 2006; Hincapie, 2016; Padilla-Romo, 2017). Comprehensive programs that extend the school day as well as strengthen teacher training, curriculum, and school management appear to be the most effective (Agüero, 2016; Alfaro, Evans, & Holland, 2015; Cruz, Loureiro, & Sa, 2017). For the universe of students participating in the extended school day as well as all of the other components of the IFTS model, the analysis assumes a benefit of a 0.40 additional year of education, or the lower-middle range in the literature. For participants in the extended day who do not benefit from many of the other components, the analysis assumes an impact of a 0.20 additional year of education. The investment cost was US$1.8 million. 5. Minor investments and other activities: The last component of the IFTS model includes funds for equipment and materials for sports and art as well as a transportation stipend, school feeding, and the cancellation of fees for extended day workshops. These interventions may have encouraged some participants to attend school who otherwise would not have enrolled or would have dropped out. In their review of education interventions in developing countries, Ganimian and Murnane (2016) and Krishnaratne, White, and Carpenter (2013) found evidence to support this. In Latin America, Krishnaratne, White, and Carpenter (2013) found that eliminating school fees boosted enrollment by 10 percent, corresponding to an additional 0.10 year of education. In Colombia, Ganimian and Murnane (2016) found that reducing fees for low-income families boosted enrollment by three to six percent. Krishnaratne, White, and Carpenter (2013) found that school feeding programs in Peru and other developing countries can also have a positive impact on enrollment, attendance, and dropout rates. And Ganimian and Murnane (2016) found cases where reducing commuting costs increases enrollment, particularly for girls, up to 30 percent. However, it is unclear whether the students who received the Project’s transportation stipend were the ones who needed it the most, and whether they would otherwise not have been able to go to school. Moreover, interviews with teachers and directors suggest the Project’s school feeding component may not have been well-managed. Therefore, the analysis 31A recent economic analysis of a project to improve school infrastructure in Nicaragua assumed new buildings would last 20 years (World Bank, 2018). The newly-renovated schools in El Salvador, if properly maintained, have been estimated to last 50 to 60 years. Page 45 of 82 The World Bank Education Quality Improvement Project (P126364) uses a conservative estimate of a 0.05 additional year of education for beneficiaries of this component, some of whom also received educational materials and equipment. The investment cost was US$10.0 million. Assuming there is no overlap in the intervention universes and using El Salvador’s 2017 average expected income32 of US$3,463 and a discount rate of 5 percent, the Project’s net present value (NPV) is estimated at US$64.1 million. The associated internal rate of return (IRR), or the rate of return that brings the net present value to zero, is 8.9 percent. Furthermore, a sensitivity analysis was performed that reduced the estimated return to education by 30 percent. As shown in Table 2, even in this pessimistic scenario, the Project’s IRR is still greater than the discount rate of 5 percent, suggesting that the Project is a worthwhile investment. Table 2. Summary of Cost-Benefit Analysis Sensitivity Analysis: Baseline analysis Returns to Education Reduced by 30% NPV US$64,123,101 US$25,881,525 IRR 8.90% 6.40% Source: Author’s calculations based on Project monitoring data and enrollment data from El Salvador’s school census. The IRR obtained in this analysis is similar to the ones estimated at the Project’s appraisal (8.4 percent) and the 2015 restructuring (9.0 percent). However, this analysis takes a different approach to measuring the Project’s benefits. The previous analyses measured benefits by estimating how many more students graduated from school as a result of the Project, then calculated the additional earnings those beneficiaries were likely to receive with a diploma. By contrast, this analysis estimates the total number of students benefiting from all aspects of the IFTS model each year. It reviews the literature to estimate the impact in terms of additional years of education and uses an estimate of the return to education in El Salvador to calculate the additional earnings these students will receive in the labor market. Moreover, the previous analyses included the total investment cost of the Project, including the cost to improve institutional capacity and governance, and a significantly higher estimate of the Project’s recurring costs.33 By contrast, this analysis excludes the component to improve governance due to the difficulty of measuring the benefits, and limits recurrent costs to the estimated maintenance costs of the 36 schools receiving renovations. Despite the comprehensive approach this analysis takes to measuring the Project’s benefits, it is possible that it may still underestimate the true impact, since it does not take into consideration the potential benefits that are harder to quantify in monetary terms. For example, there is evidence that extended school day programs may reduce teen pregnancies and juvenile crime and increase the labor force participation of mothers of school-age children (Berthelon & Kruger, 2011; Contreras & Sepúlveda, 2016; Pires & Urzua, 2015). In El Salvador, an after-school program similar to the extended school day reduced violent and anti-social behavior of participants and had positive spillover effects on other students as well 32 Calculated by multiplying the average annual salary in 2017 by the employment rate. 33 The previous analyses assumed some of the Project’s activities would continue after closure. This analysis does not make that assumption, although El Salvador’s Ministry of Education has indicated it would like to continue to fund the Project. Page 46 of 82 The World Bank Education Quality Improvement Project (P126364) (Dinarte & Egana del Sol, 2018). The analysis does not attempt to measure such benefits. Additionally, efforts to strengthen school management and improve the institutional capacity and governance of El Salvador’s schooling system may also have a positive impact, but this is not measured directly in the analysis. It is also possible that the Project’s benefits may persist or even grow over time. The analysis assumes that, aside from school infrastructure, students will not continue to benefit from the Project’s interventions after closure. However, it is possible that students may continue to benefit from the improved teaching practices of teachers who were trained, or from educational materials and equipment that remain in the schools. There is also evidence that the potential positive impact of extending the school day may grow as kinks in the implementation process are worked out. In Mexico, Padilla- Romo (2017) found that the impact of extending the school day was strongest four years after its introduction. Likewise, Bellei (2009) suggests that teachers may need time to adapt to the longer school day and that it may take schools a few years to work out implementation issues. Page 47 of 82 The World Bank Education Quality Improvement Project (P126364) ANNEX 5. BORROWER, CO-FINANCIER AND OTHER PARTNER/STAKEHOLDER COMMENTS This is a summary of the Borrower’s Closing Report of the Education Quality Improvement Project “Informe de Cierre del Proyecto Mejoramiento de la Calidad Educativa,” translated into English and redacted by the Bank. Any mistakes are the sole responsibility of the Bank. 1. Project Context and Objectives 1.1 National Context 1. El Salvador was severely affected by the global economic crisis, and the country’s recovery is still fragile. The recession in the United States and the consequent fall in remittances, investment, and foreign demand contributed to a contraction in El Salvador’s real gross domestic product (GDP) of 2.1 percent in 2009. The growth rate in 2010 (1.4 percent), while positive, was the lowest in Central America. The external shocks also affected the country’s social indicators. After reaching its lowest level in 2006 (30.7 percent), the poverty rate rose to 40 percent in 2008 and remained at 37.8 percent in 2009. The crisis also contributed to relatively high levels of unemployment and insecurity, particularly among youth, and a rise in crime rates. 2. In response to these challenges, the authorities established an anti-crisis plan for 2009 to 2011. The plan included measures to create temporary employment, expand coverage of the social security system, improve access to low-income housing and basic infrastructure, and reduce violence. The plan aimed to mitigate the immediate impact of the crisis, especially among the most vulnerable segments of the population, and at the same time address long-term development challenges, such as the provision of social services. 3. Reducing poverty, social and economic inequality, social exclusion, and violence among youth are currently important priorities for the government. The government’s development plan for 2010 to 201434 emphasized programs targeting the most vulnerable, including children from poor rural and urban areas (who are particularly exposed to poverty, organized crime, and violence), women, and the elderly. The government is especially eager to reduce crime and violence, as the country is classified as one of the most violent in the world.35 Additionally, it is necessary to improve the quality of education and expand access beyond primary school in order to improve competitiveness and reduce inequality. 4. El Salvador’s economic growth reached 2.3 percent in 2017. This growth was driven mainly by agriculture, livestock, forestry, fishing, manufacturing, and mining, as well as commerce, restaurants, and hotels, which jointly represented approximately two thirds of the observed growth. The country also benefited from an improvement in the current account thanks to strong flows of worker remittances. The external deficit stood at 2 percent of GDP in 2017, compared to 2.1 percent in 2016. This improvement took place despite a deterioration in the trade deficit, which in 2017 reached US$4.8 billion.36 34 Government of El Salvador (2010). Plan Quinquenal de Desarrollo 2010-2014. San Salvador, El Salvador. 35 El Salvador has ranked second in the world in recent years in terms of homicide rates per 10,000 inhabitants, after Honduras. See UNODC (2011). 2011 Global Study on Homicide: Trends, Contexts, Data. Vienna: United Nations Office on Drugs and Crime. 36 World Bank. “Overview of El Salvador.” Updated October 5, 2018. https://www.worldbank.org/en/country/elsalvador/overview. Page 48 of 82 The World Bank Education Quality Improvement Project (P126364) 5. The country continues to exhibit low levels of GDP growth, averaging 2.6 percent between 2010 and 2016. This has placed the country among the slowest-growing economies in Central America. The growth rate for 2018 is expected to be 2.8 percent. 6. Regarding its youth, El Salvador has an historic opportunity, as it is experiencing a period in which the economically-active population is larger than the dependent population, and, therefore, investing in youth would generate a great economic and social “bonus” for the entire country. In 2017, people between 15 and 29 years old represented 30 percent of the population. Young women face additional challenges in both education and employment. Among young people who are neither studying nor working, 77.2 percent are women, many of whom perform unpaid domestic work. On average, young women devote 25.5 hours a week to this type of work, compared to only 7.2 hours a week for young men. 7. The United Nations Development Programme (UNDP) points out that rural youth in El Salvador face great gaps, as demonstrated by the fact that the Human Development Index for rural youth is 5.5 percent lower than that of their urban peers.37 On the topic of education, only 18.1 percent of the rural population 16 years or older attend school, compared to 34.9 percent in urban areas. Moreover, rural youth enter the labor market around the age of 14, almost three years earlier than their urban peers. Thus, the UNDP emphasizes the importance of focusing on education and employment measures in the country. In the transition to the labor market, only 16.6 of economically active youth find decent work. This share is even lower in rural areas, in homes with lower incomes, and for youth younger than 24. 1.2 Sectoral and Institutional Context 8. The country’s Education National Plan (Plan Nacional de Educación en Función de la Nación) establishes strategies to carry out the objectives of the 2014-2019 Five-Year Development Plan (El Salvador Productivo, Educado y Seguro: Plan Quinquenal de Desarrollo 2014-2019). The goal is to achieve high-quality and inclusive education and social equity, and continue the progress made under the Education Social Program “Let’s Go to School” (Vamos a la Escuela), which undertook a series of important steps to improve the quality of education and coverage, with an emphasis on human rights and policies that put into place a system of social protection. 9. The Education Quality Improvement Project has contributed significantly to this goal, particularly in the areas of: 1. Creating a national system of teacher professionalization, 2. Early childhood learning, 3. Creating a national system of education evaluation, 4. Constructing a pleasant school environment, 5. Equity, inclusion, quality, and relevance of education, 6. A dynamic national education curriculum based on teacher professionalization, 7. Developing and strengthening conditions for knowledge creation and innovation, 8. Deepening and strengthening adult education, and 9. Institutional reform and valid legislation. 10. El Salvador has made notable progress in expanding the coverage of primary education and is focusing its efforts on improving the transition to and completion of secondary school, as well as improving equity, efficiency, and quality. 37UNDP (2018). Informe sobre Desarrollo Humano El Salvador 2018: ¡Soy Joven! ¿Y ahora qué? San Salvador, El Salvador: Programa de las Naciones Unidas para el Desarrollo. http://www.sv.undp.org/content/el_salvador/es/home/library/informe- sobre-desarrollo-humano-el-salvador-2018-isoy-joven---y-.html. Page 49 of 82 The World Bank Education Quality Improvement Project (P126364) 11. According to the country’s National Education Council (Consejo Nacional de Educación — CONED), El Salvador has made significant progress in access to basic education, without significant gaps in terms of geographic area, sex, or income level. The Ministry of Education (Ministerio de Educación — MINED) estimates that in 2017, 80.6 percent of primary-school age children were studying in school. However, challenges remain in lower secondary education, which had a net enrollment rate of 55.9 percent in 2017. Moreover, at the level of upper secondary, net enrolment was only 36.8 percent the same year. 12. The country also faces a high dropout rate. The 2017 Multi-Purpose Household Survey (Encuesta de Hogares de Propósito Múltiples) finds that for that year the overall dropout rate for 15 to 19-year-olds was 37.7 percent. However, not all of the youth who had dropped out were at the upper secondary level; only 10.3 percent were, while 12.6 percent were in primary school and 14.9 percent were in lower secondary. Disaggregated by sex, the dropout rate is higher for women (39.6 percent) than for men (36.8 percent), and much higher in rural areas than in urban ones (50.3 percent and 27.9 percent, respectively). 13. Information from the MINED school census estimates that in 2017 alone some 63,359 children, adolescents, and youth enrolled between the first grade of primary school up to the last year of secondary education withdrew from school; of these, a total of 11,611 were in secondary school. The four main reasons for dropping out of school are: 1. Change of address (20 percent), 2. Leaving the country (14.3 percent), 3. Poor academic record (13.7 percent), and 4. Economic difficulties (9.1 percent). Other reasons related to violence are 1. Delinquency (7.0 percent), 2. Victim of gangs (1.1 percent), and 3. Victim of forced displacement (0.6 percent). 14. Improving retention is not enough if students are not properly prepared to enter the labor force. According to CONED, the country must make significant progress to improve the quality of education at all levels, which means having properly-trained teachers and directors and adequate infrastructure and physical school environments. 15. To effectively face these challenges, public investment in education must significantly increase with respect to GDP, and reach at least 7 percent, as recommended by CONED’s Plan “Plan El Salvador Educado”. In recent years, according to information from the Centroamerican Institute for Fiscal Studies (Instituto Centroamericano de Estudios Fiscales), public spending on education grew from 2.5 percent of GDP in 2007 to 3.1 percent in 2013, and is estimated to have reached 3.6 percent in 2015. However, since then the level has fallen continuously; based on 2018 budget projections, education spending will be around 3.2 percent of GDP. 16. According to CONED, one of the main challenges for the education system is to make school free from violence. School violence can occur both inside and outside of school. Inside school, various forms of violence effect the wellbeing and sense of security of students and staff. 17. According to observational data from MINED, 48 percent of public secondary schools experience instances of psychological violence, 33 percent experience physical violence, 28 percent experience bullying, and 4 percent experience sexual violence. In addition, risk factors have also been reported, including gang presence (34.8 percent), drugs (25 percent), theft (20.9 percent), robbery (16.4 percent), extortion (10.9 percent), blade weapons (10.5 percent), firearms (3.1 percent), and trafficking (1.2 percent). Page 50 of 82 The World Bank Education Quality Improvement Project (P126364) 18. The UNDP proposes adopting the concept of an education community made up of stakeholders committed to the proper functioning of a school. This community can be divided into: 1. A nuclear community composed of students, teachers, parents and guardians, school directors, and other personnel working at the school, and 2. An enlarged community composed of other actors interested in well- functioning schools, such as community organizations, Community Development Associations (Asociaciones de Desarrollo Comunitario), municipal governments, and other organizations (such as private companies and international cooperation agencies). The Project’s Integrated Systems (IS) governance component is an example of adopting such a concept. 19. Since 2009, MINED has promoted the Inclusive Full-Time Schools (IFTS) education model and the IS management structure, as outlined in the National Policy of Inclusive Education (Política Nacional de Educación Inclusiva). This strategy aims to create networks of nearby schools that can share opportunities and provide education services throughout a region. 20. The purpose of the IFTS model is to empower citizens to participate in the creation of a more equitable, democratic, and developed country. Achieving this requires that schools be based on the principle of inclusion, and promote access, retention, and completion of studies irrespective of disability, creed, race, socioeconomic status, or political choice. Inclusive education puts each student at the center of the teaching-learning process, and is mindful of students’ cultural, social, economic, political, and environmental needs, as well as the needs of their families and communities. The IS form the management structure of the IFTS model, to be established throughout the diverse socioeconomic and cultural regions of the country. The IS-IFTS model is thus meant to unify the country’s pedagogical, territorial, and organizational components. 21. The IS-IFTS model is a strategy to gradually establish schools that can respond to the demand for education in their region, where the population has complete access to high-quality education services, from initial education up to secondary. This strategy is coordinated by an organizational structure that promotes a shared vision between a network of education centers and communities. 22. The IS-IFTS model unifies three components: pedagogical, territorial, and organizational. The three are equally important and complementary. 23. The degree of fragmentation in the education system makes it difficult to improve efficiency and quality. The administrative structure consists of MINED and the Departmental Directorates (Direcciones Departamentales — DD), which operate at the department level. The lack a governance structure that is closer to the school-level makes monitoring of student performance difficult. In addition, the social environment in which schools operate is not conducive to learning. About 40 percent of adolescents between 13 and 17 years old live alone or without one of their parents, and they are often vulnerable to violence, gangs, substance abuse, and teen pregnancy.38 The absence of extracurricular activities, along with the weak link between the community and secondary schools, reinforces the tendency to drop out. 24. To improve this fragmentation, in 2015 MINED implemented a new organizational structure that will impact the different levels of the education system and the role that the school plays in its 38World Bank (2010). “Youth Development and Economic Opportunities in El Salvador.” Technical paper prepared for the non- credit technical assistance operation entitled El Salvador: Human Development for Poverty Reduction. Page 51 of 82 The World Bank Education Quality Improvement Project (P126364) environment. It will improve the participation of different stakeholders (students, parents, families, teachers, etc.) and facilitate project management throughout the DD and at the central level. This will be regulated by the Project Operating Manuel, which will standardize procedures for project management and result in a greater capacity to achieve objectives. 1.3 Project Objectives 25. The Project’s objective is to improve access, retention, and graduation rates for students in lower secondary (7th to 9th grade) and upper secondary (10th to 11th grade) in the Borrower’s public schools that adopt the IFTS model. An additional objective is to strengthen education institutions and governance. 2. Project Description 26. The Project has two components: 1. Adoption of the IFTS model, and 2. Improvement of MINED’s institutional capacity and the governance of the education system. Component 1: Adoption of the IFTS model 28. This component supported the expansion of the IFTS model in 29 municipalities throughout the 14 departments of the country. It involved working in 902 schools over the course of the Project, of which 213 were prioritized. The prioritized schools were organized into IS and developed their own governance system. 29. Component 1 carried out activities to support the provision of teaching materials and training of teachers, directors, and training specialists in eligible schools. In particular, it: • Supported the provision of teaching material and equipment for classrooms, laboratories, libraries, and staff rooms in participating schools • Supported the improvement of learning facilities in participating schools, including the renovation and construction of education facilities (classrooms, staff rooms, dining rooms, laboratories, specialized classrooms and workshop facilities, and arts and sports facilities), the acquisition of new furniture, and the provision of technical assistance for the design and supervision of such activities • Supported extending the teaching schedule in lower and upper secondary schools by extending the school day and creating new class sections • Carried out investments and activities in participating schools, including the acquisition of equipment and material for recreation, sports, art, culture, and education; the provision of transportation for students in lower and upper secondary; minor maintenance; and the financing of teaching extra hours and materials for extended-day workshops Component 2: Improvement of MINED’s Institutional Capacity and the Education System’s Governance 30. This component aimed at strengthening MINED’s capacity by improving education planning, and monitoring policies, including: Page 52 of 82 The World Bank Education Quality Improvement Project (P126364) • Developing new policies and regulations to make the Borrower’s education management systems more efficient, including developing a policy aimed at regulating the recruiting, appointment, and evaluation of teachers • Developing and publishing standards for school management, teaching activities, and teacher training • Improving monitoring and evaluation systems, including the design of web-based education indicators that can be accessed by the general public • Creating new strategies and instruments to assess student performance, including adopting annual standardized tests to collect data on education quality • Designing and implementing a communication strategy to increase the support and participation of local communities in school activities 32. Additionally, Component 2 supported: • The execution of a pilot program in eligible schools located in the departments of Usulután, Sonsonate, and La Libertad, which introduced a new model of governance based on school clusters to oversee the operation of the Borrower’s public schools • The provision of technical assistance to MINED and the DD to strengthen their monitoring and evaluation systems • The design and implementation of an impact evaluation system for the Project 3. Project Results 3.1 Relevance of the PDOs 33. The Project aimed to improve the coverage and quality of education in schools adopting the IFTS model. The Project objectives address three important aspects: access for students to the education system, retaining students in this system, and improving graduation rates in lower and upper secondary schools. An additional objective was to strengthen MINED’s institutional capacity and the education system’s governance. 34. The Project is framed by the drive to improve the quality of education and make the country’s public schools relevant. It follows the strategy of the Education Social Program “Let’s Go to School.” 35. The Project established a strategy to form the IS, which are key elements of the IFTS model and the Education Social Program. This strategy is at its core an effort to develop an inclusive education policy, and is guided by a long-term vision to improve policies, regulations, education practices, cultural guidelines, and school environment and organization so that everyone can have a quality education. It calls for strategic investments in infrastructure, the adoption of technology, and the provision of essential resources so that those who are at risk of missing out on opportunities are included. The plan creates a new concept and strategic framework for schools: the school as a Cultural Center, whose impact transcends the walls of the classroom and reaches families and the entire community. Page 53 of 82 The World Bank Education Quality Improvement Project (P126364) 36. The Project has contributed to altering school design and infrastructure and realigning education policies to allow for the implementation of IFTS under different methods of organization, where the principal stakeholders come from and are familiar with local communities and context. 3.2 Effectiveness 37. The Project’s effectiveness would be measured by the extent to which it achieved its objectives, keeping in mind that the results have been affected by the social and security issues afflicting the country, including violence, extortion, budgetary problems, etc. 38. However, an implementation report released by the World Bank in early 2019 rated progress towards achieving the PDOs as “moderately satisfactory,” noting that “the latest available figures on access (2016), retention (2017) and graduation (2017) show that four of the indicators have already surpassed the targets and a fifth one is almost at target level. Additionally, the preliminary results of the impact evaluation indicate that the implementation of the IFTS model had a positive effect on all three education outcomes.”39 The Project’s effectiveness will ultimately be considered when the evaluation results are finalized. 3.3 Efficiency 39. The Project has to-date completed its training component with great success, with 585 training specialists and 1,619 trained teachers. The Project has also constructed 36 integrated schools; repaired 230 schools through minor works that provide better space; equipped 434 schools with furniture; provided 195 schools with educational materials, computers and laboratory equipment, and musical instruments; assisted 9,759 students with school transport; and reached 15,801 students with the extended school day. All of this has contributed substantially to improving coverage, retention, and graduation rates for students in lower and upper secondary schools. 40. Through Component 2, mechanisms and guidelines for project implementation have been strengthened, improving such areas as accounting practices, timely payments, and the capacity to provide technical assistance. 41. To carry out these actions, procurement methods specified by the World Bank have been followed, awarding for quality and cost and guaranteeing the optimal use of resources. Of the US$60 million loaned from the World Bank, by Project closure a total of US$59,313,690.03 has been disbursed, representing 98.86 percent of the total. 39 World Bank (2019). “Education Quality Improvement Project Implementation Status and Results Report.” Report No. ISR34864. Jan. 7, 2019. http://documents.worldbank.org/curated/en/937671546884296912/pdf/Disclosable-Version-of-the-ISR-Education- Quality-Improvement-Project-P126364-Sequence-No-14.pdf. Page 54 of 82 The World Bank Education Quality Improvement Project (P126364) 3.4 Factors Effecting Project Results 42. The principal factors effecting the Project’s implementation and results include: • The repeated restructuring of the Project: there were four restructurings over the Project’s seven-year life. Given the amount of time needed to prepare each one, a significant amount of effort has thus been devoted to restructurings • Changes in the initial execution strategy of the IS-IFTS pilot: initially, a pilot project reaching 88 schools in 7 municipalities was proposed. However, this later morphed into an effort to reach 902 schools in 101 IS simultaneously, without the legal framework to support the new model nor the capacity to provide timely technical assistance • The delay in the provision of new infrastructure (in 36 integrated schools) required to adequately execute the extended-day program • The limited communication and coordination between the various units implementing the Project • The lack of clarity at the Project design regarding the pedagogical, administrative, and technical assistance needed to implement the IFTS model • The delay in the allocation of counterpart funds of US$10.4 million, which were urgently needed for the timely execution of the different components of the Project 4. Performance of the World Bank, Compliance with Standards, and Risks to the Project’s Continuation 4.1 Performance of the World Bank in the Execution of the Project 43. The management and technical staff of the World Bank assisted with the execution of the Project. This included technical assistance, video conferences to discuss various processes and aspects of the Project, discussions about Project documents and information, official supervision missions to verify the execution of Project activities on-site, and analysis of proposals of addenda to the Project and reprogramming. These activities were carried out with expediency and consensus. 4.2 Performance of the Government in the Execution of the Project 44. The Borrower, through MINED, requested four amendments to the Project (including the last one, whose purpose was to extend the implementation period of the Project). The Project was carried out by the technical, procurement, and financial units of MINED. These units have executed the Project’s various activities in accordance with the loan agreement, the Project Operating Manual, and World Bank regulations, using implementation systems (SEPA and later STEP for acquisitions and Client Connection for the disbursement of funds). Accounting processes were carried out through MINED’s Government Accounting System, with the Government also completing timely technical reports and audits regarding the Project. 45. To-date, 99.75 percent of the Project’s activities have been executed. The 0.25 percent that has not been executed corresponds to the activity “Develop a web portal or blog with education and Page 55 of 82 The World Bank Education Quality Improvement Project (P126364) monitoring and evaluation indicators and other data for the IS-IFTS to provide information to the public,” for an amount of US$30,000. This was reprogrammed for the activity “Hire technical coordinators and administrative support for the 14 regional Education DD and for the activity “Design and implement an information system, the National Education Evaluation System.” 46. A total of 98.86 percent of funds have been disbursed, supporting: • 36 integrated schools with renovations • 195 schools furnished with equipment and educational materials • 9,842 students attending the extended school day in 2018 • 9,759 students benefiting from school transportation • 1150 master teacher trainers • 1,619 teachers with subject-specific training • 441 schools with school furniture • 230 schools with funds for minor repairs 47. The table below shows the appraised distribution of resources by component, their final allocation after the multiple Project restructurings and the total executed funds by Project closure, both for the loan and the counterpart funds. Appraised Final allocation of funds Executed funds Component Counterpart Counterpart Counterpart Loan Total Loan Total Loan Total funds funds funds Component 1 54.9 10.4 65.3 55.2 4.7 59.9 55.0 6.6 61.6 Sub-component 1.1 4.5 - 4.5 7.7 - 7.7 7.7 - 7.7 Sub-component 1.2 4.2 - 4.2 3.5 - 3.5 3.5 - 3.5 Sub-component 1.3 33.6 - 33.6 38.6 - 38.6 38.6 - 38.6 Sub-component 1.4 0.7 1.4 2.1 0.5 1.1 1.6 0.5 1.3 1.8 Sub-component 1.5 11.9 9.0 20.9 4.8 3.6 8.4 4.8 5.3 10.0 Component 2 5.0 - 5.0 4.7 - 4.7 4.5 - 4.5 Sub-component 2.1 0.9 - 0.9 0.4 - 0.4 0.4 - 0.4 Sub-component 2.2 0.3 - 0.3 0.1 - 0.1 0.1 - 0.1 Sub-component 2.3 2.3 - 2.3 2.7 - 2.7 2.7 - 2.7 Sub-component 2.4 0.5 - 0.5 0.3 - 0.3 0.2 - 0.2 Sub-component 2.5 - - - 1.1 - 1.1 1.1 - 1.1 Unallocated 1.0 - 1.0 - - 0 - - - SUB TOTAL 59.9 10.4 70.3 59.9 4.7 64.6 59.5 6.6 66.1 Front-end fee 0.15 - 0.15 0.15 - 0.15 0.15 - 0.15 TOTAL 60.0 10.4 70.4 60.0 4.7 64.7 59.7 6.6 66.2 Page 56 of 82 The World Bank Education Quality Improvement Project (P126364) 48. The following graph shows the timeline for the disbursement of the Project funds as of December 31st, 2018. Of the funds received from the World Bank, US$59,313,670.03 or 98.86 percent was disbursed. Disbursement of Project funds by year 70 59.3 60 50 40 30 21.1 20 11.6 11.1 8.6 10 1.3 1.4 4.1 0 2012 2013 2014 2015 2016 2017 2018 Total 4.3 Compliance with standards 49. The Education Quality Improvement Project, approved through Loan Agreement No. 8110-SV, was executed in accordance with this agreement and conforms to the provisions of the General Conditions and to Annex I: Project Description, and Annex II: Execution of the Project, which were ratified by the Legislative Assembly of El Salvador. 50. The main guide for the execution of the Project was the Project Operations Manual, approved by the World Bank, which incorporates procurement standards for goods and services, the selection and contracting of consulting and training services, the use of SEPA and STEP for acquisitions, and the observation of the Disbursement Manual of Bank Funds. The remission of funds was strictly observed by Project reports and established audit reports. 4.4 Risks to the Project’s continuation 51. The biggest risk to the continuation of the Project that can be foreseen at this moment is the change in Government in 2019, which undoubtedly will lead to changes in education policy. These short- term changes occur with each administration. However, there are very visible results from the Project that have contributed and strengthened the education system, including 36 schools undergoing renovations and infrastructure improvements and around 210 schools receiving funds for minor repairs. It is worth noting that the transfer of funds for minor repairs is an almost permanent program, with Government funding. There is also a felt need for the continuation of the teacher training program, for which MINED already has allocated funding. 52. In adopting the IFTS model, it is known that the results of the Project funded by the World Bank are very limited, but they form a foundation that will be strengthened by FOMILENIO II, although that action is also limited to only 345 schools. Page 57 of 82 The World Bank Education Quality Improvement Project (P126364) 4.5 Sustainability 53. Now that the IFTS model has matured, it is important to ensure its sustainability by: • Involving all the model’s stakeholders in its implementation • Adjusting the in-service teacher training so that it more fully aligns with the Project’s education model. It is important to mention that teacher training is in the process of being standardized, with the creation of the Teacher Training Institute and resources allocated to its budget • Having MINED present a transition and sustainability plan to be implemented after the Project’s closure. This plan will highlight school maintenance plans; the use of infrastructure, teacher training, and technical assistance for pedagogy and administration to strengthen and expand the education model; strategies to support the extended day; and the development of a social conscience for the model • Making a legal plan to strengthen governance in public schools 5. Lessons learned and recommendations 54. The Project was prepared in the following context: • The Project was designed within the framework of political transition • MINED was committed to expanding the IFTS model • The scope of the Project was defined at the central level • The structure and legal regulations of the new education model were not fully defined • There was no previous study of the costs (human and financial) and technical assistance needed to implement such a model • The difficult budgetary situation of the counterpart • School centers were to be repaired or built without a clear indication of land ownership 5.1 Bottlenecks 55. Bottlenecks in the Project’s implementation include: • The elaboration of terms of reference, technical specifications for works, and goods and services to be acquired • Little clarity and consensus on the education model • The definition of the governance model to address management issues • Little communication between the different actors involved in Project activities, such as central and department-level technicians, and between providers, MINED technicians, school directors, and the community • The bureaucracy of the administrative and technical processes • Inexpediency of counterpart funds Page 58 of 82 The World Bank Education Quality Improvement Project (P126364) 56. These bottlenecks were due to: • Excessive project restructuring • Lack of leadership in the coordination of the Project (IFTS Deputy Director, Coordinator of MINED’s Coordinating Unit of Strategic Projects, etc.) • Poor planning of Project activities and actions • Coordination and communication channels that were not fluid • Difficult Government budget situation • Bureaucratic mentality 57. Many of these bottlenecks were not resolved, and the Project timeline had to be extended by 12 months. Others were resolved in specific coordination meetings. 5.3 Recommendations 59. In order to push for the extended school day, additional school activities need to address the areas of art and culture, job readiness, family life, etc. Financial resources to hire workshop instructors and purchase materials should be allocated to the Government budget each year. The contribution of workshops to improving student performance should be assessed, and efforts to improve the quality of learning should be re-oriented, with adequate provision of teacher training focused on the IFTS model and learning materials and other resources that permit students to improve their academic and citizenship skills and productivity. 60. Other recommendations include: • The strategy for school transport and feeding should be analyzed very carefully in terms of sustainability, due to the impact on MINED’s budget. • The management training that began in 2017, and that will continue in the following years, should support the needs that arise in implementing the IFTS model. • The National Directorate of Education Management should consider a strategic plan to enhance: (a) Maintenance plans for newly constructed infrastructure (b) Technical assistance for pedagogy and administration to strengthen the IFTS model through the DD (c) The continuation of the National Teacher Training Plan, focused on strengthening the IFTS model, after December 31, 2018 (d) Jointly establishing with the Education DD guidelines to take advantage of the additional learning facilities, furniture, and equipment, as well as a strategy to share available resources Page 59 of 82 The World Bank Education Quality Improvement Project (P126364) ANNEX 6. REVISION HISTORY OF PROJECT INDICATORS PDO Indicators PDO Indicator 1- Revised Original Original Original Indicator Original Description Original Universe Reason Baseline Target Number of students enrolled in 7th This PDO indicator was revised in the Improvement in grade in year t (net of 7th-grade July 2015 restructuring to monitor the Access Rate All schools in the repeaters in year t-1) as a proportion changes in the access rate not only for for lower 96.3% 98.8% 29 targeted of the total number of 6th-grade lower but also for upper secondary secondary municipalities students who were promoted to 7th education. In addition, the education grade at the end of year (t-1) methodology to calculate the access Revised indicator Revised Revised rate changed to align it with the Revised Description Revised Universe UNESCO definition of effective (July 2015) Baseline Target The number of students enrolled in 7th transition rate and the targets were grade in year (t+1) minus the 7th grade revised downward to reflect the repeaters from year (t+1) as a Lower Lower declining trend of this indicator at the proportion of the number of 6th grade secondary: secondary: national level. Finally, the universe of All schools in the measure for this indicator was modified students enrolled in year (t) minus the 96.6% 96.6% Improvement in 40 targeted to reflect the changes in the number of number of 6th grade repeaters from the Access Rate Integrated schools that would benefit from the full year (t+1) multiplied by 100 for lower and Systems, out of implementation of the IFTS model. The number of students enrolled in upper secondary which 213 offer 10th grade in year (t+1) minus the 10th education secondary grade repeaters from year (t+1) as a Upper Upper education proportion of the number of 9th grade secondary: secondary: students enrolled in year (t) minus the 78% 78% number of 9th grade repeaters from year (t+1) multiplied by 100 Page 60 of 82 The World Bank Education Quality Improvement Project (P126364) PDO Indicator 2- Revised Original Original Original Indicator Original Description Original Universe Reason Baseline Target Improvement in Number of students in grades 7-9 In the Project restructuring of July 2015 the Retention that remain in school during the All schools in the this PDO indicator was revised to Rate for lower academic year as a proportion of 95.1% 98.4% 29 targeted monitor changes in the retention rate secondary the total number of students of municipalities not only for lower but also for upper education grades 7-9. secondary education. In addition, the Revised indicator Revised Revised targets were revised downward to Revised Description Revised Universe reflect the declining trend of this (July 2015) Baseline Target Number of students in grades 7-9 indicator at the national level and the enrolled at the end of year t universe of measure was modified to divided by the total number of reflect the changes in the number of 91.0% 91.0% schools that would benefit from the full students in grades 7-9 enrolled at the beginning of year t multiplied implementation of the IFTS model. by 100 All schools in the Improvement in Number of students in general 40 targeted the Retention upper secondary education plus Integrated Rate for lower number of students in technical Systems, out of and upper secondary education enrolled at which 213 offer secondary the end of year t divided by the secondary education students in general upper 92.5% 92.5% education secondary education plus number of students in technical secondary education enrolled at the beginning of year t multiplied by 100 Page 61 of 82 The World Bank Education Quality Improvement Project (P126364) PDO Indicator 3- Revised Original Original Original Indicator Original Description Original Universe Reason Baseline Target Number of students that graduate In the Project restructuring of July Improvement in from 11th grade at the end of 2015 this PDO indicator was revised to the Graduation All schools in the school year t as a proportion of the monitor changes in the graduation rate Rate for upper 75.0% 91.7% 29 targeted number of students enrolled in not only for upper but also for lower secondary municipalities 10th grade at the beginning of secondary education. In addition, education school year (t – 1) students in technical upper secondary Revised indicator Revised Revised were included in the calculations, the Revised Description Revised Universe targets were revised downward to (July 2015) Baseline Target Number of students that graduate reflect the declining trend of this from 9th grade at the end of indicator at the national level and the school year t as a proportion of the universe of measure was modified to 69.0% 69.0% reflect the changes in the number of number of students enrolled in 7th grade at the beginning of school schools that would benefit from the year t-2 full implementation of the IFTS model. All schools in the Improvement in Number of students that graduate 40 targeted the Graduation the second year of General Upper Integrated Rate for lower Secondary in year t as a proportion Systems, out of and upper of students enrolled in the first which 213 offer secondary year of General Upper Secondary secondary education (t-1) plus number of students that 71.5% 71.5% education graduate the second year of Technical Upper Secondary in year t as a proportion of students enrolled in the first year of Technical Upper Secondary in year (t-1) multiplied by 100 Page 62 of 82 The World Bank Education Quality Improvement Project (P126364) PDO Indicator 4- Revised Original Original Original Original Original Description Reason Indicator Baseline Target Universe A cluster will be considered effectively implemented in a given municipality The PDO indicator changed to if, at least, 4 out of the 6 factors described below are in place for all schools capture the number of schools, within all clusters in that given municipality: as part of an Integrated Systems, Number of • New pedagogical approach implemented which adopt the IFTS instead of municipalities The 29 • Longer school day implemented the number of municipalities. The with clusters 0 29 targeted • School directors and teachers of all grades trained in the features of the universe of measure was effectively municipalities new model modified to reflect the changes in implemented • Schools earmarked for civil works fully completed, including furniture. the number of schools that would • Educational equipment and material delivered benefit from the full • Territorial reorganization of IFTS schools within the cluster completed. implementation of the IFTS Revised model. Revised Revised Revised indicator (July Revised Description Baseline Target Universe 2015) It is considered that a school has fully adopted the IFTS model if it completes at least 4 of the 6 factors which are: • Be in an IS with a Pedagogical Plan completed. • Longer school day implemented with 30 or more hours of class time All 213 offered. schools that Number of • School directors and lower and upper secondary teachers in the school offer schools that trained in the features of the IFTS model. secondary fully adopt the • School has access in the IS to improved hub school infrastructure with civil 0 213 education in Inclusive Full- works fully completed, including furniture. the 40 time School • Educational equipment and material have been delivered to schools targeted Model earmarked for this in the IS; Integrated • School participates in the territorial reorganization of IS as evidenced by at Systems least one teacher instructing in more than one school in the IS; at least one shared space used by more than one school; and students receiving a transport stipend. Page 63 of 82 The World Bank Education Quality Improvement Project (P126364) Intermediate Results Indicators IRI Indicator 1 - Revised Original Original Original Indicator Original Description Original Universe Reason Baseline Target Direct beneficiaries of the Project will be This indicator was revised to calculated only for those 29 selected capture the changes in the Eligible schools municipalities and will be: scope of the project (482 schools with - 34,017 students in lower secondary education interventions and the teacher Direct Project 43,871 primary and/or (grades 7 to 9) in Participating schools; training strategy as well as to beneficiaries, of 0 (49% secondary - 7,038 students in upper secondary education include parents and students which female female) education) in the (grades 10 to 11) in Participating schools; from technical upper 29 targeted - 2,334 full-time teachers from grades 1-11 in secondary education into the municipalities Eligible schools; calculations. - 482 directors of the Eligible schools Revised indicator Revised Revised Revised Description Revised Universe (July 2015) Baseline Target Beneficiaries in all 902 schools in the 101 Integrated Systems: - 59,576 students in grades 7-9 All schools in the - 12,440 students in general upper secondary 101 Integrated Direct Project education. 96,549 Systems beneficiaries, of - 15,634 students in technical upper secondary 0 (49% benefitting from which female education. female) the Project - 6,544 teachers in all levels of education. interventions - 814 school directors. - 761 parents - 780 teacher trainers Page 64 of 82 The World Bank Education Quality Improvement Project (P126364) IRI Indicator 2 - Dropped Original Original Original Original Indicator Original Description Reason Baseline Target Universe Number of full-time At the Project restructuring of July equivalent teachers of 2015, the teacher training strategy was Number of additional grades 1-6 who were redesigned. As a result of this change, it qualified secondary school accredited through was decided that the Project funds will All schools in the teachers and number of passing all mandatory only be used to train secondary 0 1,216 29 targeted additional qualified school IFTS courses (modules). education teachers and therefore this municipalities directors resulting from - The mandatory modules indicator was dropped. Project intervention for teachers will be of two types: a) pedagogical; b) subject matter based. Revised indicator (July Revised Revised Revised Revised Description 2015) Baseline Target Universe Indicator dropped Page 65 of 82 The World Bank Education Quality Improvement Project (P126364) IRI Indicator 3 - Revised Original Original Original Indicator Original Description Original Universe Reason Baseline Target Number of full-time equivalent teachers of grades 7-11 This indicator was Number of additional and number of school directors who were accredited disaggregated into two qualified secondary school through passing all mandatory IFTS courses (modules). indicators to be able to teachers and number of All schools in the 29 targeted - The mandatory modules for teachers will be of two 0 943 monitor the progress additional qualified school municipalities types: a) pedagogical; b) subject matter based. of the teacher and directors resulting from - The mandatory modules for school directors will be of directors' training Project intervention two types: a) pedagogical; b) management. separately. Revised indicator (July Revised Revised Additionally, the Revised Description Revised Universe 2015) Baseline Target targets were revised to The National Plan for Teacher Training in the Public capture the changes in All schools with secondary the scope of the Sector for 2015-2019 (Plan Nacional de Formación de Number of certified education in the 101 project interventions Docentes en Servicio en el Sector Público 2015-2019) teachers from lower and 0 1,301 Integrated Systems and, in June 2017, the defines the certification criteria for secondary teachers. upper secondary schools benefitting from the Project target for the number - Subject areas: Mathematics, Social Studies, Language interventions of teachers certified and Literature, Biology, Physics and Chemistry. The National Plan for Teacher Training in the Public All schools with at least 5 was revised to capture Sector for 2015-2019 (Plan Nacional de Formación de teachers in the 101 the inclusion of art, Number of certified physical education, Docentes en Servicio en el Sector Público 2015-2019) 0 285 Integrated Systems directors and indigenous defines the certification criteria for directors. benefitting from the Project interventions cultures as additional subject areas for which Revised indicator (June Revised Revised Revised Description Revised Universe teachers will be 2017) Baseline Target certified under the The National Plan for Teacher Training in the Public Project financing. Sector for 2015-2019 (Plan Nacional de Formación de All schools with secondary Number of certified Docentes en Servicio en el Sector Público 2015-2019) education in the 101 teachers from lower and defines the certification criteria for secondary teachers. 0 1,500 Integrated Systems upper secondary schools - Subject areas: Mathematics, Social Studies, Language benefitting from the Project and Literature, Biology, Physics, Chemistry, Physical interventions Education, Arts, English and Indigenous Cultures. Page 66 of 82 The World Bank Education Quality Improvement Project (P126364) IRI Indicator 4 - Revised Original Original Original Indicator Original Description Original Universe Reason Baseline Target Number of additional All technical staff from This indicator qualified MINED MINED's headquarters and 4 was revised in technical staff (at the Number of MINED technical staff (central and from all 14 Departmental the Project central and departmental level) who pass all mandatory modules 0 82 Directorates involved with restructuring of departmental level) of training workshops designed for this purpose. the direct implementation of July 2015 to resulting from Project the IFTS model in the 29 capture the intervention targeted municipalities. changes in the Revised indicator (July Revised Revised teacher training Revised Description Revised Universe strategy. Then, 2015) Baseline Target The National Plan for Teacher Training in the Public in June 2017 the Sector for 2015-2019 (Plan Nacional de Formación de targets were Docentes en Servicio en el Sector Público 2015-2019) increased to Number of certified reflect the defines the certification criteria for teacher trainers 0 624 Teachers at the national level teacher trainers inclusion of art, - Subject areas: Mathematics, Social Studies, Language and Literature, Biology, Physics and physical Chemistry. education, English and Revised indicator Revised Revised Revised Description Revised Universe indigenous (June 2017) Baseline Target cultures as The National Plan for Teacher Training in the Public additional Sector for 2015-2019 (Plan Nacional de Formación de subject areas for Docentes en Servicio en el Sector Público 2015-2019) which teachers Number of certified defines the certification criteria for teacher trainers will be certified 0 750 Teachers at the national level teacher trainers - Subject areas: Mathematics, Social Studies, under the Language and Literature, Biology, Physics, Chemistry, Project Physical Education, Arts, English and Indigenous financing. Cultures. Page 67 of 82 The World Bank Education Quality Improvement Project (P126364) IRI Indicator 5 - Revised Original Original Original Indicator Original Description Original Universe Reason Baseline Target Number of IFTS that are Number of IFTS that are The indicator was revised to just being renovated and/or being renovated and/or Selected schools offering measure educational materials equipped with equipped with secondary education in and equipment transfers to 0 201 schools, as the number of educational material and educational material and the 29 targeted equipment according to equipment according to municipalities schools renovated with small the IFTS standards the IFTS standards works was already captured in IRI 8. Revised indicator (July Revised Revised Revised Description Revised Universe 2015) Baseline Target Selected schools offering Number of IFTS that are Educational material and secondary education in being equipped with equipment transfers 0 195 the 101 Integrated educational material and delivered to schools. Systems benefitting from equipment the Project interventions Page 68 of 82 The World Bank Education Quality Improvement Project (P126364) IRI Indicator 6 - Revised Original Original Original Indicator Original Description Original Universe Reason Baseline Target Number of new This indicator was revised to better capture the Selected schools classrooms built Number of new works done in the rehabilitated schools, since offering secondary under the Project, classrooms built under schools benefitted not only from new 0 723 education in the 29 including the Project, including classrooms but also from other educational targeted necessary necessary furniture facilities such as kitchens, libraries, bathrooms, municipalities furniture dining areas, etc. The methodology to monitor Revised indicator Revised Revised the progress in this indicator was revised during Revised Description Revised Universe the June 2017 Project restructuring. (July 2015) Baseline Target Facilities include new Selected schools classrooms and other Number of newly offering secondary educational facilities built or renovated 0 882 education in the 40 such as kitchens, facilities targeted Integrated libraries, bathrooms, Systems dining areas, etc. Revised indicator Revised Revised Revised Description Revised Universe (June 2017) Baseline Target Facilities include new Selected schools classrooms and other Number of newly offering secondary educational facilities built or renovated 0 697 education in the 40 such as kitchens, facilities targeted Integrated libraries, bathrooms, Systems dining areas, etc. Page 69 of 82 The World Bank Education Quality Improvement Project (P126364) IRI Indicator 7 - Revised Original Original Original Original Original Universe Reason Indicator Description Baseline Target Number of Selected schools The targets for this indicator were revised to reflect the Number of IFTS changes in the infrastructure strategy agreed in the IFTS being offering secondary being renovated Project restructuring of 2015 which entailed doing more renovated 0 201 education in the 29 under the comprehensive and in-depth infrastructure renovations under the targeted Project in fewer schools. Then, in June 2017 the targets were Project municipalities reduced from 46 to 36 as a result of the higher than Revised budgeted infrastructure costs and the poor state of Revised Revised Revised indicator (July Revised Universe many of the schools targeted for this activity, which Description Baseline Target 2015) required more extensive works than originally planned. Number of Selected schools Number of IFTS IFTS being offering secondary being renovated renovated 0 46 education in the 40 under the under the targeted Integrated Project Project Systems Revised Revised Revised Revised indicator (June Revised Universe Description Baseline Target 2017) Number of Selected schools Number of IFTS IFTS being offering secondary being renovated renovated 0 36 education in the 40 under the under the targeted Integrated Project Project Systems Page 70 of 82 The World Bank Education Quality Improvement Project (P126364) IRI Indicator 8 - Added in July 2015 Original Original Original Indicator Original Description Original Universe Reason Baseline Target This indicator was added to Revised indicator Revised Revised monitor the progress in the Revised Description Revised Universe transfers made to the schools (July 2015) Baseline Target Number of IFTS that Number of schools that Selected schools offering for repairs and other minor receive transfers for receive transfers for secondary education in the 101 works 0 195 repairs or other repairs and small Integrated Systems benefitting minor works equipment purchases from the Project interventions IRI Indicator 9 - Revised Original Original Original Indicator Original Description Original Universe Reason Baseline Target Number of students in Number of students in This indicator was revised during the All schools offering Project restructuring of July 2015 to grades 7 to 9 that have grades 7 to 9 that have lower secondary capture the changes in the scope of the 40 weekly hours of 40 weekly hours of 0 34,017 education in the 29 project interventions and be consistent pedagogical activities pedagogical activities targeted with MINED's regulation, according to coordinated by the coordinated by the municipalities which the IFTS model extended the school school pedagogical activities an additional 5 hours Revised indicator (July Revised Revised Revised Description Revised Universe per week (30 weekly hours in total). 2015) Baseline Target Number of students in Number of students in All schools offering grades 7 to 9 that have grades 7 to 9 that have lower secondary 30 or more weekly 30 or more weekly 0 12,604 education in the 40 hours of pedagogical hours of pedagogical targeted Integrated activities coordinated activities coordinated Systems by the school by the school Page 71 of 82 The World Bank Education Quality Improvement Project (P126364) IRI Indicator 10 - Revised Original Original Original Original Indicator Original Universe Reason Description Baseline Target Number of Number of All Eligible schools This indicator was revised during the Project students students (offering primary restructuring of July 2015 to capture the changes in transported to transported to 0 11,906 and/or secondary the scope of the project interventions and school school in the school in the education) in the 29 clustering and then, in June 2017, the targets were cluster cluster targeted municipalities reduced due to both budgetary constraints in MINED, challenges in monitoring the use of Revised indicator Revised Revised Revised stipends, and reduced need for movement of Revised Universe (July 2015) Description Baseline Target students as more schools in integrated systems offer workshops on their premises. Number of Number of students students All schools offering transported to transported to secondary education in 0 16,000 school in the school in the the 40 targeted Integrated Integrated Integrated Systems System System Revised indicator Revised Revised Revised Revised Universe (June 2017) Description Baseline Target Number of Number of students students All schools offering transported to transported to secondary education in 0 10,000 school in the school in the the 40 targeted Integrated Integrated Integrated Systems System System Page 72 of 82 The World Bank Education Quality Improvement Project (P126364) IRI Indicator 11 - Revised Original Original Original Original Description Original Universe Reason Indicator Baseline Target This indicator was revised The monitoring system will have to comprise, at Municipalities during the Project least, a) the adoption of annual standardized tests with a restructuring of July 2015 for students to collect data on education quality All 29 targeted monitoring 0 29 to capture the changes in and learning outcomes, and b) the adoption of a municipalities system at the the geographic targeting, system of indicators to monitor evolution of access, cluster level from municipalities to retention and graduation rates at the cluster level. Integrated Systems, and Revised indicator Revised Revised in the universe that Revised Description Revised Universe would benefit from the (July 2015) Baseline Target full implementation of the IFTS model. The monitoring system will have to comprise, at least, a) the adoption of annual standardized tests for students to collect data on education quality All 40 Integrated Number of and learning outcomes, and b) the adoption of a Systems targeted to Integrated system of indicators to monitor evolution of access, benefit from the from Systems with a 0 40 retention and graduation rates at the cluster level. the full monitoring implementation of the system Each monitoring system will include ICT equipment, IFTS model such as a computer, software, and connectivity capabilities. Page 73 of 82 The World Bank Education Quality Improvement Project (P126364) IRI Indicator 12 - Revised Original Original Original Indicator Original Description Original Universe Reason Baseline Target The communication strategy will This indicator was revised during have to comprise, at least, a) a the Project restructuring of July publicity spot in the local media; b) 2015 to capture the changes in Municipalities that a communication campaign the geographic targeting, from implemented the All 29 targeted (posters, brochures, etc.) within 0 29 municipalities to Integrated communication municipalities each eligible school; c) one Systems, and in the universe that strategy for the IFTS workshop for the community in would benefit from the full each cluster to be created within implementation of the IFTS each municipality. model. Revised indicator Revised Revised Revised Description Revised Universe (July 2015) Baseline Target The communication strategy includes a national and local focus. National strategy: advertising spot, radio slots, informative pamphlet All 40 Integrated Number of or brochure, and a point of contact Systems targeted to Integrated Systems in each of the 14 departmental benefit from the from that implemented 0 40 directorates. the full the communication At a local level, there will be at implementation of strategy for IFTS least one point of contact for the IFTS model communications in each IS, social network promotion and use, and a web page for each IS. Page 74 of 82 The World Bank Education Quality Improvement Project (P126364) IRI Indicator 13 - Revised Original Original Original Indicator Original Description Original Universe Reason Baseline Target Municipalities that Selected municipalities The focus of this indicator The process evaluation will be a qualitative have undertaken a within the 29 targeted shifted from undertaking to assessment of the performance of the new process evaluation 0 7 where the new starting the evaluation of the governance model piloted in 7 selected of the governance governance model was governance model for the IFT municipalities. system going to be piloted schools to account for delays Revised indicator Revised Revised resulting from MINED's decision Revised Description Revised Universe (July 2015) Baseline Target to review the national level Municipalities that Selected municipalities governance of schools, to which The process evaluation will be a qualitative the IFTS governance plan was have started a within the 29 targeted assessment of the performance of the new subject. process evaluation 0 7 where the new governance model piloted in 7 selected of the governance governance model was municipalities. system going to be piloted IRI Indicator 14 - Revised Original Original Original Original Indicator Original Description Reason Baseline Target Universe Baseline information for the Impact This indicator was Number of municipalities with an All 29 targeted Evaluation will be collected in the 1st 0 29 disaggregated into two impact evaluation started municipalities year of the Project. indicators easier to Revised Revised Revised measure, which monitor Revised indicator (July 2015) Revised Description the two most important Baseline Target Universe Baseline survey for impact evaluation milestones of the impact Baseline survey for impact evaluation All 29 targeted has been collected No Yes evaluation: the data has been collected municipalities collection for the Follow-up survey for impact evaluation Follow-up survey for impact All 29 targeted baseline and the follow- evaluation has been collected No Yes has been collected municipalities up survey Page 75 of 82 The World Bank Education Quality Improvement Project (P126364) ANNEX 7. THEORY OF CHANGE WITH SUPPORTING LITERATURE Figure 7. 1. Overview of the Project’s Theory of Change The logic of the results chain of the Project is well-supported by evidence in the literature: • Improved physical environment for learning: There is a consensus that the physical school environment is linked to access to education and retention. In a recent review of the evidence, Barrett et al. (2019) find that classroom light, air quality, temperature, and acoustics play a role in a student’s ability to learn. Schools should be soundly built and able to protect students from the elements, particularly rain. Classrooms that are damp and have poor ventilation can create respiratory problems that lead to higher absences. Improving infrastructure is particularly important in Central America, where the physical condition of schools is generally poor (Duarte, Jaureguiberry & Racimo, 2017). In a review of 39 studies from developing countries, including 16 studies from Latin America, Cuesta, Glewwe & Krause (2016) find that sturdy walls, floors, and roofs, and access to basic services such as water and sanitation, have a positive impact on test scores and, to a lesser extent, on enrollment and attendance. Similarly, analyzing results from SERCE40, Treviño et al. (2010) find that additional elements of infrastructure, such as a sports field or dining room, correspond to higher test scores. Glewwe et al. (2013) and Krishnaratne, White & Carpenter (2013) find that constructing new facilities can increase enrollment and attendance by expanding capacity and reducing the distance to school. To improve access to education and retention and graduation rates, the project financed the expansion and renovation of 26 schools and 718 educational facilities, including classrooms, staff rooms, dining rooms, kitchens, laboratories, specialized classrooms and workshops, water and sanitation facilities, and arts and sports facilities. Additionally, 441 schools received classroom furniture, including desks, chairs, and bookshelves. These investments will allow 40 The Second Regional Comparative and Explanatory Study, implemented in 2006 in 16 Latin American countries. Page 76 of 82 The World Bank Education Quality Improvement Project (P126364) participating schools to implement the IFTS model and provide students with a comfortable physical environment that is conducive to learning. • Increased supply of materials for learning: Educational materials and equipment can encourage students to learn and help them graduate from school. In their reviews of education interventions in developing countries, Glewwe & Muralidharan (2016), Krishnaratne, White & Carpenter (2013), and McEwan (2015) all find some evidence that learning materials and ICT equipment can have a positive impact on test scores and graduation rates. The project provided 195 schools with educational materials and equipment, including computers and projectors, library books, sports equipment, and art supplies. • Additional class time: Extending the school day generally appears to have positive effects on education outcomes, including test scores and retention and graduation rates. Analyzing the impact of Uruguay’s Full-Time Schools program, Cerdan-Infantes & Vermeersch (2007) find statistically significant positive effects on test scores, and studies have found similar evidence in Brazil, Chile, Colombia, Mexico, and Peru (Agüero, 2016; Bellei, 2009; Berthelon, Kruger & Vienne, 2016; Cabrera- Hernández, 2015; Cruz, Loureiro & Sa, 2017; García Marín, 2006; Hincapie, 2016 and Padilla-Romo, 2017). In Brazil and Uruguay, Dias Mendes (2011) and Machado (2016) find that additional school hours improve passing rates, while in Chile, Pires & Urzua (2015) find that drop-out rates declined by three percent. In Argentina, Llach, Androgué & Gigaglia (2009) find that students who attended full- day primary schools were more likely to graduate from secondary school. In general, comprehensive programs that extend the school day as well as strengthen teacher training, curriculum, and school management appear to be more effective (Agüero, 2016; Alfaro, Evans & Holland, 2015; Cruz, Loureiro & Sa, 2017). In addition to expanding school hours with education workshops for 9,842 lower secondary students, the project sought to improve teaching practices and school management by training teachers and school directors. An impact evaluation of the project found that dropout rates in participating lower secondary schools declined by three percent, keeping 470 students in school who would have otherwise dropped out41. • Lower opportunity cost of studying: As a complement to the extension of the school day, some students received a transportation stipend and school feeding, the latest provided with government funds. Providing school feeding and student transportation can reduce hunger and commuting costs, improving access to education and retention rates for economically disadvantaged students. Krishnaratne, White & Carpenter (2013) find evidence that school feeding programs in Peru and other developing countries have a positive impact on enrollment, attendance, and dropout rates. Ganimian & Murnane (2016) and Glewwe & Muralidharan (2016) find evidence of a positive impact on test scores in some countries, including Argentina. Ganimian & Murnane (2016) also find cases where reducing commuting costs increase school enrollment, particularly for girls. • Improved teaching and school management practices: Effective teaching is one of the most important factors influencing a student’s ability to learn. Well-trained teachers can significantly improve test scores and provide students with the support they need to stay in school and complete their studies successfully. In a review of 77 randomized experiments measuring impact on learning 41The treatment group constructed for the impact evaluation included upper and lower secondary schools that received varying levels of exposure to the Project’s different components. Not all schools in the tr eatment group extended school hours. Page 77 of 82 The World Bank Education Quality Improvement Project (P126364) in developing countries, McEwan (2015) finds that all successful interventions include at least some effort to train teachers. Popova, Evans & Arancibia (2016) find that teacher training programs are particularly effective when they focus on subject content and are paired with educational materials. The project financed the training of 1,619 teachers in subject content and implementing the IFTS curriculum. Teachers also received educational materials, including reference books and ICT equipment, and were trained on how to use them in the classroom. There is also evidence in the literature that school management plays an important role in improving student outcomes (Adelman & Lemos, n.d. and Bloom et al., 2014). Therefore, in addition to training teachers, the Project also trained 474 school directors in an effort to improve school management practices in the IFTS. • Improved capacity to implement the IFTS model: The Project financed different activities to support the implementation of the IFTS model, such as the technical assistants in the departments, and evaluate it, such as the governance and the impact evaluations. All of these supported the institutionalization of the model. Page 78 of 82 The World Bank Education Quality Improvement Project (P126364) ANNEX 8. SUPPORTING DOCUMENTS Adelman, M., & Lemos, R. (n.d.). Managing for Learning: Measuring and Strengthening Education Management in Latin America and the Caribbean (Unpublished World Bank Working Paper). Washington, DC: World Bank Group. Agüero, J. M. (2016). Evaluación de Impacto de la Jornada Escolar Completa (Proyecto FORGE Documento Técnico). Lima, Peru: Fortalecimiento de la Gestión de la Educación en el Perú. Retrieved from http://www.grade.org.pe/forge/descargas/Evaluaci%C3%B3n%20de%20impacto%20de%20la%20jornad a%20escolar%20completa.pdf. Alfaro, P., Evans, D., & Holland, P. A. (2015). Extending the School Day in Latin America and the Caribbean (World Bank Policy Research Working Paper No. 7309). Washington, DC: World Bank Group. Retrieved from http://documents.worldbank.org/curated/en/477421467986293530/Extending-the- school-day-in-Latin-America-and-the-Caribbean. Barrett, P., Treves, A., Shmis, T., Ambasz, D., & Ustinova, M. (2019). The Impact of School Infrastructure on Learning: A Synthesis of the Evidence. Washington, DC: World Bank Group. Retrieved from http://documents.worldbank.org/curated/en/853821543501252792/pdf/132579-PUB-Impact-of- School.pdf. Bellei, C. (2009). Does lengthening the school day increase students’ academic achievement? Results from a natural experiment in Chile. Economics of Education Review 28(5), 629-640. Retrieved from https://ac.els-cdn.com/S0272775709000405/1-s2.0-S0272775709000405-main.pdf?_tid=3c4ccaaf- 17d4-42c4-af50-30f4a30f6a58&acdnat=1548787096_bb43021cee263f68c7ad55e57c88948c. Berthelon, M. E., & Kruger, D. I. (2011). Risky behavior among youth: Incapacitation effects of school on adolescent motherhood and crime in Chile. Journal of Public Economics 95, 41-53. Retrieved from https://ac.els-cdn.com/S0047272710001210/1-s2.0-S0047272710001210-main.pdf?_tid=5707098f- 466c-4411-b371-4409173ab46f&acdnat=1548787002_ce7898d3b008887792e07deb8ca41c42. Berthelon, M., Kruger, D., & Vienne, V. (2016). Longer School Schedules and Early Reading Skills: Effects from a Full-Day School Reform in Chile (IZA Discussion Paper No. 10282). Bonn, Germany: Institute for the Study of Labor. Retrieved from http://ftp.iza.org/dp10282.pdf. Bloom, H. S., Hill, C. J., Black, A. R., & Lipsey, M. W. (2008). Performance Trajectories and Performance Gaps as Achievement Effect-Size Benchmarks for Educational Interventions. Journal of Research on Educational Effectiveness 1(4), 289-328. Bloom, N., Lemos, R., Sadun, R., & Van Reenen, J. (2014). Does Management Matter in Schools (NBER Working Paper 20667). Cambridge, MA: National Bureau of Economic Research. Retrieved from https://www.nber.org/papers/w20667. Page 79 of 82 The World Bank Education Quality Improvement Project (P126364) Cabrera-Hernández, F. (2015). Does lengthening the school day increase students’ academic achievement? Evidence from a natural experiment (University of Sussex Working Paper Series 74-2015). Sussex, England: University of Sussex. Retrieved from https://www.sussex.ac.uk/webteam/gateway/file.php?name=wps-74-2015.pdf&site=24. Cerdan-Infantes, P., & Vermeersch, C. (2007). More Time Is Better: An Evaluation of the Full-Time School Program in Uruguay (World Bank Policy Research Working Paper No. 4167). Washington, DC: World Bank Group. Retrieved from http://documents.worldbank.org/curated/en/814001468124480715/pdf/wps4167.pdf. Contreras, D., & Sepúlveda P. (2016). Effect of Lengthening the School Day on Mother’s Labor Supply (World Bank Policy Research Working Paper No. 7599). Washington, DC: World Bank Group. Retrieved from http://documents.worldbank.org/curated/en/985891467995073286/pdf/WPS7599.pdf. Cruz, T., Loureiro, A., & Sa, E. (2017). Full-Time Teachers, Students, and Curriculum: The Single-Shift Model in Rio de Janeiro (World Bank Policy Research Working Paper No. 8086). Washington, DC: World Bank Group. Retrieved from http://documents.worldbank.org/curated/en/675871496683432879/pdf/WPS8086.pdf. Cuesta, A., Glewwe, P., & Krause, B. (2016). School Infrastructure and Educational Outcomes: A Literature Review, with Special Reference to Latin America. Economia 17(1), 95-130. Dias Mendes, K. (2011). O Impacto do Programa Mais Educação no Desempenho dos Alunos da Rede Pública Brasileira. São Paulo: University of São Paulo. Duarte, J., Jaureguiberry, F., & Racimo, M. (2017). Sufficiency, equity and effectiveness of school infrastructure in Latin America according to TERCE. Santiago, Chile: United Nations Educational, Scientific and Cultural Organization. Retrieved from https://unesdoc.unesco.org/ark:/48223/pf0000247571_eng. Evans, D. K., & Popova, A. (2016). What Really Works to Improve Learning in Developing Countries? An Analysis of Divergent Findings in Systematic Reviews. World Bank Research Observer 31(2), 242-270. Retrieved from https://academic.oup.com/wbro/article/31/2/242/2418055. Ganimian, A. J., & Murnane, R. J. (2016). Improving Education in Developing Countries: Lessons From Rigorous Impact Evaluations. Review of Educational Research 86(3), 719-755. García Marín, A. (2006). Evaluación del Impacto de la Jornada Escolar Completa (Unpublished Working Paper). Santiago, Chile: Universidad de Chile. Retrieved from https://sites.google.com/site/afgarciama/research. Glewwe, P., Hanushek, E. A., Humpage, S., & Ravina, R. (2013). School Resources and Educational Outcomes in Developing Countries: A Review of the Literature from 1990 to 2010. In P. Glewwe (Ed.), Education Policy in Developing Countries (pp. 13-64). Chicago, IL: University of Chicago Press. Page 80 of 82 The World Bank Education Quality Improvement Project (P126364) Glewwe, P. W., & Muralidharan, K. (2016). Improving Education Outcomes in Developing Countries: Evidence, Knowledge Gaps, and Policy Implications. In S. Machin, L. Woessmann, & E. A. Hanushek (Eds.), Handbook of the Economics of Education, 2016 (pp. 653-743). Amsterdam, Netherlands: Elsevier. Hincapie, D. (2016). Do Longer School Days Improve Student Achievement? Evidence from Colombia (IDB Working Paper Series No. 679). Washington, DC: Inter-American Development Bank. Retrieved from https://publications.iadb.org/publications/english/document/Do-Longer-School-Days-Improve-Student- Achievement-Evidence-from-Colombia.pdf. Kremer, M., Brannen, C., & Glennerster, R. (2013). The Challenge of Education and Learning in the Developing World. Science 340, 297-300. Krishnaratne, S., White, H., & Carpenter, E. (2013). Quality education for all children? What works in education in developing countries (3ie Working Paper). New Delhi, India: International Initiative for Impact Evaluation. Llach, J. J., Adrogué, C., & Gigaglia, M. E. (2009). Do Longer School Days Have Enduring Educational, Occupational, or Income Effects? A Natural Experiment in Buenos Aires, Argentina. Economia 10(1), 1- 43. Machado, A. (2016). Trayectorias educativas y desempeños escolares: principales determinantes e incidencia de la política de Escuelas de Tiempo Completo. Mimeo. McEwan, P. J. (2015). Improving Learning in Primary Schools of Developing Countries: A Meta-Analysis of Randomized Experiments. Review of Educational Research, 85(3), 353-394. Montenegro, C. E., & Patrinos, H. A. (2014). Comparable Estimates of Returns to Schooling Around the World (World Bank Policy Research Working Paper No. 7020). Washington, DC: World Bank Group. Retrieved from http://documents.worldbank.org/curated/en/830831468147839247/pdf/WPS7020.pdf. Padilla-Romo, M. (2017). The Short and Long Run Effects of Full-Time Schools on Academic Performance (Unpublished Job Market Paper). College Station, TX: Texas A&M University. Retrieved from https://www.academia.edu/34134241/The_Short_and_Long_Run_Effects_of_Full- Time_Schools_on_Academic_Performance. Paxson, C., & Schady, N. R. (2002). The Allocation and Impact of Social Funds: Spending on School Infrastructure in Peru. World Bank Economic Review 16(2), 297-319. Retrieved from http://documents.worldbank.org/curated/en/754171468293403751/pdf/773840JRN020020llocation0a nd0Impact.pdf. Pires, T., & Urzua, S. (2015). Longer School Days, Better Outcomes? (Unpublished Working Paper). Chapel Hill, NC: University of North Carolina. Retrieved from https://sites.google.com/site/tiagomcccpires/. Page 81 of 82 The World Bank Education Quality Improvement Project (P126364) Popova, A., Evans, D. K., & Arancibia, V. (2016). Training Teachers on the Job: What Works and How to Measure It (World Bank Policy Research Working Paper No. 7834: Background Paper to the 2018 World Development Report). Washington, DC: World Bank Group. Retrieved from http://documents.worldbank.org/curated/en/122031474915232609/pdf/WPS7834.pdf. Treviño, E., Valdés, H., Castro, M., Costilla, R., Pardo, C., & Donoso Rivas, F. (2010). Factores asociados al logro cognitivo de los estudiantes de América Latina y el Caribe. Santiago, Chile: Organización de las Naciones Unidas para la Educación, la Ciencia y la Cultura. Retrieved from https://unesdoc.unesco.org/ark:/48223/pf0000186769. Page 82 of 82