Document of The World Bank Report No: ICR00003655 IMPLEMENTATION COMPLETION AND RESULTS REPORT (IBRD-78870, IBRD-81320, IBRD-83140) ON A LOAN IN THE AMOUNT OF US$1,301,503,759.40 TO THE UNITED MEXICAN STATES FOR A SERIES OF PROGRAMMATIC DEVELOPMENT POLICY LOANS ON UPPER SECONDARY EDUCATION September 30, 2016 Education Global Practice Latin America and the Caribbean Region CURRENCY EQUIVALENTS (Exchange Rate Effective August 30, 2016) Currency Unit = Mexican Peso MX$1.00 = US$0.05 US$1.00 = MX$18.63 FISCAL YEAR January 1 – December 31 ABBREVIATIONS AND ACRONYMS ANUIES National Association of Universities and Institutions of Higher Education (Asociación Nacional de Universidades e Instituciones de Educación Superior) CBC Competency-Based Curriculum CONAEDU National Council of Education Authorities (Consejo Nacional de Autoridades Educativas) CONALEP National College of Professional Technical Education (Colegio Nacional de Educación Profesional Técnica) CONAPO National Population Council (Consejo Nacional de Población) CONEVAL National Council for the Evaluation of Social Development Policy (Consejo Nacional de la Evaluación de la Política de Desarrollo Social) COPEEMS National Council for Assessment in Upper Secondary Education (Consejo para la Evaluación de la Educación Media Superior) COSDAC Academic Development Unit (Coordinación Sectorial de Desarrollo Académico) CPS Country Partnership Strategy DPL Development Policy Loan EMS Upper Secondary Education (Educación Media Superior) ENADEMS National Survey on Dropout in Upper Secondary Education (Encuesta Nacional de Deserción en la Educación Media Superior) ENLACE National Evaluation of Academic Achievement in Schools (Evaluación Nacional del Logro Académico de Centros Escolares) EXCALE Education Quality and Achievement Test (Examen para la Calidad y el Logro Educativo) GDP Gross Domestic Product FCL Flexible Credit Line FM Financial Management FY Fiscal Year IBRD International Bank for Reconstruction and Development ICR Implementation and Completion Results Report IDA International Development Association IDB Inter-American Development Bank i IEISNB Index of entry into the SNB IFC International Finance Corporation IMF International Monetary Fund INEE National Institute for the Evaluation of Education (Instituto Nacional para la Evaluación de la Educación) M&E Monitoring and Evaluation MCC Common Curricular Framework (Marco Curricular Común) MUSE Mexico Upper Secondary Education NAFIN National Financial Development Bank (Nacional Financiera, S.N.C I.B.D) NDP National Development Program (Plan Nacional de Desarrollo) OECD Organisation for Economic Co-operation and Development PBIS Positive Behavioral Interventions and Support PDO Program Development Objectives PISA Program for International Student Assessments PLANEA National Plan for the Evaluation of Learning (Plan Nacional para la Evaluación de los Aprendizajes) PRF Program Results Framework PROFORDEMS In-Service Teacher Capacity-Building Program for Upper Secondary Education (Programa de Formación Docente de Educación Media Superior) PROFORDIR School Director Capacity-Building Program for Upper Secondary Education (Programa de Formación de Directores para Educación Media Superior) PROFORHCOM Skills-Based Human Resources Training Program (Programa de Formacion de Recursos Humanos Basada en Competencias) PSE Education Sector Program 2013-18 (Plan Sectorial de Educación) RIEMS Integral Upper Secondary Education Reform (Reforma Integral de la Educación Media Superior) SEMS Under Secretariat of Upper Secondary Education (Subsecretaría de Educación Media Superior) SEP Ministry of Public Education (Secretaría de Educación Pública) SHCP Ministry of Finance and Public Credit (Secretaría de Hacienda y Crédito Público) SIAT Early Warning System (Sistema de Alerta Temprana) SINATA National Academic Tutoring System (Sistema Nacional de Tutoría Académica) SISEEMS School Services System, Sistema de Servicios Escolares de la Educación Media Superior SPD Servicio Profesional Docente SNB National Upper Secondary Education System (Sistema Nacional de Bachillerato) TERCE Third Regional Comparative and Explanatory Study (Tercer Estudio Regional Comparativo y Explicativo) ii Senior Global Practice Director: Amit Dar Practice Manager: Reema Nayar Project Team Leader: Ciro Avitabile ICR Team Leader: Ciro Avitabile iii UNITED MEXICAN STATES Series of Development Policy Loans on Upper Secondary Education CONTENTS Data Sheet A. Basic Information B. Key Dates C. Ratings Summary D. Sector and Theme Codes E. Bank Staff F. Results Framework Analysis G. Ratings of Program Performance in ISRs H. Restructuring 1. Program Context, Development Objectives and Design ............................................................ 1 2. Key Factors Affecting Implementation and Outcomes .............................................................. 8 3. Assessment of Outcomes .......................................................................................................... 15 4. Assessment of Risk to Development Outcome ......................................................................... 28 5. Assessment of Bank and Borrower Performance ..................................................................... 28 6. Lessons Learned........................................................................................................................ 30 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners........................... 31 Annex 1. Bank Lending and Implementation Support/Supervision Processes............................. 32 Annex 2. Beneficiary Survey Results ........................................................................................... 36 Annex 3. Stakeholder Workshop Report and Results ................................................................... 37 Annex 4. Summary of Borrower's ICR ......................................................................................... 38 Annex 5. Comments of Cofinanciers and Other Partners/Stakeholders ....................................... 53 Annex 6. List of Supporting Documents ...................................................................................... 54 Annex 7. Scheme of the RIEMS ................................................................................................... 56 MAP .............................................................................................................................................. 57 iv A. Basic Information Program 1 Upper Secondary Country Mexico Program Name Education (MUSE) Development Policy Loan Program ID P112262 L/C/TF Number(s) IBRD-78870 ICR Date 08/05/2016 ICR Type Core ICR UNITED MEXICAN Lending Instrument DPL Borrower STATES Original Total USD 700.00M Disbursed Amount USD 700.00M Commitment Implementing Agencies Ministry of Public Education (Secretaría de Educación Pública) Cofinanciers and Other External Partners Program 2 Second Programmatic Upper Secondary Country Mexico Program Name Education Development Policy Loan IBRD-78870,IBRD- Program ID P126297 L/C/TF Number(s) 81320 ICR Date 08/05/2016 ICR Type Core ICR UNITED MEXICAN Lending Instrument DPL Borrower STATES Original Total USD 300.75M Disbursed Amount USD 300.75M Commitment Implementing Agencies Ministry of Public Education (Secretaría de Educación Pública) Cofinanciers and Other External Partners Program 3 Third Upper Secondary Country Mexico Program Name Education Development Policy Loan IBRD-81320,IBRD- Program ID P147244 L/C/TF Number(s) 83140 ICR Date 08/05/2016 ICR Type Core ICR Lending Instrument DPL Borrower UNITED MEXICAN v STATES Original Total USD 300.75M Disbursed Amount USD 300.75M Commitment Implementing Agencies Ministry of Public Education (Secretaría de Educación Pública) Cofinanciers and Other External Partners B. Key Dates Upper Secondary Education (MUSE) Development Policy Loan - P112262 Revised / Actual Process Date Process Original Date Date(s) Concept Review: 12/15/2009 Effectiveness: N/A 12/20/2010 Appraisal: 03/11/2010 Restructuring(s): N/A N/A Approval: 05/11/2010 Mid-term Review: N/A N/A Closing: 12/31/2010 06/30/2011 Second Programmatic Upper Secondary Education Development Policy Loan - P126297 Revised / Actual Process Date Process Original Date Date(s) Concept Review: 10/06/2011 Effectiveness: N/A 12/26/2012 Appraisal: 01/11/2012 Restructuring(s): N/A N/A Approval: 03/13/2012 Mid-term Review: N/A N/A Closing: 11/29/2013 11/29/2013 Third Upper Secondary Education Development Policy Loan - P147244 Revised / Actual Process Date Process Original Date Date(s) Concept Review: 09/10/2013 Effectiveness: 08/15/2014 07/18/2014 Appraisal: Restructuring(s): N/A N/A Approval: 12/16/2013 Mid-term Review: 11/24/2014 10/23/2014 Closing: 09/30/2015 09/30/2015 C. Ratings Summary C.1 Performance Rating by ICR Overall Program Rating Outcomes Satisfactory Risk to Development Outcome Low or Negligible Bank Performance Moderately Satisfactory Borrower Performance Satisfactory C.2 Detailed Ratings of Bank and Borrower Performance (by ICR) vi Overall Program Rating Bank Ratings Borrower Ratings Moderately Quality at Entry Government: Satisfactory Unsatisfactory Implementing Quality of Supervision: Satisfactory Satisfactory Agency/Agencies: Overall Bank Overall Borrower Moderately Satisfactory Satisfactory Performance Performance C.3 Quality at Entry and Implementation Performance Indicators Upper Secondary Education (MUSE) Development Policy Loan - P112262 Implementation QAG Assessments (if Indicators Rating: Performance any) Potential Problem Program No Quality at Entry (QEA) None at any time (Yes/No): Problem Program at any Quality of Supervision No None time (Yes/No): (QSA) DO rating before Satisfactory Closing/Inactive status Second Programmatic Upper Secondary Education Development Policy Loan - P126297 Implementation QAG Assessments (if Indicators Rating: Performance any) Potential Problem Program No Quality at Entry (QEA) None at any time (Yes/No): Problem Program at any Quality of Supervision Yes None time (Yes/No): (QSA) DO rating before Moderately Closing/Inactive status Satisfactory Third Upper Secondary Education Development Policy Loan - P147244 Implementation QAG Assessments (if Indicators Rating: Performance any) Potential Problem Program No Quality at Entry (QEA) None at any time (Yes/No): Problem Program at any Quality of Supervision No None time (Yes/No): (QSA) DO rating before Moderately Closing/Inactive status Satisfactory vii D. Sector and Theme Codes Upper Secondary Education (MUSE) Development Policy Loan - P112262 Original Actual Sector Code (as % of total Bank financing) Secondary education 92 92 Tertiary education 8 8 Theme Code (as % of total Bank financing) Education for all 84 84 Education for the knowledge economy 8 8 Participation and civic engagement 8 8 Second Programmatic Upper Secondary Education Development Policy Loan - P126297 Original Actual Sector Code (as % of total Bank financing) Secondary education 70 70 Vocational training 30 30 Theme Code (as % of total Bank financing) Education for all 60 60 Education for the knowledge economy 40 40 Third Upper Secondary Education Development Policy Loan - P147244 Original Actual Sector Code (as % of total Bank financing) Secondary education 100 100 Theme Code (as % of total Bank financing) Education for all 100 100 E. Bank Staff Upper Secondary Education (MUSE) Development Policy Loan - P112262 Positions At ICR At Approval Vice President: Jorge Familiar Pamela Cox Country Director: Gerardo M. Corrochano Gloria M. Grandolini Practice Manager/Manager: Reema Nayar Chingboon Lee Task Team Leader: Ciro Avitabile Mario Cristian Aedo Inostroza ICR Team Leader: Ciro Avitabile ICR Primary Author: Abril Ibarra viii Second Programmatic Upper Secondary Education Development Policy Loan - P126297 Positions At ICR At Approval Vice President: Jorge Familiar Hasan A. Tuluy Country Director: Gerardo M. Corrochano Gloria M. Grandolini Practice Manager/Manager: Reema Nayar Chingboon Lee Task Team Leader: Ciro Avitabile Erik Bloom ICR Team Leader: Ciro Avitabile ICR Primary Author: Abril Ibarra Third Upper Secondary Education Development Policy Loan - P147244 Positions At ICR At Approval Vice President: Jorge Familiar Hasan A. Tuluy Country Director: Gerardo M. Corrochano Gloria M. Grandolini Practice Manager/Manager: Reema Nayar Reema Nayar Task Team Leader: Ciro Avitabile Peter Anthony Holland ICR Team Leader: Ciro Avitabile ICR Primary Author: Abril Ibarra F. Results Framework Analysis Program Development Objectives (from Program Document) The proposed programmatic DPL aims to support the Government in the implementation of the Mexico Upper Secondary Education Reform to improve the internal efficiency of upper secondary education and its responsiveness to the labor market. The programmatic series would support actions in the following three areas addressing the key constraints of the EMS system: (i) improving flexibility of the upper secondary education; (ii) enhancing quality and relevance of education; and (iii) reducing opportunity costs of upper secondary education. This will be accompanied by the development of robust monitoring mechanism providing up-to-date information on the performance of schools and individualized data for parents on the learning achievement of their children. Revised Program Development Objectives (as approved by original approving authority) The PDO was revised in DPL 2 (February 8, 2012) to read ‘The proposed programmatic DPL aims to support the Government in the implementation of the Mexico’s Integral Upper Secondary Education Reform (Reforma Integral de la Educación Media Superior, RIEMS) to improve the internal efficiency of EMS and its responsiveness to the labor market. The programmatic series would support actions in the following three areas addressing the key constraints of the EMS system: (i) improving flexibility of EMS; (ii) enhancing quality and relevance of education; and (iii) reducing opportunity costs EMS.â€? The PDO was further revised in DPL 3 (November 11, 2013) to read ‘The proposed Development Policy Loan (DPL) aims to support the Government in the implementation of the Mexico’s Integral Upper Secondary Education Reform (Reforma Integral de la Educación Media Superior, RIEMS) to improve the internal efficiency and quality of EMS. The programmatic series would support actions in the following three areas addressing the key ix constrains of the EMS system: (i) improving flexibility of the upper secondary education; (ii) enhancing quality of EMS; and (iii) reducing opportunity costs of upper secondary education.â€? (a) Outcome Indicators Upper Secondary Education (MUSE) Development Policy Loan - P112262 Original Target Actual Value Formally Baseline Values (from Achieved at Indicator Revised Value approval Completion or Target Values documents) Target Years Percent of the upper secondary schools that, having adopted the Marco Común Indicator 1: Curricular (MCC), accept student transfers and validate equivalencies Value (quantitative or Less than 10 percent 40 percent 70.6 percent Qualitative) Date achieved 04/30/2010 09/30/2015 09/30/2015 Target exceeded. The actual value for this indicator is calculated with the percentage of schools that have adopted the MCC that are expected to accept students’ transfers Comments and validate equivalences; however, a national registry that records individual level (incl. % information does not yet exist. This indicator was replaced in DPL 3 with the Index of achievement) entry into the Sistema Nacional de Bachillerato (SNB), since students enrolled in SNB schools can transfer freely from one school to another. Unemployment rate ratio of upper secondary graduates ages 18-21 over the Indicator 2: unemployment rate of those upper secondary graduates with ages 25-30 Value (quantitative or 1.8 1.6 1.9 Qualitative) Date achieved 12/31/2009 09/30/2015 09/30/2015 Target not met. This indicator was dropped in DPL 2 because it was not suitable for Comments measuring Upper Secondary Education (Educación Media Superior, EMS) (incl. % responsiveness to the labor market. Changes in the indicator are likely to reflect the achievement) differential effect of macroeconomic conditions on different cohorts. Indicator 3: Reduction in dropout rates in upper secondary education Value (quantitative or 41.7% 34.3% 36.8% Qualitative) Date achieved 12/31/2008 09/30/2015 09/30/2015 Target 93% achieved. This indicator is calculated based on EMS terminal efficiency, and shows the percentage of students that did not complete upper secondary Comments education within the stipulated period of time. This indicator was replaced in DPL 3 (incl. % by the average annual dropout rate in EMS to align it with Ministry of Public achievement) Education (Secretaría de Educación Pública, SEP) official indicators. The average annual dropout rate was considerably reduced from 16.3 at the beginning of the DPL Series to 12.6 at the end of the Series. x Indicator 4: Value 99.2% (quantitative or 91% 95% (2014-15 school Qualitative) year) Date achieved 12/31/2010 09/30/2015 09/30/2015 Target exceeded. This indicator shows that most students transfer from lower to upper secondary education in Mexico. This indicator was dropped in DPL 2 as it was no Comments longer considered a major challenge for the reform and because transition rates are (incl. % more likely to be affected by macroeconomic conditions than by the policies achievement) supported by the DPL Series. However, the DPLs supported policies related to quality of education and the universalization of EMS that should have indirectly contributed to increased students transition. The percentage of students scoring in the categories good or excellent in the Indicator 5: ENLACE assessment Value 39.3% (quantitative or 15% 18% (ENLACE 2013-14) Qualitative) Date achieved 12/31/2008 09/30/2015 09/30/2015 Comments Target exceeded. This indicator was calculated as the average good or excellent (incl. % scores for Mathematics only. achievement) Improved teaching quality in upper secondary education due to massive in Indicator 6: service training and certification of teacher competences, as measured by the baseline and final results of the planned Impact Evaluation Value Final results from Baseline from the impact No impact evaluation (quantitative or the impact evaluation was conducted Qualitative) evaluation Date achieved 09/30/2015 This indicator was dropped in DPL 3. Before conducting a full-scale impact Comments evaluation, the Government decided to redesign the program to make it more relevant (incl. % for improving classroom practices and to address major enrollment, certification, and achievement) quality challenges that emerged at implementation. Indicator 7: Increased completion rates in upper secondary education for lowest quintile 47% for the entire 49% for the entire 50% for the entire population population population Value (quantitative or 29.9% for the lowest 38.7% for the lowest Qualitative) quintile (Revised at the quintile ICR) (Data collected at the ICR) Date achieved 12/31/2008 09/30/2015 Shows significant improvement. Baseline and target reflect data for the entire population and not for the lowest quintile. Data reflects that the increase in Comments graduation rates in recent years has been pro-poor. While the overall (incl. % completion rate grew by 7 percent, the rate for students in the lowest income achievement) quintile grew by almost 30 percent. This indicator was replaced in DPL 3 with the Gross Upper Secondary Education enrollment for students from households in the first four deciles of income per capita to align it with official SEP indicators. xi Indicator 8: Transition from lower to upper secondary school for lowest population quintile Value 57% (quantitative or 64% 75% 59.9 % (Revised at ICR) Qualitative) Date achieved 12/31/2008 09/30/2015 09/30/2015 Target exceeded. Information on the transition rate for the lowest population quintile is not available. However, data on the percentage of students in households under Mexico’s poverty line (benefiting from the Conditional Cash Transfer Program PROSPERA) and transferring from Lower to Upper Secondary Education is available. While the baseline and target of the original indicator are not necessarily Comments comparable to the revised indicator, the latter suggests that considerable (incl. % improvements have been achieved in this area. While the gap between the entire achievement) population and the poor is still high (24.2 percent difference), it is encouraging that three out of four poor students are transferring to EMS. Changes in this indicator are likely to reflect the differential effect of macroeconomic conditions on different cohorts. Therefore, and to align it with SEP official indicators, this indicator was replaced in DPL 3 by the gross upper secondary education enrollment for students from the households in the first four deciles of income per capita. Second Programmatic Upper Secondary Education Development Policy Loan - P126297 Original Target Actual Value Formally Baseline Values (from Achieved at Indicator Revised Value approval Completion or Target Values documents) Target Years Percentage of the upper secondary schools that accept student transfers and Indicator 1: validate equivalencies End of DPL 2: Value 25% (quantitative or Less than 10% 70.6% Qualitative) End of program: 40% Date achieved 04/30/2010 09/30/2015 09/30/2015 Target exceeded. The actual value for this indicator is calculated with the percentage of schools that have adopted the MCC that are expected to accept students’ transfers Comments and validate equivalences; however, a national registry to ensure they do does not yet (incl. % exist. This indicator was replaced in DPL 3 with the Index of entry into the Sistema achievement) Nacional de Bachillerato (SNB), since students enrolled in SNB schools can transfer freely from one school to another. Indicator 2: Reduction in dropout rates in upper secondary education Value (quantitative or 41.7% 34.3% 33.0% 36.8% Qualitative) Date achieved 12/31/2008 09/30/2015 02/30/2012 09/30/2015 Target 93% achieved. This indicator is calculated based on EMS terminal efficiency, which shows the percentage of students who completed upper secondary education Comments within the stipulated period of time. This indicator was replaced in DPL 3 by the (incl. % average annual dropout rate in EMS to align it with official SEP indicators. The achievement) average annual dropout rate dropped from 16.3 at the beginning of the DPL Series to 12.6 at the end of the Series. xii The percentage of students scoring in the categories good or excellent in the Indicator 3: ENLACE assessment in Spanish and Mathematics. End of DPL 2: End of Program: Math: 28% Math: 39.3% Value Math: 15.6% Spanish: 59% Spanish: 44.7% (quantitative or Spanish: 52.3% Qualitative) End of Program: ENLACE (school Math: 35.0% year 2013-14) Spanish: 63.0% Date achieved 12/31/2008 09/30/2015 Comments (incl. % Mathematics target exceeded. Spanish target not met. achievement) Increased completion rates in upper secondary education for the lowest Indicator 4: population quintile 50% for the entire 47% for the entire population End of DPL 2: 50% population Value End of Program: 38.7% for the lowest (quantitative or 52% 29.9% for the lowest quintile Qualitative) quintile (Revised at the ICR) (Data collected at the ICR) Date achieved 12/31/2008 09/30/2015 Comments Original baseline and target reflect data for the entire population and not the lowest (incl. % quintile. achievement) Indicator 5: Transition from lower to upper secondary school for lowest population quintile 57% End of DPL 2: Value 63% (quantitative or 75% 59.9% (Revised at the End of Program: Qualitative) ICR) 64% Date achieved 12/31/2008 09/30/2015 Target exceeded. Information on the transition rate for the lowest population quintile is not available. However, data on the percentage of students in households under Comments Mexico’s poverty line (benefiting from the Conditional Cash Transfer Program (incl. % PROSPERA) and transferring from Lower to Upper Secondary Education is achievement) available. Therefore, and to align it with SEP official indicators, this indicator was replaced in DPL 3 by the gross upper secondary education enrollment for students from the households in the first four deciles of income per capita. xiii Third Programmatic Upper Secondary Education Development Policy Loan – P147244 Original Target Actual Value Formally Baseline Values (from Achieved at Indicator Revised Value approval Completion or Target Values documents) Target Years Indicator 1: Index of entry into the Sistema Nacional de Bachillerato Value (quantitative or 0% 20% 37.18% Qualitative) Date achieved 12/31/2008 12/31/2013 09/30/2015 Target exceeded. This indicator is more reliable than the previous indicator “percent Comments of upper secondary schools that accept student transfers and validate equivalencesâ€? (incl. % because the percentage of students in SNB schools can transfer freely from one achievement) school to another. Indicator 2: Average annual dropout rate in EMS Value (quantitative or 16.3% 12% 12.6% Qualitative) Date achieved 12/31/2008 12/31/2013 09/30/2015 Comments Target 95% achieved. School dropouts were a major challenge for the Reform and (incl. % were considerably reduced during the DPL Series. achievement) Percentage of students scoring “goodâ€? or excellent on the ENLACE assessment Indicator 3: in Spanish and Math 44.7% Spanish Value Spanish 52.3% Spanish 53% 39.3% Math (quantitative or Math 15.6% Math 37% ENLACE (school Qualitative) year 2013-14) Date achieved 12/31/2008 12/31/2013 09/30/2015 Comments (incl. % Mathematics target exceeded. Target for Spanish not met. achievement) Percentage of students attending schools in very highly marginalized localities Indicator 4: scoring good or excellent on the ENLACE assessment in Spanish and Mathematics. Value Spanish 22.6% Spanish 31% Spanish 20.31% (quantitative or Math 6% Math 29% Math 23% Qualitative) Date achieved 12/31/2008 12/31/2013 09/30/2015 Comments (incl. % Mathematics target 79.3% achieved. Spanish target not met. achievement) Indicator 5: Gross Upper Secondary Education enrollment (youth ages 15-17) Value (quantitative or 58.6% 68.7% 71.5% Qualitative) Date achieved 12/31/2008 12/31/2013 09/30/2015 Comments Target exceeded. xiv (incl. % achievement) Gross Upper Secondary Education enrollment for students from households in Indicator 6: the first four deciles of income per capita. Value (quantitative or 61.8% 64% 67.8% Qualitative) Date achieved 12/31/2008 12/31/2013 09/30/2015 Comments Target exceeded. This is a major achievement since enrollment of low-income (incl. % students was one of the main challenges faced by the Reform. achievement) (b) Intermediate Outcome Indicators Upper Secondary Education (MUSE) Development Policy Loan - P112262 Original Target Actual Value Formally Baseline Values (from Achieved at Indicator Revised Value approval Completion or Target Values documents) Target Years Competency based curriculum implemented in at least 50 percent of all federal Indicator 1: schools Value (quantitative or 0 50% 100% Qualitative) Date achieved 04/30/2010 06/30/2011 Comments (incl. % Target exceeded. achievement) Indicator 2: A school evaluation system for entry in the SNB is operating Value (quantitative or NO YES YES Qualitative) Date achieved 04/30/2010 06/30/2011 Comments Target met. The standards for accreditation were agreed. The school evaluation (incl. % system was operating, fully staffed, and trained the necessary external evaluators to achievement) conduct the in situ school assessments for accreditation. Mexico has implemented its first sample-base assessment (EXCALE) for 12th Indicator 3: grade Value (quantitative or NO YES YES Qualitative) Date achieved 04/30/2010 06/30/2011 th Target met. The first EXCALE for 12 grade covering writing and citizenship was administered successfully by the National Institute for the Evaluation of Education Comments (Instituto Nacional para la Evaluación de la Educación, INEE) to a representative (incl. % sample of 13,118 students from May 12-19, 2010. The database has been made achievement) available to the public and INEE published several reports with the analysis of the main results. xv Number of teachers trained in PROFORDEMS (In-Service Teacher Capacity Indicator 4: Program for Upper Secondary Education) 72,763.00 (May 2011) Value 97,869 (quantitative or 0 120,000 (June 2011) Qualitative) 171,844 (September 2015) Date achieved 04/30/2010 04/30/2010 Target met at the end of the Series, not at expected year. Data reflects number of teachers enrolled (not graduates or certified teachers) in the In-Service Teacher Capacity-Building Program for Upper Secondary Education (Programa de Comments Formación Docente de Educación Media Superior, PROFORDEMS). The training of (incl. % teachers in PROFORDEMS was challenging due to several factors, including high achievement) demand outpacing funding and diffusion of responsibility in monitoring, evaluating, and following up. These issues were addressed and the indicator was actually exceeded by the Series closing date. Based on the new assessment instrument and existing administrative data, INEE Indicator 5: publishes for the first time the Indicadores del Sistema de Educación Media Superior Value (quantitative or NO YES YES Qualitative) Date achieved 04/30/2010 04/30/2010 07/31/2011 Comments Target met. The Indicadores del Sistema de Educación Media Superior were (incl. % published in the “Panorama Educativo de Méxicoâ€? Report in July 2011. achievement) The number of scholarships has increased in upper secondary education for the Indicator 6: 2009-10 school year. 426,642 (2009-10 school Value year) (quantitative or 280,000 320,000 Qualitative) 757,183 (2014-15 school year) Date achieved 09/30/2008 04/30/2010 06/30/2011 Comments (incl. % Target exceeded. achievement) SEMS has initiated a comprehensive evaluation study of the scholarship Indicator 7: program. Value (quantitative or NO YES YES Qualitative) Date achieved 04/30/2010 04/30/2010 06/30/2011 Comments Target met. The Under Secretariat of Upper Secondary Education (Sub-Secretaría de xvi (incl. % Educación Media Superior, SEMS), in coordination with the Bank, conducted an achievement) impact evaluation of the scholarship program. The study was finalized and the results were discussed with the Government. The conclusions of the study provided useful suggestions for the redesign of the scholarship program. Second Programmatic Upper Secondary Education Development Policy Loan - P126297 Original Target Actual Value Formally Baseline Values (from Achieved at Indicator Revised Value approval Completion or Target Values documents) Target Years The four service providers (federal, state, autonomous university and private) Indicator 1: have adopted the complete CBC Value (quantitative or NO YES YES Qualitative) Date achieved 02/30/2012 02/30/2012 02/30/2013 Comments Target met. Indicator revised to better reflect the achievements of the reform. The (incl. % original wording for this indicator was “Competence based curriculum implemented achievement) in all federal schools.â€? Indicator 2: COPEEMS has evaluated at least 100 schools Value (quantitative or NO YES YES Qualitative) Date achieved 02/30/2012 02/30/2012 02/30/2013 Comments Target exceeded. 171 schools were evaluated at DPL 2’s closing date and 1,926 were (incl. % accepted into the SNB by the end of the DPL Programmatic Series. achievement) Indicator 3: The publication by INEE of the EXCALE results Value (quantitative or NO YES YES Qualitative) Date achieved 02/30/2012 02/30/2012 02/30/2013 Comments (incl. % Target met. achievement) Indicator 4: 120,000 teachers have enrolled in PROFORDEMS training Value (quantitative or NO YES YES Qualitative) Date achieved 02/30/2012 02/30/2012 02/30/2013 Comments Target exceeded. 127,161 teachers were enrolled in PROFORDEMS in 2012. This (incl. % indicator was adjusted to respond to the challenges faced by teacher training achievement) programs mentioned earlier. xvii Indicator 5: The number of states participating in SIGUELE is at least 19 Value (quantitative or 0 19 32 Qualitative) Date achieved 02/30/2012 02/30/2012 02/30/2013 Comments Target exceeded. All states were participating in SIGUELE (an umbrella program (incl. % including several EMS initiatives to target students at risk of dropping out). achievement) Third Programmatic Upper Secondary Education Development Policy Loan – P147244 Original Target Actual Value Formally Baseline Values (from Achieved at Indicator Revised Value approval Completion or Target Values documents) Target Years There were no intermediate/output indicators for DPL 3. G. Ratings of Program Performance in ISRs Upper Secondary Education (MUSE) Development Policy Loan - P112262 Date ISR Actual Disbursements No. DO IP Archived (USD millions) 1 06/18/2010 Satisfactory Satisfactory 0.00 2 02/09/2011 Satisfactory Satisfactory 698.25 3 07/07/2011 Satisfactory Satisfactory 698.25 Second Programmatic Upper Secondary Education Development Policy Loan - P126297 Date ISR Actual Disbursements No. DO IP Archived (USD millions) 1 07/31/2012 Satisfactory Satisfactory 0.00 2 06/14/2013 Moderately Satisfactory Moderately Unsatisfactory 300.00 3 10/15/2013 Moderately Satisfactory Moderately Satisfactory 300.00 Third Upper Secondary Education Development Policy Loan - P147244 Date ISR Actual Disbursements No. DO IP Archived (USD millions) 1 04/07/2014 Satisfactory Satisfactory 0.00 2 01/26/2015 Moderately Satisfactory Satisfactory 300.00 H. Restructuring (if any) N/A xviii 1. Program Context, Development Objectives and Design 1.1 Context at Appraisal Macroeconomic Background 1. The Mexican economy rapidly recovered from the initial impact of the global financial and economic crisis. As a relatively open economy, Mexico was hard hit by the collapse of international trade during the last quarter of 2008 and the first quarter of 2009. After a severe contraction, resurgence in demand for Mexican manufactured exports led to a sharp rebound of economic activity. By the end of 2010, output had returned to its pre-crisis level and the output gap generated by the 2008-09 contraction was almost closed towards the end of 2012. 2. Modest global growth and heightened financial volatility weighed on the country’s economic performance. A global economic slowdown and weak external demand led to a dip in the expansion of economic activity in 2013 and 2014, whereas manufacturing export performance and, more recently, of private consumption managed to bring growth back again to its modest longer term annual average of about 2.5 percent in 2015 and 2016. Lower oil prices, a gradual tightening of monetary policy in the United States, and a slowdown of growth perspectives in emerging market economies are posing renewed challenges to economic stability and growth as the monetary and fiscal policy response to these adverse external shocks will weigh down on aggregate demand. 3. The authorities are gradually withdrawing fiscal support to reassure markets of Mexico’s fiscal sustainability. The slow and anemic recovery in global economic activity following the 2008 global financial crisis contributed to a delay in the withdrawal of countercyclical fiscal stimulus in Latin America’s six large financially integrated emerging market economies, including Mexico. A persistent primary deficit, a steady increase in the public debt-to- Gross Domestic Product (GDP) ratio between 2009 and 2016, and a sharp decline of oil prices as of late 2014 diminished the available fiscal space and necessitated consolidation. A significant decline in oil related fiscal revenue by about 3-4 percent of GDP due to the continued slump in oil prices over the past two years is being compensated by a hike in tax revenue as a result the revenue-enhancing tax reform enacted end-2013, an increase of the excise tax on domestic sales of gasoline and diesel, and significant non-recurrent revenue from an annually contracted oil price hedge. 4. Mexico’s strong macroeconomic policies and policy frameworks have been endorsed by the country’s continued access to the International Monetary Fund’s (IMF) Flexible Credit Line (FCL). The FCL was established in 2009 as part of a major reform of the IMF’s lending framework. The FCL is designed for crisis prevention purposes as it provides the flexibility to draw on the credit line at any time. Disbursements are not phased nor conditioned on compliance with policy targets. This flexible access is justified by the very strong track records of countries that qualify for the FCL, which gives confidence that their economic policies will remain strong. Mexico’s first FCL arrangement was approved in 2009, and successor arrangements were approved in 2010, 2011, 2012, 2014, and recently in 2016. The most recent FCL to an amount of US$88 billion has been contracted for a two-year period and, together with a significant international reserves at US$177 billion, provides insurance against 1 the tail risk of a sudden capital outflow in a climate of increased downside risks to global growth and volatility in global financial markets. 5. Implementation of an ambitious structural reform agenda introduced by the current administration has the potential to raise productivity and unleash growth in the medium term. During the first two years of the current administration, major progress has been achieved on the enactment of legislative changes in the areas of labor market regulation, education, telecommunication and competition policy, financial sector regulation, energy, and fiscal policy. The Government currently faces significant challenges to implement the reforms in these different areas with numerous new regulatory agencies being created. Reforms of this nature take time to implement and while several beneficial effects have already been observed, the overall impact on economic growth remains thus far elusive. 6. Progress in poverty reduction has been limited over the past few years. In addition to weak economic growth there has not been a strong connection between growth and poverty reduction. The cause of this can be found in the job market as there have not been enough jobs nor enough jobs that pay adequate wages created in recent years. There has also been an increase in the labor force—due to demographic change, net-zero migration to the United States and increased female labor force participation—that the economy has not been able to absorb. Labor incomes and hourly wages have fallen in many occupations. On the positive side, both public transfers, particularly in urban areas, and a lower dependency ratio have contributed to lower poverty. Sector Background 7. At the time of the Development Policy Loan (DPL) Program preparation, Mexico had made significant progress in education coverage, particularly in basic education, but national and international tests highlighted low learning outcomes of Mexican students across education levels. Coverage for basic education was of 94 percent in 2009-10, while it was only 60.8 percent and 25.6 for upper secondary and higher education, respectively. Dropout rates for basic education were quite low, with less than one percent of primary and 6 percent of secondary students leaving school, whereas in upper secondary the dropout rate was as high as 14.9 percent for the same school year. Most students that entered basic education successfully completed this school cycle, as reflected by a terminal efficiency rate estimated at around 90 percent in 2009-10, while upper secondary education had the lowest terminal efficiency rate (62 percent for the same school year), even lower than higher education with a 67.6 terminal efficiency rate. Finally, student learning, as measured by international standardized tests, lagged behind other Organisation for Economic Co-operation and Development (OECD) countries. Mexico’s 2012 Program for International Student Assessments (PISA) scores placed it in the last position among OECD countries and in 53rd place out of the 65 participating countries. Mexico slightly improved its Mathematics results in PISA over time (from 387 in 2000 to 413 in 2012), but reading results stagnated (from 422 in 2000 to 424 in 2012). The National Evaluation of Academic Achievement in Schools (Evaluación Nacional del Logro Académico de Centros Escolares, ENLACE), now the National Plan for the Evaluation of Learning (Plan Nacional para la Evaluación de los Aprendizajes, PLANEA) also highlighted low learning outcomes of Mexican students across education levels. 2 8. Despite important gains in education in recent decades, Mexico’s education system did not produce the human capital and skills demanded by the labor market. The upper secondary education system was, and still is, crucial to improving Mexico’s productivity and competitiveness. As result of the increased international competition and speed of the skill- biased technological change, Mexico needed more robust policies to make the transition from a labor-intensive industrial economy to a knowledge-based economy (Farrell, D., A. Puron, and J.K Remes, 2005). At the beginning of the DPL Programmatic Series, the education level of the Mexican workforce was one of the lowest in the OECD, as was the country’s performance on international education assessments. The percentage of the population with upper secondary education was one of the lowest for any country and was below that of some other countries in Latin America (See Table 1). More than at any other level, it was the low coverage at the upper secondary level that reduced the average education attainment of Mexico compared to its international peers. Preparing Mexico’s workforce for the knowledge economy required a different kind of Education System – one that emphasized new competencies and analytical ability while responding to the demands from the productive sector. Table 1: Education Attainment of Adults, age 24 to 64 (percentages) Country Basic or Upper Tertiary Less Secondary Mexico 78 7 15 Brazil 71 22 8 Chile 50 37 13 OECD Average 31 42 27 United States 13 48 40 Korea 23 44 32 Source: Education at a Glance 2008, OECD. Upper Secondary Education (Educación Media Superior, EMS) in Mexico and the gestation of the Reforms. 9. EMS is highly diverse. Students enter EMS (grades 10 to 12) at age 15 and after completing basic education (at least one year of preprimary, six years of primary, and 3 years of lower secondary). Approximately 80 percent of the students in EMS are between 15 and 17 years old. Four types of providers administer EMS schools: a) federal; b) state; c) publicly-financed autonomous universities; and d) private (with authorization from a public system, typically an autonomous university). These providers (called “sistemasâ€?) are divided into different systems (called “sub-sistemasâ€?), which have the authority to grant diplomas. There are a total of 200 to 300 systems, depending on how they are counted. The Federal Government serves 26 percent of the EMS student population through four major systems, typically in cooperation with the states. Each of the 31 states operates at least one system, accounting for 43.8 percent of enrollment. In addition, 27 autonomous universities operate upper education systems that account for 12.5 percent of enrollment. There is no national teachers’ union for EMS and most systems have at least one union. The course of study ranges from two to five years, although around 90 percent of students are in a three-year program. 10. Enrollment rates in EMS were high, but graduation rates remained low. At the 3 beginning of the Series, most students successfully completed basic education and an estimated 91 percent continued to EMS. Those that did not transition to EMS were disproportionately poor. Despite overall high enrollment, the dropout rate at the EMS level was high, as is shown in Figure 1. Most of the dropout occurred in the first year of EMS and only about 60 percent of students who enrolled in EMS graduated from a complete EMS program. Most of those who dropped out struggled to find a job. In 2006, according to the National Institute for the Evaluation of Education (Instituto Nacional para la Evaluación de la Educación, INEE), 21 percent of youth between 15 and 17 years old were neither working nor studying, a group known as the NiNis, which was a concern to policymakers that focused on crime and social disorder. Figure 1. Enrollment Rate by Grade for Basic and Upper Secondary Education 11. Among those who finished, the quality of learning was low. Students’ academic achievement was also a big concern. Only 15.6 percent of students achieved good or excellent scores in Mathematics in the census assessment (ENLACE) for students in their third and last year of EMS. Overall, EMS was a major bottleneck in the Education System that was limiting the number of skilled workers available to the Mexican economy. 12. The Under-secretariat of Upper Secondary Education (Subsecretaría de Educación Media Superior, SEMS) was created in 2005 to establish national standards and policies for the planning, organization, and academic and administrative evaluation of all EMS modalities and providers. Before the creation of SEMS, there was no central education authority guiding the upper secondary level, resulting in a disorganized and uncoordinated set of subsystems with important variations in quality. 13. In 2007, two years after its creation, SEMS embarked on an ambitious reform (Reforma Integral de la Educación Media Superior, RIEMS) to develop a national system to improve the relevance, quality and equity of EMS. The Reform established the National Upper Secondary Education System (Sistema Nacional de Bachillerato, SNB) to bring coherence 4 among the four different types of providers (federal, state, autonomous university, and private). The main elements of the reform were: a) the introduction of a Competency-Based Curriculum (CBC), which moved teaching away from the memorization of content into the development of skills and abilities; b) Coordination and Regulation, which set up common bodies to guide a diverse EMS system; c) Accreditation and Certification as a quality seal acknowledging the adoption of the Reform by systems and schools; d) New Tools and Institutions that supported capacity building and monitoring and evaluation; and e) Mentoring and Student Welfare, aimed at ensuring equity (see Annex 1 for the Reform scheme). The reform showed early progress in its implementation. At this point in time, the Bank’s involvement was crucial in providing resources and technical assistance to support the newly created SEMS and, particularly, key elements of the RIEMS to improve flexibility, relevance, quality and equity of education. 1.2 Original Program Development Objectives (PDO) and Key Indicators 14. The Development Objective of the DPL series as stated in the first and second DPLs’ Program Documents was to “support the Government in the implementation of Mexico Upper Secondary Education Reform (the second DPL explicitly mentions the Mexico’s Integral Upper Secondary Education Reform, RIEMS) to improve the internal efficiency of EMS and its responsiveness to the labor market.â€? 15. The Key Development Indicators defined to assess achievement of the program outcomes, in the three policy areas included in the DPL series, are listed in Table 2 below. Table 2. Key indicators of the DPL Series as Stated in the First DPL Program Document Policy Area 1. Improving Flexibility of EMS ï‚· Percent of upper secondary schools that, having adopted the Marco Curricular ComuÌ?n (MCC), accept student transfers and validate equivalencies. Baseline: Less than 10 percent End of program: 40 percent Policy Area 2. Improving quality and relevance of EMS ï‚· Unemployment rate ratio of upper secondary graduates ages 18-21 over the unemployment rate of those upper secondary graduates with ages 25-30. Baseline: 1.8 in 2009. End of program: 1.6. ï‚· Reduction in dropout rates in upper secondary education. Baseline: 41.7 percent in 2008. End of program: 34.3 percent. ï‚· Transition from lower to upper Secondary School. Baseline: 91 percent. End of program: 95 percent. ï‚· Improved student learning. Measured by: The percentage of students scoring in the categories good or excellent in the ENLACE assessment. Baseline: 15 percent in 2008. End of program: 18 percent. ï‚· Improved teaching quality in upper secondary education due to massive in service training and certification of teacher competences, as measured by the baseline and final results of the planned Impact Evaluation. Policy Area 3. Reducing opportunity costs of EMS ï‚· Increased completion rates in upper secondary education. Baseline: 47 percent in 2008 for lowest quintile. End of program: 49 percent for lowest quintile. ï‚· Transition from lower to upper Secondary School Baseline: 57 percent in 2008 for lowest quintile. End of program: 64 percent for lowest quintile 5 1. 3 Revised PDO and Key Indicators, and Reasons/Justification 16. A change was made to the PDO in the third operation. DPL 3 replaced EMS responsiveness to the labor market with EMS quality. This decision was taken because the set of supported polices under the DPL series was focusing more broadly on EMS quality issues, as opposed to exclusively focusing on EMS relevance for the labor market. The latter left out important dimensions of quality that the RIEMS was trying to address; for instance, the development of generic competencies aimed at fostering citizenship and cultural awareness. In addition, there was no easily available data to measure the relationship between EMS and its responsiveness to the labor market given the complex set of variables (apart from EMS relevance) affecting this relationship. 17. In addition, five indicators were deleted, three added, and three revised to: (i) respond to the adjustment in the PDO; (ii) reflect the faster achievement of outcomes due to the rapid and unexpected advancement of the reform and; (iii) align DPL indicators with similar official Government indicators. Table 3 describes the changes made to key indicators and the reasons for modifications. Table 3. Revised Key Indicators and Reasons Policy Area 1. Improving Flexibility of EMS Index of entry into the Sistema Nacional de Revised in DPL 3 to adjust to Ministry of Public Bachillerato IISNB (baseline 2007-08 school year): Education (Secretaría de Educación Pública, SEP) 0% End of Program (2014-15 school year): 20%. indicators. The IISNB captures the percentage of students enrolled in schools that entered the SNB which should accept transfers and validate equivalences. The unemployment rate ratio of upper secondary Deleted in DPL 3 to respond to the adjustment to the graduates ages 18-21 over the unemployment rate of PDO. those upper secondary graduates with ages 25-30. Policy Area 2. Improving Quality and Relevance of EMS Average annual dropout rate in EMS Baseline Revised in DPL 3 to adjust to SEP indicator and to (2007-08 school year): 16.3%. End of program better illustrate the main challenge of the reform. (2014-15 school year): 12. Transition from lower to upper Secondary School. Deleted in DPL 3 because transition rates are quite Baseline: 91 percent. End of program: 95 percent. high and the real problem in EMS is not entry into, but permanence in this education level. Improved teaching quality in upper secondary Deleted in DPL 3 as it was unclear what the education due to massive in service training and indicator would measure and how. certification of teacher competences, as measured by: the baseline and final results of the planned Impact Evaluation. Percentage of students scoring “goodâ€? or “excellentâ€? Revised in DPL 2 to specify disciplines and on the ENLACE assessment in Spanish: Baseline disaggregate Spanish and Mathematics. (2007-08 school year): 52.3%. End of program (2014-15 school year): 53%) Mathematics: Baseline (2007-08 school year) 15.6%. End of program (2014-15 school year): 37%. Percentage of students attending schools in very Added in DPL 3 for both Math and Spanish, to 6 highly marginalized localities scoring “goodâ€? or reflect the equity dimension of improvements in “excellentâ€? on the ENLACE assessment in Spanish. quality of education. Baseline (2007-08 school year): 22.6%. End of program, (2014-15 school year): 31% Mathematics. Baseline (2007-08 school year): 6% End of program (2014-15 school year): 29%. Policy Area 3. Reducing Opportunity Costs of EMS Increased completion rates in upper secondary Deleted in DPL 3 since completion is strongly education. Baseline: 47 percent in 2008 for lowest correlated with not dropping out, especially in the quintile. End of program: 49 percent for lowest first year of EMS. quintile. Transition from lower to upper Secondary School Deleted in DPL 3 since it was not part of the official Baseline: 57 percent in 2008 for lowest quintile. End statistics calculated by SEP. of program: 64 percent for lowest quintile. Gross Upper Secondary Education enrollment for Added in DPL 3 to update previous indicators with students from households in the first four deciles of SEP official indicator, which focuses on the bottom income per capita. Baseline (2012-13 school year): 40 percent. 61.8%. End of program (2014-15 school year): 64%. Gross Upper Secondary Education enrollment Added in DPL 3 to adjust to SEP indicators. It (youth ages 15 to 17). Baseline (2007-08 school represents SEP’s current overarching target for this year): 58.6%. End of program (2014-15 school administration. This indicator will indirectly account year): 68.7%. for the poorest population that consistently have less access to upper secondary education. 1.4 Original Policy Areas Supported by the Program: 18. Policy Area 1. Improving flexibility of Upper Secondary Education. Prior to the introduction of the RIEMS, there was no overarching framework for EMS. Each program and system had its own approach and curriculum, which were often incompatible. Therefore, there was little possibility to transfer among systems without starting over. This lack of flexibility contributed to the inefficiency of EMS, resulting in a high level of school dropout. This policy area supported initiatives under RIEMS to establish the SNB.1 In particular, the development of the Common Curriculum (Marco Curricular Común, MCC) supported under DPL 1, was key for facilitating the transfer of students among and across the different systems, such that students could adjust their schooling pathways without starting over, thereby rendering the system more efficient. 19. Policy Area 2. Enhancing Quality and Relevance of Education. There was widespread concern that EMS did not produce graduates that were well prepared for further studies or for the labor market. With no overall body evaluating and accrediting schools against common standards, virtually all schools that offered EMS relied on disarticulated curricula, usually teacher-centered, out-of-date, and focused on memorization. Before the RIEMS, 70 percent of EMS teachers were university graduates, but there was little uniformity in their pedagogical training. This was one of the reasons why student achievement was low, as measured by national 1 Mexico created the SNB based on defined competences, to ensure a minimum standard for all upper secondary education graduates and to facilitate transfers of students between different sub-systems. 7 and international assessments. In addition, students’ lack of interest in EMS due to their concerns about its quality was estimated to account for almost 42 percent of the dropout rate (Encuesta Nacional de la Juventud, 2005) and the unemployment rate was high for graduates. This Policy Area supported actions to improve the quality and relevance of EMS, including the establishment of the National Upper Secondary Evaluation System, the full autonomy of INEE, and the provision of relevant professional development for all EMS teachers. 20. Policy Area 3. Reducing Opportunity Costs of Upper Secondary Education . Poor and vulnerable students (especially males) dropped out of EMS due to a host of economic, social, and psychological reasons. An important reason for dropping out of EMS was the foregone income for the duration of studies at this level. Since not all poor students qualified for income support cash transfer programs like Oportunidades (now PROSPERA), a well-targeted scholarship program was necessary to keep students in school. In addition, students from low- income families often missed the socio-emotional background to cope with the challenges of the EMS environment – e.g. larger and more diverse schools and more demanding curricula, and they tended to underestimate the returns to EMS. As a result, students from the bottom quintile of the income distribution were less likely to transition from lower to upper secondary and more likely to drop out. This Policy Area supported RIEMS aspects related to increased access to EMS among the poor and vulnerable youth, such as the consolidation, expansion, and redesign of its scholarship program to reach students who were at greatest risk of abandoning their studies. It also supported the policy to consolidate a package of programs to support at risk students at the state level and the Constitutional amendment that made EMS mandatory. 1.5 Revised Policy Areas 21. Policy Area 2 was rephrased to “Enhancing quality of the National Upper Secondary Education Systemâ€? in DPL 3 to fit with the change in the PDO. The reference to the relevance of education was deleted to align it with the revised PDO. This revised Policy Area focused more broadly on actions to improve the quality of EMS so students complete EMS with the skills to continue into higher education and for eventual labor market success. Specifically, it supported student assessments, teacher training, and school accreditation aspects of the RIEMS. As a result of the actions supported under this Policy Area, it was expected that EMS graduates would improve their skills (as measured by standardized test scores) to ultimately increase wages and employability. 1.6 Other significant changes 22. The triggers and indicative triggers for DPL 2 and 3 were adjusted to actual prior actions, mainly to respond to the faster achievement of outcomes due to the rapid and unexpected advance of the Reform. 2. Key Factors Affecting Implementation and Outcomes 2.1 Program Performance 23. Disbursements for the three operations were carried out as planned in the schedule established in DPL 1. Table 4 summarizes disbursements for each operation in the DPL Series. 8 Table 4: Disbursements for Each Operation Actual Release Operation Amount (US$) Release Date MUSE DPL I $700,000,000.00 12/27/2010 Regular MUSE DPL II $300,751,879.70 12/28/2012 Regular MUSE DPL III $300,751,879.70 07/25/2014 Regular 24. All policy actions under the three policy areas were met before Board approval for the three DPLs. Some changes were introduced to the indicative triggers for DPL 2 and 3 mainly to adjust the Program to the pace of the Reform. Table 5 briefly summarizes the status and evolution of the policy actions for the three operations. DPL 1. First Operation in the Programmatic Series Prior Actions Status Policy Area 1. ï‚· SEP has established a National Upper Secondary Education System and Completed established the institutional basis of said system. ï‚· SEP has established the set of minimum competences by subject that Completed students need to achieve by the end of upper secondary education in the National Upper Secondary Education System. Policy Area 2. ï‚· SEP has established a national evaluation system for the National Upper Completed Secondary Education System. ï‚· SEP, in collaboration with UPN and ANUIES, has developed an Upper Completed Secondary Education Teacher Training Program (PROFORDEMS), which is benefiting teachers from all states. ï‚· SEP has decided to establish a civil association (COPEEMS) to evaluate Completed the entry of schools into the National Upper Secondary Education System and has issued the guidelines for the by-laws of said civil association. Policy Area 3. SEP has consolidated its scholarship program targeting poor students who Completed are not covered by the Oportunidades Program or by any other scholarship program. DPL 2. Second Operation in the Programmatic Series Prior Action Status Policy Area 1. Indicative Triggers as in DPL 1 Project Document ï‚· Competence-based curriculum (CBC) is implemented in at least 50 Completed and exceeded. All percent of all federal schools. federal schools implemented the CBC. Dropped in DPL 2 since it did not reflect the rapid implementation of the Reform. 9 ï‚· SEP has completed the definition of generic, disciplinary and Completed. professional competences to be achieved by the end of upper secondary education. Prior Actions as in DPL 2 Project Document ï‚· In order to improve the employment prospects for graduates of the Completed. Trigger National Upper Secondary Education System, COPEEMS, has approved rephrased in DPL 2 to reflect 85 technical programs for the National Upper Secondary Education the rapid implementation of System covering, among others, the fields of agriculture, services, the Reform. administration and information. Policy Area 2. Indicative Triggers as in DPL 1 Project Document ï‚· Mexico has carried out and disseminated a 12th grade learning Completed. Dropped at the assessment nationwide based on basic competencies that includes time of DPL 2 preparation mathematics, language, and at least one other area (natural science or since the national assessment social science) did not consider a third area; however, the new assessment (PLANEA) introduced in 2015 includes it. ï‚· First sample-based education examination (EXCALE) in Upper Completed. Secondary Education is completed. The sample is statistically representative at the national level and for major EMS sub-systems. Results of EXCALE disseminated to the public and analysis and diagnostic of EMS prepared. ï‚· COPEEMS has begun evaluation and analysis of schools for entry into Completed. the SNB. ï‚· 70 percent of all federal school directors have been selected using the Completed and exceeded. competitive selection process. Dropped because all federal directors are now selected competitively. Prior Actions as in DPL 2 Project Document ï‚· In order to improve the quality of Upper Secondary Education, promote Completed. Trigger greater transparency in evaluation and facilitate the development of new rephrased to better reflect the curricula for the National Upper Secondary Education System, INEE has status of the Reform and links disseminated the results of the EXCALE test administered to students in among the assessment system, the last year of the National Upper Secondary Education System between transparency and quality. March and May of 2010. ï‚· In order to improve the administration and implementation of Completed. Introduced in PROFORDEMS, SEP has transferred the functions of supervision and DPL 2 given the slow evaluation of PROFORDEMS to SEP’s Academic Development Unit. progress in the number of teachers trained. ï‚· In order to establish clear rules regarding equal opportunity for all Completed. Triggered schools to join the National Upper Secondary Education System, the rephrased in DPL 2 to reflect Executive Committee for the National Upper Secondary Education System the faster implementation and has approved a modification of the rules, which govern the school improvements of the accreditation system. accreditation process. 10 Policy Area 3. Indicative Triggers as in DPL 1 Project Document ï‚· Results of evaluation of scholarships program are released and Completed. disseminated to general public and discussed within the sector. Prior Actions as in DPL 2 Project Document ï‚· In order to expand the coverage of the SIGUELE Program at the State Completed. Introduced in level, SEP has established the guidelines to implement the SIGUELE DPL 2 to reflect the Program at the State level, including the provision of technical support to Government’s effort to the Borrower’s States to implement the program, as evidence by the State improve coordination of Agreements. different federal and state programs to support students at risk. DPL 3. Third Operation in the Programmatic Series Prior Actions Status Policy Area 1. Indicative Trigger as in DPL 2 Project Document ï‚· SEP issues rules for the Certificado Nacional de Bachillerato. Dropped. The Government has delayed the introduction of the Certificado Nacional de Bachillerato, using instead the accreditation of schools as the basis for establishing coherence system-wide. Prior Actions as in DPL 3 Project Document ï‚· The Government has recognized equivalences of subjects across schools Completed. Introduced in to facilitate the transfer of students between systems. DPL 3 to reflect the rapid implementation of the Reform. Policy Area 2. Indicative Triggers from DPL 2 Document ï‚· SEP and INEE enter into partnership agreements with some states to Completed. INEE has now disseminate results from the National Evaluation System. agreements with 28 out of the 32 states. ï‚· SEP has developed a policy for training part-time public teachers. Completed ï‚· SEP has developed a mechanism to officially recognize alternative Completed teacher training programs. Prior Actions as in DPL 3 Project Document ï‚· The Government has (i) established INEE’s mandate, functions and Completed. Introduced in responsibilities and internal organization; and (ii) granted INEE full legal, DPL 3 to reflect new law technical and financial autonomy and new responsibilities, including those towards the full establishment related to the evaluation of Upper Secondary Education. of the National Assessment System. 11 ï‚· The Government has issued regulations to officially recognize Completed. Two triggers alternative teacher training programs under the National Upper Secondary were consolidated to reflect Education System. the Government’s strategy to accelerate relevant teacher training. Policy Area 3. Indicative Triggers from DPL 2 Document ï‚· SEP carries out an evaluation of SIGUELE. Dropped. The Government discontinued the SIGUELE name.2 ï‚· Mexico has developed a policy framework that assures universal EMS. Completed. Prior Actions as in DPL 3 Project Document ï‚· The Government, through SEP, has introduced a new modality to award Completed. scholarships under SEP scholarship program in order to improve the targeting of students at risk of dropping out. ï‚· The Government has made Upper Secondary Education compulsory. Completed. The trigger was rephrased to strengthen wording. 2.2 Major Factors Affecting Implementation 25. The Commitment of the Government of Mexico was a key factor in the timely and adequate implementation of the Reform. Despite the 2009 economic crisis and its effects in the following years, the Government maintained the education budget for EMS to keep the pace of the Reform going. SEMS also continued supporting the implementation of the Reform even though there was a change in administration and ruling party in December 2012. Moreover, in 2013, the Amendment to the Constitutional Article 3 and the General Education Law made explicit the State mandate of guaranteeing quality of compulsory education (which by then also included EMS), making the case of the RIEMS stronger. 26. The main challenge faced by the RIEMS has probably been the uneven adoption of the Reform by different education providers. In particular, federal providers, directly dependent on the Federal Government, adopted the Reform more quickly and completely than state providers did. This was expected since federal schools received direct funding to implement the policy actions promoted by the RIEMS. However, the Federal Government has also worked closely with state providers and has mobilized a considerable amount of resources to ensure that they also benefit from the RIEMS. 27. The implementation of the in-Service Teacher Capacity-Building Program for Upper Secondary Education (Programa de Formación Docente de Educación Media Superior, PROFORDEMS) also faced important implementation challenges including: (i) 2 SIGUELE was an overarching strategy to support students at risk through the consolidation of many pre-existing programs. SIGUELE featured 6 dimensions: i) an early warning system for students at risk of dropping out (Sistema de Alerta Temprana, SIAT); ii) a national academic tutoring system; iii) career counseling; iv) a youth development program (CONSTRUYE-T); v) scholarship programs; and vi) a reading program. SEMS’ current leadership discontinued SIGUELE, but the programs under the strategy remain. 12 high demand outpacing funding (particularly challenging was to reach 60 percent of all teachers who worked part-time and taught only a few hours a week); (ii) a diffusion of responsibility in monitoring, evaluating and following-up on the implementation of the training program; and (iii) a series of logistical challenges, such as assigning teachers to their closest training location and providing them with adequate equipment to allow for virtual options. To address these issues: (i) SEMS transferred the coordination and supervision functions of the program to SEMS’ Academic Development Unit (Coordinación Sectorial de Desarrollo Académico, COSDAC); (ii) the SNB’s Steering Committee recognized other existing training programs that met certain quality standards as acceptable alternatives to PROFORDEMS, diversifying teacher training options to better fit teachers’ needs; and (iii) the Government improved the content and methodology of the teacher training program. By the closing date of the DPL Series, however, most implementation issues were solved and approximately 90 percent of teachers were trained in PROFORDEMS. 28. The impact evaluation of PROFORDEMS was not conducted. At the beginning of the DPL Series, the demands of the daily administration of the PROFORDEMS prevented follow-up on the impact evaluation of the program. In the third DPL, the impact evaluation was cancelled because the Government decided to redesign PROFORDEMS to make it more relevant for improving classroom practices and to address major enrollment, certification, and quality challenges. Although the impact evaluation of PROFORDEMS was not conducted, the Bank team engaged with the new administration on the topic of teacher training. This collaboration included a workshop in June 2013 to share international best practices and strategies to improve teachers’ practices in the EMS classroom. Based on the outputs of the workshop, the Bank prepared a note to help guide the restructuring of PROFORDEMS that included recommendations to make the program more flexible and responsive to particular teaching needs and to revise the content and delivery to focus more sharply on those elements linked to improving teacher practices. 29. Gaining full accreditation to enter the National Upper Secondary Education System, SNB, proved to be very challenging for most schools. Hardly any school complied with the accreditation criteria (including adequate number of qualified teachers, school improvement plans, suitable infrastructure, etc.) and many lacked funding to finance the accreditation process. To address this issue, the National Council for Assessment in Upper Secondary Education (Consejo para la Evaluación de la Educación Media Superior, COPEEMS), the entity responsible for school accreditation, allowed schools to enter the SNB with a set of minimum criteria of quality, conditional to specific quality improvements in a certain period of time. From 2012, schools could enter the SNB with varying levels of quality (from level 4 with minimum quality criteria to level 1 with extremely high quality criteria). This change allowed more schools to participate in the SNB, but affected the quality seal function of the proposed SNB Certificate for individual students. Therefore, SEMS postponed the introduction of the SNB Certificate and instead decided that would continue to signal quality at the school level by the use of the school accreditation credentials/levels. 2.3 Monitoring and Evaluation (M&E) Design, Implementation and Utilization 30. Design. DPL1 Program Results Framework (PRF) had significant shortcomings. There 13 were 3 indicators linked to the differential effect of macroeconomic conditions on different cohorts that were not directly linked to the policies introduced by the RIEMS. One of the indicators (unemployment rate ratio of upper secondary graduates ages 18-21 over the unemployment rate of those upper secondary graduates with ages 25-30) was removed in DPL 2 and the remaining two (increased completion rates in upper secondary education for lowest income quintile and the transition from lower to upper secondary school for lowest population quintile) were deleted in DPL 3. The original PRF also confused baseline and target data on completion rates in EMS for the entire population with the lowest quintile. In addition, it did not have enough information on how and when the data to feed the PRF should be collected. In particular, it had two indicators (i. the percentage of students scoring in the categories good or excellent in the ENLACE assessment; and ii. the improved teaching quality in upper secondary education) that, although linked to the quality aspect of the PDO, were vaguely explained in the Program Document. In the case of ENLACE results, it was unclear if the indicator was referring to results in Mathematics, Spanish or the average of both. This drawback was corrected in DPL 2. The indicator to measure teaching quality was more problematic since its wording was confusing and did not specify a baseline or target. Information to measure teacher quality was to be provided by an impact evaluation; however, the impact evaluation was never conducted and thus this indicator was deleted in DPL 3. 31. Implementation. Although there was no clear specification on the data collection methods in DPLs 1 and 2, the Government monitored the progress of the Reform with several indicators, including most DPL Programmatic Series indicators. The original PRF was also adjusted in DPL 3 to be aligned with the Government’s indicators. This decision reduced the burden of data collection and analysis to feed into the PRF while ensuring more timely and reliable monitoring and evaluation (M&E). The Bank team conducted regular implementation support missions (at least twice a year) and worked closely with the Government to ensure timely and reliable data collection to feed the PRF and to provide enough information to understand the status of the DPL Series. However, education relevance and responsiveness to the labor market was not measured or evaluated until the Implementation Completion and Results Report (ICR). 32. Utilization: The Implementing Agency and the Bank constantly assessed and used Program data to inform policy-making. Based on the indicators’ data, SEMS policies were continuously revised to ensure better impacts on flexibility, quality and reduction of opportunity costs. The M&E material was also useful to provide just-in-time support and information to SEMS on a regular basis. For instance, information on the number of teachers enrolled in PROFORDEMS raised concerns about the training program operation, leading to the organization of a seminar with the Bank to analyze the teacher-training program in light of best international practices. The Government then tackled design and implementation problems identified at the seminar. In addition, the Bank’s technical support helped conduct two robust impact evaluations, one on the impact of information about labor market returns to EMS on students’ performance (Avitabile and De Hoyos, 2015) and one on the impact of the EMS scholarship program (De Hoyos et al. unpublished). Avitabile and de Hoyos (2015) found that providing students with information about the expected earnings by education level and the scholarships to attend higher education has no impact on the probability of completing upper secondary in time, but significantly on the performance in a standardized exam. Interestingly, the 14 study finds a differential effect by gender, with a statically larger effect on girls than on boys. De Hoyos et al. shows that the scholarship program that was based on a centralized allocation system had no impact, but the effect varies significantly according to the household income. The study highlights how one of the possible reasons behind the lack of statistically significant impact of the program has to be found in the mistargeting of the program: the online application system prevents students from the poorest households from applying due to the lack of a computer or a connection. Since then, the scholarship program has been replaced by a school based assignment system, where a local committee identifies students who are most at risk. While the new system has not been evaluated, it is likely to have addressed the targeting issues mentioned above. The Government used the results of both evaluations to scale up the role of information and begin redesigning the latter. 33. Overall, the M&E design, implementation and utilization of the DPL Series is Moderately Satisfactory. This rating is justified since the M&E shortcomings in DPL 1 were addressed in the second and third operations, except the EMS responsiveness to the labor market aspect of the PDO was never measured or evaluated for DPL 1 and DPL 2 (this objective was dropped for DPL 3). 2.4 Expected Next Phase/Follow-up Operation 34. The Bank team has built a strong relationship with SEMS and continues to provide technical assistance and policy advice whenever possible. For instance, the Bank is currently providing technical assistance to adapt to the Mexican context a successful initiative in the United States to reduce dropouts and improve learning outcomes of EMS students through intensive tutoring and the application of Cognitive Behavioral Therapy. 3. Assessment of Outcomes 3.1 Relevance of Objectives, Design and Implementation 35. Objectives. The relevance of objectives is deemed to be High throughout the Series. The original and revised PDOs are relevant to the World Bank Group’s Country Partnership Strategy (CPS) for FY14-FY19 (Report No. 83496) and fit under the CPS theme II “Increasing Social Prosperity,â€? under the fifth area of engagement “Promoting Labor Markets for Inclusive Growth.â€? This operation is expected to contribute to more people (and particularly the poor) completing upper secondary and with the right skills, to increase productivity and boost shared prosperity. To do so, this Series focused on improving the skills and preparedness of young Mexicans entering the labor market, with activities aimed at reducing the dropout rate of those in the bottom quintile. One of the main priorities for Mexico’s growth and development, emphasized both in the reform process and in the National Development Plan (NDP) for 2013- 18, is improving the quality of education, which is consistent with the Series’ PDO of improving internal efficiency and quality of EMS. The NDP also focuses on reducing the achievement gaps between rich and poor to increase productivity and long-term growth while reducing social inequalities, which is consistent with the PDO and the supported policy area of reducing the opportunity costs of EMS. In addition, the commitment of the Government in promoting equal opportunities resulted in a Constitutional amendment that made EMS mandatory (Government Decree modifying the General Education Law and published in the Official Gazette on June 10, 2013). This is in tune with the Series’ purpose to increase enrollment and reduce dropout rates in 15 EMS. Finally, the DPL series objectives are aligned with the Education Sector Program 2013-18 (Plan Sectorial de Educación, PSE) that underlines as one of the main objectives for this administration to consolidate the SNB, universalize the MCC, and strengthen the professionalization of teachers and head teachers at the EMS level. 36. Design and Implementation. The relevance of design and implementation of the DPL Programmatic Series is deemed to be Modest. The design was consistent with the objectives and was critical in supporting the implementation of the Reform. To improve the internal efficiency of the system (mainly due to the lost investment on students dropping out of EMS), it was crucial to include improvements in EMS flexibility and policies for reducing opportunity costs. Supporting policies and actions to make EMS more flexible was critical in helping retain students that dropped out of EMS because they were not able to transfer from one system or school to another. Similarly, the policies for reducing opportunity costs (such as consolidating and improving the targeting of the scholarships program) were directed at students that were unable to continue their studies due to their socioeconomic circumstances, particularly those at the bottom quintile. In parallel, including in the design policies to improve the EMS evaluation and accreditation system and teacher quality was key for improving the quality of EMS education, as measured by improvements in dropout rates and students’ academic achievement. The programmatic DPL implemented in a series of three operations allowed adapting policy actions in the second and third operations based on the experience gained, the rapid advance of the RIEMS, and the implementation challenges faced during early implementation. The technical assistance was also instrumental for providing just-in-time support to the Government to refine policies and address, in a timely and informed manner, implementation challenges of the Reform. However, the design had considerable shortcomings related to inadequacy of some indicators to measure some elements of the PDO (for more information please refer to section 2.3), and weak explanations of the links between some policy actions, key outcome indicators, and certain elements of the PDO. 3.2 Achievement of Program Development Objectives 37. The RIEMS created the institutional basis for the National Upper Secondary Education System (SNB) and introduced concrete policy actions to ensure that the four EMS providers (federal, state, private and autonomous universities) benefited from said system. Prior to the RIEMS, each of the EMS providers, and the many EMS subsystems within them, worked independently and without clearly defined national policies and quality assurance mechanisms that prevented student transferring from one subsystem to another and hindered quality and relevance of education. After less than 10 years of its implementation, more than one third of EMS students are now enrolled in schools that have gained SNB accreditation. In other words, SNB schools that: (i) share a Competency-Based-Curriculum for developing the skills and abilities required from EMS graduates as citizens, employees and university students; and (ii) have (or are working towards achieving) quality school leadership and teaching, school improvement plans, adequate group sizes and pupil-teacher-ratios, suitable infrastructure and equipment, and quality mentoring and student services. The Reform also consolidated the different EMS scholarships into a single Scholarship Program, reducing the transaction costs of having to apply to multiple and complicated processes to obtain a scholarship and introduced a scholarship targeting students at risk of dropping out. 16 38. Even at this early stage, the Reform has shown remarkable results. Due to the above- mentioned changes, the EMS system has improved its internal efficiency. Despite large increments in enrollment rates (from 58.6 in school year 2007-08 to 71.5 percent and 74.2 percent in school years 2014-15 and 2015-16, respectively), dropout rates have decreased considerably from 16.3 percent in 2008 to 12.6 percent in the 2014-15 school year. Szekely (2013) created an index that varies between 0 and 1, where a higher score is assigned to schools that have adopted a larger number of the RIEMS elements3 and he finds that graduates from schools that have adopted the Reform are more likely to be employed and have a lower probability of being NiNis. His results suggest that the RIEMS is likely to have improved the EMS responsiveness to the labor market. In terms of quality, and as stated previously, dropout rates have significantly improved and students’ academic achievement shows mixed results with impressive gains in Mathematics, but no improvements in Spanish. Previous evidence suggests that high school curricula that are more Math intensive can have a long term impact on the labor market outcomes (Altonji, 1995; Rose and Betts, 2004), and the lack of improvement in Spanish might be the result of students focusing more on Mathematics.4 Nevertheless, it is important to stress that reform processes take time to show convincing results in learning outcomes. Finally, in terms of equity, the Reform has helped reduce the opportunity costs of EMS by shortening the gaps in transition, enrollment, and graduation rates between students in the lower and upper ends of the income distribution. 39. Table 5 shows the links between the achievement of results, the policy actions and key indicators relying on evidence from the PDO Indicators and on supplementary indicators introduced at the Implementation Completion and Results Report (ICR) stage. Table 5. Links between PDO, Policy Areas and Indicators (including ratings) PDO/Ratings Policy Area Indicators/Ratings Explanation Improve the Improving 1. Percent of the Meeting the targets for indicator 1 internal efficiency Flexibility of upper secondary and 2 has improved the flexibility of upper secondary Upper Secondary schools that, having of EMS because more students can education (original Education adopted the transfer from one school/program to PDO) (original PDO) Competency-Based- another without the need to start Overall Rating: Curriculum (CBC), over. Flexibility has helped reduce High accept student transfers dropout rates (a stand-alone and validate indicator for internal efficiency) equivalencies. since students are now more likely Rating: High to find a school type that better fits their interests and talents without 2. Index of entry the need to start over. into the SNB (IEISNB) Rating: High In addition, the CBC and the SNB introduced national policies and 3. Average quality assurance mechanisms, 3 The index takes into account the six possible areas that the RIEMS was supposed to affect. 4 For instance, Avitabile and de Hoyos (2015) find that students are more likely to shift to math-intensive subtracks as a result of information about labor market returns of upper secondary. 17 Annual Dropout Rate bringing coherence to the system Rating: Substantial and improving its internal efficiency. Reducing 4. Gross Upper Completion and transition Opportunity Secondary Education indicators are not used in the ICR to Costs of Upper enrollment (youth ages measure reductions in the Secondary 15-17) opportunity costs of EMS since Education. Rating: High they are linked to the differential effect of macroeconomic conditions 5. Gross Upper on different cohorts that were not Secondary Education directly linked to the policies enrollment for students introduced by the RIEMS. from households in the However, achieved targets in gross first four deciles of EMS enrollment and gross EMS income per capita enrollment from households in the Rating: High first four deciles of income per capita are good proxies for reductions in the opportunity costs of EMS. Results from the impact evaluation conducted by de Hoyos et al. (unpublished) shows that the scholarship program, based on a centralized assignment system, had no impact on the dropout rate. However, students from the lowest income quintiles display a moderate reduction in dropout. The study suggests that because of the online application system, the scholarship failed to target the poorest students. Improve EMS Enhancing 6. Correlation The unemployment rate ratio of responsiveness to relevance of between the RIEMS upper secondary graduates ages 18- the labor market Education implementation index 21 over the unemployment rate of (original PDO) and labor market those upper secondary graduates Rating: Substantial indicators. ages 25-30 was removed in DPL 2 Rating: Substantial and is not used to measure enhanced relevance and improved EMS responsiveness to the labor market in the ICR since it is linked to the differential effect of macroeconomic conditions on different cohorts that were not directly linked to the policies introduced by the RIEMS. Improve quality of Enhancing quality 3. Average Annual The reduction in the dropout rate EMS (Revised and Relevance of Dropout rate. could be a good proxy of PDO) Education Rating: Substantial improvements in the quality of Rating: Substantial education. The national survey on 7. Percentage of dropout rates showed that one of students scoring in the the main reasons for students to categories good or drop out was the lack of quality and 18 excellent in the irrelevance of the education ENLACE assessment. received. Similarly, improvements Rating: Substantial in the transition rate from lower to upper secondary schools might be a 8. Improved proxy of improved quality. teaching quality in upper secondary As the literature points out, education improved teaching is also a good Rating: Modest proxy for improved education quality. 9. Index of entry into the SNB (IEISNB) The most conventional indicator to Rating: High measure improved quality of education is student academic 10. Percentage of achievement. students attending schools in very highly marginalized localities scoring good or excellent on the ENLACE assessment in Spanish and Mathematics. Rating: Modest Improve the internal efficiency of upper secondary education. Overall Rating: High 40. The overall rating for improved internal efficiency of upper secondary education is deemed High since 4 out of the 5 indicators that measure it were rated as High and 1 as Substantial. EMS improved its internal efficiency by increasing the system’s flexibility (Policy Area 1 supported by the DPL Series). Allowing more students to transfer from one school/program to another without the need to start over helped reduce dropout rates, a stand- alone indicator for internal efficiency. In addition, reducing the opportunity costs of EMS (Policy Area 2 supported by the DPL Series) also contributed to improving its internal efficiency as it allowed poorer students to enroll and stay in EMS. Percent of the upper secondary schools that, having adopted the Competency-Based- Curriculum (CBC), accept student transfers and validate equivalencies. Overall Rating: High (DPL1: High, DPL2: High, DPL3: N/A) 41. More than two thirds of the schools have adopted the CBC. The proportion of schools that have adopted the CBC and could potentially accept students’ transfers and validate equivalencies was less than 10 percent in 2008 and reached 70.6 percent in 2015, largely surpassing the target of 40 percent. COPEEMS ratifies the adoption of the CBC through the adjustment of study plans and programs from a traditional content-based curriculum to a competency-based approach that develops skills and abilities to respond to the societal, academic and labor demands for EMS graduates. The adoption of the CBC is the first step for undertaking the accreditation process. That more than 70 percent of schools have taken the first step for the accreditation process is indicative of the progress of the Reform. However, this does not 19 necessarily mean that all of these schools are accepting student transfers and validating equivalences, since these schools are not yet part of the SNB (have completed the accreditation process). That is why this indicator was replaced in DPL 3 for the indicator described in the following paragraphs. Index of entry into the SNB (IEISNB) Overall Rating: High (DPL1: NA, DPL2: NA, DPL3: High) 42. More than a third of students (1,789,427) 5 enrolled in 1,926 schools are able to adjust their schooling pathways without starting over, thus completing their degrees in fewer years and increasing the flexibility and internal efficiency of the EMS system. The proportion of students enrolled in schools that are part of the SNB in relation to the total enrollment of EMS increased from 0 in 2008 to 37.18 percent in 2015, exceeding the 20 percent target. Schools within the SNB are required to accept student transfers and validate equivalences. 43. The greater institutional capacity and flexibility of the system due to the development of the CBC and the creation of the SNB, prior actions supported under the DPL series, contributed to achieving the target of schools that adopted the CBC and the IEISNB. In particular, the adjustment of the curriculum to generic, subject area, and professional competencies and the development of minimum standards for upper secondary graduates (as part of the CBC) was crucial to provide a common ground to the diverse group of EMS subsystems. 44. In addition, the DPL series supported modifications of the General Education Law and a resolution among state and federal education authorities6 to recognize equivalences of subjects across schools to facilitate the transfer of students between systems. As a result of these policies, SEMS has strengthened the School Services System (Sistema de Servicios Escolares de la Educación Media Superior, SISEEMS). The SISEEMS has two main features: (i) it allows switching between the same subsystem campuses and careers depending on the number of subjects undertaken and passed (subsystems using this feature are DGETA, DGECYTM and DGETI); and (ii) allows external portability through (a) the revalidation of studies (when studies are performed abroad and contained in Certificates, Diplomas or Degrees provided they are comparable to studies conducted within the SNB) and (b) equivalences (when studies are conducted within the national education system and are comparable to each other), consisting of certificates. Students can now revalidate a full semester or specific subjects. Average Annual Dropout Rate Overall Rating: Substantial (DPL1: Substantial, DPL2: Substantial, DPL3: Substantial) 45. Dropouts were the major challenge faced by the reform since they stagnated at 17 percent for five consecutive years (from 2000 to 2005) and at 16 percent from 2006 to 2008. Dropout rates were higher in Mexico than in any other OECD country and many Latin American countries (OECD, 2008). The average annual dropout rate in EMS in 2008 was of 16.3 and in 5 There are a total of 4,813,164 EMS students in Mexico. 6 Represented by the National Council of Education Authorities (Consejo Nacional de Autoridades Educativas, CONAEDU). 20 2014, it decreased to 12.6, achieving the 12 percent target by 95 percent. This is a major achievement of the Reform and the policies introduced to tackle this problem. 46. The DPL series supported concrete actions to reduce dropout rates under the three policy areas. For instance, under Policy Area 1, actions to help students transfer from one school/program to another contributed to reducing dropouts by helping students stay in the system without starting the EMS cycle from scratch. Efforts to increase quality through improvement in the evaluation of the EMS system and teacher quality under Policy Area 2 also contributed to reducing dropouts. Similarly, Policy Area 3 supported the introduction of a new modality of scholarships for students at risk of dropping out and supported the Government mandate for compulsory Upper Secondary Education, which put pressure on all stakeholders to enroll and, more importantly, maintain students in school. 47. In particular, during DPL 3, SEMS launched a campaign to reduce dropouts, Movimiento Contra el Abandono, providing school committees autonomy to grant scholarships to students locally identified as being at risk of dropping out and to equip school leaders with a toolkit to identify and better target disadvantaged students to help them stay and complete their EMS studies. The Movimiento Contra el Abandono campaign has benefited almost 200,000 students. 48. DPL 2 also supported the implementation of the SIGUELE Program at the state level, including the provision of federal technical support to implement 6 programs to target students at risk through: i) an early warning system for students at risk of dropping out (Sistema de Alerta Temprana, SIAT) later converted into the Movimiento contra el Abandono; ii) the National Academic Tutoring System; iii) Career Counseling; iv) CONSTRUYE-T, a youth development program; v) the Scholarships Program; and vi) a Reading Program. All of these programs continue to target students at risk of dropping out that, in many cases, are also low-income students. 49. Equally important was the technical assistance that accompanied this DPL Series. The Bank team prepared several policy notes, reports, and seminars focusing on supporting the Government strategies to decrease dropouts. The Bank also provided technical assistance to conduct a qualitative evaluation of the Movimiento Contra el Abandono initiative that provided concrete recommendations to improve it based on a detailed study of some of the most successful and unsuccessful schools. Gross Upper Secondary Education enrollment (youth ages 15-17) Overall Rating: High (DPL1: N/A, DPL2: N/A, DPL3: High) 50. Gross upper secondary education enrollment increased from 58.6 in 2008 to 71.5 percent in 2015, widely exceeding the 68.7 percent target. The DPL Series helped improve EMS enrollment mainly by supporting the policy to make EMS compulsory. 7 As the improvement in this indicator shows, this policy put pressure on both the supply and demand sides to ensure that more students enroll in EMS. Granting the right to all Mexicans to receive free public education through grade 12 (about the age of 18) is a critical step for increasing shared prosperity in 7 Recent evidence (e.g. Clark and Royer (2013)) shows that increases in compulsory schooling age has a positive impact on educational attainments. 21 Mexico since it has a major impact on the poor, who have considerably fewer opportunities for accessing upper secondary education and thus for increasing their productivity and wages. Gross Upper Secondary Education enrollment for students from households in the first four deciles of income per capita. Overall Rating: High (DPL1: N/A, DPL2: N/A, DPL3: N/A) 51. EMS enrollment for students from households in the first four deciles of income per capita increased from 61.8 percent to 67.8 percent from 2008 to 2015, exceeding the target of 64 percent. This is a major achievement of the Reform since enrollment of low-income students was one of the main challenges faced by EMS. The DPL Series contributed to this achievement by supporting the constitutional amendment rendering EMS mandatory, supporting the consolidation of the scholarship program, and providing scholarships specifically to target students at risk of dropping out of school. In the 2014-15 school year, SEMS and PROSPERA supported almost 2 million EMS students, equivalent to 50 percent of the entire EMS population. As a result of a better-targeted scholarship program, 8 out of every 10 EMS students coming from the first four income deciles have been granted a SEMS or PROSPERA scholarship. Improve EMS responsiveness to the labor market (original PDO). Overall Rating: Substantial 52. The overall rating for the EMS responsiveness to the labor market is deemed Substantial since the only indicator to adequately measure the achievement of this part of the PDO (introduced at the ICR stage) was rated as Substantial. EMS improved its responsiveness to the labor market mainly by enhancing the relevance of education (Policy Area 2 supported under DPL 1 and 2). Correlation between the RIEMS implementation index and labor market indicators (indicator added at the ICR) Overall Rating: Substantial (DPL 1: Substantial; DPL 2: Substantial; DPL 3: N/A) 53. As mentioned earlier, education relevance and EMS responsiveness to the labor market were hard to measure. The indicator set to measure the relevance of EMS (unemployment rate ratio of upper secondary graduates ages 18-21 over the unemployment rate of those upper secondary graduates with ages 25-30) could not be used because variations might be driven by the differential impact of macroeconomic conditions on different cohorts. Therefore, this ICR adds a new indicator that attempts to capture the relationship between EMS relevance and its responsiveness to the labor market. The indicator was constructed by Székely (March 2014) 8 and correlates the degree to which the RIEMS has been implemented with labor market indicators for EMS graduates. This indicator shows positive and statistically significant differences on the probability of being employed among graduates from schools (especially for federal schools) with higher levels of EMS implementation. Consistently, in schools with a higher implementation of RIEMS, there is a lower probability of being a NiNi. 8 This indicator was created for the Evaluation of the Inter-American Development Bank (IDB) Project, Component 1. Skills-Based Human Resources Training Program (Programa de Formación de Recursos Humanos Basada en Competencias, PROFORHCOM) in 2014. The author shared the relevant information from the original document with the Bank team on March 28, 2016. 22 54. The Series helped improve employment rates of EMS graduates by supporting the aspects of the RIEMS that made EMS more relevant. In particular, the development and introduction of the CBC was crucial to better respond to employer demands. As empirical evidence shows, what employers value most in Mexico are: teamwork, communication, problem solving, innovation, and conflict resolution (Villasenor 2013 in Cunningham and Villasenor 2016). In other words, there is a greater labor-market demand for socio-emotional and higher- order cognitive skills than for basic cognitive or technical skills. Supporting the shift from a content curriculum that priced memorization and was more tuned to basic cognitive skills to a CBC valuing acquisition of higher-order skills was necessary to better respond to the labor market requirements. The CBC provided students with a broad spectrum of skills through three sets of competencies: generic (comprising socio-emotional and citizenship skills), subject- specific, and professional competencies (skills required in specific occupations). Supporting the creation of a National Committee to create National Competence Standards for technical programs and the approval of 85 technical programs covering, among others, the fields of agriculture, services, administration and information, also helped to better respond to the needs in the economy. 55. In addition, the Bank supported CONSTRUYE-T, the national youth development program to foster socio-emotional skills and improve school climate to reduce risky behavior of EMS students. In particular, the Bank team supported the implementation of policies fostering socio-emotional earning by organizing several knowledge exchange experiences with the Government and providing technical assistance in the design of an online training course on Positive Behavioral Interventions and Support (PBIS) for school directors that, among others, incorporated piloting of a School Climate Survey. 9 It is expected that the actions to foster students’ socio-emotional skills have improved those skills in graduates and, therefore, allowed them to find jobs more easily. However, there is no empirical evidence to confirm this has been the case. Improve quality of EMS (Revised PDO). Overall Rating: Substantial 56. The improved quality of EMS is deemed Substantial since one out of the 5 indicators was rated as High, 2 were rated as Substantial and 2 were rated as Modest. The quality of EMS improved by enhancing the evaluation and accreditation systems and teacher quality (actions supported under the DPL Series). Percentage of students scoring in the categories good or excellent in the ENLACE assessment. Overall Rating: Substantial (DPL1: High, DPL2: Substantial, DPL3: Modest) 57. Students’ academic achievement increased considerably in Mathematics, from 15 percent in 2007-08 to 39.3 percent in 2013-14, significantly exceeding the 18 percent original target and even exceeding the 35 and 37 percent revised targets for DPLs 2 and 3, respectively. This is even 9 The training course was developed by the Bank with the support of socio-emotional learning specialists from the University of Virginia and feedback from SEMS. It was comprised of 17 interactive modules to sensitize Head Teachers on the importance of a positive school climate and provide them with a basic methodology to measure and improve it. 23 more remarkable considering that the higher enrollment rate and lower dropout should have led to a selection of weaker students into EMS. In contrast, Spanish (language and communication/reading comprehension) ENLACE results decreased from 52.3 in 2008 to 44.7 in 2013-14. The mixed results in Mathematics and Spanish do not allow a full diagnostic of students’ academic performance through the DPL series. ENLACE is a reliable examination, but experienced some structural changes10 in 2011 that might have compromised the comparability of the results. In 2015, the assessment was further revised to be aligned with the new national standardized test, PLANEA, introduced by INEE in the school year 2014-15 for EMS.11 58. This ICR complements ENLACE results with two robust international assessments to provide more information on students’ academic achievement in Mexico: the PISA and the Third Regional Comparative and Explanatory Study (Tercer Estudio Regional Comparativo y Explicativo, TERCE). As in ENLACE, the PISA results for 15 year old students (those expected to be in the first year of EMS) improved considerably in Mathematics, from 385 points in 2003 to 413 points in 2012,12 where Mexico achieved the third highest improvement in all OECD countries. This result suggests that the pool of potential applicants to upper secondary might have improved over time. PISA’s disaggregated data for Mexico shows that only 4 percent of student attained the highest results (within levels 4 to 6), 41 percent achieved intermediate results (levels 2 and 3), and the majority, 55 percent, obtained poor results (below level 2). In Reading, as in ENLACE, results were modest and practically stagnant from 422 in 2000 to 400 in 2003, 425 in 2009, and 424 in 2012. The disaggregated results for PISA Reading 2012 still show that a small amount of students, 5 percent, attained the highest levels, 54 percent were in the intermediate levels, and 41 percent were below level 2. In the TERCE, a regional assessment for students in the third and the sixth grades or primary schooling, the trends in Mathematics and Spanish are more similar than those in PISA. For instance, Mexican sixth graders obtained better results in Mathematics (768) than in Reading (735), with 53.1 and 57.6 percent of students within the lower marks (levels 1 and 2) and 46.9 and 42.5 percent achieving higher ratings (levels 3 and 4), respectively. In TERCE, as opposed to PISA, Mexico fared considerably better than the Latin American average of 700. 13 Mexican students’ academic achievement, as measured by these tests (PERCE and SERCE), has also showed improvements through the years for both Mathematics and Spanish. 59. In sum, students’ academic achievement in EMS appears to have improved, particularly in Mathematics, but stagnated or even worsened in Spanish. 14 This trend is consistent for primary, lower secondary, and upper secondary students. Education reform and policies take time to impact students’ academic achievement, and so the years in which the DPL Series took 10 The main changes consisted of adjusting the test to the MCC to evaluate students’ reading competencies instead of reading content. 11 Although PLANEA has a different rating scale than ENLACE, the latter was rescaled for comparability purposes. The rescaled results for ENLACE 2013-14 are 19.3 and 33.1 percent of students scoring good or excellent in Mathematics and Spanish, respectively. PLANEA results for Mathematics and Spanish in 2014-15 were of 18.8 and 36 percent of students scoring in the categories good or excellent, respectively. These results show that in the last year, students had a slight improvement in Spanish and maintained their scores in Mathematics. 12 The PISA average for OECD countries is 500 points and the standard deviation is of 100 points. 13 The mean of the scale was fixed at 700 points, and the standard deviation at 100 points, corresponding to the mean and standard deviation of the 15 Latin American countries analyzed. 14 Although if the last academic year PLANEA results are taken into account, scores in Spanish improve slightly. 24 place might have not been enough to reverse the negative trend observed since basic education. A longer time span is needed to measure the real impact of the EMS Reform on students’ academic achievement. Nevertheless, a brief analysis of the international evidence linking the reforms supported by the DPL Series and improvements in the quality of education is presented in the following two paragraphs. 60. Evaluation and Accreditation. The DPL Series supported policies related to establishing a National Evaluation System for EMS, including the administration and dissemination of sample-based and census-based standardized assessments for students (EXCALE and ENLACE) and the creation and operation of COPEEMS, a civil association in charge of school accreditation. As evidence shows, the process of gathering and evaluating information on what stakeholders (students, teachers, etc.) know and are able to do is crucial for making informed decisions to improve the educational process (Clarke 2012, OECD 2012, OECD 2013). In particular, a sound and sustainable student assessment system is essential for improving curricula and focusing on students’ weaknesses (Clarke, 2012). School accreditation by COPEEMS put in place strong quality assurance mechanisms (such as school leadership and teaching quality, school plans, group size, adequate infrastructure and equipment, and student services) crucial for improving the quality of education in the long run (McEwan 2013, OECD 2012, OECD 2013). 61. The Series also supported efforts to better understand the performance of the entire upper secondary education system by supporting INEE’s autonomy. Before the Reform, all evaluation activities were the responsibility of SEP, creating a potential conflict of interest and reducing the credibility of results. The establishment of INEE as a fully autonomous entity has helped strengthen the quality and utility of the information produced by the assessment system, which should have a positive impact in the quality of education (Clarke 2012). Percentage of students attending schools in very highly marginalized localities scoring good or excellent on the ENLACE assessment in Spanish and Mathematics. Overall Rating: Modest (DPL1: N/A, DPL2: N/A, DPL3: Modest) 62. ENLACE Mathematics results for students living in very highly marginalized localities increased considerably from 6 percent of students attaining good or excellent results in 2008 to 23 percent in 2013-14. The set target of 29 percent was achieved by 79 percent. ENLACE Spanish results for the same group of students stagnated from 22.6 percent in 2008 to 20.31 percent in 2013-14, and was 10 percentage points behind the target of 31 percent. As in the case of the aggregate ENLACE results, it is difficult to draw any conclusions on disadvantaged students’ learning outcomes given the mixed results (see explanation on ENLACE possible pitfalls in Policy Area 2). Therefore, this ICR also analyzes PISA data in terms of student family and socioeconomic background. In PISA, the gap between rich and poor Mexican students is the smallest among OECD countries, at only 38 points compared to 78 for the OECD average. This is not necessarily a big achievement since results are quite low for the entire sample backgrounds. In addition, the percentage of resilient students (those in socioeconomic disadvantages that despite their context achieve the best marks) is lower for Mexico (3.8 percent) than for the OECD average (6.5 percent). However, Mexico has been successful in reducing the variations linked to socioeconomic factors from 17 percent in 2003 to 10 percent in 2012. 25 Index of entry into the SNB (IEISNB) Overall Rating: High (DPL1: NA, DPL2: NA, DPL3: High) 63. As previously mentioned, the IEISNB increased from 0 in 2008 to 37.18 percent in 2015, exceeding the 20 percent target. The IEISNB represents the number of students that have benefited from important quality gains through the SNB, one of the most relevant aspects of the Reform. In particular, the adjustment of the curriculum to generic, subject area, and professional competencies and the development of minimum learning standards for upper secondary graduates (as part of the CBC) was crucial to improve the quality of education at the upper secondary level. Before the reforms introduced by SEMS the curriculum was content-based and focused on memorization as opposed to fostering meaningful learning and problem solving skills. Improved teaching quality in upper secondary education, due to massive in-service training and certification of teacher competences, as measured by the baseline and final results of the planned Impact Evaluation. Overall Rating: Modest (DPL 1: Modest; DPL 2: Modest; DPL3: NA) 64. The PROFORDEMS impact evaluation was not conducted. However, as empirical evidence increasingly shows, teacher effectiveness is one of the most important school determinants of student learning (Leu 2004, Hanushek 2002, 2005, 2009, Hanushek and Rivkin, 2005, Craig 2010). Countries that have invested in recruiting, grooming and motivating teachers show positive learning outcomes (Bruns et al 2014). Effective in-service training based on observed classroom practices can contribute to teacher quality. The DPL Programmatic Series supported the development and implementation of PROFORDEMS, the first nationwide EMS teacher-training program, and the regulations to officially recognize alternative teacher training programs. In addition, the Bank’s technical collaboration with SEMS allowed redesigning and improving PROFORDEMS, and more broadly teacher professional development, to align it with international best practices. These policy actions provided EMS in-service teachers with quality training opportunities while giving them the flexibility of subscribing to other training options that better addressed their particular training needs. Although PROFORDEMS faced some challenges (as explained in the implementation section), it was able to train a fair number of teachers (116,892) and, more importantly, to certify a few (38,504). 15 The low number of certified teachers is explained by the high standards of the certification process. 16 Since the quality seal of certified PROFORDEMS teachers is quite high, it is expected that when more teachers obtain their certification, the academic achievement of students will improve. SEMS is concerned about those teachers that have not been able to achieve certification and is considering different pathways to help them do so (i.e. retraining). 65. In addition, the Government of Mexico passed a new law establishing the professionalization of the teaching career (Servicio Profesional Docente, SPD). The SPD requires that all teachers undertake regular evaluations and, based on their performance, receive 15 The total number of EMS public teachers is 193,482. 16 To obtain a certificate, teachers need to put together an original and innovative portfolio to improve student learning within the RIEMS framework. The portfolio is thoroughly assessed and graded by an Independent Evaluation Committee (IEC). In addition, participants undertake an oral examination (conducted by the IEC) to assess the degree to which they have obtained the necessary skills for teaching under the competency-based scheme introduced by the RIEMS. Teachers need to obtain a satisfactory grade in both aspects to obtain their certification. 26 tailored professional development and training. About 63,000 SEMS teachers have already been evaluated under the SPD scheme. One in every three SEMS teachers has been evaluated and will receive tailored training. As empirical evidence shows, a coherent teacher evaluation system is also crucial to improve teacher quality (Burns et al 2014). 66. Overall, there is evidence linking teacher quality, as well as robust accreditation and assessment systems, to learning outcomes (OECD, 2012). Therefore, the support to policies to strengthen student assessments, teacher training, and school accreditation are expected to contribute to sustained improvements in the skills of EMS graduates in the medium term, and to their wages and employability when they eventually enter the labor market. 3.3 Justification of Overall Outcome Rating Satisfactory 67. Relevance ratings for both the original and revised Program Development Objectives are High. The relevance rating for design and implementation is Modest. Efficacy ratings are Substantial for the whole DPL Series given that: (i) improved internal efficiency of upper secondary education is rated as High, (ii) upper secondary education responsiveness to the labor market is rated as Substantial, and (iii) improved quality of education is rated as Substantial (with i and iii weighting 40 percent each and ii weighting 20 percent since it was taken out for DPL 3). For a summary of the ratings for each indicator, please refer to Table 5. The combination of Relevance (Substantial) and Efficacy (Substantial) yields an overall Satisfactory rating for the overall outcome rating for the Programmatic DPL Series. 3.4 Overarching Themes, Other Outcomes and Impacts (a) Poverty Impacts, Gender Aspects, and Social Development 68. The Series contributed to the Bank’s twin goals of eliminating extreme poverty and boosting shared prosperity by helping improve retention rates among EMS students, particularly those from the bottom 40 percent on the income distribution. Even when controlling for a set of socioeconomic and demographic factors, finishing EMS has a positive association with the possibility of finding a job and with higher earnings (Székely, 2013). 69. At the macro level, a society with more human capital–measured, for instance, by years of schooling or learning outcomes—enjoys higher rates of long-term economic growth. Conversely, from a micro perspective, more human capital translates into higher personal or household incomes, less poverty and, if the increases in human capital take place among disadvantaged households, less income inequality. 70. There is no evidence that the RIEMS had a differentiated effect by gender. However, there is some suggestive evidence that the increased curricular flexibility might have helped girls more than boys. For instance, Avitabile and de Hoyos (2015) find that in response to the information on the labor market returns to upper secondary completion, girls’ results in standardized tests increased more than boys’ and this was partly due to girls being more likely to choose sub-tracks with higher expected labor market returns. These results are consistent with the hypothesis that the increased flexibility introduced by the RIEMS might benefit girls more 27 than boys since girls’ school choices were more likely to be constrained by social norms and information gaps. (b) Institutional Change/Strengthening 71. Through support of the RIEMS, the DPL Series played a key role in strengthening SEMS’ institutional capacity and contributing to developing strong coordination and communication among the federal and the state education authorities. The DPL Series also assisted in establishing the legal and regulatory framework for the EMS system and the SNB in particular, enhancing the institutional capacity of federal, autonomous, state and particular EMS providers. In addition, the Bank team worked in coordination with the Government to produce important pieces of analytical work that have been key in the implementation of the RIEMS. (c) Other Unintended Outcomes and Impacts 72. The RIEMS policies for changing from a traditional content-based curriculum to a competency- based approach permeated to the lower secondary education system, which is now undertaking a similar reform process. In addition, the DPL Series provided a space for engaging with the Mexican Government, and particularly SEMS, in a solid technical cooperation beyond the DPL that has created a wide range of analytical products, including just-in-time analysis on priority topics for SEMS, such as scholarships, teacher training and policies and strategies for reducing dropouts, among others. 3.5 Summary of Findings of Beneficiary Survey and/or Stakeholder Workshops N/A 4. Assessment of Risk to Development Outcome Ratings: Negligible to Low 73. The current administration is particularly keen to continue strengthening the policies to improve EMS internal efficiency and quality supported by this DPL Series. SEMS has been the Under Ministry with the most staffing and management stability in recent years. Even after the elections, the reform is likely to continue since the implementation of the Reform has been strongly supported by all political parties and was not impacted by the change of ruling party and administration in December 2012. The current fiscal situation is putting pressure on the overall SEP budget, which can hamper the sustainable improvements in EMS. However, it is expected that SEMS will receive an adequate budget provision from the Ministry of Finance and Public Credit (Secretaría de Hacienda y Crédito Público, SHCP) to continue implementing the policy changes at least until the next administration takes office in 2018. 5. Assessment of Bank and Borrower Performance 5.1 Bank Performance (a) Bank Performance in Ensuring Quality at Entry Ratings: Moderately Unsatisfactory 74. The quality at entry is deemed Moderately Unsatisfactory because the DPL Series design had significant shortcomings. The Bank could have prevented the inclusion of vague indicators without clear baselines and targets, particularly in the first operation. In addition, the PRF was 28 only aligned with the official Government indicators until the third operation. The design could have also been less ambitious in terms of the gains in learning outcomes, since reforms take an average of 10 years to render significant gains in this area. On the other hand, the DPL Series provided support to relevant and strategic reforms in EMS for the Mexican context. The Program design also took into account poverty and social development aspects, putting greater emphasis on vulnerable students. Finally, the Bank was flexible in adjusting policy actions to reflect the Reform’s accelerated pace, demonstrating flexibility in revising triggers and prior actions to recognize areas where progress was made faster than initially planned and in integrating new areas that could benefit from inclusion in the program. Analytical work was also set up throughout the program to inform later policy decisions. (b) Quality of Supervision Ratings: Satisfactory 75. The team composition and level, as well as the team combination of analytical and operational skills throughout the program (and particularly for DPL 3) ensured an adequate supervision effort for the DPL Series. The Bank conducted regular implementation support missions to maintain positive dialogue with the Government and SEMS on the policies and actions that were needed to ensure full implementation of the Reform. Policy and implementation challenges were adequately addressed by providing just-in-time technical support. SEMS emphasized that the Bank’s technical support to develop practical and relevant products that could be put in practice fairly easily was extremely useful. A minor shortcoming was the lack of regular reviews of progress on indicators from previous operations. Progress on indicators removed for DPL 1 and 2 had to be reviewed at the ICR stage. (c) Justification of Rating for Overall Bank Performance Ratings: Moderately Satisfactory 76. Overall Bank performance is considered Moderately Satisfactory, since Quality at Entry was Moderately Unsatisfactory (with significant shortcomings) and Quality of Supervision was Satisfactory with minor shortcomings. 5.2 Borrower Performance (a) Government Performance Ratings: Satisfactory 77. The Government showed ownership of the Reforms and commitment to achieving the DPL Series development objectives. SHCP provided sufficient budgets to SEP and SEMS throughout the DPL Series, and ensured that the loans were disbursed on schedule. SHCP continuously followed up with the Bank, the implementing agency, and NAFIN to ensure that the Program worked as expected, including overseeing the implementing agency’s capacity and addressing challenges to the successful achievement of the Program objectives. Despite a change in the administration (and the ruling party) in December 2012, the Government ensured continuity of the RIEMS. NAFIN also performed well in its role of fiduciary agent of the Government for the loans, ensuring that Financial Management (FM), governance and covenants were complied with and followed. 29 (b) Implementing Agency Performance Ratings: Satisfactory 78. SEP, through SEMS, executed the Reform policies and ensured that the actions supported by the DPL Series were executed in a timely manner. SEMS continuously showed its strong commitment to keeping the program on track and seeking interim solutions that would enable achievement of the targets set. SEMS ensured that policies supported by the Bank were implemented as planned, particularly in the federal system that is under its direct administration. SEMS efforts to coordinate actions and information flows across several actors and levels of government was remarkable. For instance, SEMS effectively coordinated state authorities and other service providers to ensure that the new EMS education policies were implemented nationwide and students in all states benefited from them. The implementing agency performance is rated Satisfactory since there was only one minor shortcoming related to the implementation of the PROFORDEMS impact evaluation. (c) Justification of Rating for Overall Borrower Performance Ratings: Satisfactory 6. Lessons Learned 79. The DPL Programmatic Series was the most suitable Bank lending instrument to provide general budgetary support while accompanying a complex reform. The DPL Series allowed adjustment to the pace in which policies were implemented and was also helpful for policy continuity despite changes in the administration and ruling party in December 2012. 80. In countries with strong technical capacity such as Mexico, DPLs should focus on those elements of the reforms that will benefit most from Bank engagement. In this operation, the Bank tried to respond to the client’s technical demand while being selective in choosing in which areas to support the Government based on team composition and availability of resources. 81. The complementary technical assistance provided by the Bank in parallel to the DPL Series helped inform the Government reform and achieve the DPL Series objectives. The technical collaboration between the Bank and the Government helped shape the policies introduced by the EMS reforms based on evidence and international best practices. For instance, the impact evaluation of the scholarship program provided useful information to better target scholarships to the poor. In addition, the provision of just-in-time technical assistance helped overcome the main obstacles faced during implementation of the reform. For instance, the outcomes of the workshop on international best teaching practices were key to improve the design of PROFORDEMS, the teacher training program introduced by the RIEMS. 82. It is crucial that indicators are directly linked to the policies introduced by the reform supported under DPL Programs and not linked to the differential effect of macroeconomic conditions on different cohorts; otherwise, it is complicated to measure the relationship between the Bank’s intervention and outcomes. In this DPL Programmatic Series, the original PRF had three indicators with this shortcoming, preventing measurement of improvements in the responsiveness of education to the labor market and, in some cases, on equity aspects. For instance, the selection of unemployment rates as an indicator was not a wise choice for measuring the achievement of the PDO since they do not solely depend on improving internal efficiency and relevance of education. 30 83. Designing indicators based on planned impact evaluations is not recommended unless there is certainty that the evaluations are going to be undertaken. The original PRF considered an indicator whose baseline and target were supposed to be set with data from an impact evaluation of a teacher-training program. However, the impact evaluation was not conducted, preventing data collection on teacher quality. 84. Aligning DPL Program indicators to official indicators helps decrease the M&E burden of the Program for the Government and allows the team to obtain more reliable and timely indicators. When the Bank team aligned its indicators to the official indicators in DPL 3, M&E was greatly improved because the implementing agency had more pressure to periodically collect and report data on official indicators. 85. The main lessons learned from PROFORDEMS, the teacher training program introduced by the Reform are: (i) the design and implementation of national in-service teacher training programs to change classroom practices need to be practical and flexible enough to respond to the wide variety of teachers’ needs; (ii) clearly defined leadership at the national level is crucial for program adjustments and M&E; (iii) adequate support is required not only for core program elements, but also for logistics and materials; (iv) and, since teacher professional development usually entails high investments, robust certification processes and, if viable, an impact evaluation should be built into the program design from the outset. 86. Introducing school accreditation processes requires taking into account schools’ starting conditions and their means to improve the education services they provide. Gaining full accreditation to enter the SNB proved to be very challenging for most schools in Mexico. Hardly any schools complied with the full accreditation criteria and many lacked the necessary resources to complete the accreditation process. To tackle these issues, as in the Mexican case, policies might consider allowing schools to gain temporary accreditation if they comply with a set of minimum criteria of quality and commit to specific quality improvements in a certain period of time. 7. Comments on Issues Raised by Borrower/Implementing Agencies/Partners (a) Borrower/Implementing agencies (b) Cofinanciers N/A (c) Other partners and stakeholders N/A 31 Annex 1. Bank Lending and Implementation Support/Supervision Processes (a) Task Team members P112262 - Upper Secondary Education (MUSE) Development Policy Loan Responsibility/ Names Title Unit Specialty Lending LCSHE - Cristian Aedo Senior Education Specialist Task Team Leader HIS Co-Task Team Erik A. Bloom Senior Economist IEGPS Leader Mariangeles Sabella Senior Counsel LEGEN CTRFC - Jose C. Janeiro Senior Finance Officer HIS Xiomara A. Morel Lead Financial Management Spec GGODR LCSHE - Tatiana Proskuryakova Senior Operations Officer HIS Christoph Kurowski Sector Leader LCC1C HDNED - Javier Luque Senior Education Economist HIS Maria E. Colchao Senior Program Assistant GHNDR LCSHE - Isy Faingold Vigil E T Consultant HIS LCSHE - Carla Cutolo Program Assistant HIS Antonella Novali Program Assistant GEDDR Supervision LCSHE - Cristian Aedo Senior Education Specialist HIS Erik A. Bloom Senior Economist IEGPS Maria E. Colchao Senior Program Assistant GHNDR LCSHE - Isy Faingold Vigil E T Consultant HIS Antonella Novali Program Assistant GEDDR P126297 - Second Upper Secondary Education Development Policy Loan Responsibility/ Names Title Unit Specialty Lending Erik A. Bloom Senior Economist IEGPS Task Team Leader Co-Task Team Peter Anthony Holland Senior Education Specialist GEDDR Leader Christoph Kurowski Sector Leader LCC1C Fabiola Altimari Senior Counsel LEGLA LCSDE- Janet K. Entwistle Senior Operations Officer HIS 32 Wendy Cunningham Sector Leader LCC1C LCSFM- Xiomara A. Morel HIS LCSFM- Juan Carlos Serrano Machorro HIS LCSHE- Abril Alicia Ibarra Castaneda Consultant HIS Antonella Novali Program Assistant GEDDR Supervision Erik A. Bloom Senior Economist IEGPS Peter Anthony Holland Senior Education Specialist GEDDR LCSHE- Rafael E. de Hoyos Navarro Economist HIS Fabiola Altimari Senior Counsel LEGLA LCSDE- Janet K. Entwistle Senior Operations Officer HIS Wendy Cunningham Sector Leader LCC1C LCSFM- Xiomara A. Morel HIS LCSFM- Juan Carlos Serrano Machorro HIS LCSHE- Abril Alicia Ibarra Castaneda Consultant HIS Antonella Novali Program Assistant GEDDR P147244 - Third Upper Secondary Education Development Policy Loan Responsibility/ Names Title Unit Specialty Lending Peter Anthony Holland Senior Education Specialist GEDDR Task Team Leader Co-Task Team Rafael E. de Hoyos Navarro Senior Economist GEDDR Leader Ciro Avitabile Senior Economist GEDDR Fabiola Altimari Senior Counsel LEGLA LCSDE- Janet K. Entwistle Senior Operations Officer HIS Wendy Cunningham Sector Leader LCC1C CTRLN- Victor Ordonez Senior Finance Officer HIS LCSFM- Xiomara A. Morel HIS LCSFM- Juan Carlos Serrano Machorro HIS LCSHE- Abril Alicia Ibarra Castaneda ET Consultant HIS Antonella Novali Program Assistant GEDDR Supervision 33 Peter Anthony Holland Senior Education Specialist GEDDR Rafael E. de Hoyos Navarro Senior Economist GEDDR Ciro Avitabile Senior Economist GEDDR Fabiola Altimari Senior Counsel LEGLA LCSDE- Janet K. Entwistle Senior Operations Officer HIS Wendy Cunningham Sector Leader LCC1C CTRLN- Victor Ordonez Senior Finance Officer HIS LCSFM- Xiomara A. Morel HIS LCSFM- Juan Carlos Serrano Machorro HIS LCSHE- Abril Alicia Ibarra Castaneda ET Consultant HIS Antonella Novali Program Assistant GEDDR (b) Staff Time and Cost Staff Time and Cost (Bank Budget Only) USD Thousands (including Stage of Project Cycle No. of staff weeks labor, travel and consultant costs) P112262 - Upper Secondary Education (MUSE) Development Policy Loan Lending FY09 3.29 13,728.60 FY10 47.75 268,998.50 Total: 51.04 282,727.10 Supervision/ICR FY11 19.58 64,849.91 FY12 7.13 16,385.63 Total: 26.71 81,235.54 Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) P126297 - Second Upper Secondary Education Development Policy Loan Lending FY11 6.55 50,585.27 FY12 23.43 146,123.18 Total: 29.98 196,708.45 Supervision/ICR FY12 0.00 4,891.53 FY13 16.10 93,345.04 34 FY14 1.75 8,203.30 Total: 17.85 106,439.87 Staff Time and Cost (Bank Budget Only) Stage of Project Cycle USD Thousands (including No. of staff weeks travel and consultant costs) P147244 - Third Upper Secondary Education Development Policy Loan Lending FY14 21.89 110,571.89 Total: 21.89 110,571.89 Supervision/ICR FY14 21.58 101,552.32 FY15 23.63 101,483.98 FY16 6.14 44,717.86 Total: 51.35 247,754.16 35 Annex 2. Beneficiary Survey Results N/A 36 Annex 3. Stakeholder Workshop Report and Results N/A 37 Annex 4. Summary of Borrower's ICR INTRODUCTION The last decade experienced the convergence of three major reforms to improve the internal efficiency of Upper Secondary Education (EMS) in Mexico, as well as the relevance in this educational level. First, the Integral Upper Secondary Education Reform (RIEMS) of 2008 promoted teaching based on competencies and the regulation and integration of more than 30 educational subsystems through the establishment of a Common Curricular Framework (MCC) and the National Upper Secondary Education System (SNB). A second reform making EMS mandatory and universal was introduced in 2012. Finally, in 2013 a third reform sought to ensure quality of compulsory EMS. Thus, the near-simultaneity and convergence of these three reforms has promoted actions that seek to transform EMS to provide a more inclusive, relevant and quality education. In this context, the implementation of reforms has been supported by assistance from the World Bank since 2010. Through a Series of Development Policy Loans (DPLs) for Upper Secondary Education (7887-MX, 8132-US and 8314-MX), this International Financing Agency has provided technical support in order to assist the Government's efforts to improve the performance of the Mexican education system with special emphasis on three public policy areas: Increase the Flexibility of Upper Secondary Education; Improve the Quality of Upper Secondary Education; and Reduce the Opportunity Cost of Upper Secondary Education. This document reports the findings and conclusions of the implementation of the Series of DPLs for EMS. It analyzes the performance of the Federal Government within the framework of the three loans disbursed since 2010, as well as the performance of the World Bank, the execution of the operation, and the results that were obtained. OPERATION SUPPORTED BY A SERIES OF DPLS FOR EMS With the objective of supporting the Government’s actions in the implementation of the RIEMS to improve the internal efficiency of EMS, its response to the needs of the labor market, and its quality, the World Bank provided technical assistance to the Ministry of Public Education (SEP) through the DPL Series. As mentioned, the operation of the Series supported activities in three areas covering major limitations of EMS within the consolidation process of the RIEMS and the reform making EMS mandatory: increasing the flexibility of EMS, improving its quality and relevance, and reducing the opportunity cost of EMS. This Series included three lending operations. The first one backed the creation of the SNB, supported the creation of PROFORDEMS, and the consolidation of the scholarship program. The second promoted the continuity of the RIEMS. The third supported the constitutional amendment to make EMS mandatory. The objectives of the operations supported by the DPLs were real, attainable and useful because it supported the efforts that the Federal Government undertook as part of two transformation processes in the upper secondary level. The first process was the RIEMS in 2008, which sought to endow EMS with an identity, an order, an articulation, and a relevance amid a context of diverse modalities and subsystems, through four axes: the construction of a Common Curricular 38 Framework; the definition and recognition of the options offered by the EMS system; the professionalization of educational services; and the certification of the SNB. It must be noted that before the RIEMS, EMS was composed by a number of subsystems that not only operated independently, without any correlation to an articulated overview and with insufficient communication between them, but also suffered from low graduation rates, poor quality and, in some cases, a dubious relevance to the labor markets. The challenge was finding common goals among these subsystems to maximize their scope and equip students with the necessary knowledge and skills to continue their successful development in university and/or in their workplace. The second process was the constitutional amendment to make EMS mandatory and to aim for universalization of EMS by the 2021-22 school year. Along with the present Government program to achieve 80% coverage by the end of its 6-year term and the recent reforms that mandate a Mexico with quality education, this justifies the relevance and importance of the objectives of the operations. In terms of design, the spheres of policy mentioned above, which were the base to define the actions supported by each DPL, are relevant because they group the challenges of EMS that the Government has sought to address with the RIEMS and with mandatory and EMS. On one hand, the flexibility dimension responded to the need to facilitate the movement of students between subsystems in a context of diversity. Before the RIEMS, each program and subsystem had its own approach, which meant students had no possibility of transferring to another subsystem or program without having to start their studies over, which encouraged school dropouts. On the other hand, the dimension of quality and relevance of EMS framed the need to define an evaluation system that could grade the quality of learning in the classroom and to promote the quality of teaching practices and of managers’ leadership. This considering that nothing determines the performance of students more than having a teacher who is able to implement practices that have a positive and significant effect on his/her students’ learning experience. Finally, the dimension of reducing the opportunity cost covers one of the main challenges of the education policy agenda: ensuring equality and inclusion. It is known that huge gaps persist in EMS coverage in terms of income. For the population in the two lowest income quintiles, for example, coverage is significantly reduced compared to the highest. This, coupled with the fact that socioeconomic difficulties are one of the causes of higher dropout rates, makes this dimension a particularly interesting one for the construction of a Mexico with quality education for all. World Bank support in the implementation of the operations was fundamental. First, it is important to point out that the Federal Government directly or indirectly finances an important percentage of public education expenditure. In the upper secondary level, it finances six centralized subsystems under the jurisdiction of the SEP, through its Under Secretariat of Upper Secondary Education (SEMS). At the state level, it co-finances with state governments other decentralized organizations such as CECYTE and the States’ Colegios de Bachilleres. Furthermore, the implementation of the RIEMS and the consequent reforms that make EMS mandatory represent a large budgetary effort to finance public policy interventions aimed at giving the needed coherence, relevance and quality to this educational level throughout the country. Second, the World Bank’s technical assistance was important because it provided information on international best practices and strategies on public policy areas that align with the priorities of SEP and helped shape the interventions in EMS. Such is the case of the 39 Movimiento contra el Abandono and the teacher training program PROFORDEMS, which are detailed in the following sections. SEP PERFORMANCE WITHIN THE FRAMEWORK OF THE DPL SERIES Based on the major reforms that have sought to transform the upper secondary level, SEP, through SEMS and with Bank support in strategic areas, has made great efforts and implemented public policy interventions to fulfill the constitutional mandate. The following describes the interventions developed in recent years to transform EMS (Bachillerato) in accordance with the RIEMS and the subsequent reforms that make this educational level mandatory to reach universality by 2021. The essential purpose is to describe the interventions promoted under the Series of DPLs and identify both the progress and the remaining road to give EMS the coherence, relevance and quality that it has not enjoyed historically. 1.1 Interventions promoted by the Federal Government to expand educational opportunities Within the framework of the "Decree declaring the reform of paragraph one; subsection c) of section II and section V of article 3, and section I of article 31 of the Constitution of the United Mexican States," compulsory EMS consists of the States’ duty to provide a place to pursue this educational level to anyone having reached the typical age and completing Basic Education. In 2012-13, about 5.4 million out of 6.7 million youth in the typical age to attend EMS (between 15 and 17 years old), had a middle school certificate. Meanwhile, 4.4 million students were enrolled in EMS. The gross coverage rate of EMS was thus of 65.9%; today the gross coverage rate is equivalent to almost four out of five youth within this age range (78.6%). The challenge is to achieve 80% coverage by 2018. To increase the enrollment in EMS, SEMS promotes the following four public policy interventions: A) Increase the number of Telebachilleratos Comunitarios (TBC) as a strategic priority to attend communities with less than 2,500 inhabitants. The Population and Housing Census shows that in the 2010-11 school year, the gross coverage in urban areas was of 70.4%, while in rural areas it barely recorded 43.7% of the population between 15 and 17 years old. At the beginning of the 2013-14 school year, Mexico had 10,276 middle secondary schools (28% of the country’s total) without EMS services in a 5km radius, 97% of them located in rural areas with less than 2,500 inhabitants. To address this, SEMS, in coordination with State Governments, began a pilot program in 2013 with a total of 253 TBC. This public policy intervention aims to build on existing infrastructure and the social and cultural capital created by rural middle schools to bring EMS to rural areas, especially those that are isolated and dispersed. In 2014, SEMS authorized the creation of 1,490 TBC across the country, so that in 2014-15 1,743 services operated in middle school facilities with no EMS services in a 5km radius (and exceptionally in locations with those education services, but with an unmet demand). Currently, about 3,000 TBC are operating throughout the national territory, benefiting almost 100,000 students. 40 B) Promote expansion of the Fondo Concursable de Inversión en Infraestructura para la Educación Media Superior ( FCIIEMS). In Mexico, the allocation of public resources to educational institutions through competitive funds has gone from an innovation in education funding to an established practice. The implementation of competitive funds has had a positive impact on both the organizational culture of the institutions and the performance indicators. In 2013, SEMS created the Competitive Fund for Investment in Infrastructure Fund (Fondo Concursable de Inversión en Infraestructura, FCIIEMS) to meet the needs of construction, expansion, rehabilitation and equipment provision of educational facilities in EMS. The objective of this mechanism was to accelerate the achievement of the coverage goal and promote greater transparency and accountability in the exercise of resources. Unlike its predecessor, the Infrastructure Program for Upper Secondary Education, which operated until 2012 with budget ceilings ex-ante project presentation, the FCIIEMS directs resources towards cost-attractive projects that contribute most to achieve the coverage goal, promote equality, and improve the relevance and quality of EMS. Since 2013, SEP has invested MX$5,200 million in developing 1,868 construction, expansion, rehabilitation, or equipment projects of educational spaces in 1,537 EMS public schools, i.e. almost one in seven public schools at the beginning of the current administration. Figure A4.2. Number of supported projects and allocated budget (in MX$ millions) C) Expand the virtual educational offering through the creation and scaling up of the Online High School (Preparatoria en Línea) "Prepa en Línea" offers an educational alternative for those who, for social, economic, work or other needs, cannot opt for the in-school modality. It is a student-centered, competency-based approach with attractive and interactive educational materials that is characterized by a curricular structure that can be completed in two years and four months. This educational service helps develop a strong sense of belonging and responsibility through the integration of Virtual Learning Communities. Its platform allows students to interact in a Virtual Campus, share experiences, and provide support and feedback. Currently, around 50,000 students are actively enrolled in the Prepa en Línea. It is expected to have an enrollment of over 150,000 students by 2018. D) Modernize the Open High School program (Preparatoria Abierta) 41 The Preparatoria Abierta is a highly flexible program that allows schooling accreditation in an order and rhythm set by the student; it also helps meet the demand in all kinds of communities in rural or urban areas. Its target population includes new graduates from middle schools, students with partial studies at the upper secondary level, and those who cannot continue their in-school education due to restrictions. While the Preparatoria Abierta is an educational modality that has been operating for several decades, SEMS has undertaken its modernization to improve the quality of the service. 1.2 Interventions to promote quality in Upper Secondary Education E) Incorporation to the National Upper Secondary Education System (Sistema Nacional de Bachillerato, SNB) The SNB constitutes a quality standard to be met by EMS establishments. Incorporation into the SNB involves a comprehensive assessment, in accordance with the standards established by its Executive Committee, of learning plans and curricula, student support, teaching, facilities and equipment, management, and the institution as a whole. The SNB’s Executive Committee is composed of 12 members; 6 of which are state educational authorities selected by peers each year. Of the more than 15,000 public and private schools, 1,926 belong to the SNB. In December 2012, this number was just 226. Since then, the proportion of enrollment in this quality standard registry increased from 4.2% in 2012 to over 30% in 2015. Table A4.1. Evolution of the number of schools in the SNB per level 2011 2012 2013 2014 2015 Total 63 226 707 1,437 1,926 Level I 0 0 3 4 7 Level II 3 10 62 89 106 Level III 60 216 631 971 1,128 Level IV 0 0 11 373 685 F) Training and professionalization of teachers The Professional Teaching Service (Servicio Profesional Docente, SPD) is the set of activities and mechanisms established for the incorporation, promotion, recognition, and permanence of human resources in the public education service and the promotion of their continuous training, and it is considered one of the most visible elements of the 2013 reform to provide higher quality education. Its objective is to guarantee the suitability of the knowledge and capabilities of teaching staff and managerial and supervisory personnel. Anyone performing these tasks must meet the personal qualities and professional skills that, within different social and cultural contexts, promote student learning achievements. The first contest to get into the SPD in EMS took place in July 2014 and involved more than 34,000 candidates; just over 11,000 were determined to have the ideal profile and were thus integrated into priority lists for the allocation of teaching positions. Of these, permanent positions were allocated to nearly 5,000 qualified teachers and another 5,000 received temporary 42 positions. The second contest was held in 2015 with similar figures to those of the previous year. Regarding performance evaluation, during the first stage in 2015, over 29,000 teachers from all the subsystems were evaluated (about one in five of the teachers required by law). The second stage will run its course by the end of 2016, with a second group of approximately 20,000 teachers. To ensure that all teachers and administrators are able to respond to the priorities set by the Education Reform and the requirements demanded by the General Law of the Professional Teaching Service (Ley General del Servicio Profesional Docente), SEP, through SEMS, has supported actions to strengthen the skills of teachers and management personnel through two programs: Teacher Training (PROFORDEMS and the set of teaching skills for the upper secondary level) and Managerial Training (Program for Managerial Professionalization and Updating and the Development of Managerial Competencies in EMS). Such models of continuous training and professional development are being reviewed, as they only focus on one area: the development of teaching skills. This approach sought to address the fact that, although almost 95% of teachers in EMS have undergraduate or graduate degrees, these teachers would need to develop new teaching abilities and methods in line with the model of competency-based education established in the MCC for EMS. However, there is evidence to corroborate that this approach has proved insufficient. Not only was the content of this kind of courses insufficient to effectively influence teaching activity in the classroom and strengthen interactions with students, it did not support the updating and expansion of knowledge of their disciplines and other relevant educational aspects. Therefore, SEMS is working on a new training program based on the needs of each teacher according to their situation with regard to the SPD. For beginner teachers, a course on teaching skills in EMS has been launched and, so far, 80% of newly hired teachers for the 2014-15 cycle have been trained. Another course that will be offered to all new teachers focuses on improving their communication skills. This aspect is considered of great importance because teachers require strong abilities to communicate with students and provide feedback on their learning progress. As a complement, SEMS is working on a course for tutors to strengthen the technical and methodological aspects of mentoring delivery to new teachers by their peers. Tailored courses have been planned to address the needs of teachers who took part in the performance evaluation, in accordance with the results of the evaluation. For teachers with the lowest levels of achievement in performance evaluations, two courses have been planned to help those teachers to remain in service, by achieving a better performance level in both their teaching practice and in the evaluation they must submit in a year, according to the Ley General del Servicio Profesional Docente. The first of these courses corresponds to the dimension of teaching skills. This course has been designed to provide teachers with the necessary tools to perform the teaching-learning activities under the competency-based approach that prevails in EMS and within the provisions of the MCC, which will let them identify the real needs of their students and implement relevant methods for the learning and integral development of youth. In parallel, a second group of courses will be offered as part of the regularization offer to improve the performance of teachers in any of the 20 disciplines covered by the first performance evaluation. This offer of discipline-specific courses represents one of the most important innovations to comply with the continuous learning requisite derived from the educational 43 reform because it addresses a key issue in the training of teachers that did not receive an appropriate and systematic response in the past. It is worth mentioning that the work model considers conducting sessions within these discipline-specific courses with in-person assistance, with the purpose of reinforcing the learning experience. This assistance will be provided in more than one hundred sites strategically located throughout the country so that they can be accessed by the participating teachers. To meet the needs of teachers with sufficient achievement in their performance evaluations, an innovative course has been designed to improve their work in the classroom, based on the review of classroom teaching practice videos and their feedback. The videos, as a learning tool for teachers, offer evidence of interactions between teachers and students that improve the quality of education. Additionally, this group of teachers will be able to participate in courses directed toward the "Pedagogical strengthening of the content for each discipline-specific area." In EMS, the subjects of the MCC are grouped in five discipline-specific areas (mathematics, experimental sciences, humanities, social sciences, and communication). These courses focus on giving teachers a greater mastery of specific educational elements for the disciplines they teach by analyzing relevant teaching techniques for each area. A third type of planned courses is aimed at teachers with the highest results in the performance evaluation. In this case, the proposal consists of the integration of several specialization courses in EMS. On one hand, courses with a more in-depth treatment of topics related to the teacher competencies are being considered, such as didactic planning, innovative teaching methods, evaluation of the students’ skill development, consolidation and improvement processes of collegiate teacher work, development of socioemotional skills in youth, methods for the vocational guidance, and mechanisms for building learning communities. Performance evaluations had never been applied to EMS. In these first experiences, the instruments used brought about doubts and questions for many teachers. For this reason, and based on the experiences of various activities that were performed as part of the follow-up with the participating teachers, two courses will be launched. The first will focus on the development of a portfolio of evidence of teaching, which teachers can use to document and discuss their own performance and which will also build evidence in relation to the work they do in the classroom. Additionally, a course on planned teaching approaches with arguments will be offered, which will let teachers register the teaching and pedagogical strategies and elements they use to promote learning among their students, to establish how they organize their educational practice and how they take into account the elements of the context in which they develop their function. These teachers may choose to participate in a course on "Knowledge and teaching competencies for learning," which reinforces the mastering of elements of the competencies-based educational model. Thus, the models described above for the continuous training and professional development of teachers seek to expand the availability of courses so teachers can meet their discipline-specific and pedagogical needs, including on-site training and observation of real-life situations. G) Supported interventions to improve learning results The results from different standardized exams (PLANEA, ENLACE, PISA) indicate that students who attend compulsory education have insufficient levels of achievement in knowledge 44 structuring areas such as communication and mathematics. In the 2015 PLANEA test for EMS students, four out of ten students placed at the lowest level of performance in language and communication and five out of ten did the same in mathematics. To change this situation, an intervention is being implemented to improve the learning outcomes of youth in the short and medium terms, which focuses on: ï‚· Promotion of the SPD to favor that only merit and the suitability of the teacher are the factors determining who should be at the front of the group, as well as to support the continuing education and professional development of in-service teachers to permanently expand their capabilities. ï‚· Extension of frequent collegiate work to all schools in EMS, because evidence indicates that this promotes more effective teaching practices. ï‚· Development of socioemotional skills through CONSTRUYE-T, since some abilities, such as perseverance and planning for the future, may have a positive impact on better student performance, as shown in Figure A4.3. As observed, students who placed in the highest level of perseverance performed better compared to those that placed in the lowest. ï‚· Consolidation of the minimum normalization of schools to guarantee a basic set of organizational conditions in the schools. ï‚· Promotion of parents’ involvement in their children’s education. Figure A4.3. Proportion of students in Level IV of the PLANEA test by level of perseverance and socioeconomic status (SC) 1.3 Interventions promoted by the Federal Government to increase the relevance of EMS It is often said that EMS does not sufficiently prepare youth to succeed in the workplace, resulting in high rates of youth unemployment and underemployment, low productivity, and precarious incomes. Against this background, many subsystems are reviewing the models used to develop the professional skills of youth and SEP, through SEMS, is promoting two interventions: The Mexican Model of Dual Training and the Model of Entrepreneurs. The first seeks to alternate classroom time with training in the workplace to strengthen the development of professional skills. At the beginning of the 2015-16 school year, more than 1,000 students participated in the model. The second seeks to develop entrepreneurial skills. It currently operates in 2,600 public schools and will be expanded to an additional 2,400 in the next three years. 1.4 Interventions promoted by the Federal Government to reduce the opportunity cost of EMS: school dropout rates, promotion of the continuity of educational trajectories, and scholarships. 45 H) Movimiento contra el Abandono In 2011-12, more than 650,000 students dropped out of EMS. This problem is explained by multiple causes of economic, social, and academic order. The Movimiento contra el Abandono Escolar is a SEP initiative designed to encourage students to successfully complete their educational trajectory through three components: a) Training of school managers on the use of the Caja de Herramientas, a set of manuals and procedures available to administrators, teachers, and parents to prevent dropouts. b) Establishment of an early warning system, implementation of mentoring programs, and mechanisms to facilitate the transition between educational modalities and services, as well as prevention of risk conditions, preventive communication with parents, and induction and integration programs for youth. c) Scholarship Program against School Desertion. These interventions are contributing to promote a relevant change in the culture of the educational communities. It is now recognized that school dropout is not an inevitable fact, that its causes are not only outside the school and that the educational communities thus can and should do more to ensure the permanence of their students. Due to the different efforts by the educational communities, the dropout rate decreased from 15% in 2012 to 12.6% in 2015, a reduction of 2.4%. It is necessary to keep working with the educational communities so schools can develop enough capabilities to offer guidance to youth throughout their educational trajectory. Figure A4.4. Dropout rate in the EMS I) Increase transition rates from middle school to EMS among PROSPERA beneficiaries In Mexico, the probability of access to EMS for youth from households in the highest decile of per capita income is three times greater than the probability recorded in the lowest decile. To reverse this situation, the Programa de Inclusión Social (PROSPERA) of the Secretariat of Social Development (SEDESOL) and SEMS supports the transition of middle school graduates from the lower income segments to EMS through the following interventions: ï‚· Provision of more information on the available supply of EMS and the enrollment process to facilitate the transition to this educational level, especially for families benefiting from PROSPERA. ï‚· Increased visits to middle schools with high concentrations of scholarship recipients to promote their continuation of studies in EMS. ï‚· Implementation of an action plan, with the involvement of the PROSPERA State Committees to encourage this transition. 46 During the 2014-15 school year, SEP and PROSPERA gave scholarship support to nearly 2 million youth in public schools at the upper secondary level, representing almost 50% of the students. Of all EMS scholarship recipients, 63% receive their support through PROSPERA and currently, as observed in Figure A4.5, 75% of the recipients transition from middle school to EMS. Figure A4.5. Percentage of scholarship recipients that transition from middle school to EMS in the school year J) PROBEMS Scholarships The education policy prioritized in its agenda issues of equality and inclusion and has directed its attention to the expansion of educational opportunities for groups with a socioeconomic disadvantage. As part of this effort, SEMS currently provides a set of scholarships to accompany students in lower income segments along their educational journey. As mentioned earlier, an important part of these scholarships is operated through PROSPERA. The rest are operated through the Scholarship Program for Upper Secondary Education (Programa de Becas de Educación Media Superior, PROBEMS). The objective of this program is to contribute to ensuring higher educational coverage, inclusion and equality among all groups of the population by providing scholarships to avoid dropout and promote the continuation of studies, as well as scholarships for vulnerable groups, targeted to improve participation in the labor market, and to promote the academic excellence of students at the upper secondary level. As a result, almost 8 out of 10 EMS students from the first four income deciles now have a scholarship from PROSPERA or SEMS. PERFORMANCE OF THE BANK, CO-FINANCIERS AND OTHER PARTNERS IN THE PREPARATION AND IMPLEMENTATION OF THE OPERATION The technical assistance provided by the Bank to the Mexican education authorities was always relevant and complete. Relevant because the issues on which it provided assistance were always aligned with the public policy priorities defined by each administration. While the consolidation of the RIEMS is an ongoing process, the change in administration implies a redefinition of priorities in the matter of public policy. In the case of the last operation, for example, the reduction of school dropout rates was the general priority in the new government program. As a consequence, the Bank provided technical assistance, including a seminar on international strategies to reduce dropout in EMS, a policy note on the causes of dropout by region, and a qualitative evaluation of the Movimiento contra el Abandono. Another important issue for the 47 new administration, which also supported by the Bank, was the attention to youth who do not study, do not work, and are not being trained – Ni estudian, ni trabajan (NINIS). The cooperation was complete because it was not limited to one instrument. The assistance included monitoring missions, organization of international conferences, seminars and workshops, and preparation of analytical studies and notes on best international practices in this field. Some examples include: Analysis of the RIEMS: Presentation on the RIEMS implementation process; a seminar and note to improve the training of EMS teachers. The seminar was attended by experts from the Bank, Chile and Brazil among others; Seminars and note for the development of socioemotional skills in EMS students; Design of 17 online training modules for EMS directors on Positive Behavioral Interventions and Support, Assessment of the perceptions of the returns of secondary education and school desertion. It is worth mentioning that SEP valued the delivery of practical products during the development of the operation. The Bank delivered concrete and simple tools that could be implemented in a short period of time. RESULTS OF THE OPERATION COMPARED TO THE AGREED OBJECTIVES The following presents information on the Indicators for the Series of DPLs. It reports the baselines and targets defined for the indicators and whether or not targets were reached. It is worth mentioning that there are indicators whose baseline does not correspond with the records of the educational authorities responsible for the follow-up (SEMS). In these cases, it provides the available information and a comparison in the exchange rates of the Bank data and the estimates by SEMS. In most cases, the established goals were met. However, there are areas with a significant lag and which are already being addressed, especially regarding ENLACE/PLANEA results. Although the objectives were achieved when considering the percentage of students with good or excellent performance in Mathematics, this was not the case for results in Spanish. The challenge is even greater in marginalized communities. A. Indicators for the first and second loans Indicator 1: Percent of the upper secondary schools that, having adopted the Marco Común Curricular (MCC), accept student transfers and validate equivalencies Baseline value Original Target Value Achieved current value (2015) Target status Less than 10% End of the DPL2: 25% 70.6% Met End of the program: 40% Notes: Even though schools that adopted the MCC accept student transfers and validate studies, there is still no national registry to ensure that schools are validating studies. The closest effort is the School Services System for Higher Secondary Education (Sistema de Servicios Escolares de la Educación Media Superior, SISEEMS), which is used by three centralized subsystems and allows the revalidation of studies completed inside the country and abroad. This indicator was replaced in the last DPL by the Index of entry into the SNB. Indicator 2: Unemployment rate ratio of upper secondary graduates ages 18-21 over the unemployment rate of those upper secondary graduates with ages 25-30 48 Baseline value Original Target Value Achieved current value (2015) Target status 1.8 1.6 1.9 Not met Notes: Although this indicator attempts to measure the relevance of EMS in the labor market, it must be considered that unemployment in the country is affected not only by the quality of the graduates, but also by external variables other than the efforts made by SEP to train people with the skills and competences demanded by the labor market. Indicator 3: Reduction in dropout rates in upper secondary education Baseline value Original Target Value Achieved current value (2014-15) Target status 41.7% 34.3% 12.6% Not met in the defined terms Notes: The baseline does not match SEMS records. Since it does not have the technical data sheet to review its construction, the available information corresponding to the official indicator used by SEMS to track the dropout rate is provided. It is important to note that when comparing the percentage change between the baseline and target for the indicator of the DPL and that of the SEMS, the SEMS indicator is visibly higher: -20.8% vs. -17.7%. Estimated baseline value for the 2008-2009 school year: 15.9%. Indicator 4: Baseline value Original Target Value Achieved current value (2014-15) Target status 91% 95% 99.2% Met DPL1 Indicator 5: The percentage of students scoring in the categories good or excellent in the ENLACE assessment Baseline value Original Target Value Achieved current value Target status (ENLACE 2013-14) 15% 18% 39.3% Met Observations: ENLACE was last applied in the 2013-14 school year. The first evaluation with the PLANEA test, its replacement, was performed in 2015. DPL2 Indicator: The percentage of students scoring in the categories good or excellent in the ENLACE assessment in Spanish and Mathematics Baseline value Original Target Value Achieved current value (2013-14) Target status 15.6% Mathematics End of the DPL2: Mathematics 39.3% Met in 52.3% Spanish Mathematics 28% Spanish 44.7% mathematics Spanish 59% End of the program: Mathematics 35.0% Spanish 63.0% Observations: ENLACE was last applied in the 2013-14 school year. The first evaluation with the PLANEA test, its replacement, was performed in 2015. 49 Indicator 6: Improved teaching quality in upper secondary education due to massive in service training and certification of teacher competences, as measured by the baseline and final results of the planned Impact Evaluation Baseline value Original Target Value Achieved current value Target status Baseline of the impact Final results of the - Assessment not assessment impact assessment performed Observations: The Impact Assessment of the Teacher Training Program (PROFORDEMS) was not performed because the program was transformed and a new training model is currently being promoted to meets the needs of teachers under the SPD requirements. DPL1 Indicator 7. Increased completion rates in upper secondary education for lowest quintile Baseline value Original Target Value Achieved current value (2014) Target status 47% 49% 38.7% Not met in defined terms Notes: The baseline for this indicator does not match SEMS records. Since it does not have the technical data sheet to review its construction, the available information corresponding to the indicator built by SEMS is provided. It is important to note that when comparing the exchange rates between the baseline and target for the indicator of the DPL and that of the SEMS, the one for SEMS is visibly higher: 29.4% vs. 4.3%. Estimated baseline value for 2008: 29.9%. DPL2 Indicator. Increased completion rates in upper secondary education for the lowest population quintile Baseline value Original Target Value Achieved current value (2014) Target status 47% End of the DPL2: 50% 38.7% Not met in the defined End of the program: terms 52% Notes: The baseline for this indicator does not match SEMS records. Since it does not have the technical data sheet to review its construction, the available information corresponding to the indicator built by SEMS is provided. It is important to note that when comparing the exchange rates between the baseline and target for the indicator of the DPL and that of the SEMS, the one for SEMS is visibly higher: 29.4% vs. 4.3%. Estimated baseline value for 2008: 29.9%. Indicator 8. Transition from lower to upper secondary school for lowest population quintile Baseline value Original Target Value Achieved current value (2014-15)* Target status 57% End of the DPL2: 63% 75% Met, but not in the End of the program: 64% defined terms Observations: *These numbers only consider scholarship recipients from PROSPERA that transition from middle school to EMS. This indicator is presented because it is the one used by the SEMS to measure the transition rate and, although it does not display information disaggregated by income quintiles, it does quantify the transition from middle school to EMS among the poorest households, taking their income as a reference point. It should be noted that PROSPERA is a social inclusion program designed exclusively for the population living in poverty. Its target population consists of households with an estimated monthly income per capita lower than the Minimum Line for Wellness ( Línea de Bienestar Mínimo). This line, defined by CONEVAL, is the monetary value of a basic food basket and allows quantification of the population that, even when using all its income for food, still cannot acquire the essentials for an adequate nutrition. The estimated baseline value for the 2008-2009 school year was 59.9%. B. Indicators of the third loan 50 Indicator 1. Index of entry into the Sistema Nacional de Bachillerato Baseline value Original Target Value Achieved current value (2015) Target status 0% 20% 37.1% Met Indicator 2. Average annual dropout rate in EMS Baseline value Original Target Value Achieved current value (2014-15)* Target status 16.3% 12% 12.6% Advance of more than 90% Observations: Achievement rate of 97% considering the projected figure and 95% considering the preliminary one. Indicator 3: Percentage of students scoring “goodâ€? or excellent on the ENLACE assessment in Spanish and Math Baseline value Original Target Value Achieved current value (2013-14) Target status Mathematics 15.6% End of the program: Mathematics 39.3% Met in mathematics Spanish 52.3% Mathematics 37% Spanish 44.7% Spanish 53% Observations: ENLACE was last applied in the 2013-2014 school year. The first evaluation with the PLANEA test, its replacement, was performed in 2015. Indicator 4: Percentage of students attending schools in very highly marginalized localities scoring good or excellent on the ENLACE assessment in Spanish and Mathematics Baseline value Original Target Value Achieved current value (2013-14) Target status Mathematics 6% Mathematics 29% Mathematics 23% Not met Spanish 22.6% Spanish 31% Spanish 20.3% Observations: ENLACE was last applied in the 2013-2014 school year. The first evaluation with the PLANEA test, its replacement, was performed in 2015. Indicator 5: Gross Upper Secondary Education enrollment (youth ages 15-17) Baseline value Original Target Value Achieved current value (2014-15) Target status 58.6% 68.7% 71.5% Met Indicator 6: Gross Upper Secondary Education enrollment for students from households in the first four deciles of income per capita. Baseline value Original Target Value Achieved current value (2014-15) Target status 61.8% 64% 67.8% Met LESSONS LEARNED AND SUSTAINABILITY OF THE OPERATION As has been shown throughout this report, SEP, through SEMS, promotes actions articulated around a government plan to increase educational opportunities for young Mexicans, to ensure the quality of learning in the classroom and the relevance of the education. All of them are derived from the RIEMS and the subsequent reforms. To advance in the consolidation of the reforms to EMS, the Government, within the framework of the Series of DPLs, has worked with the World Bank in defining public policy interventions 51 that use international best practices as a reference point and a solid justification based on assessments and research. The efforts made thus far are not minor. As shown by the indicators in the previous section, there has been significant progress since the approval of the first DPL. On one hand, EMS is serving an increasing number of young Mexicans and is being delivered to the most vulnerable sectors of the population. The coverage among the households in the first four income deciles has been increasing since 2008. The goal established for this indicator was not only reached, but also exceeded. In addition, the dropout rate maintains a decreasing trend, the completion rate of EMS is increasing, and the transition rate to EMS for those located in the poorest households is growing. While the advances in terms of the objectives proposed within the DPLs are visible, there are still important areas of opportunity to attend and a constitutional mandate to consolidate, which makes the operation sustainable. The fact that quality of learning in the classroom is still poor cannot be overlooked and, while there has been progress in terms of coverage, work must continue to achieve universality in 2021 and to incorporate the most vulnerable sectors. The challenges are significant; however, the DPLs laid the foundation that will drive the actions of the Government in the construction of a Mexico with quality education. As for lessons learned, the most important are concentrated around the performance evaluation of the operation, particularly in the indicators measuring the progress of the objectives. Because it was a long-term operation that transcended the change in administrations, providing information for the follow-up of the indicators was difficult because, for those corresponding to the first DPLs, there was no technical datasheet specifying how the data was constructed. It should be noted that these were not part of the indicators measured by EMS and for which it has historical information. Additionally, the construction of indicators should consider variables on which the impact of public policy interventions is not mediated by external factors, as happens with the unemployment rate. Considering the above, relevant and feasible indicators can be defined to monitor operations that include a change in Government administration. 52 Annex 5. Comments of Cofinanciers and Other Partners/Stakeholders N/A 53 Annex 6. List of Supporting Documents Altonji, Joseph 1995. he E§ect of high school Curriculum on Education andLabor Market Outcomes, Journal of Human Resources, 30 (3): 409-438. Avitabile, Ciro & De Hoyos Navarro, Rafael E., 2015. "The Heterogeneous effect of information on student performance: evidence from a randomized control trial in Mexico," Policy Research Working Paper Series 7422, The World Bank. B. Bruns, Luque J. 2014. Great Teachers. How to raise students learning in Latin America and the Caribbean. World Bank. Clark, D. and H. Royer. 2013. The effect of Education on Adult Mortality and Health: Evidence from Britain. American Economic Review, vol. 103, no 6: pp. 2087-2120. Clarke, M. 2012. What Matters Most for Student Assessment Systems: A Framework Paper. World Bank. Washington DC. Available at https://openknowledge.worldbank.org/bitstream/handle/10986/17471/682350WP00PUBL0WP1 0READ0web04019012.pdf?sequence=1 Cunningham, W. and Villaseñor P. Employer Voices, Employer Demands, and Implications for Public Skills Development Policy Connecting the Labor and Education Sectors. Farrell, D., A. Puron, and J.K Remes. 2005. Beyond Cheap Labor: Lessons for Developing Economies. McKinsey Quarterly (1): 99-109. Rose, H. and J. R. Betts (2004), The E§ect of high school Courses on Earnings, The Review of Economics and Statistics, 86(2): 497-513. Santiago, P., et al. 2012. OECD Reviews of Evaluation and Assessment in Education: Mexico 2012, OECD Publishing. Available at <
http://dx.doi.org/10.1787/9789264172647-en> Székely M. Evaluación Final del Componente 1 del Programa de Formación de Recursos Huamnos Basada en Cometencias (PROFORHCOM) FASE III. Extracto para envío al Banco Mundail, 28 de marzo de 2016. Székely M. World Bank. June 2013. Después de la educación media superior: un análisis para el Estado de Oaxaca. World Bank. 2010. Mexico - Development Policy Loan for Upper Secondary Education Reform Program. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/499881468057260979/Mexico-Development-Policy- Loan-for-Upper-Secondary-Education-Reform-Program World Bank. 2012. Mexico - Second Upper Secondary Education Development Policy Loan Project. Washington, DC: World Bank. 54 http://documents.worldbank.org/curated/en/839141468049873355/Mexico-Second-Upper- Secondary-Education-Development-Policy-Loan-Project World Bank. 2013. Mexico - Third Upper Secondary Education Development Policy Loan Project. Washington, DC: World Bank Group. http://documents.worldbank.org/curated/en/165621468299124174/Mexico-Third-Upper- Secondary-Education-Development-Policy-Loan-Project 55 Annex 7. Scheme of the RIEMS  Common Curriculum  Specialized Tracks SIGUELE (overarching Competency- CONAEDU (multi- program for students at risk) based stakeholder body)  National Academic Curriculum  Coordination and Tutoring system regulation of (SINATA) Mentoring Coordination federal, state,  Career Counseling and and university and  Afterschool youth Student Regulation private providers Welfare development program RIEMS and their systems  Definition and (CONSTRUYE-T) consolidation of  Early warning system programs for vulnerable students (SIAT)  Teacher training New Tools (PROFORDEMS)  School accreditation and Certification  Director training Institutions (COPEEMS) (PROFORDIR)  Student certification  Evaluation System  Student Assessments Objective: Strengthen EMS for labor market responsiveness Focus: • Improving Flexibility (transferability of students among EMS systems and subsystems) • Enhancing Quality • Reducing Opportunity Costs 56 MAP 57