45271 Quality of Public Expenditure NOTE No. 3 ­ JUNE 2008 Harnessing Evaluation to Improve Upper Secondary Education Although Mexico has made great strides in improv- major investments in education and has shown ing the coverage of education at all levels, it still consistent increases in educational investment, not has a major gap in enrollment at the upper sec- just in absolute terms, but also in terms of a rising ondary level: less than half of the youth population share of GDP being dedicated to education. graduates. The education level of adults in Mexico is far behind that of other middle-income countries. The Sub-Secretary of Upper Secondary Edu- In addition, the results from international learning cation (SEMS) is introducing a series of reforms assessments give rise to concerns about the qual- intended both to increase coverage and to improve ity and relevance of upper secondary education. the quality and relevance of the education that The Sub-Secretary of Upper Secondary Education students receive. As part of these reforms, SEMS is is leading a major reform of the sector with the supporting the establishment of a National Degree twin objectives of increasing coverage and quality System (Sistema Nacional de Bachillerato), which within the framework of a more integrated system. will create a set of common standards for upper Evaluation has been an important tool in designing, secondary education and allow transferability implementing, and monitoring the reform. within the system. It is also introducing a universal Upper Secondary Education in Mexico Box 1. The Upper Secondary System in a Nutshell In recent decades, Mexico has made substantial Whereas basic education in Mexico is provided by state governments, progress in education. At the primary education upper secondary education is complicated: it is provided by the federal (primaria) level, coverage has remained around government, autonomous public universities, state governments, 90% since 1970 and the system's efficiency has and a large private sector. Nationwide, 28% of enrollment is in improved significantly. The coverage of lower the federal system, 37% in the state system, 15% in the university secondary (secundaria) has risen from 30% in system, and 21% in the private sector. The proportions vary greatly 1970 to around 90% in 2006, and the cover- from state to state. age of upper secondary (media superior) 10% to 55% over the same period. However compared With the variety of different providers, it is estimated that 200 to to basic education, there are significant problems 300 different subsystems are in operation, providing both techni- with efficiency. In essence, Mexico can get youth cal and academic education. Upper secondary education depends into the upper secondary system but cannot get heavily on contract and part-time teachers. Transfer among the them to graduate. different subsystems is difficult, and both learning and regulatory standards vary greatly. Measuring quality is difficult, but the interna- tional PISA learning assessment shows that Mexico is one of the lowest-performing countries in the OECD.1 While its scores have improved, it is still 1PISA, the Program for International Student Assess- behind other middle-income OECD countries. ments, is applied every three years to fifteen year old At the same time, the OECD report Education at youth in school in all OECD and many partner countries. a Glance 2007 indicates that Mexico has made It covers mathematics, science, and language. MEXICO: QualIty Of PublIC EXPEndIturE student assessment to complement existing as- sible to compare the efficiency of different types sessments in the basic education system. of upper secondary school systems. This is where the second element of the new quality assessment system plays an important role. SEMS is using Role of Evaluation information from the Secretary of Public Education to update its databases and correct information it has on schools and education finance so that it Along with the rest of Mexico's pubic admin- can analyze what it is spent in upper secondary istration, SEMS is moving toward results-based education and what the results are. With the as- management and performance-based budgeting sessment information about the operation of the approaches. A fundamental aspect of these ap- upper secondary system, SEMS will then be able proaches is emphasis on monitoring and evalu- to improve the allocation of resources. ation of policies and programs--a challenge that involves a range of activities, from monitoring and assessing progress in the implementation of programs, to measuring changes in outcomes and Using Impact Evaluations evaluating the impact of specific interventions on those outcomes. Impact evaluation assesses the specific outcomes attributable to a particular intervention. It is a There is broad consensus that improving qual- powerful instrument to determine "what works and ity in education requires establishing transparent what does not work" in designing development and well-known quantitative standards, accom- interventions. For example, impact evaluation panied by a strong assessment system to follow can measure the increase in student learning that up on student learning and help guide action by results from a change in the curriculum. This is the school system.2 With transparent standards done by using a counterfactual that represents the and an assessment system, the education system hypothetical state the beneficiaries would have can target resources where they are needed--for experienced without the intervention. example, to teacher training, materials, or other interventions. Over the past few years some significant im- provements have made impact evaluations easier to implement systematically. First, information is Expanding the Integrated Quality more readily available because countries are Assessment System collecting more micro data through household surveys. Second, specialists have developed a number of evaluation techniques to construct the SEMS is developing an integrated quality assess- counterfactual, ranging from randomized experi- ment that both builds on existing resources and ments to quasi-experimental techniques. introduces new tools in the sector. At the center of the quality assessment system is a new learning Mexico has been a leader in developing assessment (test) that is being given to all students and applying impact evaluation techniques to at the end of 12th grade. This complements the policymaking. The national transfer program, Exámenes de la Calidad y Logro Educativo (EX- Oportunidades, started with a detailed impact CALE) assessment that is currently administered evaluation of a pilot program that was brought in the basic education system to all students in to national scale. Building on that experience, 3rd, 5th, 7th, and 9th grades. The new assessment Mexico has introduced impact evaluation tech- was first given at the upper secondary level dur- niques in different sectors to develop and improve ing the 2007­2008 school year. Although SEMS new programs. only has "authority" over federal school systems, it was able to persuade other school systems to 2This approach is discussed in L. Crouch (2006) Por participate, and participation was universal. una Educación de Calidad para el Perú. Washington: World Bank and E. Vegas and J Partow (2008) Incre- With repeated results (probably after the mentar el Aprendizaje estudiantil en America Latina. assessment is administered twice), it will be pos- Washington: World Bank. 2 NOTE No. 3 ­ JUNE 2008 SEMS has been a pioneer in using impact governments also contribute to the budgets of evaluation as part of its overall reform program. their own autonomous universities.) In many On introducting new programs to improve the ways, the upper secondary education system is coverage and quality of the upper secondary more flexible than other areas of the education education, SEMS prepared a series of impact system. It relies heavily on contract teachers and evaluations to measure the impact of these new has been quite open to experimenting with new programs. At the center of SEMS' impact evalua- approaches and new systems. tion strategy was the creation of an international impact evaluation consultative group. The consul- The overall evaluation system that SEMS is tative group comprises a number of world experts establishing will allow more comparisons among in impact evaluation who have published widely the different systems. Some systems are more ef- on impact evaluation and education, and most of ficient than others, and some have a bigger equity them have worked extensively in Mexico. impact. The combination of statistical analysis and impact evaluation will help the federal government Working with the consultative group, SEMS target additional resources to the best-performing is carrying out impact evaluations on the new systems. As the evaluation system goes beyond upper secondary education scholarship program, the systems that the federal government influences the use of computers in upper secondary training directly, other school systems may make similar programs, and the selection of school principals. use of the available information. Given the in- Data from these evaluations will help improve the creasing attention that the Ministry of Education is design of these programs. Future impact evalua- giving to results and outcomes, the establishment tions are planned for other SEMS programs. Introduction of Box 2. Use of Education Evaluation in OECD Countries Performance-Based Budgeting Objective assessments are playing a growing role in education sectors throughout the OECD. This includes both international as- The Mexican public administration is moving sessments, which provide international benchmarks, and national forward in adopting results-based management assessments. and performance-based budgeting through the new Performance Evaluation System or Sistema de In the United States, secondary students' performances on the PISA Evaluación al Desempeño (SED). This framework and on the Trends in International Mathematics and Science Study will promote the generation and use of perfor- have raised concerns about the state of science and mathematics mance information throughout the budget cycle, education and led to greater emphasis on math and science educa- and the line ministries (Secretarias) will play a tion. The federal government has required states to introduce their critical role in implementing it. SEMS's develop- own assessment systems and minimum standards. Schools that ment of rigorous evaluations and quantitative have low performance receive additional support to help students standards is an example of relevant performance meet state standards. However if a school continues to "fail," the information that will feed into the system. state can take control of the school or shut it down. This approach does not reward well-performing schools but ensures a minimum The federal government finances the upper standard for all schools. secondary system through several mechanisms. First, it provides direct financing to its own schools, In Germany, PISA results have given rise to intense debate on ways whether decentralized or autonomous; most of the country can improve students' performance. Policymakers used the decentralized school systems are operated the PISA results to support their efforts to introduce new initiatives by SEMS. Second, it provides financing to the to improve quality. Germany does not have a strong testing tradition, state systems, and it often plays a significant and PISA forced the public to see the significant weaknesses in the role in the management of these state systems. education system. As a result of reforms and targeted investment, Third, it provides a significant part of the budget Germany saw increases in its PISA scores. Thus in Germany, PISA directly to federal- and state-level autonomous served as a analytic database, a benchmarking instrument, and universities, which allocate their resources freely advocacy tool. to their own secondary education system. (State 3 MEXICO: QualIty Of PublIC EXPEndIturE of performance budgeting in Mexico can benefit To meet the challenges of the growing and from this experience and support this already very diverse upper secondary system, the federal positive development. government will need additional information to track decisions and make key decisions. It will need to share this information with states and Challenges autonomous universities to allow them to make better decisions, as well. The newly introduced evaluation system will help both the state and In Mexico the education sector is the single largest federal governments meet the growing demand recipient of public resources and one of the larg- for upper secondary education. est employers. It plays a key role in the economy, particularly because, as a middle-income country, Mexico is increasingly dependent on the services produced by educated workers. The upper second- ary education system is growing quickly and will Erik Bloom, Senior Economist, Human Development continue to grow in the coming years. Network (World Bank) Mexico: Quality of Public Expenditure is a series of notes produced in collaboration between the Mexican Government's Secretaría de Hacienda y Crédito Público and the Public Sector and Governance unit in the World Bank's Latin American and Caribbean Region.