RESULTS-BASED FINANCING RBF EDUCATION EVIDENCE PERU What are the challenges in introducing a results-based approach for the selection of school principals? APRIL 2021 An evaluation of a large scale reform in Peru assessed a results-based approach for selecting and paying principals to improve student learning outcomes. Photo courtesy of World Bank/Domingo Giribaldi The Results in Education for All Children (REACH) Trust Fund supports and disseminates research on the impact of results-based financing on learning outcomes. The EVIDENCE series highlights REACH grants around the world to provide empirical evidence and operational lessons helpful in the design and implementation of successful performance-based programs. The quality of school management the performance of public servants,2 can have a large impact on student meaning that introducing more merit- School principals are of learning. Evidence from a wide range based selection of principals may be central importance to the quality of teaching of countries demonstrates that an effective way to improve school and learning. improving the management practices performance. of principals is associated with gains in student achievement.1 However, To explore the efficacy of a results- in many countries, the selection of based system for choosing and school principals is not based on compensating principals, the achievement or merit but on seniority Results for Education for All Children Some countries or even favoritism and political (REACH) Trust Fund financed the are piloting results- evaluation of a large-scale reform based approaches for affiliation. Evidence from a number principal selection and of countries suggests that selecting of principal selection in Peru. Under compensation. public sector managers on the basis the original system, principals were of merit has a positive impact on often selected through favoritism This note was adapted from Lemos, Renata; and Piza, Caio. 2019. “Manager selection and student learning: Evidence from n Peru.” World Bank Group. Unpublished Manuscript. 2 RFB EDUCATION | EVIDENCE or seniority, and their salaries were would receive an increase in pay. had only a short-term negative (yet not based on merit. In 2013, the The intention was to ensure that persistent) impact in rural areas. Government of Peru decided to remuneration in the school system These results can be partly explained introduce a reform that introduced the became more results-based by by the low levels of managerial merit-based selection of principals. making access to higher-paying skills and poor time management Existing principals were required managerial positions meritocratic. of newly appointed principals in to pass a performance evaluation rural areas, while such differences to remain in their positions, while Despite the successful were not found in urban areas. The teachers who passed a national implementation of the reform, the government recently updated the examination were eligible to become study found that the meritocratic reform to provide principals in rural principals or vice- principals. Both selection of principals did not have areas with more training and support, existing principals and teachers any significant effects on student and to offer promotion opportunities promoted through this examination achievement in urban areas and to skilled principals. CONTEXT Prior to the principal-selection reform, local authorities selected of their performance. The amount of funding provided to each region teachers to become principals by the Ministry of Finance to pay Peru has achieved near-universal and vice-principals without teachers and principal salaries access to schooling, but levels of clear guidance from the central was based simply on the numbers learning are generally low and vary government. All principals were of teachers and principals in the widely, particularly between rural paid on a similar scale regardless system. and urban areas. Although a large majority of students (more than 70 percent) attend schools in urban areas, most schools in the country are small and located in rural Rural schools in areas, and many have only a single Peru achieve worse teacher to cover multiple grades. Rural schools achieve worse outcomes in national outcomes in national examinations. examinations. Peru For example, in 2014, half of all urban students achieved a “satisfactory” score for reading in a second-grade examination, the Census-based Student Evaluation Only 17% (Evaluacion Censal de Estudiantes, of students score ECE), while only 17 percent of rural students met this standard and satisfactorily on could be considered ready for the a second grade third grade. Rural students also lag behind in math. reading exam. PERU 3 Photo courtesy of CIFOR/Yoly Gutierrez WHY WAS THE INTERVENTION CHOSEN? The intervention was part of a of teachers’ pay and to provide them in schools where the principal had wider set of reforms, the Education with incentives to increase their changed as a result of the reform. Program for Learning Achievements efforts in the classroom; and (iii) a Such evaluations are typically (Programa de Educación para los system of competitive, merit-based conducted for a pilot intervention Logros de Aprendizaje, or PELA), evaluations and appointments was before a reform is rolled out which aimed to improve learning introduced for school principals. nationwide, but this study conducted outcomes in schools by introducing Overall, the reform package was a rigorous evaluation of a reform that more results-based elements into associated with a 6 to 8 percent had already been implemented across education financing. The reforms increase in average student learning an entire school system, with the aim were triggered by Peru’s poor outcomes on PISA tests between of generating recommendations on performance in the 2012 Programme 2012 and 2015.3 how to improve the policy. for International Student Assessment (PISA), an international learning While a package of reforms is assessment; with the country often the best way to achieve rapid achieving the lowest score of all improvements in outcomes, it can be participating countries. Under PELA, difficult to isolate the effects of the government took three actions: individual components of the reform (i) The number of standardized package. To evaluate the specific student assessments was increased impact of the change in how principals and their quality was improved to were selected, REACH supported provide clearer data to evaluate the a quasi-experimental study that performance of schools; (ii) changes used existing data on the selection were made to the process for of principals along with student appointing and promoting teachers examination results to measure to increase the results-based nature the change in learning outcomes Photo courtesy of Rotaria de Peru SAC/Heike Hoffmann 4 RFB EDUCATION | EVIDENCE HOW DID THE was called the Evaluacion de Acceso (access evaluation) and which tested posts, while fewer than 10 percent of principals who failed the first INTERVENTION candidates on a broader range of managerial skills and behaviors. evaluation were still in their posts. WORK? Officials opened the examination to Because the reform had already been implemented nationwide, the teachers whose pay ranked in the The reform required all 14,000 upper half of the salary scale, as a researchers could not evaluate the school principals to undergo a way to target those with the most impacts of the reforms using a performance evaluation to remain experience and highest suitability randomized controlled trial. Instead, in their positions. A nationwide for leadership positions. Existing they compared schools where the examination was used as a second principals who had failed the first original principals had failed the performance evaluation to select and evaluation or who had not taken it examination and should have been appoint replacements for principals were also eligible to take the second replaced (the treated group), which who had failed the first performance test. Some 43,000 teachers and schools where the original principals evaluation. The entire evaluation 5,618 vice principals and principals passed the examination and should process had two parts, beginning sat for the Evaluacion de Acceso. have remained (the nontreated with an initial multiple-choice test, Principals who passed this second group). To measure the impact of the after which those who passed test could be assigned to new reform on learning, the researchers moved on to a broader evaluation posts or remain in their existing used students’ test scores in math by local education officials. The posts, but any principals who failed and reading in the Evaluacion Censal first performance evaluation, the both evaluations were reassigned de Estudiantes (ECE), a national Evaluacion Excepcional (exceptional to regular teaching positions. The standardized student evaluation evaluation), which included some highest-performing candidates were conducted annually in specific questions about managerial skills, then invited to take up principal grades. This enabled the researchers was conducted in mid-2014. The posts to replace those who had failed to compare the historical trend in test principals who passed remained in the exam. Finally, the government scores with any changes in the years their jobs; that number totaled only redesigned the principal salary following the reform. about 3,000. The remainder either scheme to include additional pay for failed or did not turn up to undergo all the new principals who had been Test data for some students were the evaluation. To replace the failing selected as well as existing principals available over several years, which principals, the government then who had been reconfirmed. Those enabled the researchers to measure held a second assessment, which principals who were reassigned to the impact of the reform on individual teaching positions were paid under students as they progressed to the teacher salary scheme. higher grades. The team also used The reform required all surveys to learn more about how The level of compliance with the the reform affected schools, how principals to undergo reform appears to have been high. the new principals used their time, a national exam to There was full compliance with the and what constraints they faced in law’s requirement to administer the their daily work. The researchers keep their jobs and examinations and to select principals also compared how these dynamics then a second exam as based on their results. After two years, differed between rural and urban implemented to replace about 90 percent of the candidates with results good enough to be offered schools. This information was used to shed more light on the those who had failed. a principal’s job were still in their mechanisms underlying the results. PERU 5 WHAT WERE THE RESULTS The reform led to lower test Figure 1: Lower Student Test Scores in Treated Schools than in Nontreated Schools scores overall, primarily driven Learning Gains, Levels (Grade 2, Math) Learning Gap, s.d. Treated Relative to Nontreated by students of rural schools. (Grade 2, Math) Learning scores improved in both urban and rural schools between 2014 and 2016, which reflected general improvements in learning in Peru’s schools. However, scores rose more slowly in schools run by a new principal with the Learning Gains, Levels (Grade 2, Reading) Learning Gap, s.d. Treated Relative to Nontreated improvement in their scores being (Grade 2, Reading) about 0.1 standard deviation lower in both math and reading than for schools that retained their original principals.4 Analyzing the change over time, the researchers found that the negative impact was strongest in the first year following the reform, and that the negative impact of the change in principal Table 1. Similar Scores in Urban Areas but Worse Scores for Treated Schools in Rural Areas continued, albeit to a lesser extent, in the second year (Figure 1). z(Grade 2 Math) z(Grade 4 Math) Panel A: Urban (1) (2) (3) (4) Disruptions in leadership changes ITT*Post -0.001 0.013 (0.037) (0.035) at rural schools were more -0.022 persistent than in urban schools. ITT*Post (2015) (0.042) The difference in the trend in average 0.020 -0.017 ITT*Post (2016) learning scores between schools that (0.047) (0.044) changed their principals and those ITT*Post (2018) 0.043 (0.043) that did not was almost twice as Obs 11320 11320 11210 11210 large for rural schools, at 0.14 to 0.15 standard deviation in math scores Panel B: Rural (1) (2) (3) (4) (Table 1), than for urban schools. -0.137** -0.153** ITT*Post This finding is further confirmed (0.062) (0.067) when looking only at the scores of -0.191** ITT*Post (2015) (0.074) students who were evaluated twice -0.082 -0.208** ITT*Post (2016) between 2012 and 2018 (Figure 2). (0.077) (0.085) The graph shows that the impact ITT*Post (2018) -0.098 (0.081) was not influenced by differences Obs 9490 9490 8760 8760 between cohorts of students enrolled Note: ***, **, * sig. at 1, 5, 10 %. 2007-2014 Grade 2 inputted for Grade 4 regressions (2016 corr .61-.68). in schools. 6 RFB EDUCATION | EVIDENCE These results are surprising given that, Figure 2:Worse Scores for Treated Schools in Rural Areas Consistent Across while a change of principal might be Cohorts and Time associated with temporary disruption, Urban Schools those negative effects tend to fade as 2009 (Gr 2) to 2015 (Gr 8) the new principal settles into the job. 2010 (Gr 2) to 2016 (Gr 8) These estimations suggest that the 2012 (Gr 2) to 2018 (Gr 8) gap between rural and urban schools 2014 (Gr 2) to 2016 (Gr 4) was equivalent to approximately 3 to 6 months of schooling. Rural Schools The negative impacts on rural 2009 (Gr 2) to 2015 (Gr 8) schools can be explained by the lower-quality performance of 2010 (Gr 2) to 2016 (Gr 8) replacement principals compared 2012 (Gr 2) to 2018 (Gr 8) to the performance of urban school 2014 (Gr 2) to 2016 (Gr 4) principals who remained in their posts. To understand the reason behind this weak performance, the Note: ITT coefficients for individual regressions, 90% confidence interval. researchers first looked at the test scores of the principals who were demoted, of those who remained in appointee than those who did not, this This suggests that spending more post, and of the teachers who were was not the case for rural schools. time on management (or time promoted.5 They found that both away from teaching duties) is rural teachers who were promoted Newly appointed principals in rural key for rural principals to improve to principal and rural principals who schools were less likely than urban learning outcomes for their schools. retained their post had significantly school principals to devote all of However, this difference in test lower scores than their urban their time to management. Under scores was not apparent in rural equivalents. This suggests that the the Peruvian system, principals in schools that had acquired new pool of rural teachers from which small schools with fewer than nine principals or in urban schools. This to choose replacements had fewer classrooms are required to spend suggests that the new principals management skills than the pool of some of their time teaching, while in rural schools did not take full urban teachers. Furthermore, the in larger schools, the principals can advantage of being able to allocate examination scores of teachers who spend all of their time running the more of their time to managerial successfully applied and were offered school. The researchers compared tasks aimed at improving learning. a position in a treated school through the learning performance of schools the Evaluacion de Acceso in 2014 (the just below the threshold (of nine These findings are reinforced by data incoming principal) compared to the classrooms) with those just above. from a survey conducted of principals examination scores of the principals Among all rural schools that had who were appointed following they replaced (outgoing principals) not changed their principals, the the reform. In their responses, the was smaller in rural areas than researchers found that those schools principals cited a lack of relevant skills urban areas. It therefore appears that, where the principals did not have and problems with time management while urban schools that changed additional teaching duties registered as the key challenges they faced when their principals gained a more skilled substantially higher test scores. first appointed (Figure 3). PERU 7 Figure 3: Key Challenges Cited by First-Time School Principals Perception/Inspiration Motivation Persuasion skills time mgmt Strongly disagree Likert response (%) Strongly agree Notes: Phone survey with 120 first-time principals in 2015 in both urban and rural areas. Based on recall information as survey was collected in the first half of 2019. WHAT WERE is in line with evidence from the U.S. that suggests that changing school unforeseen impact due to a lack of candidates with the relevant skills. THE LESSONS principals typically has negative Despite having passed the evaluation, LEARNED? effects for the first couple of years.6 However, some of the evidence from the new principals did not have substantially more skills than those this study in Peru suggests that whom they replaced. Having often Interventions that aim to improve these negative impacts may have worked as teachers in smaller schools school management by replacing been sustained over time. Follow-up where the principal also had teaching school principals can have negative research will explore whether and duties, these new principals appear to effects in the short to medium term. to what extent these effects are have been slow to transition into full- The Peruvian reforms were unusual mitigated over time as the new time management, further reducing in that they not only introduced principals settle into their roles. the potential benefits of the reform. merit-based appointments for The findings of this research suggest future principals but also required Results-based financing may need that results-based financing may need existing principals to undergo to be implemented in different ways to be implemented in particular ways retroactive merit-based testing and for different groups of schools. at the local level or rolled out differently removed those who did not meet Even if results-based financing is in different areas of a country to the standard. The findings of this implemented unilaterally throughout achieve the same outcomes. The research suggest that the negative a country, its impact may be Government of Peru is in the process impacts of a change in principal different in different areas. In Peru, of introducing amendments to the may be enough to counteract the the introduction of a results-based policy to provide additional support to potential positive effects of more approach for selecting and paying principals in rural areas (as discussed effective management. This finding principals in rural areas had an in the next section). CONCLUSION effectively in transitioning to full-time management. further opportunities and incentives to principals, the network system also The quality of school management provides mentorship and training to can affect how much students learn. In October 2019, the Government school leaders in rural areas. As of Researchers evaluated a nationwide of Peru announced a series of 2020, the new system has already reform in Peru that introduced a amendments to the policy that been piloted in 300 newly created rural results-based selection and payment were designed to provide more networks across the country covering system for school principals. Under career opportunities for successful approximately 3,500 schools, 10,000 this system, all existing principals principals. In the original reform, teachers, and 150,000 students. The were subject to an evaluation and teachers who were promoted to goal of the pilot is to gather further those who failed it were replaced. The principal had no opportunity for evidence to inform the proposed evaluation found that the reform did further promotion and received limited amendments to the results-based not significantly improve test scores training. Under the new system, two policy to be submitted to Congress. of students from those rural schools new positions were created: network whose principals had changed, and principal and network vice-principal. also had no significant effect on Each is meant to oversee a network the slightly improved test scores of of five to 10 rural schools. School The evaluation showed students from urban schools. Data vice-principals can be promoted to that the meritocratic from the evaluations, surveys, and network vice-principal or principal with an associated increase in salary, and selection of principals did analysis of principals in smaller and larger schools all suggest that the principals can be promoted to network not have any significant newly appointed principals in rural principals. Each step up requires the effects on student areas had lower levels of skills than candidate to take the Evaluacion de those in urban areas and did not have Acceso again. These results-based achievement in urban substantially higher levels of skills rewards provide further incentives for areas and had only a short- than those whom they replaced. newly appointed principals to improve The new principals also appear to the learning outcomes of students in term negative impact in have struggled to use their time their schools. In addition to providing rural areas. 1 Bloom, N., R. Lemos, R. Sadun, and J. Van Reenen. 2015. “Does Management Matter in Schools?” The Economic Journal 125 (584): 647–74. 2 Finan, F., B. Olken, and R. Pande. 2015. “The Personnel Economics of the State.” National Bureau of Economic Research Working Paper No. 21825, NBER, Cambridge, MA. 3 Chen, L., F. Child, E. Dorn, and R. Morales. 2019. “An Interview with Former Peruvian Minister of Education Jaime Saavedra.” https://www.mckinsey.com/industries/social- sector/our-insights/an-interview-with-former-peruvian-minister-of-education-jaime-saavedra. 4 The negative impact was 0.1 SD in reading and 0.09 SD in math. 5 The researchers used data from the first, multiple-choice phase of the evaluations, which were strongly predictive of the candidate’s overall score in the evaluation. 6 Miller, A. 2013. “Principal Turnover and Student Achievement.” Economics of Education Review 36: 60–72. RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) worldbank.org/reach REACH is funded by the Government of Norway through NORAD, the Government of the United States of America through USAID, and the Government of Germany reach@worldbank.org through the Federal Ministry for Economic Cooperation and Development.