RESULTS-BASED FINANCING RBF EDUCATION EVIDENCE INDONESIA Can Performance-Based School Grants Improve Learning? JANUARY 2018 REACH funded an evaluation to assess whether bonuses distributed to schools lead to improved student learning. 2006 The Results in Education for All Children (REACH) Trust Fund supports and disseminates research on the impact of results-based financing on learning outcomes. The EVIDENCE series highlights REACH grants around the world to provide empirical evidence and operational lessons helpful in the design and implementation of successful performance-based programs. Despite large investments The Results in Education for All receiving the bonus for top-performing via grants, there have Children (REACH) Trust Fund at The schools. Announcing the performance been no significant improvements in learning World Bank funded an evaluation incentive had different impacts on in Indonesia since 2006. that assessed the early impact primary and junior secondary schools. of a performance-based school Student test scores improved in grants program on student learning all junior secondary schools, with in Indonesia. While all Jakarta the largest gains being made in government schools receive a fixed schools that were already the highest Performance-based Intervention Learning grant per student, under the new performing. However, in primary school grants (such as training) outcomes program top performing schools schools the impact on test scores also receive an added bonus. This was slightly negative, with modest Studies have shown that performance-based school evaluation focused on two separate improvement in low-performing grants can improve learning effects in the first two years of schools offset by losses in high- when combined with other the new program: the effect of performing schools. While the gains interventions. announcing the performance-based at low-performing schools were incentive to schools, and the effect of similar in magnitude in both primary Al-Samarrai, Samer, Unika Shrestha, Amer Hasan, Nozomi Nakajima, Santoso Santoso, and Wisnu Harto Adi Wijoyo. “Introducing a performance-based component into Jakarta’s school grants: What do we know about its impact after three years?.” Economics of Education Review 67 (2018): 110–136. 2 RFB EDUCATION | EVIDENCE and junior secondary schools, these The Jakarta school grants program that could inform the design of mixed results highlight the risk that is distinct from many other pilots other interventions at scale. Future performance-based competition can of performance-based grants in programs could be improved by actually have a demotivating effect that it introduced RBF at scale, using other measures of school for some schools. On the other hand, across an entire education system. performance in addition to test there is little evidence that schools The evaluation of this program scores, considering alternative that received performance bonuses took advantage of pre-existing designs of the formula to performed better than those that administrative data, which made it determine grant allocations, and did not. The effect of the program possible to include all government allowing schools more flexibility on learning was largely due to the schools at relatively low cost, in experimenting with ways to change in incentives created by without separate data collection improve learning. announcing the performance-based for the evaluation. As such, there grants, rather than by the additional are several lessons learned from grant funding itself. the implementation of this program CONTEXT In 2014, junior secondary schools Indonesia since 2006. Furthermore, received $111 per student from despite higher spending than most the Jakarta city government and provinces, schools in Jakarta scored Indonesia has made large strides in $60 per student from the national barely above average in student improving educational attainment, program. With strict limits on the test scores. In response to the increasing average years of fees they can collect from parents, limited impact of this spending, schooling from six to eight years per schools are heavily reliant on these there has been a renewed focus adult between 2000 and 2012. At the grants, receiving more than 80% of on improving the effectiveness of same time, government education their discretionary funding from the school financing. This has led to spending tripled from 2001 and national and Jakarta city programs. increased appetite in Indonesia 2011, including the introduction of for trying new RBF approaches to the Bantuan Operasional Sekolah Despite these large investments, incentivize schools to use resources (BOS) school grants program in there have been no significant more effectively and align spending 2005. Although responsibility for improvements in learning in with learning outcomes. primary and secondary schools is devolved to the provincial and district level in Indonesia, under the BOS program the federal government supplements local funding with school grants of a fixed amount per student. In the nation’s capital, the Jakarta city government also began giving school grants to all government primary schools in 2005 through a program modeled on the national BOS program. The Jakarta program evolved Jakarta significantly between 2005 and 2014, expanding to include all government Despite higher spending than most provinces, secondary schools and more than schools in Jakarta scored barely above doubling per-student funding levels. average on student test scores. INDONESIA 3 WHY WAS THE INTERVENTION CHOSEN? Recognizing that Jakarta’s school While many countries have seen showing that tying financing to grants program represented a large similarly disappointing results of the achievement of desired results and important part of the education school grants alone, grants can can be effective at various levels of budget, the city government improve learning when combined service delivery. was intent on improving the with other interventions. In the effectiveness of this spending. To Philippines, school grants raised The new performance-based incentivize improved learning at the test scores when combined with component of the grant was school level, in 2014 the Jakarta school-based management to give designed to improve student government adjusted the formula schools more autonomy in spending learning by creating stronger for determining school grant decisions.1/2 In Tanzania, school incentives to improve at the school allocations to include a component grants improved learning when level. It was expected to improve based on learning performance. The combined with teacher incentives learning through three main new performance-based component to improve student performance.3 channels. First, the announcement of Jakarta’s school grants program Indonesia itself found in a previous of the performance incentive was was implemented at all government tweak of its school grants program expected to motivate teachers, schools, including 1,629 primary that grants raised student learning school principals, and other schools and 291 junior secondary when combined with improved stakeholders to exert greater effort schools. The introduction of a linkage between the school to improve learning. Second, it was performance-based component into committee and the village council expected to encourage schools this program builds on successes in to strengthen school oversight.4 In to align their use of funding more several other countries using RBF Senegal, a program that allowed closely to the objective of improved to make education spending more schools to apply for grants and learning, for example by shifting effective. A large body of evidence compete with each other on school funding away from uses not proven shows that how resources are performance had a positive effect to improve learning in Indonesia, spent, how they are tied to results, on learning.5 While there has been such as hiring contract teachers. and how incentives are designed no other assessment of the effect Third, it was expected that schools can be more important than of tying school grant allocations that receive performance bonuses resources themselves. directly to school performance, would improve further by using the there are many examples from resources to support additional other sectors—particularly health— learning-enhancing activities. 4 RFB EDUCATION | EVIDENCE HOW DID THE ensure the new incentives were well understood. Under the new program, school ranking was conducted separately in each of Jakarta’s INTERVENTION all government schools continued to receive the basic grant allocation, six districts so that schools only competed against other schools WORK? but top performing schools received an added per student bonus serving students from similar backgrounds. However, despite these The new program was announced equivalent to 20 percent of the efforts to achieve equity, schools to all Jakarta government schools basic grant—a significant increase with higher initial performance were after the 2014 national exams, with in total discretionary funding. The much more likely to receive the a formal regulation posted on the size of the bonus was determined to performance bonus. Of the schools education department website. In be the maximum amount that was that received the performance bonus addition, a socialization meeting affordable within Jakarta’s education in 2015, 62% of primary schools and was scheduled by each of Jakarta’s budget, making sure to keep the 75% of junior secondary schools six district education offices for basic grant allocation unchanged. were already in the top quartile all school principals to attend to School performance was judged on before the program. Furthermore, two dimensions: average student performance on national exams over bottom quartile primary schools the last two years (beginning with needed to improve 7 percentage New Program 2013 and 2014), and percentage points to receive the bonus, while top quartile schools could drop 2 point improvement in exam scores Basic grant allocation + percentage points and still earn the over the same period. Each school per student bonus for best bonus. This imbalance was likely was given an average ranking across performing schools based on: these two dimensions, and schools in due to a combination of factors: the top quartile (25%) were awarded the design of the bonus formula 1) Average national exam scores the performance bonus the next year. mechanically favored schools with Percentage point 2)  higher initial performance; some improvement in exam scores To make the program more equitable schools may have felt less motivated and incentivize more schools, the to achieve the bonus based on how much improvement was required to earn it; and schools may have differed in their ability to act on the increased incentive. The impact of the redesigned program was measured by comparing changes in test scores before and after the introduction of performance bonuses between eligible government schools and ineligible non-government schools. In addition, the impact of receiving the performance bonus itself was measured by comparing changes in test scores between schools just above and just below the required performance threshold. INDONESIA 5 WHAT WERE THE RESULTS? Telling schools that a component of materials, school grants, or other schools increasing from 83 percent their grant would be based on their types of incentives.6 in 2014 to 90 percent in 2016, and performance resulted in higher test the bottom quartile of schools only scores for students in all government However, test scores improved the improving from 70 to 72 percent. junior secondary schools. After most at junior secondary schools that This may be an indication that the performance-based grant were already the highest performing, better performing schools believed component was announced after widening the gap between high they were more likely to receive the the 2014 national exams, test scores and low-performing schools. performance bonus and therefore increased by 2.6 percentage points Schools were assigned to quartiles exerted more effort to improve, or in 2015 and 4.6 percentage points based on their average student test that these schools simply had greater in 2016 for all government junior scores in 2013 and 2014, before the capacity to improve learning. secondary schools. This is equivalent performance-based component was to an increase from 72.5 percent in introduced. Test scores for the top On the other hand, the 2014 to 75 percent in 2015 and 77 quartile of junior secondary schools announcement of the performance- percent in 2016, suggesting that the improved by 6.9 percentage points based component resulted in impact may have spread across more (0.96 standard deviations) in 2016, slightly lower test scores overall schools over time. As this represents compared to only 2.2 percentage for students in government primary the early impact of just the first two points (0.30 standard deviations) schools. After the announcement in years of the program, it is possible for schools in the bottom quartile. 2014, student test scores remained that these gains will continue to This led to growing inequality across unchanged in 2015 and decreased by grow as the program becomes schools, with the top junior secondary 1.3 percentage points (0.11 standard established and stakeholders gain a better understanding of its incentives. These gains of 0.36 and Table 1. Impact on Student Exam Scores in 2016 (percentage points)7 0.63 standard deviations in 2015 and 2016, respectively, are large in magnitude compared to other Bottom quartile 2nd quartile 3rd quartile Top quartile Overall interventions to improve student test Primary 1.6 -1.3 -2.8 -2.9 -1.3 scores, such as computer-assisted schools learning, teacher training, class size Junior 2.2 3.8 5.2 6.9 4.6 reductions, provision of instructional schools 6 RFB EDUCATION | EVIDENCE deviations) in 2016 for all government conclusions on the cause of this schools, which may have led to more primary schools. This is equivalent to negative impact at primary schools, incentive for top schools to improve a reduction in scores from 71 percent it may be due to ineffective decision- to stay ahead of other schools trying in 2014 to 70 percent in 2016. making at the school level. There to narrow the gap. is some suggestive evidence that While the impact was slightly school principals shifted spending While announcing the performance- negative overall at the primary level, away from hiring temporary teachers based grant component had an the program benefited students and towards renovating classrooms, impact on some schools, actually at the lowest performing primary which may have reduced student receiving the performance bonus schools. However, these gains were learning by disrupting class instruction. yielded no additional impact. offset by negative impacts on test scores at the schools that initially There are several potential Comparing schools just above performed higher. Test scores for the explanations for the difference in and just below the threshold for bottom quartile of primary schools impact between primary and junior receiving a performance bonus, rose by 1.6 percentage points (0.15 secondary schools. First, junior there is little evidence that receiving standard deviations), from 63 percent secondary schools have more additional resources caused any in 2014 to 65 percent in 2016. These qualified and experienced staff, improvement in student test scores gains are similar in magnitude to which may have led to greater in 2015, the year the first bonuses those of the lowest performing junior capacity to improve learning. Second, were awarded. One possible secondary schools. However, test while the performance bonus was 20 explanation is that the amount scores at primary schools in the top percent of the basic grant allocation of the bonus was not sufficient quartile decreased by 2.9 percentage for all schools, in absolute terms it to impact learning for schools at points (0.27 standard deviations), was much larger for junior secondary the margin, or that the additional from 83 percent in 2014 to 80 percent schools, which may have led to spending may not have translated in 2016. This suggests that staff at stronger incentives to improve. Many quickly enough into improved previously high-performing primary primary schools felt they needed learning. In fact, receiving the schools were either unmotivated or a larger basic grant allocation, and performance bonus had no impact unable to improve learning. were unmotivated by the bonuses. on intermediate outcomes such as Third, the initial gap between high the quality of teachers or school While it is not possible with the and low-performing schools was infrastructure, so it is not surprising available data to reach definitive much smaller at junior secondary that test scores were unaffected. INDONESIA 7 WHAT WERE teacher training and procurement of learning equipment. This limited schools) to induce greater effort. Developing alternative designs THE LESSONS flexibility in the type of spending allowed may have made it difficult for for the formula to determine grant allocations—perhaps with LEARNED? some schools to respond effectively to the new incentives created by the different weights for level of performance and improvement— The evaluation of Jakarta’s program, and may have caused some could help ensure that both high performance-based school grants school grants to go unspent. This and low-performing schools are program took advantage of could be improved by making grants motivated to improve, and could pre-existing administrative data, more flexible, and encouraging schools potentially narrow the gap between and therefore was able to include to experiment with other interventions high and low-performing schools. all government schools in a scaled shown to improve learning such as up program at relatively low cost. increased teaching time and improved However, this data had limitations. pedagogy. Furthermore, the program’s impact may have been limited by What is needed? With the data available, few potential channels could be empirically tested delays in the disbursement of both School Plan to explain the observed results. the basic grant allocations and the There are many ways that schools bonuses, which may have prevented may have reallocated funding in schools from taking advantage of resources they were entitled to. response to the program’s incentives, but only data on hiring teachers and 1. Support in effective There are also concerns in Indonesia infrastructure spending could be that national exams may not be the resource management tested. Strengthening these data best indicator of school quality, due systems or supplementing them with to issues of cheating, “teaching to focus group discussions could help the test,” and exam questions that draw more concrete conclusions and do not reflect students’ learning. create more robust feedback loops to Using alternative measures of inform future policy. school performance instead of or in 2. Flexibility in regards to addition to student test scores—for type of spending allowed There are several potential lessons example school self-evaluations and learned for improving future school Indonesia’s well-established quality grants programs that stem from assurance system that assesses the challenges encountered in schools across eight dimensions— implementation. The evidence could provide a more holistic view of suggests that ineffective decision- school quality and more actionable 3. More reliable indicators making in how to reallocate spending guidance to schools on areas of may have adversely affected student improvement. learning at some schools. This could be improved by providing schools On the other hand, some schools with support in effective resource may have been able to improve but management, which is already being were unmotivated to do so because addressed through a new system of the incentives were too weak. 4. More appropriate mandatory electronic school plans In particular, the improvement to make schools more accountable. requirements and required to receive the performance At the same time, only certain types bonus may have been either too incentives depending of expenditures were allowed under large (for low-performing schools) on the school the school grants program, such as or too small (for high-performing CONCLUSION learning, including lack of ability to effectively allocate spending, limited effectively. Using alternative measures of school performance The Jakarta city government in flexibility in the spending allowed in addition to test scores could help Indonesia is a pioneer in using RBF under the program, and weak to mitigate the risk of cheating on to improve student learning. Two incentives to improve. Nevertheless, national exams and reward schools years after the introduction of a Jakarta’s performance-based school based on a more holistic view of performance-based school grants grants program has shown the school quality. Lastly, developing program at all government schools, potential for large learning gains, alternative grant allocation formulas the early results are mixed. While at least for some schools. These could help to motivate schools at the performance bonuses led to results also suggest several potential all levels of initial performance to improved learning at junior secondary avenues for improving the program improve, and ensure greater equity. schools, the mixed results at primary in the future. Making the grants Further experimentation would be schools show that incentives do more flexible would allow schools valuable to refine the program and not necessarily benefit all schools. to customize their spending to their determine which design features Several possible factors explain specific needs, and help to ensure and supporting interventions could why some schools failed to improve that all schools spend their resources maximize its positive impact. 1 Khattri, N., et al (2010). “The Effects of School-based Management in the Philippines: An Initial Assessment Using Administrative Data.” Policy Research Working Paper Series. No. 5248, The World Bank. 2 Yamauchi, F. (2014). “An alternative estimate of school-based management impacts on students’ achievements: evidence from the Philippines.” Journal of Development Effectiveness. 3 Mbiti, I., et al. (2015). Inputs, Incentives, and Complementarities in Primary Education: Experimental Evidence from Tanzania, process. 4 Pradhan, M., et al. (2014). “Improving educational quality through enhancing community participation: Results from a randomized field experiment in Indonesia.” American Economic Journal. Applied Economics 6(2): 105. 5 Carneiro, P. M., et al. (2016). “School grants and education quality: experimental evidence from Senegal.” Policy Research Working Paper Series. No. 7624, The World Bank 6 McEwan, P. J. (2015). “Improving Learning in Primary Schools of Developing Countries: A Meta-Analysis of Randomized Experiments.” Review of Educational Research 85(3): 353-394. 7 The values in the table are coefficients from a difference-in-differences framework using full controls, comparing changes in test scores between eligible government schools and ineligible non- government schools before and after the announcement of the performance-based school grants program. PHOTO CREDITS: Cover: Students at a secondary school in Jakarta. Photo courtesy of World Bank. Page 3: Students at a primary school in Indonesia. Photo courtesy of World Bank. Page 4: Students at a secondary school in Jakarta. Photo courtesy of World Bank. Page 5: Students at a secondary school in Jakarta. Photo courtesy of World Bank. Page 6: Students at a secondary school in Jakarta. Photo courtesy of World Bank. RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) worldbank.org/reach REACH is funded by the Government of Norway through NORAD, the Government of the United States of America through USAID, and the Government of Germany reach@worldbank.org through the Federal Ministry for Economic Cooperation and Development.