RESULTS-BASED FINANCING RBF EDUCATION EVIDENCE ZANZIBAR Can Goal-Setting and Incentives Improve Student Performance? FEBRUARY 2019 REACH funded an evaluation that measured how two different incentive approaches affected the academic performance of grade nine students. The Results in Education for All Children (REACH) Trust Fund supports and disseminates research on the impact of results-based financing on learning outcomes. The EVIDENCE series highlights REACH grants around the world to provide empirical evidence and operational lessons helpful in the design and implementation of successful performance-based programs. Although many developing countries rates, student test scores, or other = have made great progress in education outcomes. Providing increasing primary school enrollment financial incentives to students While it can be expensive, and completion rates, few students has been shown to be effective in financial rewards can improve student attendance make the transition to secondary increasing school attendance and and achievement. school, and even fewer complete achievement and, in some cases, secondary school. Results-based more effective than providing financing (RBF) has been used in incentives to parents or teachers. many developing countries in an However, these programs are attempt to incentivize students, expensive and difficult to implement = ? parents, teachers, and others for countries with limited capacity. to achieve better results. RBF On the other hand, the effect of Less is known in developing mechanisms work by linking financial other types of incentives such as countries about the impact of non-financial incentives. (or in some cases non-financial) personal goal-setting or non-financial incentives to measurable results, for incentives has been less extensively example school attendance, dropout studied in developing countries. This note was adapted from Islam, Asadul, Sungoh Kwan, Eema Masood, Nishith Prakash, and Shwetlana Sabarwal (2018). ‘All Pain and No Gain’: When Goal Setting Leads to More Effort but No Gains in Test Scores, (mimeo). 2 RFB EDUCATION | EVIDENCE Furthermore, as a precondition to in school and (b) whether this effect establishing financial incentives is strengthened when combined with for students, it is first necessary to non-financial incentives. ensure that students know how to act on these incentives to improve The REACH evaluation found that their performance. personal goal-setting—with or without non-financial incentives—had The Results in Education for All no discernible impact on student Children (REACH) Trust Fund at the achievement. However, goal-setting World Bank funded an evaluation did have a significant positive that measured how two different impact on students’ effort in school, incentive approaches affected the and this impact was not affected academic performance of grade nine by the existence of non-financial students in Zanzibar (Tanzania). The incentives. The students’ inability to first approach allowed students to translate their increased effort into set personal goals at the beginning improved performance may stem of the school year regarding their from significant over-estimation of performance by the end of the year. their own baseline performance and The second approach combined over-optimistic goal-setting, both this goal-setting exercise with of which were done after baseline non-financial rewards such as testing but before the students had certificates or in-kind prizes for received their results. These findings students who met their goals. These suggest that another precondition of approaches were designed to answer establishing effective RBF incentives the following questions: (a) whether at the student level is providing students setting goals for themselves students with timely feedback about has any effect on their performance their own performance. CONTEXT from lower secondary to upper secondary in grade 11, which occurs at roughly 16 years of age.1 The Zanzibar is a semi-autonomous majority of these student dropouts region of the United Republic of Tanzania, with a population of 1.4 are due to their poor performance million—nearly half of whom live in the lower secondary school exit below the poverty line. A key driver examinations. Therefore, to reduce of poverty in Zanzibar is the limited student dropouts, the Ministry of Zanzibar Education and Vocational Training economic prospects for young Nearly people, due in part to high rates of (MoEVT) of Zanzibar intends to half student dropouts before secondary implement programs that will improve school completion. Almost half of all student performance in the short run of the population lives students entering secondary school and increase students’ aspirations, below the poverty line drop out before completion, and only access to information, and motivation 8.4 percent of students transition in the long run. ZANZIBAR 3 WHY WAS THE INTERVENTION CHOSEN? After making considerable input- students is at least as effective as of financial or non-financial incentives driven investments in the education providing them to parents. by helping students to build the system, the MoEVT is keen to shift tools to effectively respond to these its focus away from “more of the However, as a precondition to a incentives. From a policy perspective, same” school inputs and towards student-targeted RBF program, it is goal-setting is a low-cost, scalable results. In particular, the government first necessary to establish whether option that also has intrinsic merit is interested in undertaking an students know how to convert beyond its instrumental value in evidence-based scale-up of a student extrinsic incentives into actions that promoting student achievement. incentive program as part of its improve their achievement. Intuitively, Similarly, it is important to assess US$50 million World Bank-financed performance-based incentives will how the intrinsic incentives of goal- education project. To improve be effective only to the extent that setting can be enhanced by extrinsic student performance, several studies actors (whether students, teachers, or incentives, which may be particularly in Kenya,2 Cambodia,3 Benin,4 and parents) have control over achieving effective for children who are more the United States5 have found that the intended results, have the ability motivated by short-run rewards than incentivizing students directly is to achieve the intended results, less tangible long-run rewards. a promising option. Furthermore, and understand how much effort studies in Mexico6 and Tanzania7 is needed to achieve the intended In assessing how well students set have shown that incentives aimed results. The act of proactively setting goals for themselves and respond to at students could potentially be goals and working towards those non-financial incentives, the REACH more effective at improving student goals can help students to motivate evaluation was designed to build an performance than incentives aimed at themselves to increase their effort, evidence base to inform the future teachers. Similarly, studies in Malawi,8 particularly when self-control is scale-up of a student-targeted RBF India,9 and Mozambique10 have found a constraint.11/12 Goal-setting can approach to improving secondary that providing incentives directly to therefore increase the effectiveness student performance in Zanzibar. 4 RFB EDUCATION | EVIDENCE HOW DID THE randomly assigned to the following three groups. The students in the ceremony, so this had no effect on the results. The third group of students INTERVENTION first group completed an interactive exercise with enumerators in which constituted the control group. WORK? they estimated their expectations for their own performance on a baseline Teachers and head teachers of the students in both of the treatment In February 2016 at the beginning test and set a goal for themselves groups were asked to give them of the 2016 school year, 187 on a similar test to be taken in nine periodic reminders of the goals they secondary schools with a total of months’ time at the end of the school had set throughout the school year 18,281 grade nine students were year. The students made these (February to November 2016). In estimations of both their baseline addition, the students were given a and goal-setting scores after the systematic reminder of their goals in baseline testing had been conducted August 2016, and the endline tests Treatment Group 1 but before the students had received were administered in November 2016. Students estimated their their results. The students in the The main outcome of interest for performance on a test and set second group completed the same comparing the three groups was the goals for a future test. goal-setting exercise and, in addition, endline test score for each student, received non-financial awards after controlling for the baseline test Treatment Group 2 including certificates and in-kind score. In addition, the effect of the Students estimated their prizes at a public ceremony at the end two approaches on student effort performance on a test, set goals for of the school year if they achieved was also assessed as a secondary a future test, and received awards if their goals. The team running the outcome. Effort was measured goals were met. intervention chose the type of in-kind mainly using students’ self-reports of prize to be given based on focus how much time they spent studying Control Group 3 group discussions with the students but also using principal components No interventions used but did not disclose the type of analysis to construct an “effort index” prize to them until the day of the based on multiple survey questions. ZANZIBAR 5 WHAT WERE Table 1: Impact on Student Test Scores (Standard Deviations) Table 2: Impact on Student Effort THE RESULTS? Math English Study more than 30 min. Effort in school The evaluation found that personal Goal-setting 0.06 0.06 Goal-setting 0.04** 0.06 goal-setting had no discernible (0.07) (0.07) (0.02) (0.04) impact on student achievement— Goal-setting + 0.03 0.07 Goal-setting + 0.04** 0.03 with or without non-financial Incentives (0.07) (0.06) Incentives (0.02) (0.06) incentives. Neither goal-setting alone nor goal-setting combined Baseline 0.50*** 0.49*** Baseline 0.21*** 0.28*** Effort (0.05) (0.02) Effort (0.01) (0.01) with non-financial incentives had a significant impact on student Constant -0.03 -0.03 Constant 0.53*** -0.01 (0.05) (0.04) (0.02) (0.03) achievement. The differences between the test scores of students N 13,426 13,426 N 13,241 13,360 in the two treatment groups and the scores of those in the control group Notes: Standard errors in parentheses. * p<0.1; Notes: Standard errors in parentheses. * p<0.1; were positive, ranging from 0.03 to **p<0.05; ***p<0.01 **p<0.05; ***p<0.01 0.07 standard deviations in Math and English, but they were small and the slight increase in self-reported reduced the amount of effort that they insignificant (Table 1). effort was a result of the students thought they needed to make in order However, goal-setting had being included in the two treatment to improve (Figure 1). significant positive effects groups and feeling the need to improve their school achievement Partly because of these inflated on students’ effort in school, or to report having made high levels baseline estimates, the students also regardless of the existence of of effort to the study observers. set overly ambitious improvement non-financial incentives. Students However, the REACH evaluation did goals for themselves relative to their who did this goal-setting exercise not disentangle these factors from actual endline performance, and were 4 percentage points more the effect of the goal-setting and these goals were unrealistic even likely to spend at least 30 minutes non-financial incentives themselves. after controlling for the initial over- studying per day outside school, estimation on the baseline tests which was the primary outcome The students significantly over- (Figure 2). Goals were defined in related to student effort (Table 2). estimated their own baseline three different ways: unchallenging Adding the non-financial incentives performance and set over-optimistic goals, challenging and realistic to the goal-setting exercise did goals, both of which may have goals, and challenging but unrealistic not lead to any additional student contributed to the intervention’s goals. Thirty-nine percent of treated effort. The extra 30 minutes of lack of impact.Only 11 percent students set unchallenging goals studying was concentrated among of students were able to provide a which are lower than their expected students with low-mid baseline reasonable estimate (within a 20 test scores at the baseline (Group 1). test scores and those who aspired percent range) of their actual baseline Thirty-two percent of students set to go to college or university. performance immediately after taking challenging and realistic goals which However, the goal-setting exercise the baseline test. Almost half of the is within 25 percent of their expected had no discernible impact on the students overestimated their baseline scores on baseline test (Group 2). composite “effort index,” either with performance by 50 percent or more. Lastly, 29 percent of students set or without non-financial incentives. It is possible that students’ over- challenging but unrealistic goals which Furthermore, it is possible that estimates of their own baseline ability are more than 25 percent above 6 RFB EDUCATION | EVIDENCE their expected scores at the baseline regulate their own study habits and suggest that more than 85 percent of test (Group 3). It is possible that convert their increased effort into them had difficulty understanding or the students’ inability to accurately improved learning outcomes. Lastly, even recalling their goals from earlier predict how much improvement was the interviews conducted with the in the school year, which may have realistic contributed to their failure to students at the midpoint of the study reduced the intervention’s impact. Figure 1: Over Estimation 2250 2000 1750 Number of students 1500 1250 1000 750 500 250 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Baseline math score Expected baseline score Figure 2: Over Aspiration 2750 2500 2250 2000 Number of students 1750 1500 1250 1000 750 500 250 0 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 Endline math score Goal ZANZIBAR 7 WHAT WERE THE LESSONS LEARNED? A key insight gained from this incentives are most effective. While getting students to understand their evaluation study is that one of the impact of financial incentives own academic performance was a the preconditions of an effective may depend heavily on the size of the challenge. Future research may show performance-based incentives incentive, the impact of non-financial that goal-setting increases student program is students’ capacity to incentives may be highly sensitive performance in the long run, not only know how much effort is needed to to the choice of whether to reward by increasing effort but also by giving achieve a goal and to self-regulate students with recognition or in-kind students a better understanding of their own effort. Most students set prizes and what type of prize is their own ability and the link between highly unrealistic goals, which was offered. In this intervention, the effort and performance. a function of both their significant students were told only that the over-estimation of their baseline non-financial incentives would be “a performance and their over- certificate and a prize,” and therefore One of the optimistic endline goals, even after it is unlikely that the type of prize had preconditions controlling for their inflated baseline any effect on the results. However, in expectations. The results of this cases where students will know the of an effective study suggest that having more type of prize ahead of time, careful performance-based accurate information about their attention should be paid to assessing own performance may help students student preferences and selecting incentives program to set and achieve more realistic incentives that are likely to be useful is students’ capacity goals. Students’ over-estimations to students and provide them with of their own level of achievement sufficient motivation. to know how much and their inability to self-regulate their own effort appear to be It is also possible that the increased effort is needed to binding constraints to improving effort made by the goal-setting achieve a goal and students in this intervention simply student achievement. It is therefore critical for schools to provide timely did not have time to improve their to self-regulate their feedback to students about their academic achievement in the own effort. performance and advice on how short time period of this study, they can improve over time. New which was just one school year. interventions focused on overcoming Therefore, designers of future financial or non-financial incentives this information constraint should be developed and tested before RBF and other supporting interventions 30% mechanisms aimed at students are should be careful to ensure that the scaled up. Other interventions may timelines that they set for measuring also be needed to address gaps in outcomes reflect a realistic theory teacher skills or motivation or other of change regarding how student aspects of the learning environment effort translates into improved that may be preventing student effort performance. Zanzibar is a place from becoming improved learning. where many students are unfamiliar and uncomfortable with the act of This evaluation may also help to shed test-taking and receive very little Teacher feedback light on what types of non-financial feedback from teachers, so simply is crucial. CONCLUSION ensuring that students have the required tools to respond to financial scores in other subjects that were not covered by this intervention. The Government of Zanzibar or non-financial incentives to improve In addition, the study found that (Tanzania) has an ambitious agenda their own performance. This study non-financial incentives did not for implementing a performance- tested the impact of personal enhance the effectiveness of goal- based incentives program aimed student goal-setting with and without setting for students. This suggests directly at students with the goals non-financial performance incentives that further interventions, such of improving their academic and found that, while setting goals as providing regular performance performance and increasing rates increased students’ academic feedback to students or increasing of secondary school completion. effort in school, it had no impact on students’ ability to convert effort However, before the government their performance. However, goal- into learning, are needed to establish embarks on this incentives program, setting may have value in and of the preconditions for effective RBF it will be necessary to establish itself and may yield other benefits to mechanisms before performance- several preconditions for RBF students that were not captured in based incentives at the student level mechanisms to work, which include this evaluation, such as higher test are scaled up in Zanzibar. 1 Unpublished MoEVT administrative data. 2 Kremer, Michael, Edward Miguel, and Rebecca Thornton (2009). “Incentives to Learn.” MIT Press Journals, 91 (3), pp. 437–456. 3 Barrera-Osorio, Felipe and Deon Filmer (2013). “Incentivizing schooling for learning: Evidence on the impact of alternative targeting approaches”. Journal of Human Resources, 51(2), pp. 461–499, University of Wisconsin Press. 4 Blimpo, Moussa (2014). “Team incentives for education in developing countries: A randomized field experiment in Benin.” American Economic Journal: Applied Economics. 6(4), pp. 90-109 5 Levitt, Steven, John List, Susanne Neckermann, and Sally Sadoff (2011). “The impact of short-term incentives on student performance.” University of Chicago Press Working Paper. 6 Berhman, Jere, Jorge Gallardo-Garcia, and Viviana Velez-Grajales (2012). “Are conditional cash transfers effective in urban areas? Evidence from Mexico.” Education Economics, 20(3), pp. 233–259. 7 Sabarwal et al (forthcoming). 8 Baird, Sarah, Craig McIntosh, and Berk Ozler (2011). “Cash or Condition? Evidence from a Cash Transfer Experiment.” The Quarterly Journal of Economics, 126(4), pp. 1709–1753. 9 Berry, James. (2015) “Child Control in Education Decisions. An Evaluation of Targeted Incentives to Learn in India.” Journal of Human Resources 50.4: 1051–1080. 10 De Walque, Damien and Christine Valente (2017). “Preventing Excess Female School Drop Out in Mozambique: Conditional Transfers and the Respective Role of Parent and Child in Schooling Decisions.” World Bank Working Paper. 11 Duckworth, A. L., and M.E. Seligman (2005). “Self-discipline outdoes IQ in predicting academic performance of adolescents.” Psychological Science, 16(12), 939–944. 12 Duckworth, A. L., P.D. Quinn, and E. Tsukayama (2012). “What No Child Left Behind leaves behind: The roles of IQ and self-control in predicting standardized achievement test scores and report card grades.” Journal of Educational Psychology, 104(2), 439. PHOTO CREDITS: Cover: Photo courtesy of the World Bank Page 2: “Education in Zanzibar, Tanzania” by GPE/Chantal Rigaud, license: CC BY-NC-ND 2.0 Page 3: “Oppression” by Giorgio Montersino, license: CC BY-SA 2.0 Page 4: Project photos of students receiving certificates, courtesy of the World Bank RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) worldbank.org/reach REACH is funded by the Government of Norway through NORAD, the Government of the United States of America through USAID, and the Government of Germany reach@worldbank.org through the Federal Ministry for Economic Cooperation and Development.