RESULTS-BASED FINANCING RBF EDUCATION EVIDENCE THE DEMOCRATIC REPUBLIC OF CONGO Can Incentives to Take Home Textbooks Increase Learning? JANUARY 2019 REACH funded an evaluation that measured the effectiveness of both financial and non- financial incentives at the student, classroom, and school levels. The Results in Education for All Children (REACH) Trust Fund supports and disseminates research on the impact of results-based financing on learning outcomes. The EVIDENCE series highlights REACH grants around the world to provide empirical evidence and operational lessons helpful in the design and implementation of successful performance-based programs. Significant progress has been made evidence that distributing textbooks in school enrollment over the last to schools increases learning. One Significant investments have been made towards several decades in many low-income possible explanation for this is that the provision of textbooks countries (LICs), but student learning it is not sufficient to make textbooks to combat the learning levels remain low, particularly in available—they also must be used, crisis in the Democratic Republic of Congo fragile settings. Governments and which may require both teachers donors have invested considerable and students to change their resources in the provision of school behavior. Results-based financing inputs and specifically in textbooks (RBF) mechanisms have been used to try to combat this learning crisis. in many developing countries in It makes intuitive sense that having an attempt to incentivize teachers, Solely distributing textbooks school materials such as textbooks students, and other stakeholders does not necessarily mean in classrooms would be an important to achieve better results by that they are being sent home or used. part of the education production encouraging such desired behavior function. However, the existing changes. RBF mechanisms work by research has found only very weak providing rewards—either financial This note was adapted from Falisse, Jean-Benoit, Marieke Huysentruyt, and Anders Olofsgård (2018). Textbooks for Homework: Impact on Learning in a State of Fragility (mimeo). 2 RFB EDUCATION | EVIDENCE or non-financial—that are given only both financial incentives at the group school. The intervention also affected on the condition that the potential level and non-financial incentives at students’ job aspirations in favor recipient takes specific actions or the individual level. These incentives of jobs requiring more education achieves measurable results such were implemented by Cordaid (Caritas rather than those that require manual as student test scores or other Netherlands), a Dutch NGO that has labor and convinced more students intermediate education outcomes. operated an RBF project in the South and teachers of the usefulness of Kivu region of the DRC since 2008. textbooks. These results suggest The Results in Education for All that allowing students to use Children (REACH) Trust Fund at the The evaluation found that the textbooks outside the classroom World Bank funded an evaluation that incentives given to encourage and supporting them to do so by measured the effectiveness of both students to take home textbooks providing modest financial incentives financial and non-financial incentives raised their French language test or non-financial incentives can at the student, classroom, and school scores by 0.27 to 0.30 standard increase the effectiveness of existing levels in the Democratic Republic of deviations (SD) but had no significant resources at a relatively low cost Congo (DRC). A new classroom routine impact on math test scores. The and with limited complexity. This is was designed to encourage all grade intervention also increased the likely to be particularly useful in the five and six students to take home likelihood of the students taking— case of fragile countries with limited a classroom textbook and use it to and passing—the end of year resources and administrative capacity study for a weekly quiz. Students and national exam at the conclusion of where more demanding interventions schools were incentivized to adopt grade six, which is a prerequisite are unlikely to be feasible. the routine through a combination of for continuing on to secondary CONTEXT every year since it began compiling a list of such countries in 2006.1 Fragile situation countries such as This evaluation took place in the the DRC often suffer from violence DRC, one of the poorest and most that makes it difficult for students Democratic conflict-ridden countries in the Republic of to access education, entrenches Congo world. More than 80 percent of inequalities in primary education, the population lived on less than and threatens teaching and learning US$1.25 in 2012. In South Kivu, a quality. Furthermore, it is not clear province in eastern DRC, 60 percent that the most binding constraints to of households were living below education in fragile settings are the the national poverty line in 2012, same as in other poor but relatively putting South Kivu roughly in the Violence in the DRC makes middle of the distribution of DRC stable and peaceful environments or that the relevant actors such as it difficult for students provinces. Furthermore, South Kivu students and teachers will respond in to access education, has been the center of several bouts the same way to incentives. As such, entrenches inequalities of armed conflict since the 1990s. the South Kivu region of the DRC The World Bank has designated provides an interesting case study to in primary education, and the DRC as a fragile and conflict- explore what interventions may be threatens learning quality. affected situation (FCS) country effective in fragile settings. THE DEMOCRATIC REPUBLIC OF CONGO 3 The DRC has achieved gross primary the country in an attempt to raise enrollment rates in excess of 100 low levels of learning. However, an percent over the last decade, but, as evaluation of this program found in many developing countries, this dismal results, in part because has not been matched by substantial the program was only partially learning gains.2 Standardized test implemented. This was followed by scores for primary students in the another US$100 million World Bank DRC remain among the lowest in grant that funded the distribution of the world, even compared to other 22 million textbooks between 2012 LICs.3 Funding for education is and 2017. However, recent reports severely inadequate, with the DRC suggest that, while 93 percent of the government spending the least on textbooks made it to the schools, Therefore, comprehension of French education as a percentage of GDP most are not being used in the in primary schools is often weak, among 28 countries in Sub-Saharan classroom, and students are very sometimes even among teachers. Africa.4 However, with the support rarely allowed to take them home of a US$180 million World Bank because of concerns about damage The REACH evaluation was grant, in 2008 the Government and theft. Furthermore, the current conducted in 90 primary schools in of the DRC launched a five-year textbooks are written in French, the Walungu and Shabunda districts program to improve public service which is the official language of of South Kivu province, where delivery in education. As part of this education in the DRC but often Cordaid operates its education RBF project, the government distributed only the third language spoken by program, which reaches roughly 18 million textbooks throughout children in rural parts of the country. 64,000 primary school students. WHY WAS THE INTERVENTION CHOSEN? During the past few decades, the world are not learning even the evaluations of textbook distribution substantial progress has been made basics while in school.6 projects in low-income countries in boosting school enrollment in have found very little positive impact. developing countries. Gross primary One possible explanation for For example, Sabarwal et al (2014) why increasing enrollment has enrollment rates in Sub-Saharan found that distributing textbooks not led to increased learning is Africa increased from 50 percent to in Sierra Leone had no impact on that learning depends not just on 98 percent between 1970 and 2014, students’ test scores, in part because spending time in the classroom and net primary enrollment rates the intervention merely increased but also on a wide range of other increased from 40 percent to almost the number of textbooks stored at complementary inputs, including 80 percent.5 However, this progress school but did not increase the use learning materials. There has been in enrollment has not been matched of textbooks in the classroom or in extensive literature citing the crucial by a similar improvement in the role played by school materials students’ homes.11 Similarly, Glewwe amount of actual student learning. such as textbooks in producing et al (2009) found that distributing This so-called “global learning crisis” learning and in combating the textbooks in Kenya had no impact on has been estimated to mean that 250 learning crisis.7/8/9/10 However, average test scores, although it did million school-aged children around the few available rigorous impact increase the scores of the strongest 4 RFB EDUCATION | EVIDENCE students. The authors attributed comprehensible to students. These this to the fact that the language two challenges are of course related, used in the textbooks was English, as students and teachers are less which, for many students, was likely to use textbooks that are of low only their third language, meaning quality or in a language that they do that most students could not read not understand. or comprehend the textbooks.12 These results suggest that merely Therefore, the objective of the REACH distributing textbooks is not enough evaluation was to determine whether to have a positive impact on learning. RBF incentives could improve In addition, students and teachers learning by targeting the first of must be encouraged to use them, these two challenges by encouraging and the textbooks themselves students and teachers to make more must be of high quality and extensive use of existing textbooks. HOW DID THE INTERVENTION WORK? In each of the treatment schools, teachers become more familiar with weekly quiz. The second element a routine was put in place to the textbooks over time. Students, of the incentive scheme consisted incentivize schools and students to teachers, headmasters, and parents of in-kind material rewards such as use textbooks in mathematics and were informed about the new routine notebooks, pens, and pencils with a the French language for homework first through verbal announcements, value of roughly US$9 per student and weekly quizzes. The routine and then the teachers and students to all students in a classroom if consisted of a system by which created posters for distribution the class as a whole achieved students could check textbooks in around the school explaining the 75 percent participation in the and out twice each week and take purpose of the intervention and textbook borrowing routine over them home in order to complete their encouraging students to take their an entire trimester. Each school homework in math and the French homework seriously. was also provided with financial language. The students were then incentive consisting of a lump sum tested on the material covered in The intervention used both financial of roughly US$120 to participate in the homework in a weekly quiz. This and non-financial incentives to the project designed to compensate routine was designed to supplement motivate students to check out the teachers and school administrators the already existing routine textbooks. The first element of the for potentially lost or damaged based around weekly homework incentive scheme was an intrinsic, textbooks. If there was any money assignments in order to maximize non-financial incentive consisting left over after replacing the lost or its chances of being adopted and of a star system displayed publicly damaged books, the school could of becoming a habit for students in each classroom to showcase use it to cover its general expenses. and teachers. The intervention’s the students who borrowed and goals were to have an immediate returned textbooks and participated This intervention was randomly positive effect not only on learning in a weekly quiz. A student earned assigned to 45 primary schools in the outside the classroom but also on a star for every week that he or she Walungu and Shabunda districts of teaching within the classroom in took home a textbook, returned it in South Kivu, while another 45 schools the longer term as students and good condition, and took part in the were assigned to the control group. THE DEMOCRATIC REPUBLIC OF CONGO 5 Baseline surveys were conducted in the winter of 2016/17 on grade WHAT WERE level of difficulty, which suggests that the intervention helped to promote five students, grade five teachers, headmasters, and the students’ THE RESULTS? learning across the board. When a small number of schools that failed households. Grade five students The incentives significantly to comply with the intervention in also completed a baseline test in increased the use of textbooks both the first year were excluded from math and the French language so inside and outside the classroom. the analysis, there was a slightly that a value-added model could be At the most basic level, the financial higher impact of 0.35 SDs. Compared used with endline test results as the and non-financial incentives with other similar interventions dependent variable and baseline test designed to encourage students focused on primary school learning results as one of the control variables. and teachers to make better use of outcomes, this can be considered a Students in both the treatment and textbooks achieved what they were moderate size effect. For example, the control group schools were then designed to do—they increased it is slightly larger than the impact followed for a period of roughly one the proportion of students who of 0.22 SDs that was found in Kenya used the textbooks outside the among the highest performing and a half years. The endline surveys students in Glewwe et al (2009).13 and tests in math and the French classroom. Eighty-one percent of language were conducted in June students in the treatment schools However, the incentive scheme had 2018 when the students were in reported having taken home a no significant impact on math test grade six. Treatment schools also textbook in the previous month, scores. The financial and non-financial received two external visits in 2017 compared to only 39 percent of incentives had no significant impact and 2018 to ensure that they were students in the control schools. All on math test scores (although the complying with the intervention. teachers in the treatment schools estimated coefficients were slightly While the randomization created reported that students were allowed positive), despite the fact that French generally balanced samples, some to take textbooks home, and the and math textbooks were taken home variables were significantly different initial concerns about books being at the same rate. This may be because between the two groups—teaching lost or damaged proved to be the language taught in the French efficacy, student age, student gender, mostly unfounded. Teachers in the language textbooks is at a more frequency of students eating breakfast, treatment schools also reported rudimentary level than the level of and student time spent working—and that students were making greater language skills required to understand these observable characteristics were use of textbooks in the classroom, textbooks in specific subjects such as therefore included as control variables although these self-reported math. This suggests that the language in the analysis. measures may be biased. in which the textbooks are written may be a constraint for learning a The incentive scheme significantly subject like math, whereas the French raised French language test scores. language textbooks are closer to The financial and non-financial The intervention used incentives raised French language students’ appropriate level. To learn French, it is likely that being exposed both financial and non- test scores by an average of 0.27 to 0.30 standard deviations (SD) in the to the language may be helpful by itself, even if the textbooks are not of financial incentives to second year of the program, using the highest pedagogical quality, at the a value-added model controlling optimal level, or written in appropriate motivate students to for baseline student test scores. language. Although there is no check out textbooks. This impact was roughly the same across the four different categories conclusive evidence that this is what drives the difference in results between of questions grouped according to French language and math test scores, 6 RFB EDUCATION | EVIDENCE it is likely that textbook language teachers that self-identified as less effectiveness of interventions remains an important issue. effective teachers (0.42 SDs), those aimed at improving student test with teachers with lower French scores varies substantially due to The intervention yielded the greatest test scores themselves (0.41 SDs), large differences in both their cost benefits for the weakest students and those with teachers with less and their impact. The financial and those in classrooms with the experience (0.58 SDs). These results and non-financial incentives in least skilled and least experienced suggest that the positive impact this intervention cost roughly teachers. When the results are from textbook use is due to students’ US$17 per student and yielded an separated for students above and self-directed learning rather than improvement in French language below the median test score at the any classroom interaction with the test scores of 1.6 SD per US$100 baseline, the intervention had no teacher. The intervention also had spent. This compares favorably with significant impact on above median a positive impact on both girls and the 30 randomized controlled trials students but had a significant impact boys, with no significant difference evaluated for cost-efficiency by on below median students equal to by gender or by age. On the other Kremer et al (2013).14 Furthermore, 0.42 SDs, larger in magnitude than hand, students who had had no it is likely that the cost-efficiency of the overall impact. This suggests that previous exposure to books and this intervention could be increased encouraging textbook use helps to students who were classified as if it was scaled up if a system could reduce learning inequalities between being especially vulnerable—orphans, be put in place to decentralize high-performing and low-performing physically disabled children, children monitoring at the school level. Much students. It also suggests that of physically or mentally disabled of the cost of this intervention came textbooks and other learning inputs parents, or children displaced by war— from having to monitor the schools’ such as teachers may be substitutes did not benefit from the intervention. implementation of the textbook rather than being complementary routine, which would not be in place in the learning production function. The incentive scheme cost for most schools if the intervention Similarly, the intervention had a relatively little and ranked higher were scaled up. positive and larger impact among in cost effectiveness than other students in classrooms with similar interventions. The cost- THE DEMOCRATIC REPUBLIC OF CONGO 7 The reason why the incentives had a positive impact may have of education and strong French language skills than are required WHAT WERE been because of changes in students’ and teachers’ attitudes by manual jobs. This may help to explain the increased test scores THE LESSONS to textbooks and students’ more ambitious job aspirations. Both if the students became more LEARNED? motivated to learn in order to have the teachers and the students in better job prospects. One significant challenge in the the treatment schools reported implementation of this intervention that they were more likely after the The intervention increased the was the need to monitor schools’ intervention to say that textbooks proportion of students who compliance with the textbook are useful for learning, whereas either participated in or passed borrowing routine. During the first those in the control schools the national exam at the end of monitoring visit, the observers found had no change in their opinions. primary school. The treatment that seven of the 45 treatment Furthermore, students in the schools, where the financial and schools had failed to implement treatment schools had begun to non-financial incentive scheme the routine at all so no textbooks aspire to qualify for non-manual was implemented, had roughly 10 had been taken home, though jobs that would require higher levels this improved in the second year percent more students passing the national exam at the end of grade six of implementation. In any future implementation of this kind of than the control schools. This exam intervention, a simple system of tests students on math, the French monitoring should be designed that The intervention language, and general knowledge is owned and managed by individual and is a requirement for students increased the proportion to continue on to secondary school. schools to ensure that teachers comply with the routine as intended. of students who either This result is primarily driven by a higher proportion of students participated in or passed taking the exam rather than a higher the national exam at the end of primary school. proportion of test participants who passed the exam. It may be that CONCLUSION students became more confident and Providing students and schools with motivated to take the national exam a set of financial and non-financial in order to advance to secondary incentives was effective in school because they had begun encouraging students to take home to aspire to jobs that require more textbooks and in increasing their The intervention used education than manual labor. While French language test scores. This both financial and non- the test scores on the national exam suggests that there may be an were not significantly changed by important role for RBF mechanisms financial incentives to the intervention, this may be partly to play in increasing the use of motivate students to due to the fact that more marginal existing classroom resources students may have been induced to without the need for large additional check out textbooks. take the exam who otherwise would resources. Because this kind of not have, thus reducing the average intervention is both low in absolute test scores and offsetting any gains cost and relatively cost-efficient, it made by the students who would may be particularly useful in fragile have taken the exam anyway. and conflict-affected settings such as the DRC and other low-income remain about the effectiveness of and level of textbooks may be a countries where education systems textbooks countries where students’ barrier to achieving further learning are severely financially constrained. ability to read and understand the gains. Therefore, the results of this These results also suggest that language of instruction may be evaluation should not be interpreted RBF incentives to encourage more limited, these results suggest that as suggesting that the use of extensive use of learning materials encouraging more extensive use teaching materials is more important may reduce educational inequality by of textbooks can help to increase than their effectiveness but rather boosting learning among the weakest learning even when the language and that the two should be viewed as students and those with the least level of the learning material may complementary. Further research effective teachers. not be optimal for the students. On will be needed to establish the most the other hand, the lack of impact effective ways to increase learning The results of this evaluation also on math test scores suggests that in more complex subjects like math begin to explain the ineffectiveness the effectiveness of existing learning and science and to explore the of just distributing textbooks with materials remains an important importance of the level and language no other initiatives. While concerns constraint and that the language of the teaching material. 1 http://www.worldbank.org/en/topic/fragilityconflictviolence/brief/harmonized-list-of-fragile-situations 2 http://data.uis.unesco.org 3 Altinok, Nadir, Noam Angrist, and Harry Anthony Patrinos (2018). “Global Data Set on Education Quality (1965-2015).” Policy Research Working Paper, no. 8314. World Bank Group, Washington D.C. 4 IMF (2015). “Democratic Republic of Congo: Selected Issues.” IMF Country Report No. 15/281, International Monetary Fund, Washington D.C. 5 Bold, Tessa, Deon Filmer, Gayle Martin, Ezequiel Molina, Brian Stacy, Christophe Rockmore, and Jakob Svensson (2017). “Enrollment without Learning: Teacher Effort, Knowledge, and Skill in Primary Schools in Africa.” Journal of Economic Perspectives, 31, pp. 185–204. 6 UNESCO (2014). “Education for All Global Monitoring Report.” UNESCO. 7 Glewwe, Paul and Karthik Muralidhan (2016). “Improving Education Outcomes in Developing Countries: Evidence, Knowledge Gaps, and Policy Implications” in Eric A. Hanushek, Stephen J. Machin, and Ludger Woessmann (2016), Handbook of the Economics of Education, pp. 653-743. Elsevier. 8 Read, Tony (2015). “Where Have All the Textbooks Gone?: Toward Sustainable Provision of Teaching and Learning Materials in Sub-Saharan Africa.” Directions in Development-- Human Development, World Bank, Washington D.C. 9 Pritchett, Lant and Deon Filmer (1999). “What education production functions really show: a positive theory of education expenditures.” Economics of Education Review, 18, pp. 223-239. 10 Lockheed, M. E. and E. R. Hanushek (1988). “Improving Educational Efficiency in Developing Countries: What Do We Know?” Compare 18, pp. 21-38. 11 Sabarwal, Shwetlena, David K. Evans, and Anastasia Marshak (2014). “The permanent input hypothesis: the case of textbooks and (no) student learning in Sierra Leone.” Policy Research Working Paper, No. WPS 7021, World Bank, Washington D.C. 12 Glewwe, Paul, Michael Kremer, and Sylvie Moulin (2009). “Many Children Left Behind? Textbooks and Test Scores in Kenya.” American Economic Journal: Applied Economics, 1(1): 112-35. 13 Glewwe, Paul, Michael Kremer, and Sylvie Moulin (2009). “Many Children Left Behind? Textbooks and Test Scores in Kenya.” American Economic Journal: Applied Economics, 1(1): 112-35. 14 Kremer, Michael, Conner Brannen, and Rachel Glennerster (2013). “The Challenge of Education and Learning in the Developing World.” Science, 340, 297-300. PHOTO CREDITS: Cover: “Education in the Democratic Republic of Congo” by GPE/Guy Nzazi, license: CC BY-NC-ND 2.0 Page 3: “A student in lower secondary school open to blind students” by GPE/Guy Nzazi, license: CC BY-NC-ND 2.0 Page 4: “Education in the Democratic Republic of Congo: Mboga primary school” by GPE/Federico Scoppa, license: CC BY-NC-ND 2.0 Page 6: “Boys and girls of Kamilabi primary school welcoming the CEO of the Global Partnership for Education” by GPE/Claire Horton, license: CC BY-NC-ND 2.0 Page 7: Project photo courtesy of the World Bank RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) worldbank.org/reach REACH is funded by the Government of Norway through NORAD, the Government of the United States of America through USAID, and the Government of Germany reach@worldbank.org through the Federal Ministry for Economic Cooperation and Development.