Results in Education for All Children (REACH) ANNUAL REPORT 2018 Acknowledgements: This Annual Report was prepared by Samer Al-Samarrai, Jessica Lee, Kesha Lee, and Minna Mattero. Contributions were also made by Kathy Chen. Photo credits: World Bank/Flickr Acknowledgements: This Annual Report was prepared by Jessica Lee, Kesha Lee, Minna Mattero, and Marie Tamagnan. Contributors include Samer Al-Samarrai, Omar Arias, and Patricia da Camara. Photo credits: World Bank/Flickr Contents Results-Based Approaches for Financing Education: . . . . . . . . . . . . . . . . . . . . . . 1 Mobilizing Resources to Improve Student Outcomes Leveraging Results-Based Financing to Strengthen Education Systems. . . . . . . . . . . . .5 Growing the Global Evidence Base: Knowledge, Learning, and Innovation Grants. . . . . . 13 Contributions to the RBF Knowledge and Evidence Base . . . . . . . . . . . . . . . . . . . . . . . . . . . 15 Examples: Impacts of REACH KLI Grants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 20 Progress Update on Knowledge, Learning, and Innovation Grants By Call . . . . . . . . . . . . . . . . . . 26 Progress Update on Additional REACH Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 26 Spreading the Word on What Works and What Doesn’t . . . . . . . . . . . . . . . . . . . . 27 Collaboration with the Global Partnership for Education . . . . . . . . . . . . . . . . . . . . . . . . . . . . 28 Knowledge Products and Dissemination . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Just-in-Time Support . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 Update from REACH Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 33 The Path Ahead for REACH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Generating Knowledge and Filling in Gaps. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Disseminating Knowledge and Building Capacity . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 Providing Operational Expertise on RBF . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 35 REACH Financial Statement, 2015–19 (in US$). . . . . . . . . . . . . . . . . . . . . . . . . . 36 Results Framework 2016–2018. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 Annex 1: Description of Knowledge, Learning, and Innovation Grants. . . . . . . . . . . . .39 Annex 2: Additional Examples of Results from Closed REACH KLI Grants . . . . . . . . . . 48 FIGURES 1: New IBRD/IDA Commitments for Education (US$ millions), 2015–18 . . . . . . . . . . . . .2 2: Countries Receiving Country Program and Knowledge, Learning, and Innovation Grants . 4 3: REACH Website Traffic . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 31 TABLES 1: Status of Knowledge, Learning, and Innovation Grants . . . . . . . . . . . . . . . . . . . 15 REACH Financial Statement, 2015–19 (in US$). . . . . . . . . . . . . . . . . . . . . . . . . . 36 Results Framework 2016–2018. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 38 BOXES 1: REACH Grants . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 2: Indonesia . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 3: Mozambique . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22 4: Tanzania . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 23 5: Peru . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 24 6: Bangladesh . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25 REACH ANNUAL REPORT 2018 i ii RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Results-Based Approaches for Financing Education: Mobilizing Resources to Improve Student Outcomes Developing countries are investing heavily Countries may not be allocating resources in education and providing their youth equitably; funds may not be reaching schools with unprecedented access to schooling. or may not be used for their intended Globally, governments, households, and purpose; decisions on the use of funds may donors are estimated to spend a combined not be aligned with learning objectives. $4.7 trillion on education each year. And Government agencies may also lack the spending continues to increase at a relatively capacity to use funds efficiently. Whatever the rapid rate. Average government spending cause, the resulting spending efficiencies can on education, for example, doubled in real be substantial. For example, unequal teacher terms between 1999 and 2015. Development deployment and high rates of teacher assistance has also begun to increase after a absenteeism can each lead to spending period of relative stagnation, although it still inefficiencies equivalent to 15 percent of a represents a small fraction of total spending. country’s overall education budget. Despite this growth, countries will have Results-based financing, which rewards to make greater financial commitments in individuals or institutions after agreed- coming years to achieve national education upon results are achieved and verified, goals and ensure that all students learn. gives policy makers and donors the tools Low- and middle-income countries are to tackle such inefficiencies in the educa- expected to need an additional $191 billion tion sector. It does this by strengthening to $311 billion in total each year to achieve incentives to make better use of education quality universal primary and secondary funds, by driving improvements in moni- education and to meet the other education toring and evaluation, and ultimately, by Sustainable Development Goals by 2030. sharpening the system focus on expanding access to quality education for all children. Even if countries are able to mobilize the One example can be seen in the use of needed resources, they will need to do much teacher incentives. Teachers play a critical more with these funds to achieve the SDGs. role in determining whether and how Recent research shows relatively low returns children learn, and their salaries make up to increases in public education funding about 80 percent of education budgets in terms of improving access and learning in low-income countries. Yet in many of outcomes in basic education. Many factors these countries, teachers lack the subject contribute to the low spending efficiency: REACH ANNUAL REPORT 2018 1 knowledge or pedagogical skills to teach FIGURE 1: New IBRD/IDA Commitments effectively, while some even fail to show up for Education (US$ millions), 2015–18 for class regularly. By linking teacher remu- 4500 neration to learning outcomes or other clear goals, results-based education encourages 4000 teachers to expend more effort in the class- room and to put a greater focus on student 3500 learning, ideally increasing the efficiency of 77% 3000 spending and the quality of education. US $ MILLIONS 2500 REACH has been at the forefront of efforts to identify the ways in which results-based 2000 financing can strengthen education systems 58% and improve education outcomes. It has 1500 61% helped countries to develop reliable 1000 data systems and build other capacity in 25% government agencies and schools needed 500 to effectively implement this financing approach. It also has worked to expand the 0 2014/15 2015/16 2016/17 2017/18 knowledge base on the ways governments Total IBRD/IDA commitments and development partners can use results- Percentage of commitments that are results-based based financing to improve education Source: World Bank Projects Database. outcomes and move closer to achieving the Note: Percentages in Figure 1 show the proportion of Sustainable Development Goals for education. development financing that is linked to results. IDA = International Development Association; And REACH has supported the shift in IBRD = International Bank for Reconstruction and Development development assistance towards a greater focus on results. It has primarily assisted The three main activities under REACH teams in the World Bank and under the that have supported this shift are: Global Partnership for Education (GPE) ■■ Country Program Grants (CPG). REACH to design and implement results-based provides large funding packages for financing projects. It has done so by building results-based financing schemes that the knowledge base on the use of this strengthen country systems in specific financing strategy in education, by providing IDA1 countries. direct support to RBF projects through its Country Program Grants, and by assisting ■■ Knowledge, Learning, and Innovation many countries to build the capacity they (KLI) Grants. REACH provides smaller need to utilize results-based financing. Over grants for results-based financing schemes the lifetime of REACH, the proportion of to IDA and IBRD2 countries that contribute development assistance for education that is to the global evidence base. focused on results has grown tremendously; at the World Bank, results-based financing 1 The International Development Association (IDA) is the part of the accounted for three-quarters of new World Bank that helps the world’s poorest countries by providing loans (called “credits”) and grants for programs that boost commitments in 2017/18, compared to only economic growth, reduce inequality, and improve people’s living conditions. IDA countries are low-income countries that qualify for a quarter in 2014/15 (Figure 1). IDA assistance. 2 The International Bank for Reconstruction and Development (IBRD) is the original World Bank institution. It provides loans and other assistance primarily to middle-income countries. IBRD countries are middle-income countries that qualify for IBRD assistance. 2 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) ■■ Capacity Building and Learning around Results-Based Financing. REACH organizes capacity-building and learning events and maintains a website Box 1: that consolidates global knowledge on results-based financing. It also provides REACH Grants Just-in-Time support, an advisory service on results-based financing, to World Bank REACH currently funds results-based financing teams and clients working on projects activities through large country program grants with RBF components. (CPGs) to Nepal and Lebanon and through REACH has helped build the evidence base smaller Knowledge, Learning, and Innovation around results-based financing by funding Grants that finance activities in 22 countries. In research that looks into previously unex- each country, a World Bank task team super- plored topics and subtopics related to this vises the respective grant, with team members financing approach. Some of these grants, based at the Bank’s headquarters in Wash- even if small, have had a major impact, ington, D.C. and in-country. While the country whether by paving the way for countries to grants leverage more money, the knowledge scale up experimental programs, becoming grants are generally attached to larger World integrated with larger development projects, Bank work programs to ensure that the funded or by providing policy makers with new tools, activities are closely aligned with the country skills, and knowledge that can be applied client’s national education strategy. beyond the education sector to other areas of government and policy making (see REACH only funds activities that receive section on "Impacts of REACH KLI grants"). support from the client government and that These grants comprise part of a larger have the potential to be scaled up, ensuring global trend that is putting more emphasis that the funds invested will pave the way for on results in development financing, with long-term systemic change. the findings from REACH-funded activities contributing to a growing base of evidence on how results-based financing can help to strengthen education systems. REACH ANNUAL REPORT 2018 3 FIGURE 2: Countries Receiving Country Program and Knowledge, Learning, and Innovation Grants 4 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Leveraging Results- Based Financing to Strengthen Education Systems REACH provides Country Program Grants (CPGs) to countries that demonstrate country commitment to use RBF, readiness to implement a large program, and strong technical merits in their proposed activities, in addition to suffering from a clear financing gap in their education sector. To date, REACH has approved two country grants, to Nepal and Lebanon, totaling US$4 million each through Recipient-Executed Trust Funds.3 Below are summaries of the grants, their status, and their achievements to date. 3 A Recipient-executed Grant is a Trust Fund Grant that is provided to a third party under a grant agreement, and for which the Bank plays an operational role - i.e., the Bank normally appraises and supervises activities financed by these funds. REACH ANNUAL REPORT 2018 5 Country Program Grant: Nepal Snapshot Summary: Nepal School Sector Development Program (SSRP) Project Development Objective Increase access to and improve quality of school education, particularly (PDO) basic education (Grades 1–8) for children from marginalized groups World Bank Board approval date September 22, 2009 Listed closing date July 15, 2019 (for the GPE grant) REACH grant closing date November 30, 2018 Total project cost: 2.635 billion USD IDA grant: 58.5 million USD Borrower/Recipient: 1.885 billion USD Financing Other donors: 691.5 million USD Additional Financing: GPE grant: 59.3 million USD REACH grant: 4 million USD (Recipient-Executed Trust Fund) 7 million students, 200,000 teachers, 28,000 community schools around Beneficiaries the country Status of PDO indicators: The net enrollment rate (NER) for basic education (Grades 1–8) reached 91 percent (target: 85 percent) Completion rates for primary (grades 1–5) and basic education increased to 81 percent (target: 79 percent) and 70 percent (target: 66 percent), respectively Gender parity in net enrollment rate was achieved at all levels–primary, Results basic, and secondary (grades 9–12)–with some yearly fluctuation The national assessment of student achievement (NASA) for Grades 3, 5, and 8 was completed (target: completion) The net enrollment rate for primary education was 97 percent (target: 99 percent) An equity strategy was developed and was in the process of implemen- tation (target: completion and implementation); it will be continued in a successor program. REACH-Funded Disbursement- Total Value Total Disbursed Status Update Linked Indicator* DLI 4: Independent verification of US$2 million US$2 million Achieved EMIS data** DLI 5: Strengthening of financial US$2 million (pending management capacity at the school US$2 million Achieved internal clearances) level *A disbursement-linked indicator (DLI) is a results indicator to which a monetary sum is linked. When that indicator is met and externally verified, the pre-determined sum attached to that indicator is disbursed. **EMIS = education management information system. 6 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Progress Update To support a stronger focus on results, The World Bank has been working in Nepal the first school sector development plan on education for more than three decades, (a World Bank project) was designed but when the country decentralized its as traditional investment loan with education sector in late 2017 as part of a disbursement-linked indicators. Results- broader transition to a federalist govern- based financing was not completely new to ment, REACH’s country program grant was Nepal: The World Bank transport team was caught in this shift. This generated some implementing a Program-for-Results (PforR) challenges, but the ongoing transforma- operation, and the Asian Development Bank tion of Nepal’s education sector allowed (ADB) had had success with other results- REACH to play a pivotal role in guiding based financing projects in the country. After reforms. Specifically, REACH funding implementation began on the Bank’s educa- helped to substantially strengthen and tion program, stakeholders discussed adding validate existing evaluation and account- a REACH grant and Global Partnership for ability systems while the country was going Education variable part, and they became through fundamental change. part of an additional financing package. Nepal has made great strides in education Shortly thereafter, the government began despite its relative poverty and periods of planning for the implementation a Program- instability. Yet challenges remain, with nearly for-Results for the next school sector reform 8 percent of primary school-aged children phase. A critical factor in determining the out of school and the figure increasing for success of such projects is an accurate higher grades. The country lacks a system- education- data system. Nepal had such atic way to measure learning achievements systems in place, but it had never tested according to internationally acceptable stan- them for rigor; for example, its education dards, while available assessments show only management information system (EMIS) modest improvements in learning outcomes. used self-reported data that had never been The government has sought to address independently verified. REACH funding these issues through a school sector reform incentivized the government to take a closer program focused on expanding access to, look at its information systems and make and improving the quality of, education. necessary changes so that both the current REACH ANNUAL REPORT 2018 7 and upcoming education reform programs tive actions, which the Ministry of Education would be able to capture data accurately reviewed. This resulted in the government and efficiently. organizing targeted training in a few outlier districts that had reported unreliable data. Specifically, REACH funded two disburse- The study also found that the EMIS unit in ment-linked indicators: one that focused on the Ministry of Education had checks and validating Nepal’s education management balances in place to minimize errors in data information system (DLI 4) and a second collected at both local and central levels. that sought to strengthen financial manage- ment at the school level (DLI 5), where the Confirming that Nepal has a strong educa- quality of record-keeping tended to be very tion management information system can low. Another reason for targeting these serve two purposes. First, policy makers two systems: A Public Expenditure Tracking can make better decisions knowing they Survey (PETS) study4 had identified both have access to reliable, regular, and high- areas as in need of improvement. coverage data. Second, the system can boost efficiency in the longer term as both Nepal achieved the goals under Disburse- the government and donors can rely on ment-Linked Indicator 4 (DLI 4) in 2017, with country-level data. The EMIS unit is also an independent agency verifying the results introducing other initiatives to strengthen by spot-checking a random sample of 1,563 data collection in Nepal. These include schools to confirm that their data entry was the establishment of a web-based system accurate. The agency also produced a report to facilitate local government data input, that included recommendations for correc- with in-house checks to prevent errors. It is also taking the lead to set up a national 4 Public Expenditure Tracking Surveys were designed to identify missing government information on public spending and focuses school identification system, which will allow on collecting micro-level data. They have been primarily used in the health and education sectors. it to eventually link internal and external 8 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) databases and build a more consolidated, for an impact evaluation to assess the effects efficient, and transparent school data of the results-based financing projects. system. This will also help reduce inefficien- This funding provided vital resources to cies in data entry. All of these activities are central authorities to analyze whether and the direct result of REACH’s support to how results-based mechanisms might work Nepal through Disbursement-Linked Indi- under a newly decentralized system where cator 4 (DLI 4). the incentives for various stakeholders had changed considerably. While the impact Nepal has also met the goals under evaluation proved to be impossible to Disbursement-Linked Indicator 5 (DLI 5), conduct given the new government struc- which seeks to bolster Nepalese schools’ tures, other activities related to results-based financial management capacity. An indepen- financing were implemented. dent agency has verified the results, and the funds are waiting to be disbursed. The initial These include the administration of three intent was to provide training to 500 schools surveys of teachers, head teachers, and in basic financial management, including about 200 local leaders, respectively, on accounting for receipts and payments, inven- the type of support they felt would be most tory documentation, and the preparation of important during decentralization. The year-end financial reports. However, imple- purpose of the surveys was to explore how mentation was delayed due to uncertainty incentives could be used to secure needed arising from an October 2017 announcement support. For example, head teachers said that the government would decentralize the they were worried that local leaders may education system in mid-2018. not pay attention to poor students; incen- tives could encourage schools to support In the end, the training proved timely amid such students. Overall, the survey findings the sweeping changes taking place under showed that results-based financing could decentralization. With three-quarters of still play a key role under a decentralized Nepal’s education budget suddenly flowing education system, with incentives designed to local officials to manage, central authori- to encourage specific, desired behavior by ties and donors were concerned about how local governments. This type of information the money would be used. The REACH- proved invaluable to the Government of funded training gave school officials tools Nepal and helped officials think through how to track spending, providing basic account- results-based financing could be applied and ability on the use of the funds. As a result, reworked to fit the new country context. other donor partners provided additional funds to expand the program to 1,000 While the transformation of Nepal’s educa- schools. REACH and other stakeholders plan tion sector created some unexpected to review the results of this effort with the challenges, ultimately the team was able to Ministry of Education and local leaders in keep the government focused on fulfilling late March or early April 2019 to decide how the disbursement-linked indicators and to best to design future financial management help it initiate systemic reform of two impor- trainings for schools. tant country systems. In addition, Nepal has been chosen as a country of focus for the Besides funding the two disbursement-linked upcoming Country Assessments work, which indicators, REACH also provided financing will delve more deeply into the overall RBF for supervision and technical assistance and picture in the country. REACH ANNUAL REPORT 2018 9 Country Program Grant: Lebanon Snapshot Summary: Reaching All Children with Education in Lebanon Support Project (RACE-II) Promote equitable access to education services, enhance quality of Project Development Objective student learning, and strengthen the education systems in Lebanon's (PDO) education sector in response to the protracted refugee crisis World Bank Board approval date September 27, 2016 Listed closing date February 23, 2028 REACH grant closing date December 31, 2020 Total project cost: 204 million USD IDA credit: 100 million USD Financing Lebanon Syrian Crisis Trust Fund: 100 million USD REACH grant: 4 million USD (Recipient-Executed Trust Fund) Beneficiaries 540,000 public school students and staff Status of PDO indicators: Increase in the proportion of school-aged Lebanese and non-Lebanese children (3–18) enrolled (baseline: 400,000; current: 488,000; target: 500,000) Increase in the proportion of students passing their grades and transitioning to the next grade (disaggregated by school type, grade, nationality, and gender) (baseline: 0%; current: 0%, target: 4%) Results Timely and robust data available for evidence-informed policy making and planning (baseline: misalignment in data collection and data management functions across different ministry and Center for Educational Research and Development (CERD) units; current: Disag- gregated enrollment data for calendar year 2017–18 published before March 1, 2018; target: data and information management framework developed and adopted by Ministry and CERD; student enrollment data by March 1; passing rates data by August 31) REACH-Funded DLI Total Value Total Disbursed Status Update Baseline data for impact DLI 3: Teacher performance US$1 million (as an US$4 million evaluation to begin this measured and evaluated advance) calendar year Progress Update organizations to assist the small Mideast The flood of refugees from war-torn Syria nation to reform its education sector so that has put enormous strain on Lebanon and it can better accommodate the refugees and its limited resources and public services. better serve its own children. While REACH The education sector is no exception, with is providing a sliver of the overall funding, about 220,000 Syrian children attending it is playing an outsized role in Lebanon’s Lebanese schools alongside a similar number first foray into the use of results-based of Lebanese children, and with at least that financing, helping authorities to develop the many more refugee children not in school. fundamental knowledge, mechanisms, and The World Bank has joined other international processes needed to make it work. 10 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Specifically, REACH is working with use the results of impact evaluations for Lebanese education officials to evaluate evidence-based policymaking. Two studies and strengthen the country’s support system are being conducted under DLI 3. The first for teachers, who arguably carry the most study consists of introducing coordination weight in the classroom. The exercise is mechanisms between teacher trainers and also incentivizing and supporting capacity coaches to align both functions and create building in research and in the use of a feedback loop to improve the provision of evidence in policy making–valuable skills for support materials to teachers. To analyze the Lebanese policy makers and experts to have impact, the project will measure teachers’ that can be applied beyond the education and coaches’ satisfaction with the support sector. REACH has also played a critical role program and their knowledge of differenti- in building consensus among international ated instruction. The team expects to start and domestic stakeholders to focus on collecting baseline data in March 2019. results in designing the aid program. The second study focuses on the coaching The REACH activities are part of an interna- system for teachers. Lebanon already tional effort to support the second phase sends observers to sit in on classrooms and of Lebanon’s education reform initiative, provide feedback to teachers. But such visits called the Reaching All Children with are not systematized, nor do observers use Education (RACE-II) Sector Plan. The World standardized tools to observe teachers or Bank and other donor funding is tied to to give feedback. This intervention seeks to nine indicators. REACH funds are tied to test the impact of a systematized coaching Disbursement-Linked Indicator 3 (DLI 3), approach on teacher performance. REACH which calls for measuring and evaluating supervision and technical assistance funds teacher support programs meant to have been used to build the government’s enhance teacher performance. The full $4 capacity to conduct impact evaluations, and million REACH grant will be paid out upon the project team is working with New York completion of this indicator. University’s Global TIES for Children research As with Nepal, REACH also provided funds center and the Lebanese government to for supervision and technical assistance to design a “toolbox” of standardized class- the government to design, conduct, and room observation and coaching techniques REACH ANNUAL REPORT 2018 11 for expected rollout in September. For had to manage these inter-country politics both studies, teacher performance will be but hopes that the impacts highlighted measured through a test and performance of by the two evaluations will help serve as a a task before a jury of experts. foundation for starting a policy dialogue with the two agencies, bringing them together to The findings are expected to help inform work on improving support for teachers. policy makers on how to strengthen teacher training and coaching practices, with the REACH funds are already having an impact. goal of boosting performance. Findings This is the first time that the Ministry of from the first study could show the effec- Education or any Lebanese government tiveness and cost-effectiveness of two agency is conducting an impact evaluation, types of training follow-up: face-to-face and the World Bank has been providing meetings with the trainer versus virtual technical support to officials on topics such follow-ups using computer or mobile phone as research methodology, randomization technology. For the second study, adding and sampling, thematic capacity building, a coaching toolbox shouldn’t materially and literature review. These skills will increase classroom observation costs, improve the government’s ability to evaluate but the results of the intervention could the effectiveness of programs and ensure demonstrate the effectiveness of specific that resources are used in a more efficient observation and coaching tools, providing a and effective manner. Ultimately, this type of way to standardize and scale up such prac- capacity can help Lebanon institutionalize tices. More importantly, the support through good practices and strengthen country REACH is building capacity within the systems, leading not just to a stronger Ministry of Education to undertake impact teacher professional development frame- evaluations and use the evidence from work, but also to new and better ways of evaluations of this kind to make better deci- doing things across the government. sions on interventions to improve teaching and learning. As with any project undertaken in a fragile state, challenges have complicated imple- mentation. Besides a delay caused by a request from the UK government, one of the international donors, to conduct an enhanced due diligence assessment of the larger program, Lebanese policy makers required an enormous amount of capacity building to help them understand how to design and use impact evaluations. Mecha- nisms for coordination among different government departments didn’t exist so had to be created. Another issue arose when the two agencies in charge of training and coaching, respectively, resisted working together on one study, requiring the team to create two different studies. The team has 12 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Growing the Global Evidence Base: Knowledge, Learning, and Innovation Grants The objective of the Knowledge, Learning, work that investigates less-researched facets and Innovation (KLI) grants is to expand the of teacher incentives and their impact. evidence base on results-based financing REACH-funded studies have looked at how in education. All of the grants provided by performance-based contracts influence REACH can be categorized into six themes: teacher recruitment in Rwanda; how simple (i) Results-Based Financing and Teachers; teacher incentive schemes compare to (ii) Results-Based Financing and Students complex ones in how they affect inequali- and Families; (iii) Results-Based Financing ties in learning outcomes in Tanzania; and Schools; (iv) Results-Based Financing and whether an evaluation tool for use in and Governments; (v) Results-Based classroom observation was appropriate to Financing and Information and Data Systems; measure teacher performance for a results- and (vi) Results-Based Financing and the based scheme in China. Book Chain (which are detailed below). Results-Based Financing and These six themes focus on how results- Students and Families based financing can use incentives to shift the behavior of key actors, including school Research to date has indicated that student leaders, teachers, students and their families, and family incentives, such as conditional and central and local governments. Two of cash transfers, have a good track record of the categories examine how results-based reducing school dropout and increasing financing can be applied to influence these school attendance, though the evidence different actors in two key areas: information is more mixed in terms of its effects on systems and book chains. student learning. However, much less is known about the effects of directly Overall, REACH has increasingly taken a incentivizing students through financial or strategic approach to issuing and supporting non-financial means, although a limited grants, choosing projects that can fill crucial number of studies have shown that this gaps in the evidence base. It then helps to practice can have positive impacts on disseminate the findings from these projects learning outcomes. To better understand to practitioners, policy makers, and the wider the effects on students, REACH has funded development community to strengthen research in the Democratic Republic of understanding of results-based financing. Congo, Mozambique, and Tanzania to explore whether incentivizing students has Results-Based Financing and Teachers a greater impact than incentivizing other Since a large body of literature on teacher stakeholders such as parents or schools. incentives already exists, REACH has funded REACH ANNUAL REPORT 2018 13 Results-Based Financing and Schools Results-Based Financing and There is not much literature available on Information and Data Systems how performance-based school grants are While information and data systems don’t designed and implemented. Evidence shows technically qualify as a “stakeholder” in the that school grants that are conditional on education system, they play a critical role student attendance have generally led to in results-based financing. Proper indicator increased attendance and completion rates. selection, monitoring, and verification are Now interest is growing over how such all crucial to the success of results-based grants can be used to incentivize schools financing programs. Practitioners and policy to focus on learning. Many projects are makers alike want to better understand how including this component despite that little to identify good indicators and how to accu- evidence exists on how performance-based rately track them. REACH has funded grants school grants impact learning outcomes. in Colombia, Haiti, Niger, the Republic of Congo, and Vietnam to determine how To address this dearth of data, REACH has countries can set up reliable and efficient funded research to explicitly examine the information and data systems and use them effects of performance-based school grants to support results-based programs. in Cameroon, the Democratic Republic of Congo, Indonesia, and Mozambique. Results-Based Financing and the Book Chain Results-Based Financing and The book chain in many countries fails to Governments achieve its main objective of getting books Currently, one of the most underdeveloped to children and ensuring that teachers use research areas in results-based financing them in classrooms. Many of the problems and education involves the application and arise because actors across the book chain, impact of such an approach at the subna- from authors to publishers to local distribu- tional, or meso, level, including provincial, tors, lack incentives to fulfill their role in district and regional governments. Even the chain. Results-based financing has not as the majority of studies tends to focus commonly been used to address these on frontline actors such as teachers, it is incentive problems in the book chain. A also important to examine the role of local major focus of REACH has been to explore authorities in education and the potential how conditional financing and other incen- ways that conditional financing can be used tives can be used ensure that education to influence their behavior. This is especially systems reliably deliver the books that true for countries with decentralized political children and teachers need to improve early systems, where a goal is to ensure that local grade learning outcomes. governments implement national policies. REACH has provided grants to Bangladesh, REACH has funded projects in Cambodia, Colombia, the Dominican Republic, India, South Africa, Zambia, Sierra Leone, Bangla- Morocco, and Peru to look at how incentives desh, Mozambique, and Nepal that seek to impact subnational levels of government. use results-based financing to resolve bottle- necks along the book chain. These projects have sought to incentivize increased produc- tion of books in local languages and the timely delivery of books, among other goals. 14 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Contributions to the RBF Knowledge and Evidence Base For brief descriptions of all grants, please see Annex 1. Historical grant information can also be found in the REACH Annual Report for 2015, 2016, and 2017, and on the REACH website www.worldbank.org/reach TABLE 1: Status of Knowledge, Learning, and Innovation Grants Grant Rate of Original Listed Status Amount Disbursement Initiation Closing Closing of Deliverable Grant Name (USD) (as of Jan 17, 2018) Date Date Date* Grant Status Update RBF AND TEACHERS RWANDA: Pay- Received for-Performance policy brief for teacher $195,272.95 100% 7/1/2015 6/1/2016 10/1/2017 Delayed and expecting None recruitment and full paper in retention March, 2019 TANZANIA: Does the design of pay-for- perfor- Policy Note mance schemes $204,903.03 100% 7/1/2015 6/1/2016 10/1/2017 Closed None available matter for student learning gains? CHINA: Building the foundation to incentivize Policy Note teacher training $199,801.46 100% 1/1/2016 11/1/2017 10/1/2017 Closed None available institutes to deliver better teachers INDIA: Creating Legal issues a valid teacher Received surrounding assessment for $120,000 98% 1/1/2016 10/1/2016 12/1/2018 Delayed evaluation the activity future pay-for- proposal remain performance unresolved schemes RBF AND STUDENTS AND FAMILIES MOZAMBIQUE: Testing demand- Policy Note side incentives $198,996.86 100% 7/1/2015 6/1/2016 9/1/2017 Closed None available to keep girls in school TANZANIA: Do students who set goals for them- Policy Note selves perform $198,820.62 100% 7/1/2015 6/1/2016 3/1/2017 Closed available None better? Or do March 2019 they require financial incen- tives? REACH ANNUAL REPORT 2018 15 Grant Rate of Original Listed Status Amount Disbursement Initiation Closing Closing of Deliverable Grant Name (USD) (as of Jan 17, 2018) Date Date Date* Grant Status Update RBF AND SCHOOLS DEMOCRATIC REPUBLIC OF CONGO: Do schools that Policy Note $209,125.00 100% 3/1/2016 6/1/2018 7/1/2018 Closed None receive RBF available subsidies perform better than those that do not? INDONESIA: Piloting perfor- mance-based Policy Note $50,000 100% 7/1/2015 6/1/2016 12/1/2017 Closed None contracting in available Jakarta schools (from Call 2) MOZAMBIQUE: Designing Policy Note performance- $127,559.43 62% 7/1/2015 6/1/2016 12/1/2018 Closed available None based school March 2019 grants CAMEROON: Pre-piloting a Policy Note performance- $199,894.21 100% 1/1/2016 6/1/2017 7/1/2018 Closed available None based school March 2019 grant INDONESIA: Evaluating performance- Final paper based contracts $130,000 85%  1/1/2016 6/1/2017 3/1/2018 Delayed due April 30, None (from Call 1) and 2019 performance- based school grants in Jakarta RBF AND GOVERNMENTS MOROCCO: Developing Govern- and piloting ment performance- delays Received draft based contracts $200,000 41% 1/1/2016 12/1/2016 7/1/2018 Delayed have materials between national, prevented regional, and further local education work authorities COLOMBIA: Piloting an $200,000 50% 3/16/2018 12/1/2019 12/1/2019 Active On track None RBF- monitoring system in Bogota 16 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Grant Rate of Original Listed Status Amount Disbursement Initiation Closing Closing of Deliverable Grant Name (USD) (as of Jan 17, 2018) Date Date Date* Grant Status Update DOMINICAN REPUBLIC: Designing performance $200,000 50% 12/1/2017 2/10/2019 8/10/2019 Active On track None agreements to improve district-level performance INDIA: Incen- tivizing and empowering $200,000 28% 12/1/2017 2/28/2020 7/1/2020 Active On track None district-level officers to collect real-time data PERU: Evaluating performance- based career path and $200,000 34% 1/10/2018 10/1/2019 10/1/2019 Active On track None compensation reforms for school leaders BANGLADESH, NEPAL: Under- standing how district education officers make Waiting for $121,000 92% 11/1/2017 12/31/2018 12/31/2018 Closed None decisions in order deliverable to design better targeted RBF interventions in the future INFORMATION AND DATA SYSTEMS HAITI: Developing a Policy Note $199,982.56 100% 7/1/2015 6/1/2016 6/1/2018 Closed NONE quality-assurance available system for RBF NIGER: Additional Developing Received informa- a sustainable $99,982.73 100% 7/1/2015 6/1/2016 6/1/2018 Closed report from tion and monitoring and team analysis evaluation system required for future RBF VIETNAM: Designing a Policy Note predictive set $198,752.96 100% 7/1/2015 6/1/2016 7/1/2017 Closed None available of indicators for future RBF REACH ANNUAL REPORT 2018 17 Grant Rate of Original Listed Status Amount Disbursement Initiation Closing Closing of Deliverable Grant Name (USD) (as of Jan 17, 2018) Date Date Date* Grant Status Update COLOMBIA: Creating an RBF monitoring Policy Note system that $199,826.95 100% 1/1/2016 1/1/2017 12/1/2017 Closed available None covers multiple March 2019 dimensions of education quality REP. OF CONGO: Additional Increasing informa- accountability Received $99,590.00 100% 1/1/2016 9/1/2017 3/1/2018 Closed tion and through open report analysis data to inform an required RBF program for school finance Dropped due to TANZANIA: inability to Using RBF to $200,000 Dropped Dropped Dropped Dropped Dropped Dropped implement improve school originally inspections proposed activity BOOK CHAIN CAMBODIA: Using RBF to increase account- ability and $500,000 74% 7/1/2017 7/15/2019 7/15/2019 Active On track None transparency in the book supply chain SOUTH AFRICA: Testing how incentives work in $500,000 60% 7/1/2017 7/31/2019 7/31/2019 Active On track None a private-public partnership Govern- ment ZAMBIA: Delayed; turnover Incentivizing extension to $500,000 57% 7/1/2017 12/31/2018 4/30/2019 Delayed and more efficient be decided by procure- book delivery March 30, 2019 ment delays BANGLADESH: Using RBF to create more $500,000 73% 4/4/2018 12/1/2019 12/1/2019 Active On track None diverse reading materials in mother tongues SIERRA LEONE: Putting books (and teachers) to $500,000 3% 10/1/2018 12/31/2019 12/31/2020 Active On track None work for better reading 18 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Grant Rate of Original Listed Status Amount Disbursement Initiation Closing Closing of Deliverable Grant Name (USD) (as of Jan 17, 2018) Date Date Date* Grant Status Update MOZAMBIQUE: Delivering the Contract just right books on $500,000 2/6/2019 5/31/2020 5/31/2020 Active On track None issued time to hard to reach districts NEPAL: Uniting schools and publishers to $500,000 10% 9/28/2018 2/29/2020 2/29/2020 Active On track None bring books into the classroom Procure- RWANDA: ment Digitizing books Workshop $1,000,000 Q1 2022 Active issues for quality basic on hold require education resolution Seeking INDIA: Tech- central nology-enabled Govern- transformation Workshop $1,000,000 Q1 2022 Active ment of early-grade on hold clearances textbook supply for activi- chains ties CAMBODIA: High-quality Workshop supplementary $988,000 held 2/20– Q1 2022 Active None texts for young 21/2019 readers NIGER: Chan- Workshop neling storytelling sched- culture into $875,000 Q1 2022 Active None uled for children’s book 4/18/2019 development OTHER The nature of this activity has INDIA: changed Evaluating perfor- to mance contracts Deliverable become between second- $195,345.99 100% 1/1/2016 12/1/2017 7/1/2018 Delayed due June 30, a global chance education 2019 review of providers and the results- Government of based India financing of training and skills programs *Most REACH grants used to be delayed on average by 18 months. To address this problem, the REACH team implemented a rigorous quarterly monitoring system which flagged extension requests and delays to all relevant practice managers. As a result, delays have been reduced to an average of 12 months. The REACH team aims to further reduce delays by not permitting any grant extensions beyond the overall activity’s closing date as listed in the REACH Roadmap. REACH ANNUAL REPORT 2018 19 Examples: Impacts of REACH KLI Grants REACH-funded activities have had a broad impact in their target countries, from Mozambique to Peru to Bangladesh. Five cases studies below give a sense of the range of impacts and the sustained results of REACH-funded activities. For a complete list of examples of the impacts of closed REACH grants, please see Annex 2. 20 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Box 2: Indonesia (grant closed) In 2009, Indonesia began allocating one-fifth of its budget to education expenditures each year, a hefty increase from the past. Yet learning outcomes failed to improve. To try and boost spending efficiency, the World Bank worked with the Special Capital Region of Jakarta (DKI Jakarta) to reform an existing school grants program to make it performance- and equity- based. The program issued a higher grant value to top-performing schools based on students’ national exam scores, and as a result, scores jumped for students attending lower secondary schools (grades 7–10). But it has also had a broader impact on Indonesia’s education sector. With the REACH evaluation providing evidence that the new grant was working, the Ministry of Education and Culture and the Ministry of Religious Affairs are thinking of including a similar results-based element in the national school grants program. Meanwhile, the DKI Jakarta government is assessing whether results-based financing can be used to shift the behavior of other actors. In 2018, it tested the incentives model on teachers, and the World Bank is currently evaluating the experiment’s impact. For teachers, the hope is that perfor- mance-linked incentives will bolster their performance. Indonesia’s approach even holds promise for countries without generous education budgets. With DKI Jakarta schools vying to boost learning outcomes in response to the mere announcement of bonuses, poorer countries could create a competitive environment by using smaller bonuses or by spotlighting top-performing schools or teachers with an honorific recognition. The REACH findings did not deliver all good news, however. While test scores improved for lower secondary school students in DKI Jakarta, they did not for primary school students. A potential reason is because secondary school directors and teachers in Indonesia tend to be more experienced and better qualified than those in primary schools. Even if primary schools wanted to boost learning outcomes to secure the additional grant funding (worth 20 percent of the unconditional basic grant each school receives), leaders and teachers have limited capacity to help students do better. In such cases, performance-based grants might have more impact paired with support initiatives, such as school leadership training or teacher professional development. Another result, related to the issue of equity, underscores the importance of design in performance-based funding. Even though the DKI Jakarta’s revamped grants program sought to level the playing field between high- and low-performing schools by linking incentives to improvement rather than achievement level and by having schools of similar backgrounds compete against each other, learning outcomes still improved more for high-performing schools. Equity is a common challenge in results-based financing, which Indonesian educa- tion officials hope to redress in part through their design of teacher incentives. REACH ANNUAL REPORT 2018 21 Box 3: Mozambique (grant closed) Despite large increases in enrollment rates in lower primary school grades, most children in Mozambique are not completing primary or upper primary school. Only about 10 percent of rural children finish lower primary school by age 10, and only about 14 percent of males and 8 percent of females complete upper primary school by age 19. A 2016 REACH funded grant found that attendance rates for sixth- and seventh-grade girls rose significantly when they or their families received conditional incentives or even when parents only received information about the girls’ attendance. The World Bank is planning a follow-up survey in early 2019 to find out whether the increases in attendance seen in 2016 led to sustained impacts on completion and learning outcomes, as well as their potential effects on marriage and fertility. The interventions–providing a “report card” on student attendance to the parents; awarding cash to the parents if the student attended a set percentage of days; or providing vouchers that could be traded in for school uniforms or supplies to students who attended a set percentage of days–took place over about nine months. It remains unclear whether such short-term measures will have a longer-term effect on these adolescent girls. But the project has already had a broader implication for Mozambique as well as more generally on the practice of using conditional incentives. For Mozambique, the project demonstrated to officials both the potential of targeting incentives on the demand side (channeling funds to reward students or families, not only to pay for school buildings) and the possibility of doing so cheaply (issuing the attendance “report card”). By bringing together different government departments, the project has opened a new potential source of funding for this and other education initiatives by the social protection agency. The World Bank is currently in talks with the government on expanding the voucher program, and at least one other country in Africa is considering implementing a similar program. An impact evaluation of the project, which was financed in part by REACH, also generated lessons on the use of conditional incentives more broadly. For one, it demonstrated the potential effectiveness in certain circumstances of giving conditional incentives directly to children rather than parents. In this case, the practice of issuing vouchers to the girls wasn’t only more palatable politically, but also proved more effective in boosting student attendance. The project also showed that information alone can change behavior. Such an approach could be used by low-income countries and applied to other sectors such as health, for example with text messages being sent to women to remind them about prenatal and antenatal checkups. 22 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Box 4: Tanzania (grant closed) When Tanzania rolled out universal primary education and abolished school fees in 2001, the school system got an influx of students. But even as enrollment surged, learning outcomes fell; a dozen years later, in 2012, nearly 60 percent of secondary school students failed the national exam. As part of the World Bank’s support of government efforts to strengthen the education system, REACH helped to fund a study comparing the effects of issuing teachers bonuses linked to two different learning outcomes. These bonuses aimed to overcome the constraints of teacher motivation and low skill levels. Although the interventions have ended, their impact continues to be felt in Tanzania and in the broader world of results-based financing. The interventions, which rewarded teachers based on levels of knowledge mastered by each student or student performance compared to that of comparably ranked students, respec- tively, led to similar improvements in learning outcomes. Compared to business as usual, students whose teachers participated in the study logged the equivalent of three to four months of additional learning in one school year. Such results got the attention of officials at the Ministry of Local Government, which together with the Ministry of Education, Science, Technology, and Vocational Training, oversees Tanzania’s education sector. In December 2017, the two ministries signed a memorandum of understanding with the World Bank’s local partner, civil society organization Twaweza, to formalize their interest in implementing a potential scale-up of the program. RISE, a UK-based nonprofit group, is funding a much-larger version of the intervention that drastically reduces cost and complexity (such as by measuring student improvements through paper tests versus one-on-one testing) to pilot the potential scale-up. The ministries have also pledged to finance a bonus fund for teachers as part of any expansion. Other Tanzania authorities have expressed interest in applying this approach to parts of the civil service beyond the education sector. The Tanzania study offers several broader lessons on the use of results-based financing. One is the importance of good communication. By keeping lines open and the message simple, the project was able to establish trust among key players, especially teachers–an important factor in ensuring the program’s success. Even while teachers in some countries have resisted conditional incentives programs, Tanzania teachers generally had an overwhelmingly positive response to the study. Study organizers were also strategic in designing the study to make the most compelling argument for a future ramp-up. Instead of focusing on a few provinces, they designed a nationally representative study, sampling districts across the country. With the interventions demonstrating a positive impact in schools throughout Tanzania, the results suggest that the program could work for the nation as a whole. The Government alongside the World Bank is currently preparing a secondary education project where the results of the evaluation are helping to design a teacher recognition award program. REACH ANNUAL REPORT 2018 23 Box 5: Peru (grant ongoing) Peru has achieved some of the world’s fastest growth in overall learning outcomes since launching broad education reforms in 2011. These reforms include a shift in the appointment system for school directors to a meritocracy-based approach, which ties career development to performance with the aim of strengthening the management and organization of schools. Under this new system implemented in 2014, all school directors in basic education are reevaluated every four years and those who pass may stay in their positions while those who do not are demoted to teachers. A second track allows teachers who demonstrate excel- lency to be promoted to school directors. The World Bank is working with Peru’s Ministry of Education under a REACH grant to evaluate the impact of this meso-level change in policy. Initial findings from the study have shown slightly negative effects on student learning in the first year but signs of going back to initial levels (conversion to the mean) in the following year. These results are similar to results in the existing literature which suggests that adjustments to school leadership can affect student learning outcomes negatively in the short term. The second tranche of the REACH grant will examine the reasons behind these findings. In addition to measuring the effect of Peru’s new meritocracy-based system on learning outcomes, the impact evaluation also seeks to provide guidance through its findings on whether and how education officials might tweak the second round of school director reassessments, which are scheduled to take place this year. Specific factors that the evaluation is looking at include the new system’s effect, if any, on teacher turnover (high turnover rates could point to a need for greater communication or coordination with teachers); the appropriateness of leadership skills being assessed in terms of the needs of schools; and the possible impact of keeping a demoted school director as a teacher in the same school, including on students and the new director. The second tranche will also investigate whether schools observe improvements in learning in the longer term. The project team is also researching other countries’ selection process for school directors and conducting an analysis of the results of a 300-hour training program for school directors rolled out by Peruvian education authorities in 2015–16. Although the REACH impact evalu- ation is not expected to deliver final results until June 2019, the project team has already presented initial findings to the government and continues to hold discussions with them on the topic. With such reforms, it is important to continue evaluating the program to ensure that any unintended consequences are addressed right away. 24 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Box 6: Bangladesh (grant ongoing) Books play an important role in promoting literacy, but many developing countries lack suitable, quality books in their mother tongue language. In Bangladesh, nearly half of second graders cannot read and supplemental books are either in short supply or cost too much for the average consumer. To address the shortage especially of indigenous-language books and the low literacy rate, REACH funded a project using a results-based financing approach to incentivize teachers, authors, and other community members to create supplementary reading materials for a book contest and ultimately, for use by children and teachers. Under the Onuperona Book Challenge, contestants attended workshops to learn to use free publishing software Bloom and created books for submission to the contest, which recog- nized winners with cash awards and publicity. A panel of have already selected 30 winning books and will select the remaining 30 winning titles in April 2019. These titles will be made available for digital distribution on the National Reading Hub, an open online portal that provides teaching resources to teachers. The books can be printed on a regular printer, providing an affordable and accessible source for supplemental reading material written in mother tongue languages and reflective of local culture. Project organizer IREX, a global development and education nonprofit group, has also published and distributed 1,300 hard copies of original content books developed through the Book Challenge to schools and libraries, with another 1,300 books planned. The success of the program has attracted the interest of the Ministry of Education’s National Curriculum and Textbook Board, which is responsible for developing the country’s school curriculum. IREX plans to hold a workshop in March 2019 to train board personnel on the use of Bloom, paving the way for potential scale-up. While the project aimed to address a gap in content, IREX also designed it with other aspects of the book chain in mind. By partnering with the Access to Information (a2i) Programme–an initiative under the prime minister’s office that seeks to modernize delivery platforms for services, among other things–IREX secured a broad and free platform for distri- bution to an audience that mattered: teachers. It boosted the chance for scale-up not only by making sure the books were easy and cheap to print out, but also by having the contest include books in Bangla, the national language of Bangladesh spoken by the vast majority of the population. REACH ANNUAL REPORT 2018 25 Progress Update on Progress Update on Knowledge, Learning, and Additional REACH Work Innovation Grants By Call In 2017 the donor committee agreed to extend REACH’s mandate until June 30, Call #3: 2022. The committee approved a specific REACH and Global Book Alliance (GBA) roadmap for REACH activities through All of the grants under this call for this period. These include the following proposals are on track to close on time, activities, in addition to REACH’s standard except for the grant in Zambia, where knowledge sharing and learning efforts: government turnover has delayed book procurement. All grantees have submitted Country Assessments interim reports with progress updates. In November 2018, REACH initiated a procurement process to hire an external Call #4: firm to conduct country assessments, and Meso-Level the team had provided the Terms of Refer- The grants under the meso-level call are ence and the Expression of Interest to the underway, and each grant has produced donor committee for review. Shortlisted a short midterm report on progress. Each firms have until February 28, 2019 to submit grant is expected to close ahead of the their financial and technical proposals. The closing date, which is listed in the REACH REACH team and another World Bank staff Roadmap as the second quarter of 2020. member with expertise in assessments and monitoring and evaluation will then Call #5: score all received proposals and decide REACH and Global Book Alliance on a final awardee. Once the REACH team The grants under this call have been in selects a firm, it will begin work in the three implementation for less than a year and chosen countries: Nepal, Mozambique, and midterm reports will be due towards the Tanzania. A key feature of these assessments middle and end of calendar year 2019. will be to document country-level perspec- tives on results-based financing. Call #6: REACH and Global Book Alliance Intergovernmental Fiscal Transfers REACH sent grants under this call an “intent The concept note for the Intergovernmental to award” email in November 2018, with Fiscal Transfers paper was shared with plans to hold in-country workshops where the donor committee and their feedback potential grantees can finalize their draft incorporated into subsequent versions. The concept notes through April 2019. concept note has since been finalized and undergone peer review. Preparatory work on the global landscape of fiscal transfers is ongoing. A workshop to discuss the study and to agree country case studies and the research plan is/has taking place at the beginning of April and first draft of country case studies are expected in July 2019. 26 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Spreading the Word on What Works and What Doesn’t REACH does not only fund activities that nity of practice both inside and outside the use a results-based financing approach, but World Bank. Some highlights of REACH’s also seeks to build a knowledge base on efforts to promote knowledge include: RBF which can be used by policy makers Building momentum behind results-based and others seeking to apply results-based financing through partnerships: During its financing to education initiatives. In fact, first three years, REACH organized regular, a primary purpose for REACH awarding broadly promoted and well-attended grants is so that findings from these projects expert panels and other events to build can directly contribute to the expanding awareness around results-based financing the body of knowledge on results-based and its advantages and limitations among financing in education and build evidence different audiences. REACH has partnered on how this approach can strengthen educa- with RBF practitioners and researchers tion systems. across sectors to share their knowledge and REACH has three main objectives in views in international events, developing promoting knowledge on results-based momentum in terms of both understanding financing in education: results-based financing and using it. Its partners have included the Global Partnership ■■ Build momentum and share information for Education, Brookings Institution, the U.K.’s with a wide range of audiences about Department for International Development where and how results-based financing (DfiD), Results for Development, Instiglio, has been used successfully the Inter-American Development Bank, the ■■ Bring together a community of International Monetary Fund (IMF), and the practitioners of results-based financing Education Commission, among others. Within in education to share their operational the Education Global Practice at the World knowledge Bank, 77 percent of active operations incorpo- ■■ Build capacity amongst development rate results-based financing components. practitioners to help countries to use Capturing and sharing operational knowl- results-based financing instruments edge: One of the more difficult things to To a large extent, REACH continues to accom- do is to document operational and imple- plish these goals. The momentum behind mentation knowledge around results-based results-based financing continues to grow, financing in education. REACH has been and REACH has established a strong commu- particularly successful in capturing this REACH ANNUAL REPORT 2018 27 type of tacit operational knowledge with its Collaboration with the Global innovative knowledge events and through Partnership for Education technical assistance activities that have allowed the REACH team to gain exposure In 2018, REACH collaborated closely with to many different types of RBF projects. the Global Partnership for Education (GPE) While grants are underway, the REACH to share information about RBF. The World team focuses on acquiring operational Bank Group is the primary implementing knowledge from grantees and task teams as agency of GPE grants, many of which have they design and implement their projects. a results-based component. The knowledge Over the years, REACH has involved most of and learning teams of the two groups the organizations working on results-based communicate regularly, with REACH and financing (see previous paragraph) as well as GPE jointly organizing a number of events a large group of experts in the field to share last year. These include: their experiences. It has made such knowl- February 2018: Panel Discussion among edge publicly available through operational Five Development Organizations about notes, blogs, and videos posted on the Results-Based Financing in Education REACH website. The panel consisted of the African Devel- opment Bank (AfDB), the Inter-American Building a Community of Practice: Particu- Development Bank (IDB), the Global Part- larly through its learning activities, REACH nership for Education (GPE), The World has helped to build a strong community of Bank, REACH, and the Department for practitioners and researchers. These events International Development (DfiD), with have served as a channel to introduce the Independent Evaluation Group (IEG) different players working on results-based presenting its findings on the evaluation financing to one another. REACH continues of the World Bank’s Program-for-Results to connect teams with experts in different financing instrument. areas of RBF as needed and has also been in contact with an estimated 80 to 90 percent The objectives of the event were to: of World Bank task team leaders working on results-based financing. ■■ Share reasons why different development organizations have taken up results-based Building capacity to use results-based financing as a new financing model, financing instruments: The REACH team has especially in the education sector worked closely with other World Bank units, ■■ Understand the differences between their including the operational strategy group chosen approaches and the rationale and the Education Staff Learning Program, behind adopting them on training World Bank teams on how to use the World Bank’s primary results-based ■■ Summarize lessons learned, including financing instrument, Program-for-Results. challenges arising in using results-based These training programs continue to exist financing and practitioners’ responses to within the Bank, and REACH grantees have address these challenges participated as speakers and presenters to ■■ Identify the advantages of adopting talk about their experiences in education. results-based financing generally, as well REACH was also instrumental in ensuring as the benefits of specific approaches that Global Partnership for Education staff were included in such training programs. 28 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) May 2018: Understanding GPE’s Funding November 2018: Panel Session at the Model with a Focus on the Variable Part Comparative International Education Society Guidance (to be issued March 28, 2019) (CIES) Annual Conference (April 2019) Speakers included World Bank task team REACH and GPE submitted a joint proposal leaders who have experience designing and in 2018 for a panel session at this year’s April implementing the Global Partnership for 2019 CIES conference. Speakers will include Education’s Variable Part in Mozambique representatives from Twaweza, a Tanzania- and Ethiopia, and the GPE staff member who led the work on the revised guidance based organization that runs the KiuFunza note. The Global Partnership for Educa- teacher bonus program, and from Rwanda’s tion indicated that it plans to issue the new Ministry of Education; the task team leader guidance on the Variable Part in late March of the GPE grant’s Variable Part in Ethiopia; 2019 to development country partners. World Bank and REACH team members, who will present on Results-Based Financing The objectives of the event were to: in Education: Learning from What Works; ■■ Create a space for colleagues from the and a GPE representative, who will moderate World Bank and Global Partnership the conference. for Education Secretariat to exchange knowledge and build a community of The objectives of the event are to: practitioners ■■ Engage in a panel discussion framed ■■ Build participants’ knowledge on the around the findings of the Results-Based specifics of the Variable Part, particularly Financing in Education: Learning from on how to design, implement, and optimize it What Works to share how results-based financing works in different countries ■■ Consult World Bank task team leaders and contexts, based on input from on the operational guidance around the Variable Part so as to leverage their practitioners and policy makers experience to support its effective design and implementation REACH ANNUAL REPORT 2018 29 Knowledge Products and sented in the Forum with 95 Ministers of Dissemination Education in attendance. The REACH team held smaller events in the headquarters of In 2018, REACH primarily focused on the development agencies of Germany, disseminating the results from grants Norway and the United Kingdom to present that had closed, compiling these findings the findings and receive feedback on the along with additional research on results- REACH RBF review. REACH is planning addi- based financing into one comprehensive tional outreach and dissemination around report, Results-Based Financing in Educa- the review for 2019. tion: Learning from What Works (formerly referred to as the “Compendium”). REACH RBF Around the World also continued to publicize findings and analysis on the subject through a variety of REACH continued to produce policy notes other formats, including policy notes and a under the RBF Around the World series, revamped website. which seeks to answer key questions about the design and implementation of various results-based financing schemes in different Results-Based Financing in Education: country contexts. The policy notes detail the Learning from What Works experiences and impact of closed grants REACH’s main knowledge product in 2018 funded by REACH, with six notes produced was a review of results-based financing in in 2018, for China, Haiti, Indonesia, Mozam- education structured around the stake- bique (incentives to students and families), holders that are the most common targets Tanzania, and Vietnam. Five additional of incentives: teachers, students and their policy notes are expected to be completed families, school, and governments. This by end-March 2019: Cameroon, Colombia, review used a mixed-methods approach Democratic Republic of Congo, Mozam- to assess the existing literature around bique (school grants), and Zanzibar. results-based financing in education. The authors also conducted a qualitative survey REACH website with follow-up interviews with development agency staff. The ensuing report offers a REACH completely redesigned its website comprehensive look at how results-based to make information more easily acces- financing in education has been used and sible and more relevant to both existing what lessons can be learned from opera- practitioners of results-based financing and tional experiences. those interested in learning about it for the first time. The revamped site focuses on Jaime Saavedra, Senior Director of the different aspects of results-based financing, Education Global Practice, presented the with an overview followed by sections on report findings at the Education World interventions (teacher, school, and student Forum in London in January 2019. A panel incentives), book chain, government, and including the Director General of Education challenges. A final section introduces in the Lebanon, the Director of the Educa- REACH and its grants and provides a tion Commission and the Chief Economist database with links to additional resources. of the Department of International Develop- In 2018, roughly 4,600 visitors came to the ment discussed results-based financing at REACH website, from all over the world the Forum. Over 100 countries were repre- (see Figure 3). 30 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) REACH also shares event information and Additional DLI analysis is also in the Learning publications via email with over 250 World from What Works report. Bank staff members working in the educa- tion and other relevant sectors and shares public event information with approximately Just-in-Time Support 5,000 external contacts who have expressed Just-in-Time support is an advisory service an interest in receiving information from that REACH offers to World Bank task teams the World Bank Education group. When that are working on results-based financing. REACH co-hosts an event with other units In 2018, this service mainly focused on or organizations, these bodies also share providing hands-on technical assistance to information on results-based financing with Bank teams that were finalists for the joint their networks and on their websites. call for proposals by REACH and the Global Book Alliance (call #6). Moreover, REACH is making greater efforts to promote its knowledge products during To support the applicants in developing its learning events and presentations. Most their final concept notes, which will serve as World Bank education staff now know about the basis for whether they ultimately receive the REACH program and rely on it to provide the grant, the REACH team provided the regular updates on new research in results- following assistance: based financing. One of the most commonly used REACH products is the Disbursement- ■■ REACH developed a concept note Linked Indicators Library, a complete set template to guide applicants in of all of the disbursement-linked indicators identifying which bottlenecks in different used in World Bank projects from 2008–18. book chain areas could be tackled with results-based financing. FIGURE 3: REACH Website Traffic REACH ANNUAL REPORT 2018 31 ■■ The team held Q&A sessions to allow team also provided a list of World Bank grant applicants to ask questions during projects with teacher incentive components. the concept note stage and distributed Operations Policy and Country Services a questions-and-answers handout to (OPCS) Unit: This unit is undertaking a address common questions. review of the World Bank’s Program-for- ■■ The REACH team met with all of the Results instrument. The REACH team finalists and provided them with detailed consulted this unit while writing the Learning feedback on their concept note. from What Works report and in turn, shared ■■ REACH connected finalists to technical findings from the report with the World Bank experts including former grantees Board of Directors. and experts in results-based financing Pakistan: The Education Global Practice is and book chains, who helped them considering a new social impact bond in strengthen their proposals and address Pakistan, after the launch of the Uzbekistan problem areas. impact bond for early childhood education. ■■ The team supported finalists in organizing REACH provided materials from a past in-country workshops to gather input event on the subject along with technical from local stakeholders and partners assistance to help frame the early thinking on their draft concept notes. REACH around the design of the impact bond. team members also plan to participate in these workshops in 2019 to provide The REACH team has also reviewed and technical assistance and quality-assurance provided advice on the preparation of World oversight. Bank results-based financing operations on improving teacher quality in the Philippines, Other examples of Just-in-Time early childhood education in Guinea, perfor- Support from 2018 include: mance pay schemes in Indonesia. It has also Africa Chief Economist Office: The Chief provided support and advice to World Bank Economist’s office was doing a review of management on the use of impact bonds in teacher pay-for-performance schemes and education and advice on the design of an asked the REACH team for information about impact bonds for early childhood education REACH grants on the topic. The REACH in Uzbekistan. 32 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Update from REACH Evaluation In 2017, an external firm hired by the REACH discussion on results-based financing at donor committee conducted an indepen- the last donor meeting. dent evaluation of REACH to assess the ■■ REACH provided additional details and impact that the trust fund has had to date. analysis about its Country Program Grants The firm produced a report with several and Knowledge, Learning, and Innovation recommendations, which fall into two main grants in the Learning from What Works categories: those addressed to the donor report. Country-level information about committee regarding governance and effec- RBF will also be addressed with the tiveness, and those specifically addressed to Country Assessments work that is being REACH. All of the recommendations were contracted. This Annual Report also discussed in detail by REACH and the donor includes data on the impacts of closed committee during a teleconference and in REACH KLI Grants (see Annex 2). person at the last donor meeting. Several ■■ REACH has provided more detail on items required follow-up, and REACH which activities have served internal and the donor committee have taken the World Bank audiences and which following actions to resolve them. activities are open to the public or ■■ REACH and the donor committee collaborative in nature. In particular, have agreed on a future work plan that REACH has developed a close working addresses the use of remaining funds and relationship with the Global Partnership includes a specific timetable by which for Education (see “Collaboration with activities must close. GPE” section). ■■ In lieu of setting up an interagency group ■■ REACH has instituted a more robust on results-based financing, REACH monitoring system for grants and is has made significant collaborative keeping the donor committee informed overtures to other development partners about delays (see Table 1). such as the Asian Development Bank, ■■ REACH has sought to improve UK’s Department for International communication with donors through Development, the Global Partnership regular quarterly calls and email updates for Education, the Inter-American on activities, for example when it Development Bank, and other publishes new policy notes online or development partners to learn about when it issued the Country Assessment their RBF approaches. Several of these terms of reference. organizations took part in a panel REACH ANNUAL REPORT 2018 33 34 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) The Path Ahead for REACH REACH has been steadily growing the is critical to ensuring that RBF lessons are evidence base in results-based financing and broadly shared. That way, practitioners and has an established reputation as a leader in policy makers will have a source of informa- RBF knowledge in education both within and tion from which they can both model best outside the World Bank. REACH’s grants will practices and learn to avoid pitfalls when continue to test ways in which results-based using results-based financing. REACH financing can encourage better learning is planning a number of events around outcomes, in particular for the most vulner- results-based financing for 2019, including able children. a broader launch of the Learning from What Works report in April and a panel discussion In the future, the trust fund will continue to at the Comparative and International Educa- distill and disseminate knowledge on results- tion Society’s annual conference. In May based financing in education as widely as 2019, REACH will hold a public lunchtime possible by doing the following: seminar with a Knowledge, Learning, and Innovation grantee, and in June, it plans Generating Knowledge to host an event at Human Development and Filling in Gaps Learning Week (which will include all World Bank staff working in education, health, and REACH will continue to focus on collating and social protection). funding research on underexamined topics related to results-based financing in educa- tion, such as detailing country perspectives Providing Operational on this approach, comparing the cost-effec- Expertise on RBF tiveness of results-based financing in relation Increasingly, different groups have to traditional financing, understanding how approached REACH to seek technical performance-oriented intergovernmental support and advice on matters related fiscal transfers work, and examining ways to results-based financing in education. in which results-based financing can unlock REACH plans to maintain and strengthen bottlenecks in the book chain. relationships with external organizations such as the International Consortium on Disseminating Knowledge and Governmental Financial Management, the Building Capacity Education Commission, and UNESCO, along with internal World Bank units such As detailed in the REACH Roadmap, as the Global Partnership for Results-Based dissemination and outreach to a range of Approaches (formerly the Global Partner- partners working on results-based financing ship for Output-Based Aid). REACH ANNUAL REPORT 2018 35 REACH Financial Expense Category Total Commitments Statement, 2015–19 (in US$) PROGRAM MANAGEMENT AND ADMIN Overall $ 2,200,000.00 Knowledge Sharing & Learning Just-in-Time Support REACH Team KNOWLEDGE, LEARNING AND INNOVATION Overall $ 12,461,569.88 KLI Call 1 (FY16) $ 1,603,794.99 KLI Call 2 (FY16) $ 1,423,444.40 KLI Call 3 (FY17) $ 2,060,000.00 KLI Call 4 (FY18) $ 921,000.00 KLI Call 5 (FY19) $ 1,540,000.00 KLI Call 6 (FY20) (estimated) $ 4,000,000.00 KLI Call 6 country workshops (estimated) $ 200,000.00 Compendium $ 113,330.49 Country Assessments $ 450,000.00 Intergovernmental Fiscal Transfers $ 350,000.00 COUNTRY PROGRAM GRANTS Overall $ 9,300,000.00 Nepal DLIs (Recipient-Executed) $ 4,000,000.00 Nepal IE & Supervision^ $ 449,938.56 Lebanon DLI (Recipient-Executed) $ 4,000,000.00 Lebanon IE & Supervision^ $ 450,000.00 Total $ 23,961,569.88 DONOR CONTRIBUTIONS Total donor contributions $ 27,983,889.31 Norway $ 2,763,415.01 Germany $ 13,520,474.30 United States $ 11,700,000.00 Administration fee* $ 286,304.15 Indirect costs** $ 426,473.49 Notes: *Administration fee taken from donor contributions made before July 15, 2015 and not calculated as part of the totals ** Indirect costs are already part of the grant amounts and thus this line is not calculated as part of the totals ^ Reduced from $650,000 36 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Cumulative Disbursements Disbursement FY19 FY18 FY17 FY16 FY15 $ 1,827,577.57 $ 271,776.53 $ 469,599.96 $ 665,115.07 $ 358,821.42 $ 62,264.59 $ 563,194.54 $ 78,782.67 $ 157,452.98 $ 211,008.55 $ 109,462.78 $ 6,487.56 $ 206,798.14 $ 6,621.18 $ 32,733.00 $ 108,636.40 $ 56,946.52 $ 1,861.04 $ 1,057,584.89 $ 186,372.68 $ 279,413.98 $ 345,470.12 $ 192,412.12 $ 53,915.99 $ 5,059,678.91 $ 1,194,931.50 $ 1,743,022.26 $ 1,626,901.14 $ 420,107.34 $ 1,603,794.99 $ 50,960.45 $ 202,479.26 $ 1,011,635.36 $ 338,719.92 $ 1,320,302.72 $ 94,518.52 $ 529,131.00 $ 615,265.78 $ 81,387.42 $ 1,536,534.03 $ 693,891.03 $ 842,643.00 $ 406,302.12 $ 237,533.12 $ 168,769.00 $ 67,656.50 $ 67,656.50 $ 11,758.06 $ 11,758.06 $ 113,330.49 $ 38,613.82 $ 74,716.67 $ 3,691,173.03 $ 2,197,460.27 $ 1,407,314.00 $ 42,289.72 $ 44,109.04 $ 2,000,000.00 $ 2,000,000.00 $ 449,938.56 $ 127,710.02 $ 237,992.00 $ 40,127.50 $ 44,109.04 $ 1,000,000.00 $ 1,000,000.00 $ 241,234.47 $ 69,750.25 $ 169,322.00 $ 2,162.22 $ 10,578,429.51 $ 71,401.13 $ 197,196.09 $ 157,876.27 FY = fiscal year, June 30 - July 1 (Data as of Feb 28, 2019) REACH ANNUAL REPORT 2018 37 Results Framework 2016–2018 Objective / Indicators (cumulative) Baseline 2016 2017 2018 2018 Actual Actual Planned Actual GOAL OF THE REACH TRUST FUND: To support efforts toward more and better education services, especially to those most excluded, by helping country systems focus more sharply on results. RESULT 1. Country systems and capacity for 0 16 22 26 22 RBF in education strengtheneda/ Financial and technical support to Country 1.1  0 2 2 2 2 Programs and Pilots 1.2 Financial and technical support to activities for 0 18 24 40 29 Knowledge, Learning, and Innovation (KLI)* 1.3 Just-in-time support to WBG program teams, 0 6 11 12 12 informing RBF approaches on-the-groundb/ RESULT 2. Global evidence base for RBF in education is developed and made publicly available 2.1 Total number of KLI grant policy notes on RBF 0 0 7 14 6 approaches prepared and disseminated** 2.2 Total number of impact evaluations [ongoing] 0 6 8 10 8 RESULT 3. Evidence about RBF in education is publicly disseminated 3.1 Approach Paper on RBF in education 0 1 1 1 1 published and disseminated 3.2 Number of RBF learning events 0 2 6 10 7 (open to the public) 3.3 Number of external blogs/articles published 0 3 3 7 4 3.4 REACH website operating and regularly 0 1 1 1 1 updated with new knowledgec/ Notes: a. Represents number of countries. Some countries have benefitted from different kinds of REACH support. b. Countries that are not receiving Country Program Grants or Knowledge, Learning, and Innovation grants. c. Including externally generated evidence. * These totals differ from previous reporting due to the fact that some grants were not initiated in the calendar year in which they were approved and due to some grants that were approved but were dropped. This Results Framework has been updated to reflect the total grants that were initiated through that particular calendar year, e.g. 2017 shows the number of grants that were initiated between 2015-2017. **These totals differ from previous reporting as they have now been updated to reflect the total number of Results-Based Financing Around the World policy notes publicly available on the REACH website rather than total number of deliverables received. ^Four papers have also been published in peer reviewed journals and/or through the World Bank’s Policy Research Working Paper series 38 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Annexes Annex 1: Description of Knowledge, Learning, and Innovation Grants Teacher Incentives RWANDA: Pay-for-Performance for Teacher Recruitment and Retention The Government of Rwanda has established a system of performance contracts for public sector employees that allows for performance-based bonuses averaging 3 percent of salary. The KLI grant is funding a study that builds on the existing civil service contracts by introducing a bonus scheme that rewards teachers who score within the top 20 percent of their district on this performance measure with a merit bonus worth 15 percent of their base salary. The research will address two questions that are also relevant to other developing countries: first, whether a pay-for-performance scheme can improve teacher performance and produce student learning gains; and second, how effective are pay-for-performance contracts at attracting skilled and motivated teachers to undersupplied schools, particularly in rural areas? CHINA: Assessment of Teaching Practices for Changes in the Classroom Guangdong Province’s Department of Education has been strengthening its in-service training of teachers to make them more effective in the classroom. The KLI grant funded a pilot intervention to incentivize teacher-training institutions to strengthen their in-service training programs. In the pilot, the Classroom Assessment Scoring System (CLASS) was used to conduct classroom observations of 36 teachers and to assess the strengths and weak- nesses of their teaching practices. Teachers scored high on classroom organization but lower on emotional support and instructional support. The pilot established a baseline of teaching practices in Guangdong and demonstrated that classroom observations can be used as an outcome measure in results-based financing schemes designed to give teacher training providers incentives to change teacher behavior. The results from the pilot, provided that several preconditions are met, can be used to inform the design of RBF schemes in other contexts aimed at establishing performance-based contracts for teacher training providers. INDIA: Improving Teacher Performance through Outcome-Linked Incentives The Bihar state government is considering designing a teacher performance-pay program to improve service delivery in its schools. The incentive program will be based on a teacher composite score designed by the Indian civil society organization Pratham. The KLI grant will fund research to test the validity of an existing teacher assessment and composite score system to inform this work, and, if the assessment is found to be valid, design a pilot for this program. REACH ANNUAL REPORT 2018 39 Student and Family Incentives MOZAMBIQUE: Keeping Rural Girls in School Using Cash, Goods, and Information In Mozambique, the rate of primary school completion is low, especially in rural areas, where only 14 percent of boys and 8 percent of girls finish upper primary school. The KLI grant funded research to test the effect of demand-side incentives on school attendance for girls. Specifically, the study compared the impact of providing girls with tokens to buy school-related items such as uniforms and supplies; of providing households with cash; and of providing households with information about school attendance without any financial or in-kind incentive. The evaluation found that all three interventions significantly increased girls’ school attendance and that providing information alone had a substantial effect on attendance, even without any financial incentive. Given that providing information is less costly and complex than making financial transfers, this may be a promising and easily scalable policy option for governments in developing countries that lack the administrative and budgetary capacity to implement a conditional cash transfer program. TANZANIA: Aligning Teacher Pay with Performance of All Students In 2015 the Government of Tanzania announced its commitment to using innovative approaches to improve long-standing problems with the quality of, and access to, the coun- try’s education system. The KLI grant funded a study that examined the impact of issuing bonuses to teachers linked to the learning outcomes of students. The study compared the impact of rewarding teachers for relative and incremental gains in student learning against rewarding them based on the number of students who passed a defined threshold. The evalu- ation found that both systems raised test scores. The simple teacher incentive scheme that rewarded based on the number of students who achieved specific learning levels improved learning at least as much as the more complex scheme that rewarded teachers based on learning gains. Given the limited administrative capacity in Tanzania and other developing countries to implement complex RBF schemes, the evidence gained from this program suggests that simple incentive schemes that reward learning levels, provided certain critical design features are considered, may be suitable for wide-scale implementation. TANZANIA: Incentives for Students to Stay–and Succeed–in School In Zanzibar, almost half the students entering secondary school drop out without graduating, but the Ministry of Education and Vocational Training hopes to reverse this trend. The KLI grant is enabling the World Bank research team and the Ministry to collaborate on how to design performance-based incentive schemes for students to maximize learning impacts and reduce dropouts at the secondary school level. The project will seek to answer questions about whether individual student targets or teams of students working toward a collective goal is likely to be more effective in incentivizing poor-performing students, and how results- based financing can help overcome psychological barriers that might prevent students from responding to performance-based incentives. This research is expected to inform the body of evidence on how financing demand-side incentives can lead to better results. 40 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) School Incentives INDONESIA: Piloting Performance-Based Contracting in Schools in DKI Jakarta In DKI Jakarta, resource allocation to schools can be inefficient. At the same time, many schools have failed to effectively implement new National Education Standards, which govern such things as student learning, teacher competency, and school facilities. Education officials addressed both problems simultaneously by linking school funding to key performance and competency indicators under the National Education Standards. Under the new performance- based program, all government schools continued to receive the basic grant allocations, but top-performing schools received an added per student bonus equivalent to 20 percent of the basic grant. While the performance bonuses led to improved learning at junior secondary schools, mixed results at primary schools show that incentives do not necessarily benefit all schools. These results suggest several potential avenues for improving the results-based program and maximizing positive impact in the future including how to design more flexible grants and incentives that link local budgeting to existing national standards. MOZAMBIQUE: Learning from Performance-Based School Grants The Ministry of Education has engaged in a far-reaching reform program to upgrade teachers’ knowledge and performance and to strengthen service delivery at the local level. It is seeking to incentivize schools to improve their performance by providing direct financial incentives through a performance-based school grants program, complemented by other interventions aimed at improving local and school management. A school grants scheme is already in place, but has not proved as effective as hoped. The KLI grant will fund the pilot for a revised program based on lessons learned from the existing scheme. The World Bank team will simultaneously support the development of management tools for mid-level managers, in particular school directors and district officers, to enable them to administer the school grants effectively. This intervention will seek to generate knowledge about how conditional school grants and improved supervision by mid-level managers can help strengthen learning outcomes. CAMEROON: Results-Based Financing for Improved Education Service Delivery Building on the success of results-based financing in the country’s health sector, the Govern- ment of Cameroon is keen to experiment with the approach as a tool for increasing girls’ enrollment in schools and improving service delivery in two of its most disadvantaged school districts. The KLI grant will fund a feasibility study and pre-pilot for a performance- based school grants program in the North and Far North. The lessons learned will inform a two-year pilot, which will be scaled up beyond pilot districts if successful. DEMOCRATIC Republic of Congo: Impact Evaluation of RBF Approach in South Kivu The South Kivu provincial government, in partnership with Dutch nonprofit organization Cordaid, has been piloting results-based financing to improve education service delivery. As the end of the pilot approaches, the KLI grant will fund an impact evaluation to assess whether the approach has gener- ated better education results and whether primary schools receiving conditional subsidies have registered better learning outcomes than those that do not receive such subsidies. REACH ANNUAL REPORT 2018 41 INDONESIA: Using Performance Contracts to Address Weaknesses in Schools The Jakarta government has been experimenting with different approaches for linking education financing to results. As part of this, it has introduced a performance and equity school grants program that looks at whether merely announcing an incentive is enough to change school behavior and student performance. The government also plans to introduce performance contracts for schools, using national standards to identify areas of weakness at the school level and to develop school-improvement agreements between the school and the district office. The KLI grant will fund the evaluation of both activities and is expected to provide insights into the viability of using incentives to support the take-up of school improvement plans. Governments MOROCCO: Support for Performance-Based Contracting to Improve Governance The Government of Morocco is keen to use performance-based contracts to bring about education system and governance reform. The KLI grant will support the development and piloting of performance-based contracts among local, regional, and national governments in two regions in Morocco. BANGLADESH AND NEPAL: Improving District-Level Decision Making In many countries, district education officers play an important role in helping schools access resources and function smoothly in general. However, the ways in which district education officers make decisions and how they are evaluated for success is not transparent. The KLI grant will fund the development of a field-based experiment that aims to better understand how district education officers in two countries make decisions. Using mobile phones and gaming technology, district education officers will be shown hypothetical data on schools and be asked to make resource-allocation decisions. The experiment will test assumptions about district education officers in order to generate information on how to better align their decision making with results-based financing principles and to ensure that future results-based interventions targeted at these officers are evidence-based. COLOMBIA: Using School Information to Improve Service Delivery This KLI grant is a continuation of the second-round grant that supported the development of a results-based monitoring system in Colombia by creating a multidimensional set of indicators focused on education quality. The education quality indicators cover teachers and principals; pedagogic and academic measurements; family, school and community; school climate and well-being; administrative performance; and infrastructure and equipment. The new system provides a school-level view of these six dimensions of quality by collecting information from the school director, teachers, parents, and students. In this iteration, the system will be piloted in Bogotá, with the aim of transforming three aspects of decision making at the meso-level: targeting of programs, targeted support for school improvement plans, and allocation of resources. Results-based financing will be introduced to school directors who will receive increased funding if their schools show improvements in quality in these areas. The results of this pilot can be used to evaluate the overall effectiveness of an information system organized around dimensions of quality and results-based financing (instead of the traditional outcomes or inputs approach). 42 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) DOMINICAN REPUBLIC: Linking Funds to School District Performance The Dominican Republic has spent a decade decentralizing the distribution of financial resources and responsibilities to school districts and schools, based on the idea that schools know best what they need. As part of this process, the Ministry of Education is seeking to improve sector management at the meso-level, by linking financial transfers to performance agreements at the district level. The KLI grant will support the design and implementation of these performance agreements. The lessons learned are expected to inform the design and approach for establishing results-based financing mechanisms and performance-based contracts in other decentralized contexts. INDIA: Utilizing Technology to Strengthen Elementary School Monitoring In India, the Jharkhand state government has hired a cohort of resource persons as contract staff on fixed renumeration to visit schools, mentor teachers, and galvanize local partici- pation through school committees. These individuals often receive limited professional development and little oversight. The KLI grant will use results-based financing to support the improvement of quality-assurance mechanisms by empowering and incentivizing these resource persons through on-the-job training and by providing them with technological tools to collect real-time data. This intervention is expected to inform the body of evidence on how incentivizing meso-level actors can lead to increased transparency and better results at the school-level. PERU: Evaluating Performance-Based Career Path Reforms for School Leaders Improving the quality of the education system has been the focus of ongoing reforms in Peru. One aspect of those reforms has been focused on formulating a performance-based career path and compensation program for managerial-level staff in schools at all levels of basic education. The KLI grant will fund an impact evaluation to assess the implementation of a 2014 and 2016 system-wide, meso-level change in policy that primarily focused on improving the management and organization of schools. The results of the evaluation can help inform policy making and strengthen the country’s system for appointing school leaders. Book Chain BANGLADESH: Incentivizing Local Authors to Create Diverse Books This book creation competition seeks to generate incentives for communities to meet the need for diverse materials in mother tongue languages and align with the Prime Minister’s Access to Information project. This intervention will train local content creators on how to create high-quality supplementary readers in mother tongue languages, which can also be made available for digital distribution on the national reading database. The KLI grant will also fund a survey to assess regional and national readiness for a results-based competitive approach to content creation, the implementation of the book challenge, and a cost analysis on the use of results-based financing to support the competitive creation of books. This research is expected to inform the body of evidence on how financing demand-side incen- tives can lead to better results. REACH ANNUAL REPORT 2018 43 CAMBODIA: Enhancing Book Distribution Using Track and Trace Two key challenges to improving supply chains in Cambodia include: transparency of data along the supply chain; and the use of data by government, school-level leaders, and parents to inform decision making and track the delivery and use of books. This intervention will fund the implementation of a Track and Trace, which is a system that helps identify past and current locations of textbooks and supplemental readers along the book chain. Results- based financing will be used to incentivize the use of this data in relieving bottlenecks along the book chain and promote the timely delivery of books to schools. The lessons learned are expected to inform the design and use of Track and Trace systems in other country contexts and the ways that incentivizing actors along the supply chain might improve the timely distribution of textbooks and supplemental readers. SOUTH AFRICA: National Public-Private Storybook Development Initiative After evaluating assessments that showed that many South African children aren’t able to read at the appropriate grade level, the government launched a national campaign in 2015 aimed at improving the reading abilities of all South African children. To support this initiative, the KLI grant will fund the creation of a national public-private working group to establish best practice recommendations for the creation of early-grade storybooks, the provision of technical assistance and capacity building to publishers of early grade story- books, and the distribution of more than 100,000 books to government schools through a pooled procurement process. This intervention seeks to generate evidence on the effec- tiveness of an RBF intervention that incentives each of the three intervention steps, in a program that will ultimately benefit the South African government and children in Limpopo, Mpumalanga, and Kwa-Zulu Natal. The country’s experience is expected to produce useful lessons on how results-based financing can support the establishment of a public-private partnership for book creation and incentivize the development of pooled procurement among various supply chain actors. ZAMBIA: Evaluating Results-Based Textbook Delivery Systems The textbook delivery system in Zambia relies on centralized procurement at the national level followed by the transfer of books to District Education Board Secretaries’ offices at the regional level and final delivery by these offices to schools. This KLI grant will use results-based financing to evaluate whether stipends tied to the delivery of local language textbooks can improve the delivery process and help ensure that books reach their destina- tions. The grant will fund the rollout of two randomized results-based delivery stipends to district offices and private publishers, with the aim of identifying which group would be most effective to target in order to improve textbook delivery and reduce shortfalls at the school level. This project is expected to inform the body of evidence on how financing supply-side incentives can lead to better results and ultimately get the right books to kids in classrooms. SIERRA LEONE: Putting Books (and Teachers) to Work for Better Reading In Sierra Leone, the government, through its national reading program, set out to improve reading levels in grades 1–3 by providing reading books to every student in the early grades. However, a 2014 study in Sierra Leone showed that book availability does not always lead to use as 87 percent of students at the end of grade two could not read a single word of a short passage. This grant will test four low-cost models (professional coaching for teachers, 44 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) peer-based teacher learning circles, school-based coaching, and classroom observations) for improving “effective use” of reading books and supplementary teaching materials in the early grades by utilizing results-based financing approaches and leveraging smartphones and tablets. The interventions will incorporate incentives for all four models and provide evidence on how low-cost teacher support and supervision models might strengthen effec- tive book utilization in the classroom. MOZAMBIQUE: Delivering the Right Books on Time to Hard-to-Reach Districts In Mozambique, heavy rains and other factors often delay book delivery to schools, resulting in a shortage of learning materials for students. The Ministry of Education and Human Development is responsible for procuring books annually and currently uses paper records to track their journey through the education system. The KLI grant will fund a program that seeks to incentivize districts to get books to schools on a timely basis by linking part of their budget payment to book delivery. The project will also digitize the tracking system for books and train district staff to use the system. This intervention can provide evidence on the effectiveness of combining incentives with digital tracking in strengthening accountability and efficiency in book delivery in a low-income setting. NEPAL: Uniting Schools and Publishers to Bring Books into the Classroom Nepal’s School Sector Development Plan has suggested all grade 1–3 classrooms have at least 50 grade-appropriate books in their book corners, yet four-fifths of schools fail to meet this standard. This intervention will offer cash and other performance-based incentives to schools and publishers to help classrooms achieve the 50-book standard. A new digital platform will facilitate communications between teachers and publishers and allow policy makers to track school book purchases. This KLI grant project will test whether technology and conditional payments can unite schools and publishers to achieve a common goal of increasing quality learning materials in the classroom. Information and Data Systems HAITI: Building Capacity for Result-Based Financing Mechanisms in Fragile States The Government of Haiti has increased primary school enrollment by providing tuition waivers to children from low-income households to enroll in private schools. To enhance the Tuition Waiver Program, the government set a goal to provide financial incentives to schools that improve conditions, instruction, and learning outcomes for poor students, while reducing grade repetition and dropout rates. The activities funded by this grant successfully created a quality-assurance system in Haiti that collects information on five dimensions of learning conditions in schools, provides schools with clear standards to meet, and gives them information about their progress toward reaching these standards. The KLI grant enabled Haiti to develop the capacity and systems necessary to develop a functioning results-based financing mechanism that informs policy makers, as well as the World Bank and other future donors, about how to lay the foundations for results-based financing programs in low-income, fragile states. REACH ANNUAL REPORT 2018 45 NIGER: Resolving the Indicator Bottleneck for Results-Based Financing The Government of Niger has access to a great deal of data about its education system and young population. Such data include assessment results, social development indicators, and population census, and household survey findings. The Government is interested in using results-based financing mechanisms in its education system, but the available data in their current form don’t provide the meaningful, usable, and reliable indicators needed by policy makers and donors to accurately gauge learning outcomes. The KLI grant is enabling Niger to work with the World Bank to establish a sustainable monitoring and evaluation system that will help ensure access to more accurate and effective indicators and lay the foundation for future results-based financing operations in education in Niger. The country’s experi- ence is expected to produce useful lessons on how to establish a national data system in a resource-constrained environment. VIETNAM: Are School Traits and Teaching Practices Reliable Proxies for Learning Gains? Vietnam’s Ministry of Education and Training has been improving its teacher performance evaluation system by introducing new curricula and methods of instruction and developing a comprehensive learning assessment system for its general education program. As the country has a large amount of available data on learning, the KLI grant funded research to use existing data to identify the underlying factors that affect school quality in Vietnam. Specifically, the grant funded the development of a tool to analyze and predict which vari- ables are the key drivers of student performance in language and math. The experiment revealed that student characteristics (cognitive ability, physical factors, routines and habits, and school trajectory) and teacher characteristics were the most predictive categories of variables in determining student performance. These findings will help the Government of Vietnam to design and implement RBF incentives, teacher evaluation strategies, and other effective interventions. In addition to generating insights about Vietnam’s education reforms, this project established a predictive model that other countries could adapt to evaluate factors that impact learning in various contexts. COLOMBIA: Development of a Results-Based Monitoring System The Colombian government has been taking a results-based approach to education policy making since 2014, using a “synthetic index of education quality” to measure progress for all basic-education schools as determined by annual, preset targets. However, this index covers a limited number of indicators, and policy makers need more information to make well-informed decisions. The KLI grant will support the development of a results-based monitoring system that covers multiple dimensions of education quality, with the eventual goal of targeting fiscal transfers to areas that are shown to need improvement. This project is expected to inform the body of evidence on how establishing preconditions for results- based financing can lead to stronger incentive systems in countries with nascent data and technology systems. REPUBLIC OF CONGO: Citizen Voice for Education Only a small portion of education funds disbursed by the Ministry of Finance reach schools in Congo-Brazzaville, with leakage believed to be a main culprit. The KLI grant will finance the establishment of an open data system that will allow community members to report on funds arriving at their local school. If the system is effective, it could be used as a blueprint to set up a national results-based financing program to fund schools across the country. 46 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Other INDIA: A Review of the Global Evidence on Results-Based Financing of Education and Skills Training Programs Nai Manzil is the Government of India’s first nationwide pilot of an integrated education and skills training program. The government is currently monitoring the program, with an eye to potential scale up. In order to further inform this decision, the team has agreed to conduct a global evidence on integrated education and skills training programs, particularly those with a performance or RBF modality. REACH ANNUAL REPORT 2018 47 Annex 2: Additional Examples of Results from Closed REACH KLI Grants Teacher Incentives CHINA: Building the Foundation to Incentivize Teacher Training Institutes to Deliver Better Teachers This pilot tested a classroom assessment system (CLASS) among 36 teachers in 12 schools and found it to be an accurate tool for observers to measure teacher performance. The results could potentially be used to inform the design of a results-based financing scheme aimed at establishing performance-based contracts for teacher-training providers, with the ultimate goal of improving teaching quality in China, especially in less developed regions. The China team continues to engage with the government on these issues. TANZANIA: Does the Design of Pay-for-Performance Schemes Matter for Student Learning Gains? (See Box 4) A randomized experiment in 180 schools found that two different teacher incentive programs–a pay-for-percentile scheme and a simple proficiency threshold design–had similar rates of effectiveness. While past projects have shown the former to be more effec- tive, it is more complicated to implement and communicate. In Tanzania, students actually proved more engaged and teachers exhibited more effort under the proficiency incentive, suggesting that it may be a better option for developing countries engaging in large-scale implementation. Student and Family Incentives DEMOCRATIC REPUBLIC OF THE CONGO: Do Schools That Receive RBF Subsidies Perform Better Than Those That Do Not? This experiment–the first known for incentivizing use of textbooks around homework and quizzes, and followed by a rigorous impact evaluation–focused on more than 1,300 fifth- grade students in 90 primary schools. French-language test scores improved, along with textbook usage and student confidence and aspiration. The approach involved giving “stars” to students who took home books, and financial incentives to schools to cover potential book loss or damage. It offers a low-cost way to change classroom routines using existing resources and could also benefit other fragile states. MOZAMBIQUE: Testing Demand-Side Incentives to Keep Girls in School (See Box 3) An impact evaluation of the project, which was financed in part by REACH, also generated lessons on the use of conditional incentives more broadly. For one, it demonstrated the potential effectiveness in certain circumstances of giving conditional incentives directly 48 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) to children rather than parents. In this case, the practice of issuing vouchers to the girls wasn’t only more palatable politically, but also proved more effective in boosting student attendance. The project also showed that information alone can change behavior. Such an approach could be used by other low-income countries and applied to other sectors such as health, for example, with text messages being sent to women to remind them about prenatal and antenatal checkups. TANZANIA: Do Students Who Set Goals for Themselves Perform Better? Or Do They Require Financial Incentives? The study, which focused on 18,000 students (nearly 55 percent female) across 187 schools in Zanzibar, found that students who set goals for themselves saw an increase in effort but not performance. The addition of financial incentives did not change this result, suggesting that better communication is needed with students about their abilities, so they can set realistic goals. These findings will be used to inform current and pipeline World Bank projects in Tanzania and dialogue with its government. School Incentives CAMEROON: Pre-piloting a Performance-Based School Grant The Cameroon Education Reform Support Project, approved by the World Bank Board on May 1, 2018, will support a pilot performance-based school grants scheme targeting 400 schools in Cameroon, with a planned scale-up targeting approximately 3,000 schools during the 2019–24 period. Based on the results of the pre-pilot, the government will seek to improve the efficiency of funding flows to schools. MOZAMBIQUE: Designing Performance-Based School Grants The World Bank played an instrumental role in helping the Ministry of Education of Mozambique to considerably improve the original design of the pilot performance-based grants by providing technical assistance. The government is now likely to consider adding performance-based school grants as a component of a new International Development Association operation financed by the Global Partnership for Education. Governments BANGLADESH AND NEPAL: Understanding How District Education Officers Make Decisions in Order to Design Better Targeted RBF Interventions in the Future This game-based survey activity has successfully increased the overall understanding of how district education officers and local leaders make decisions and how their decision making aligns with decision making by head teachers and teachers at the school level. It has also highlighted ways to bring such decision making more in alignment with principles of results-based financing and equity enhancement. The findings have been used to inform technical assistance being provided by the World Bank team working on ongoing projects in both countries. REACH ANNUAL REPORT 2018 49 Information and Data Systems COLOMBIA: Creating an RBF-Monitoring System That Covers Multiple Dimensions of Education Quality The local government in Cali, and potentially also the Bogota government, will allocate program funding based on the needs identified by schools using this World Bank-designed web-based data collection and display tool. Cali will implement the tool in all schools in the city and use the collected information to redesign its portfolio of programs. The second portion of this grant (the pilot use of the data tool) is still ongoing, so the impacts of the project are yet to be determined. HAITI: Developing a Quality Assurance System for RBF The Quality Assurance System, which measures school quality, developed under REACH has been integrated into an additional financing package for a World Bank project in Haiti, resulting in the system being expanded to 120 public schools and 100 non-public schools. These schools will be receiving grants for the first time in more than a decade. Non-public schools will receive results-based transfers and public schools will receive block grants. The system’s data will be used to help schools prioritize investments. 50 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Results in Education for All Children (REACH) Education Global Practice World Bank 1818 H Street, NW Washington DC, 20433 USA Website: www.worldbank.org/reach Email: reach@worldbank.org