Results in Education for All Children (REACH) Annual Report 2016 Acknowledgements: This Annual Report was prepared by Peter A. Holland, Jessica D. Lee, Kesha Lee and Minna Mattero. Contributors include Luis Benveniste and Patricia da Camara. Photo credits: World Bank/Flickr Contents The Road to Results-Based Financing: Achievements so Far. . . . . . . . . . . . . . . . . . 2 Results-Based Financing: The Path Ahead. . . . . . . . . . . . . . . . . . . . . . . . . . . . 3 Leveraging Results-Based Financing for Systems Strengthening. . . . . . . . . . . . . . . . 5 Country Program Grant: Nepal. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5 Country Program Grant: Lebanon. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11 Growing the Global Evidence-Base: Knowledge, Learning and Innovation Grants. . . . . . 17 Updates on April 2015 KLI Grants. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 21 Updates on October 2015 KLI Grants. . . . . . . . . . . . . . . . . . . . . . . . . . . . 30 Update on REACH for Reading . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 39 Spreading the Word: Building Capacity and Confidence in RBF as a Real-World Tool. . . . 40 Knowledge Sharing and Learning Events 2016. . . . . . . . . . . . . . . . . . . . . . . 42 Knowledge Sharing and Learning Events 2017. . . . . . . . . . . . . . . . . . . . . . . 51 Communications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 Just-in-Time Support. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 52 REACH: Looking Ahead. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 55 REACH Gross Balance Sheet 2015-2017 Updated (in US$). . . . . . . . . . . . . . . . . . . 56 Results Framework. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 57 Annexes. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 58 REACH for Reading Review Committee Guidelines . . . . . . . . . . . . . . . . . . . . 58 REACH for Reading: Evaluation Rubric . . . . . . . . . . . . . . . . . . . . . . . . . . . 60 FIGURES 1 Countries Receiving KLI and CPG Grants. . . . . . . . . . . . . . . . . . . . . . . . . . 2 2 REACH at Every Step of the Policy Process. . . . . . . . . . . . . . . . . . . . . . . . . 3 3 DLI Chain in Lebanon: RACE 2 Program-for-Results. . . . . . . . . . . . . . . . . . . . 13 4 SABER Domains. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17 TABLES 1 Disbursement-Linked Indicators in Nepal School Sector Reform Project. . . . . . . . . 8 2 Disbursement-Linked Indicators for Lebanon RACE 2 Program-for-Results . . . . . . . 15 3 KLI Grants: Expected Deliverables and Timelines . . . . . . . . . . . . . . . . . . . . . 18 REACH ANNUAL REPORT 2016 i Results in Education for All Children (REACH) Annual Report 2016 This Annual Report tracks progress made by the Results in Education for All Children (REACH) initiative during 2016. The report outlines contributions to building the knowledge base around results-based financing (RBF) and developing operational expertise and capacity among World Bank Group (WBG) Education staff and country clients in applying RBF approaches. REACH’s mission is to support efforts toward more and better education services, especially to those most excluded, by helping country systems focus more sharply on results. In line with the WBG’s Education Strategy 2020, REACH supports the institution’s efforts to build evidence on what works for a systems approach to education reforms and investments. It complements the WBG’s Systems Approach for Better Education Results (SABER) diagnostic and benchmarking platform and its growing portfolio of RBF operations in the education sector. To achieve these goals, REACH has established three primary pillars of activities: ■■ Country Program Grants (CPG) – REACH provides larger financing to RBF schemes that strengthen country systems in specific IDA1 countries; ■■ Knowledge, Learning and Innovation (KLI) Grants – REACH provides smaller financing to RBF schemes in IDA and IBRD2 countries that will contribute to the global evidence base; and ■■ Capacity-Building and Learning around RBF – REACH also organizes capacity building and learning events, maintains a website on global knowledge in RBF and provides just- in-time support to WBG teams and clients on projects with RBF components. Demand from WBG teams and country clients alike to better understand RBF and its applications in various contexts is high and growing. Both staff and country clients are keen to use RBF to center policy dialogue on results. REACH has been able to offer concrete support to WBG teams and country clients by providing funding and technical assistance to help countries raise the bar toward better education quality, and reaching highly vulnerable children still excluded from education systems. 1 The International Development Association (IDA) is the part of the World Bank that helps the world’s poorest countries by providing loans (called “credits”) and grants for programs that boost economic growth, reduce inequalities, and improve people’s living conditions. IDA countries are low income countries which qualify for IDA assistance. 2 The International Bank for Reconstruction and Development (IBRD) is the original World Bank institution. It provides loans and other assistance primarily to middle income countries. IBRD countries are middle income countries which qualify for IBRD assistance. REACH ANNUAL REPORT 2016 1 The Road to Results-Based Financing: Achievements so Far In 2016, REACH received additional investment in a series of smaller activities investments from the Government of has allowed REACH to quickly identify Norway and the United States Agency for patterns of research around the world, and International Development with sustained to build relationships with those working on support from the German Federal Ministry for RBF both inside and outside of the WBG. Economic Cooperation and Development. REACH only supports activities that receive To date, REACH has achieved the following: client government support and have the potential to be scaled up; ensuring that ■■ Financing of 20 RBF activities in 16 funds invested will pave the way for long- countries that cover 5 SABER policy term systemic change. domains (see Figure 1). In addition to funding, REACH provides ■■ Generating knowledge around RBF by expertise and advisory services, either from organizing nine formal learning events in-house experience, or by connecting those with over 250 participants, and publishing seeking technical support with leading global on various RBF-related topics through experts. This has been facilitated through web and social media. learning events, just-in-time support, and REACH funds RBF activities through large informal networking with task team leaders country program grants in Nepal and at the WBG and the Global Partnership for Lebanon and through several smaller Education (GPE). The support to teams in activities under the KLI Grants. While the generating knowledge and evidence and CPGs leverage more money, KLIs are also advising clients extends to every stage of the embedded in WBG projects. This strategic policy process, as illustrated in Figure 2. FIGURE 1: Countries Receiving KLI and CPG Grants LEBANON NEPAL MOROCCO CHINA HAITI NIGER INDIA VIETNAM COLOMBIA CAMEROON RWANDA CONGO TANZANIA INDONESIA DR CONGO MOZAMBIQUE TYPE GRANTTYPE GRANT CPG CPG KLI KLI 2 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) FIGURE 2: REACH at Every Step 1 COLOMBIA 2 of the Policy Process Upstream MOZAMBIQUE Policy Preconditions Dialog for RBF 6 TANZANIA Evaluation 3 INDIA Proof of Concept 5 DR CONGO 4 Pilots LEBANON Support to Implementation Support to Design Results-Based Financing: The Path Ahead REACH enters its third year of operations Reading), KLI call Round 4 seeks to with much momentum. Having solidified the narrow the scope within RBF. Some topics conceptual basis, consolidated the team, under consideration are performance- and established the trust fund as the leading oriented school grants, teacher policies/ operational platform for results-based incentives (including monetary and financing in education, the path ahead non-monetary), and proxies for learning consists of targeting specific RBF sub-topics indicators that are suitable for RBF. for knowledge generation, distilling and 2. REACH for Reading (KLI Round 3): disseminating knowledge, and setting up Using RBF to fix the broken supply chains enabling conditions for going to scale. of books will continue to be a REACH Targeting specific areas to grow RBF focus in 2017. In addition to selecting the evidence base: finalists under the first call, which focuses 1. KLI Round 4: In contrast to the open KLI on underserved languages, REACH calls in Rounds 1 and 2, and the single expects to launch a second REACH for thematic focus of Round 3 (REACH for Reading call later in 2017. REACH ANNUAL REPORT 2016 3 Distilling and disseminating knowledge: Week (which will bring together over ■■ Products from KLI Round 1: Beginning 800 WBG staff working in Education, in April 2017, the deliverables from Health and Social Protection & Labor), the first round of KLI grants will be and the next ‘It’s Debatable’ on teacher completed. These will mainly consist of incentives (co-sponsored with the Africa policy notes and other short deliverables Chief Economist Office’s ‘Smackdown’ ready for consumption by a broader program). (See the Spreading the Word audience. In addition, REACH will set up section for more details). a series of public dissemination events to Enabling conditions to take RBF to scale: feature products (potentially by topic) as ■■ Partnerships and leverage: REACH has they become available. established solid relationships with a ■■ The KLI Compendium: Working from number of internal and external partners the products and knowledge garnered through outreach efforts. As a result, from Round 1, REACH will prepare a KLI REACH has demonstrated added value Compendium on RBF that organizes to teams working in countries using IBRD, lessons learned. Given that the KLI grants IDA, and GPE financing and is now the first have supported RBF activities at each stop for operational teams serving clients stage of the policy formulation process, that wish to apply an RBF approach. the Compendium’s structure will likely Thus far, REACH’s small envelope of CPG mimic that process, from upstream policy funds (US$8 million) has been matched dialogue to evaluation (see Figure 2). by US$100 million of IDA (Lebanon), ■■ Learning and training: Building on US$120 million of Trust Fund resources the successes and the positive brand (Lebanon Syria Crisis Trust Fund), and associated with REACH’s learning US$59.3 million of GPE funding (Nepal). and training activities to date, some This spirit of partnership will continue to highlights for 2017 will be RBF on Trial guide REACH activities and amplify the during Human Development (HD) impact of our financial envelope. 4 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Leveraging Results-Based Financing for Systems Strengthening Country Program Grants are selected on the quality secondary education; and (iii) to basis of demonstrating country ownership, improve the capacity of SSRP implementation readiness to implement a large program, agencies and its partners to enhance delivery a clear financing gap, and strong technical and monitoring of educational services and merits of the proposed operation. products. Progress under each component is further described below. Country Program Component 1: Equitable access to and Grant: Nepal3 quality of basic education In early 2016 the Nepal School Sector ■■ The program has made significant Reform Program (SSRP) received additional progress in this component. More financing of US$63.3 million from the specifically, the end-of-period targets Global Partnership for Education (GPE) and have already been met for basic level REACH, with US$21.8 million conditioned net enrollment rates (NER) and national on achieving targets for five pre-identified student assessment completion for disbursement linked indicators (DLIs). grades 3, 5 and 8. The SSRP supports the entire school Component 2: Equitable access, education sector (grades 1–12), benefiting equity, quality and relevance of more than 7 million students and 200,000 secondary education teachers in over 28,000 community schools ■■ The program has made good progress across the country. The program that in this component as well. Secondary REACH supports provides more than education NER has reached 34.7 percent, US$800 million in sector financing to surpassing the end-of-project target of improve access to education and to raise 33.2 percent, and gender parity in NER quality, with a specific focus on children at this level has also been achieved. from marginalized groups. A database of higher secondary There are three main components to the level students receiving government SSRP: (i) to ensure equitable access to and scholarships has been established, quality of basic education for all children in meeting one of the DLIs for FY 2013 –14. the 5 –12 year age group, prepare preschool- Component 3: Improve capacity of SSRP age children through early childhood implementation agencies to enhance programs for basic education, and deliver delivery and monitoring basic numeracy and literacy to youths and ■■ Notable progress in this component adults, especially women and marginalized include teacher redeployment within groups; (ii) to improve access, equity, and 65 districts, distribution of per-capita quality and relevance of secondary education salary grants to support schools with and targets children in the age 13 –16 year underserved students, and provision group, with a view to ensuring access to of professional development training to more than 95 percent of teachers in 3 This section adapted from World Bank (2009 & 2016). Nepal: School Sector Reform Program. Project Appraisal Document and community schools. Additional Financing Project Paper. Washington DC. REACH ANNUAL REPORT 2016 5 The SSRP is the outcome of high-level and financial audits are mandatory for all partnership between the Government of schools, the quality of these audits is not Nepal (GoN) and development partners adequately scrutinized by District Education and demonstrates their commitment to Offices. For these reasons, DLIs were the Paris Declaration on Aid Effectiveness. created to incentivize the Government The partnership has matured over the past to prioritize certain actions, particularly two decades. Twelve donors are supporting around Education Management this program, of which nine are financing it Information Systems (EMIS) and by pooling their funding with that of Financial Management (FM). Government of Nepal, and three through More specifically, the DLIs are set as parallel financing. Pooled donor resources “stretch” targets and were developed constitute more than 90 percent of the donor as a product of extensive consultation financing for the program. The financing with the Local Education Group and for the program accounts for over 95 percent the Government. The five DLIs are of the financing for the school sector, i.e. embedded in the Results Framework are: this program provides a holistic treatment (i) implementation of single subject of the school sector. certification in the School Leaving Certificate and Higher Secondary examinations, Program Development Objectives (PDO) (ii) number of community schools where and key results standardized classroom-based early grade reading assessments for grades 2 and 3 are The Program Development Objective is conducted by teachers, observed by parent to increase access to and improve the representatives, and results are shared and quality of school education, particularly discussed with parents; (iii) development of basic education (grades 1–8), especially for an equity index and its utilization for children from marginalized groups. The key providing targeted support to districts results are: (reflects equity at the basic level) to reduce ■■ Increase in net enrollment rates for basic the percentage of out-of-school children; education (grades 1–5, grades 1–8); (iv) implementation of independent ■■ Increase in primary and basic education verification of EMIS data on sample basis; completion rate; and (v) provision of technical and financial assistance to select schools to improve their ■■ Increase in Gender Parity Index for financial record-keeping and FM capacity. primary, basic and secondary education; REACH specifically finances DLIs 4 and 5. ■■ Completion of National Assessment of Student Achievement (NASA) rounds for Three of the five DLIs fit under the three grades 3, 5 and 8; and indicator categories that GPE requires for its variable tranche funding: equity, ■■ Development of equity strategy and its efficiency and learning. The REACH implementation for basic education. funded DLIs fall under Component 3. Results-based approach Equity: Despite good progress, overall financial ■■ Targeted interventions implemented in 10 management at the central and district most disadvantaged districts according to levels as well as in schools is still weak, and the newly developed equity index; out-of- several audit reports have raised concerns school children reduced by 20 percent in about financial irregularities. While social these 10 districts 6 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Efficiency: to the significantly decreased enrollment ■■ Single subject certification between grade 10 and grade 11. There is implemented in School Leaving no other currently available intervention to Certificate examinations and approved confront this problem, and since failing the for Higher Secondary examinations current SLC (which produces an aggregate score), leads to the large dropout rate, Learning: it was decided that a single subject SLC ■■ Standardized classroom-based Early could certify students’ proficiency in specific Grade Reading Assessment (EGRA) subjects without penalizing them with an for grades 2 and 3 are conducted with aggregate “fail.” parent observation, and results are Attaching DLIs to interventions that unlock shared and discussed with parents in critical bottlenecks, such as changing 3,000 schools/communities the SLC and fixing broken EMIS and FM These DLIs are meant to tackle systemic systems, encourages the Government of issues, for example, the implementation of Nepal to prioritize those activities to ensure the single subject certification in the School that the main challenges facing the Nepali Leaving Certificate (SLC) is a direct response school system are addressed. REACH ANNUAL REPORT 2016 7 TABLE 1: Disbursement-Linked Indicators in Nepal School Sector Reform Project Period 1 Period 2 Stretch Indicator/DLI (January 1, 2016 – (April 1, 2017 – March 31, 2017) March 31, 2018) DLI #1: Target: SLC issued to 2016 and 2017 Single subject certification SLC candidates as per single- policy approved for Implementation subject certification policy higher secondary school of single subject examinations certification in the SLC and higher Allocated secondary school 3,000,000 3,000,000 Amount(US$): examinations (Efficiency Baseline: Single subject certification policy: (i) approved for SLC, but dimension) not implemented; and (ii) not approved for higher secondary school examinations Disbursement (A) Period 1: non-scalable (B) Period 2: non-scalable Rule: for SLC issuance and for approval of policy for higher secondary school examination DLI (A) Period 1: LEG (or SE in (B) Period 2: LEG (or SE Verification the absence of LEG) reviews in the absence of LEG) Process: DOE’s report on achievement reviews the official document of DLIs and SLC mark sheets showing that the single distributed to students subject certification policy for higher secondary school examinations has been approved DLI #2: Target: 2,600 (600 in 2015 and 3,000 2000 in 2016) (cumulative 5,600) Community schools where standardized Allocated classroom-based 3,000,000 3,000,000 Amount(US$): early grade reading assessments for Baseline: 0 0 grades 2 & 3 are conducted Disbursement (A) Period 1: (i) percent of (B) Period 2: (i) percent of by teachers, Rule: achievement of target (i.e. achievement of target (i.e. observed by parent 2,600 schools = 100% of 3,000 schools = 100% of representatives, target) on or before the last target) on or before the last and results are day of the period equal to day of the period equal to shared and discussed the percent of total allocated the percent of total allocated with parents amount to be disbursed; amount to be disbursed; and (Quality/Learning and (ii) minimum of 51% (ii) minimum of 51% of the outcome dimension) achievement to be eligible target achieved to be eligible for disbursement (scalable) for disbursement (scalable) DLI Periods 1-2: (i) DLI Verification Agent reviews DOE’s report Verification on achievement of DLIs and MIS; (ii) DLI Verification Agent Process: conducts sample survey of schools that includes interviews with teachers and parents, review of school-level assessment records, report cards, and records of results 8 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) TABLE 1: Continued Period 1 Period 2 Stretch Indicator/DLI (January 1, 2016 – (April 1, 2017 – March 31, 2017) March 31, 2018) DLI #3: Target 1: Targeted interventions Targeted interventions Development of implemented in 5 most implemented in 10 most the Equity Index and disadvantaged districts disadvantaged districts its utilization for (according to Equity Index) (according to Equity Index) providing targeted Allocated support to districts 700,000 700,000 Amount(US$): (Equity dimension) Target 2: 20% reduction in the 20% reduction in the aggregate number of OOSC aggregate number of OOSC in these 5 districts in these 10 districts DLI Value 2,200,000 2,200,000 (in US$): Baseline Equity index not developed Target 1: Baseline Baseline data on number of OOSC in the 10 districts to be Target 2: collected by DOE by March 15, 2016 Disbursement (A) Target 1: Periods 1-2: non-scalable for targeted Rule: interventions implemented; (B) Target 2: Periods 1-2: (i) percent of achievement of target (i.e. 20% reduction = 100% of target) on or before the last day of the period equal to the percent of total allocated amount to be disbursed; and (ii) minimum of 51% of the target achieved to be eligible for disbursement (scalable) DLI (A) Period 1: (i) Target 1: (B) Period 2: (i) Target 1: Verification LEG (or the World Bank LEG (or the World Bank Process: in the absence of LEG) in the absence of LEG) reviews DOE’s report(s) reviews DOE’s report(s) on on achievement of DLI achievement of DLI detailing detailing the Equity Index; revised intervention plans revised intervention plans for for OOSC and dataset OOSC; dataset listing OOSC listing OOSC brought into brought into basic education; basic education; and the official document (ii) Target 2: DLI Verification showing that the Equity Agent conducts a sample Index has been approved by survey of OOSC (brought into the Recipient; and basic education in intervention (ii) Target 2: DLI Verification districts) that includes Agent conducts a sample interviews with students, survey of OOSC (brought into parents and teachers, and basic education in intervention review of school/learning districts) that includes center records interviews with students, parents and teachers, and review of school/learning center records REACH ANNUAL REPORT 2016 9 TABLE 1: Continued Period 1 Period 2 Stretch Indicator/DLI (January 1, 2016 – (April 1, 2017 – March 31, 2017) March 31, 2018) DLI #4: Target: Verification survey completed by an independent firm and Independent agreed corrective actions Verification of implemented EMIS data (Monitoring and Allocated 2,000,000 accountability Amount(US$): dimension) Baseline: Verification survey is part of the SSR Program monitoring system but not implemented Disbursement EMIS Verification survey completed and agreed corrections Rule: actions implemented (non-scalable) DLI LEG (or SE in the absence of LEG) reviews DOE’s report Verification on achievement of DLI that includes survey firm’s EMIS Process: verification report (based on survey of representative sample of schools funded by SSRP) and DOE’s implementation of agreed corrective actions from the report’s recommendation DLI #5: Target: 500 schools receive school-based FM capacity Strengthening strengthening support of Financial Management (FM) Allocated capacity at 2,000,000 Amount(US$): school-level (Financial Baseline: 0 0 management dimension) Disbursement (i) percent of achievement of target (i.e. 500 schools = 100% Rule: of target) on or before the last day of the period equal to the percent of total allocated amount to be disbursed; and (ii) minimum of 51% achievement to be eligible for disbursement (scalable) DLI (i) DLI Verification Agent reviews DOE’s report on Verification achievement of DLIs and EMIS; (ii) DLI Verification Agent Process: (third party) conducts sample survey of schools that includes interviews with teachers and review of school-level financial record-keeping Role of development partners. The ongoing and GPE. The partners are signatories to SSRP (2009-16) is implemented by the a Joint Financing Arrangement with the Ministry of Education through a Sector Government. Otherwise, the program Wide Approach (SWAp), with financial is implemented primarily using country contributions from the Government of Nepal systems. The experience with the SWAp and a group of development partners, has been a positive one, and is regarded as including the International Development a best practice example in the country. Association (IDA) of the World Bank Group 10 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Country Program Grant: Lebanon4 At the end of September 2016, the World Bank Group approved a financial package for the Government of Lebanon in support of the Reaching All Children with Education (RACE) program. Prepared in five months, the package represents a mix of grants (including from REACH) and a concessional loan from IDA awarded to Lebanon on an exceptional basis. REACH also provided Just-in-time support to the Lebanon team, with the REACH Program Manager serving as a co-Task Team Leader during the preparation of the operation. A.1 Children, youth, and their caregivers are provided with the necessary support to The RACE program (2016 – 2021) that REACH increase their demand for formal education supports is estimated to cost approximately or regulated non-formal education. US$2.1 billion over five years. The REACH A.2 Children and youth have improved grant (US$4 million) is co-financing RACE and equitable access to appropriately along with IDA (US$100 million), the Lebanon equipped public schools, especially in Syria Crisis Response Trust Fund (US$120 underserved areas. million), and many other bilateral donors and UN agencies. The program was developed Pillar II: Enhanced Quality in an open and participatory manner led 2. The second pillar strives for “enhanced by the Ministry of Education (MEHE) and quality of education services and learning including the Center for Education Research environments provided, to ensure grade- and Development (CERD) and other national appropriate learning outcomes for children and international actors. The successes and and youth.” Specifically, activities will lessons of the first phase of RACE (2013 – 2015) contribute to the following three outputs: were incorporated along the way, with the conclusion being a broadening of the scope B.1 Teachers and educational personnel to focus on issues of quality and systems have enhanced capacities to provide strengthening. RACE 2 is therefore structured learner-centered pedagogy in public around three pillars: i) Equitable Access, schools or learning spaces. ii) Enhanced Quality, and iii) Strengthened B.2 Educational personnel at the Systems. Each of these is described below. school-level are capacitated and empowered to proactively contribute Pillar 1: Equitable Access to better school governance and safe/ 1. The first pillar seeks “enhanced access to, enabling learning environments. and demand from, children, youth, and B.3 Communities are capacitated to their caregivers; for equitable formal or actively engage in the promotion of regulated non-formal education.” Within learning and well-being of students in this pillar, there are two outputs: learning spaces. 4 This section adapted from World Bank (2016). Lebanon: Support to RACE 2. Project Appraisal Document. Washington DC. REACH ANNUAL REPORT 2016 11 Pillar III: Strengthened Systems and thus depends on actors in the private 3. The third pillar is focused on “enhanced and non-profit sectors to provide education governance and managerial capacities of services as well. While MEHE and CERD will the MEHE and the CERD to plan, budget, implement many of the RACE 2 activities, deliver, monitor, and evaluate education donors, UN agencies and their partners may services.” This pillar has four outputs: be best placed to carry out some of the demand-side activities that are crucial to C.1 An effective and accurate Education achieving the RACE 2 objectives. Similarly, Management Information System (EMIS) is many of the targets in the formal education established and functional. sector in later years can only be met through support to the non-formal education sector C.2 Revised curricula for schools and learning in the current period, since the particular spaces are developed and endorsed to improve quality learning, life-skills and needs of refugee students require a multi- employability for children and youth. pronged approach. Some children have been out of school for a year or more and need C.3 Appropriate policy frameworks are instruction in non-formal settings to catch- endorsed and implemented to regulate education programs and services, up on subject matter and adapt themselves strengthen school management, and to instruction in French or English, a professionalize teaching services. characteristic of the Lebanese education system. Finally, the RACE 2 strategy is for all C.4 MEHE and CERD at the central children, Lebanese and non-Lebanese, and and regional levels are strengthened to lead and coordinate the planning, it is aligned with the long-term sustainability implementation, and evaluation of needs of the education system as a whole, in the relevant RACE 2 activities. its response to the protracted refugee crisis. Within the Government program, Program Development Objectives (PDO) REACH co-finances the RACE 2 Program and key results objectives linked to formal education.5 The Program Development Objective is The Bank-supported Program objectives to promote equitable access to education are spread across all three pillars, and services, enhance quality of student learning, exclude activities that are exclusively within and strengthen the education systems in the purview of international partners, such Lebanon’s education sector in response to the as UNICEF and UNHCR. Although these protracted refugee crisis. The key results are: activities are not part of the Bank-supported ■■ Increase in the proportion of school aged Program, they continue to be core elements Lebanese and non-Lebanese children of the overall RACE 2 program. Activities (3 –18) enrolled in formal education financed by international partners but (disaggregated by school type, education implemented by MEHE are typically within cycle, nationality, and gender); the scope of the Bank-supported Program. ■■ Increase in the proportion of students The overall RACE 2 program depends on passing their grades, and transitioning to a broad set of partnerships for success. the next grade (disaggregated by school The public education system cannot absorb type, grade, nationality, and gender); and all Lebanese and refugees of school age, ■■ Timely and robust data available for 5 To distinguish the overall RACE 2 program from the smaller evidence informed policy-making Program that the WBG will finance, a capital ‘P’ is used to identify the Bank-supported Program. and planning. 12 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) FIGURE 3: DLI Chain in Lebanon: RACE 2 Program-for-Results I. EQUITABLE ACCESS II. ENHANCED QUALITY 1 Children enrolled in 2 formal education: Students dropout decreased transitioning grade 3 Teacher performance improved 5 Community engaged to support learning 4 Learning outcomes EXAM monitored and improved III. STRENGTHENED SYSTEMS 6 Timely and robust data available 8 Foundational frameworks developed 7 9 National and adapted Capacity of curriculum institutions revised strengthened Program-for-Results (PforR) the Bank-supported Program, namely Program Scope through the use of Disbursement-Linked To achieve the Bank-supported RACE 2 Indicators (DLIs). Program result areas, a broad set of The DLI under Pillar I, “Equitable activities by MEHE and partners are Access” relates to the number of required. These activities are linked to children that are enrolled in formal the expansion of supply of education education. Importantly, the proposed services, improvements in the quality of age range (3 –18 years) extends beyond those services (including student learning the compulsory education band for outcomes), and an increase in the long-term Lebanon (6 –15 years). As agreed ability of MEHE and CERD to manage, through the RACE 2 framework, this is plan, monitor and deliver those services, to incentivize greater enrollment at the according to their institutional mandates.6 Early Childhood Education level (3 –5 RACE 2 was created in response to a crisis years), as well as the secondary level and is not a sector plan, though it forms (15 –18 years). This number represents the foundation of an eventual sector plan, all children enrolled in formal public which would include tertiary education. education. Activities under the Equitable Table 2, on page 15, presents the overall Access pillar include supply-side activities RACE 2 program, and situates some of for expanding the access to formal the key elements being supported through schooling, such as school construction, expansion, and rehabilitation. 6 For more on MEHE and CERD’s institutional mandates, please see Annexes 1 and 4. REACH ANNUAL REPORT 2016 13 There are four DLIs that contribute to There are four DLIs proposed for the Pillar II, “Enhanced Quality”. The DLIs will Strengthened Systems pillar, which will incentivize actions to help children stay in incentivize capacity development and school, measure what they are learning, and foundational policy work to ensure a support teachers to better help students lasting impact of the Program. The first most in need. The first is enhanced Systems DLI is timely and robust data quality of education services and learning available for evidence-based policy-making. environments measured through grade The second Systems DLI incentivizes the completion and transition to higher grades. updating of the national curriculum, for all The second is the improvement of teacher levels of the education system. The third performance through the measurement DLI is linked to the foundational conditions and evaluation of teacher performance for learning, rewarding key policy areas of and provision of support and training. teachers and student assessments, and The third is the measurement of student the fourth strengthens MEHE and CERD’s learning with the view to improve learning capacity to review, evaluate, and update outcomes. Finally, the fourth measures Program implementation. The activities the active engagement of community under this pillar to improve the timeliness members with schools to prevent and accuracy of data collection (which is dropouts and increase transparency. The essential for decision-making by MEHE and activities under this pillar are expected to CERD) include the procurement of data contribute to these results include targeted collection infrastructure including computers support for teachers based on classroom- and increased internet connectivity to all observations performed by MEHE’s Office schools along with training on the data of Pedagogical and Scholastic Guidance collection software and guidelines for data (DOPS) and tailored trainings offered by entry, verification and deadlines. This is CERD. This support and training linked expected to greatly improve timeliness and to specific needs identified through accuracy of education metrics, and allow classroom-observation will be further disaggregation by gender, year in school, connected to student performance in and nationality, among other key variables. formative student assessments. Students In addition, MEHE and CERD capacities identified as requiring additional will be assessed to better understand assistance will receive supplemental current functionality as compared to support. Monitoring and accountability projected needs. Systems strengthening for results will be strengthened by at the central and regional levels will be increased outreach to communities and implemented, including sustained increases stakeholders, particularly parents. This in the numbers of staff in units with current chain of interconnected activities extending shortages, such as accounting, procurement, from teachers through support services projects planning and implementation, and to individual students and outward to CERD departments. CERD will lead the communities will be articulated in school development of a new national curriculum, improvement plans to be supported by a which will include learning objectives for program of grants and other activities to each grade, and together with MEHE, will improve teaching and learning throughout develop and endorse teacher performance the public school system. standards, as well as a comprehensive learning assessment strategy. 14 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) TABLE 2: Disbursement-Linked Indicators for Lebanon RACE 2 Program-for-Results Total financing RACE 2 Program Disbursement-Linked to DLI Result Areas Indicator (DLI) (In US$ Billions) I: EQUITABLE ACCESS DLI#1 Number of Outcome A: Enhanced access to, and school aged children demand from, children, youth, and their DLI#1 (3–18) enrolled in 95 caregivers for equitable formal or regulated formal education in non-formal education participating schools II: ENHANCED QUALITY Outcome B: Enhanced quality of education DLI#2 Proportion services and learning environments DLI#2 of students 27 provided, to ensure grade-appropriate transitioning grades learning outcomes for children and youth Output B.1: Teachers and educational DLI#3 Teacher personnel have enhanced capacities to DLI#3 performance measured 6 provide learner-centered pedagogy in and evaluated public schools or learning spaces DLI#4 Number of Output B.2: Educational personnel at participating schools that the school-level are capacitated and implement formative and empowered to proactively contribute DLI#4 18.4 summative assessments to better school governance and safe/ for students in Grade 3 in enabling learning environments reading and math Output B.3: Communities are capacitated DLI#5 Proportion of to actively engage in the promotion of participating schools DLI#5 15.6 learning and well-being of students in with active community learning spaces partnerships III: STRENGTHENED SYSTEMS DLI#6 Timely and Output C.1: An effective and accurate robust data available for Education Management Information System DLI#6 15 evidence informed policy (EMIS) is established and functional making and planning Output C.2: Revised curricula for schools and learning spaces are developed and DLI#7 Curriculum endorsed to improve quality learning, DLI#7 revised to improve 28 life-skills and employability for children quality of learning and youth Output C.3: Appropriate policy frameworks DLI#8 Foundational are endorsed and implemented to Framework and Policies regulate education programs and services, DLI#8 are developed and 16 strengthen school management, and adopted to support professionalize teaching services teaching and learning Output C.4: MEHE and CERD at the central DLI#9 MEHE and and regional levels are strengthened CERD’s improved to lead and coordinate the planning, DLI#9 3 capacity to plan and implementation, and evaluation of the implement the Program relevant RACE 2 activities REACH ANNUAL REPORT 2016 15 The Bank-supported Program In addition to the IDA credit (US$100 million) beneficiaries are the students, teachers, and grant financing provided through communities of Lebanon, refugees, the REACH and Lebanon Syria Crisis and MEHE and CERD. Total cumulative Response Trust Funds (US$124 million), direct beneficiaries of the Program include other funds through development partners approximately 540,000 public school are estimated to be US$172 million of students and staff over the life of the commitments, covering many of the costs Program. This includes an estimated 500,000 for the next 18 months. It is estimated that students who will benefit from increased about US$150 million could be available for quality in service delivery, some of whom each of the four subsequent years for RACE might have dropped out or who might not 2 from current partners, based on partner have enrolled in school without the Program financing over the last three years in support support, and an estimated 40,000 teachers of the first iteration of RACE. This is a small and school administrators who will receive amount of the approximately US$12 billion support and training through the Program. that was pledged at the London conference MEHE and CERD personnel will also benefit for the Syrian Crisis Response in February from increased capacity. Policymakers and 2016, a portion of which is expected to be education sector stakeholders will benefit allocated to education in Lebanon. from increased availability of data on key Partner financing and activities beyond education system outcomes, among other the scope of the Bank-supported benefits. Investments in infrastructure and Program are also important. For example, system functionality will remain with MEHE cash transfer programs financed and and CERD long-term, regardless of the implemented by partners are expected to duration of the refugee crisis. increase demand for education, which will Role of development partners. The help reach the ultimate RACE 2 objectives Program Expenditure Framework for of enrolling all children. Similarly, technical the Bank-supported RACE 2 Program is support for curriculum development may be estimated at about US$1.8 billion, or about expected to play a role in the achievement US$369 million per year. This is a significant of important results linked to the curriculum. increase in annual program financing over Other support, including research and RACE 1 to reflect greater enrollments and a capacity strengthening activities and greater package of services to students. technical assistance from donors are expected to contribute to achievement of the overall RACE 2 results framework. In terms of coordination, the RACE Executive Committee played a coordinating role during the implementation of RACE 1 between donors, UN agencies, and the Government. The Committee is expected to continue to play this role for the duration of RACE 2 and to enable the incorporation in future years of new financing from other institutions for the currently under-financed RACE 2 goals. 16 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Growing the Global Evidence-Base: Knowledge, Learning and Innovation Grants The objective of the KLI Grants is to expand knowledge beyond the proposed country knowledge about how RBF can be used to context, strategic opportunities for long-term strengthen education systems in low and engagement on the topic, demonstrated middle-income counties. There have been client commitment, and impact on country three calls for proposals to date, two general systems. Results can be broadly defined, calls in April and October 2015, and one including both outputs and outcomes. thematic call, REACH for Reading, with a The projects selected for grants represent submission deadline of November 22, 2016. a range of situations from low-income, Under the first two general calls, 18 grants fragile and conflict-affected countries to were approved totaling approximately middle-income economies across multiple US$3.1 million. One grant that was tentatively regions. While the intention of the first two approved for Zimbabwe was rescinded calls was to be purposely broad in order to when the implementing organization capture more comprehensive knowledge was unable to secure a letter of support around RBF, there are six SABER domains from the government. that naturally lend themselves more to The general calls award up to US$200,000 RBF: School Autonomy and Accountability; per grant and each application undergoes Engaging the Private Sector; School a rigorous selection and interview process. Finance; Teachers; Student Assessment Approval criteria include the technical merits and Education Management Information of the proposed activities, methodological Systems (See Figure 4). The cross-cutting rigor, potential contribution to global themes affect any approach to education, FIGURE 4: SABER Domains LEVELS OF EDUCATION WFD WFC Workforce Development COMPLEMENTARY INPUTS AND TE CROSS-CUTTING RESOURCES Tertiary Education THEMES SF P&S ERA School Primary and Secondary Education Finance Resilience GOVERNANCE Approaches INFORMATION ECD SA&A Early Childhood Development SA School T E&I Student Autonomy Teachers Equity Assessment and and Accountability Inclusion EPS EMIS Engaging SH&SF ICT Education the Private School Information Management Sector Health and and Information School Communication Systems Feeding Technologies REACH ANNUAL REPORT 2016 17 and RBF can also be applied across all levels the first two calls into Compendiums (as of schooling (though REACH’s focus tends to mentioned in RBF: The Path Ahead). Table 3 be more toward basic education). lists all of the grants made during the two At the end of the grant cycle, each research calls launched in 2015, with the knowledge team will contribute a policy note or similar products each research activity is expected knowledge product to contribute to the to generate. More detailed descriptions of growing body of global evidence and each grant activity follow, along with more knowledge on RBF in education, with the information about REACH for Reading, as view to compiling all of the knowledge from the review process is still underway. TABLE 3: KLI Grants: Expected Deliverables and Timelines Amount Mid-term report or Expected Name Approved deliverable(s) due completion (in US$) KLI GRANTS APRIL 2015 HAITI: From Financing Access 200,000 October 2016 June 2017 Results to Learning Results Mid-term Deliverable: Mid-term report Final Deliverable: Policy note detailing approach taken to and lessons learned from introducing RBF in Haiti, outlining implications for establishing RBF in low-income, fragile contexts. INDONESIA: Piloting Performance-based 50,000 n/a June 2016 Contracting in Schools in DKI Jakarta Mid-term Deliverable: Mid-term report Final Deliverable: N/A The research feeds directly into a larger activity funded by the October 2015 grants. MOZAMBIQUE: Learning from 198,000 October 2016 August 2017 Performance-based School Grants MOZAMBIQUE: Keeping Rural Girls in School – Testing the Impact of Cash, 180,000 October 2016 June 2017 Goods, and Information Mid-term Deliverable: Mid-term report Mozambique Joint Final Deliverable: Policy note or case study integrating both approaches to RBF in Education Mozambique, and outlining implications for the global context. NIGER: Resolving the Indicator Bottleneck 100,000 July 2016, July 2017 June 2018 for Results-Based Financing Mid-term Deliverable: Mid-term report Final Deliverable: Policy note detailing the approach taken to and lessons learned from establishing national monitoring and evaluation systems and procedures in Niger, and outlining implications for laying the foundations for RBF in low- income countries with limited exposure to and experience of using data for policy making. 18 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) TABLE 3: Continued Amount Mid-term report or Expected Name Approved deliverable(s) due completion (in US$) RWANDA: Pay-for-Performance for 195,741 October 2016 June 2017 Teacher Recruitment and Retention Mid-term Deliverable: Mid-term report Final Deliverable: Policy note summarizing the findings and outlining the implications for policy makers considering introducing performance-based pay for teachers. TANZANIA: Aligning Teacher Pay with 198,500 October 2016 September 2017 Performance of All Students TANZANIA: Incentives for Students to 200,000 October 2016 March 2017 Stay – and Succeed – in School Mid-term Deliverable: Mid-term report Tanzania Joint Final Deliverable: Policy note or case study integrating both approaches to RBF in Education in Tanzania, and outlining implications for the global context. VIETNAM: Are School Characteristics and Teaching Practices Reliable Proxies for 200,000 October 2016 June 2017 Learning Gains? Mid-term Deliverable: Mid-term report Final Deliverable: Policy note commenting on the potential for replication of this research and the corresponding implications for governments wishing to undertake a similar analysis. KLI GRANTS OCTOBER 2015 CAMEROON: Results-Based Financing for 200,000 July 2016 December 2017 Improved Education Service Delivery Mid-term Deliverable: Copy of the feasibility study including a section on the similarities and differences between service delivery in health and education as they pertain to performance-based contracting. Final Deliverable: A design of the pilot; evaluation strategy for pilot, explanatory note to accompany pilot. CHINA: Assessment of Teaching Practices 200,000 October 2016 December 2017 for Changes in the Classroom Mid-term Deliverable: Implementation plan for the final pilot. Final Deliverable: Policy note outlining the approach taken to and lessons learned from introducing RBF to improve in- service teacher training in China, and outlining implications for policy makers globally considering this type of reform. COLOMBIA: Results-Based 200,000 January 2017 December 2017 Monitoring System Mid-term Deliverable: Mid-term report Final Deliverable: Policy note that details the piloting and use of the results-based monitoring system in Colombia, any subsequent RBF-related activities, and lessons on global implications of this work. REACH ANNUAL REPORT 2016 19 TABLE 3: Continued Amount Mid-term report or Expected Name Approved deliverable(s) due completion (in US$) DEMOCRATIC REPUBLIC OF CONGO: Impact Evaluation of RBF Approach 200,000 January 2017 December 2017 in South Kivu Mid-term Deliverable: Mid-term report Final Deliverable: Policy note outlining globally applicable lessons on using citizen monitoring to verify RBF indicators in education. INDIA: Improving Teacher Performance 50,000 n/a September 2016 through Outcome Linked Incentives Mid-term Deliverable: n/a Final Deliverable: A feasibility study assessing whether the teacher assessment model can be used as the basis for measuring improvements in teacher effectiveness for RBF programs. INDIA: New Horizons 200,000 January 2017 December 2017 Mid-term Deliverable: Mid-term report Final Deliverable: Policy note on the lessons learned from the evaluation of performance-based contracts between agencies and their impact on the skills development of school youth. INDONESIA: Assessing the Impact of Alternative Approaches to Linking 270,000 October 2016 December 2017 Funding to School Performance in Jakarta Mid-term Deliverable: Design for the qualitative evaluation; note detailing the final identification strategy selected for the impact evaluation accompanied by the diagnostics. Final Deliverable: Extended policy brief outlining the major findings of the different approaches to improving the effectiveness of public education resources, including a set of recommendations on potential adjustments and integration of programs; knowledge brief on the approach utilized for discussing the findings in more detail will also be prepared as the foundation for the policy brief. MOROCCO: Support to Performance 200,000 June 2016 December 2017 Contracting in Education Mid-term Deliverable: Mid-term report Final Deliverable: Two policy notes focusing on: i) How to construct a performance-based contract for education system reform; ii) How to use performance-based contracts to align incentives and actors along the service delivery chain. REPUBLIC OF CONGO: Congo Citizen 200,000 January 2017 December 2017 Voice for Education Mid-term Deliverable: Instruments for data collection and communication of data to parents and communities; evaluation design for the activity; proposal for a policy note or other evidence-based knowledge product to be delivered after the completion of the research project. Final Deliverable: Policy note on globally applicable lessons, to be determined at mid-term check in. 20 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Updates on April 2015 KLI Grants Haiti From Financing Access to Learning Results (KLI Call #1) GRANT DESCRIPTION: The Government of Haiti has successfully increased primary school enrollment through a results-based mechanism called the Tuition Waiver Program, which pays schools fees for enrolling poor children in non-public schools. It intends to build on this achievement by providing financial incentives for schools to improve conditions, instruction, and learning outcomes for poor children in these schools, while reducing grade repetition and dropouts. The KLI grant is enabling Haiti to develop the capacity and systems necessary to develop a functioning RBF mechanism so as to inform future Bank and other donor funding, and, more importantly, national policy. Haiti’s experience of establishing these preconditions will generate knowledge about how to lay the foundations for RBF in low-income, fragile situations. The REACH grant helped design a project component that focuses on a data-driven Quality Assurance System, which will serve as the basis for future financing against results. PROGRESS TO DATE: in existing Ministry of Education regulations. Despite the challenging political situation The QAS would provide a means to and the many changes in leadership at consistently track progress across schools the Ministry-level, the team has been in Haiti, support schools in developing able to mitigate implementation risks improvement plans, and measure results of by continuously engaging officials and interventions in the sector. The QAS could introducing them to RBF through a also serve as the technical foundation for a component in a new IDA financed project. renewed school accreditation process. Improving public management of the CHALLENGES AND LESSONS LEARNED: education sector is a key objective of the The main challenge thus far has been project, which centers around a Quality mentioned above, the unstable political Assurance System (QAS) that comprises climate. The team has spent considerable tools to measure key dimensions of school energy towards hosting workshops with learning conditions (including infrastructure, Ministry officials to ensure that they water and sanitation, leadership, pedagogy, understand how results-based financing and others), tools to assess student learning, works, specifically targeting technical staff and corresponding standards, all grounded whose turnover is much lower. REACH ANNUAL REPORT 2016 21 Mozambique Keeping Rural Girls in School – Testing the Impact of Cash, Goods, and Information (KLI Call #1) GRANT DESCRIPTION: The KLI Grant will fund research that tests the effect of demand-side incentives on school attendance for girls. The study compares the impact of providing girls with tokens to buy school-related items such as uniform and supplies, providing households with cash, and providing households with information about school attendance without any financial or in-kind incentive. The research will generate knowledge about whether it is more effective to incentivize students or households, and about whether information is an incentive in itself. PROGRESS TO DATE: with their parents or guardians, so that For baseline sample and experimental nearly all the sample is drawn from the balance, 173 schools were included population of eligible girls enrolled in 2015. which were stratified by district and then CHALLENGES AND LESSONS LEARNED: allocated randomly to the four experimental The end-line household survey will not groups using a random number generator take place before late fall of 2016, but spreadsheet. the team already has attendance data Given the emphasis on intra-household from the first trimester’s attendance spot relationships between parents and checks. Female attendance is highest in the daughters, and the intervention focus on parental incentives group (by 4 percentage the last two years of primary school (grades points for a baseline of 80 percent), and 6 and 7), the household survey targeted significantly so at the 10 percent significance girls who still lived with their parents and level. Attendance is nearly as high in the had attained at least grade 5 and at most girl incentives group, although it is not Ggrade 6 by the end of the previous school quite significantly different from female year (2015). attendance in the control group based on The initial strategy was to sample 20 girls the overall first trimester attendance. Female from each school at random, drawn from the attendance is also higher in the “information two lists proportionally to the size of each only” treatment group than in the control list. The household of each sampled girl was group, although again the difference is not then to be included in the sample provided statistically significant using these early that the girl in question still lived with her data. It is interesting to note that male parents or guardians. In practice, however, attendance, which was not incentivized, nearly all recent drop outs no longer lived is essentially identical across all treatment arms, with differences of less than one 22 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) percentage point, except for a (statistically (conditional on correct and complete filling insignificant) negative coefficient in the in of the attendance report cards) was the parental incentives group. reason behind the increase in attendance The team also observed that attendance observed among teachers in charge of the by teachers in charge of the lesson during specific lesson during which the spot check which the spot check takes place is higher takes place (which includes some principal in the treatment groups, especially in the teachers), then the team should observe parental incentives group. One candid a larger increase in attendance among explanation could be that parents (and, to a principal teachers. Instead, they find no lesser extent, pupils), are more likely to hold effect on attendance for principal teachers. teachers accountable when pupils’ presence In order to further probe the interpretation at school is incentivized, or even simply of the above estimates, and to try and better monitored (through report cards). predict which of these results are likely to One concern could have been that the be reinforced as more data is obtained, the compensation payment (in air time) made team also looked at the same estimates to the teachers in charge of filling in the separately for the first and for the second report cards may create an incentive in itself spot check, for the 160 schools which had for teachers to attend school. However, this two spot checks during the first trimester. interpretation is ruled out by our findings The pattern is clear: as pupil attendance for attendance of the principal teachers. starts decreasing (over time), the treatments These teachers are the ones in charge of start having an effect on female, but not filling the report cards, and the only ones male, attendance, as one would expect entitled to receive air time, directly to their given that male attendance is not monitored mobile phones. If this air time incentive using report cards or incentivized in any way. Learning from Performance-based School Grants (KLI Call #1) GRANT DESCRIPTION: The KLI Grant funds the pilot of a revised school-grants scheme, using lessons learned from an existing scheme to improve the incentive system. The team simultaneously supports the development of management tools for mid-level managers, to enable them to administer the school grants effectively. This intervention will generate lessons learned about how school grants can support improvement in learning outcomes and about the impact of strengthening capacity of mid-level management as part of a school grants program. The Ministry has learned that designing performance-based school grants requires a number of preconditions, such as continuous data availability, strong leadership, and good monitoring and evaluation systems to track progress. PROGRESS TO DATE: of Education and Human Development The team traveled to Mozambique in late (MINEDH) in the revision of the formula for September 2016 to support the Ministry the allocation of School Grants (ADE) to REACH ANNUAL REPORT 2016 23 achieve better education results through systems to track implementation and adding performance-based and equity- oversee expenditures; and (vii) inclusion based allocations. of a mix of results and process indicators, In May 2016, MINEDH created an to support implementation quality and interdepartmental Incentives Committee motivation for more immediate managerial that includes representation from all five changes (in addition to more long-term relevant departments (Quality Assurance, results in learning outcomes). Planning, Finance, Primary Education, and The mission also facilitated a meeting to Human Resources). The Committee has share the experience of using performance finalized the Regulation of Awards, which was based allocation (PBA) funds under the approved by the Minister in September 2016, Public Financial Management Program- providing a framework for the application for-Results Project between officials of of financial and non-financial awards, both the Health and Education ministries. The collective and individual, and for central, Central Medical Stores (CMAM) have provincial, district, and school levels. developed incentives schemes to better manage drug storage at warehouses, CHALLENGES AND LESSONS LEARNED: which have contributed towards meeting At the request of the Ministry, a full key disbursement-linked indicators (DLIs) day workshop was organized to foster monitored through the Project. Next direct exchanges with teams who have steps include sharing further details of the implemented school grant programs in incentive measures that CMAM have used Indonesia and Brazil. Discussions focused in the recent past, as well as proposed on how a new formula for the allocation of plans to improve allocations between school grants could be operationalized in provinces to ensure equity and incentivize the Mozambican context. Lessons learned “improved” performance (not “best from other countries have highlighted performance” alone). the following points as being essential to the success of performance-based school MINEDH will also produce a Technical Note grant models, namely: (i) continuous data about incentives and an addendum to the availability; (ii) legal backing; (iii) socialization current regulation of incentives, based on to promote awareness of incentives criteria; lessons learned from Indonesia and Brazil (iv) strong leadership and assurance of as well as the inputs of the discussions commitment at all levels; (v) targeting with the World Bank. The Technical Note allocations to address structural issues will precisely define the criteria for equity of inequality that contribute to under- grants as well as the selection criteria for performance; (vi) monitoring and evaluation performance grants in ADE. 24 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Niger Resolving the Indicator Bottleneck for Results-Based Financing (KLI Call #1) GRANT DESCRIPTION: The KLI Grant is enabling the Government of Niger, in collaboration with the WBG, to establish a sustainable monitoring and evaluation system, using data from recently conducted or upcoming surveys and assessments. This will lay the foundation for future RBF operations in education in Niger, using more effective indicators and best available data sources. The Government of Niger’s experience in moving towards results-based policy in a resource-constrained environment is expected to produce useful lessons in establishing national data systems. PROGRESS TO DATE: incentives for achieving better teacher Progress has been difficult given that deployment, improving the proportion travel was restricted to Niger in the wake of qualified teachers in schools, and of the terrorist attacks in Mali and the monitoring qualitative education service unpredictable political climate after two delivery indicators. election cycles. However, the team was CHALLENGES AND LESSONS LEARNED: still able to collaborate with the WBG Aside from logistical challenges, the Governance Global Practice on a public technical capacity of government officials sector and service delivery reform project. is relatively strong, but the cross-sectoral The project has a strong focus on work may dilute some ownership across implementation readiness, as well ministries. It will be important to ensure as increased attention on primary that all stakeholders play an active role in school service delivery by incentivizing mutually identifying results and working improvements in resource management. together to achieve them. Specifically, the project will provide REACH ANNUAL REPORT 2016 25 Rwanda Pay-for-Performance for Teacher Recruitment and Retention (KLI Call #1) GRANT DESCRIPTION: The KLI Grant is funding a study which builds on existing civil-service contracts, by introducing a bonus scheme that rewards teachers who score within the top 20 percent of their district. The research will address two questions: firstly, whether a pay-for-performance (P4P) scheme can improve teacher performance, and produce student learning gains. Secondly, how effective are P4P contracts at attracting skilled and motivated teachers to undersupplied schools, particularly in rural areas. This research will generate information about how P4P schemes can alter the demographic spread and skills of teachers. The grant has benefitted from the Rwanda Education Board introducing the requirement of an entry exam for applicants to new teaching positions – this data will provide another source about the impacts of the team’s treatments on the quality of candidates. PROGRESS TO DATE: an opportunity for policy engagement: the A large uncertainty that was resolved since Director General of REB has expressed receiving the first grant disbursement is the interest in having the study inform the design number of districts that would have new of this application exam for the cohort of new hires for the 2016 school year (there were six teachers in 2017 and beyond. districts with sufficient scale) and the level at CHALLENGES AND LESSONS LEARNED: which people would apply – and so the unit One positive outcome is that the team of randomization. Those districts with new ran what appeared to be a successful hires for 2016 conduct recruitment centrally, advertisement campaign, with some so the first stage of randomization (positions promotional materials to spread information advertised as either fixed wage or pay-for- about the P4P and fixed wage interventions. performance contracts) was conducted at Job fairs were attended by more than 200 the district-by-subject of qualification level. potential applicants, whose outcomes will This is a slightly more aggregated unit of be tracked. randomization than some of the alternative scenarios that could have arisen. A second- The greatest challenge is one of statistical stage, re-randomization of teachers into power; because REB advertised positions in contracts conditional on selection will be a limited number of districts, and because conducted at the school level as planned. those happened to be districts to which people apply centrally rather than at the Another positive update is that the Rwanda sub-district (“sector”) level, the team had Education Board (REB) introduced the to use a unit of randomization that is more requirement of an entry exam for applicants aggregate than the advertisement of P4P to new teaching positions. Access to this data vs. fixed wage positions. However, there was should provide another source of information high interest in those positions, so the team about the impacts of the team’s treatments is optimistic about potential effect sizes. on the quality of applicants. It also provides 26 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Tanzania Incentives for Students to Stay– and Succeed – in School (KLI Call #1) GRANT DESCRIPTION: In Zanzibar, almost half the students entering secondary school drop out before secondary completion. The Ministry of Education and Vocational Training Zanzibar intends to reverse this trend. The KLI grant is enabling the research team to generate clear guidance to the Ministry on how to design performance-based incentive schemes for students to maximize learning impacts and reduce dropouts at secondary level. The research will answer questions about whether directly incentivizing students improves their performance. In addition, this research will examine how results-based financing can be designed to help overcome psychological barriers that might prevent students from responding to performance- based incentives. Specifically, this research tests whether helping students to set goals for themselves increases the potential impact of performance-based incentives. This research is expected to inform the body of evidence on how financing demand-side incentives can lead to better results. The greater focus on results and on education quality has allowed policymakers to broaden the use of more creative, low-cost interventions that were originally designed as part of the REACH grant. PROGRESS TO DATE: The team was also hoping to examine The research design has been successfully nuances in the operationalization of implemented as originally proposed, with the ‘independent verification.’ This has been full cooperation of the Ministry of Education postponed due to some delays in the (MoE) and teachers/head teachers. timelines of the broader project, but the plan is still to generate insights on this aspect. The evaluation sample includes all lower secondary schools in Zanzibar (211). The CHALLENGES AND LESSONS LEARNED: team has (i) sharpened the intervention The MoE has been very enthusiastic, design based on discussions with Ministry particularly about the goal setting of Education and numerous focus group intervention. Based on the positive discussions with teachers and head teachers, feedback from teachers and head-teachers (ii) collected detailed baseline data in all 211 in treatment schools, the government is schools, including testing of about 18,000 planning a broader communication and students, and (iii) undertaken follow-up visits outreach campaign aimed at directly to sub-sample of schools. motivating students in crucial grades where REACH ANNUAL REPORT 2016 27 drop-outs tend to be higher. They also which also have intrinsic merit, regardless plan to use the anonymous goal-setting of their impacts on student performance, data to generate insights about the design present scalable and valuable strategies of a guidance program aimed at students for a small and fiscally constrained but with high risk of drop-outs. The MoE feels highly-committed and centralized education that these types of low-cost interventions, system like Zanzibar. Aligning Teacher Pay with Performance of All Students (KLI Call #1) GRANT DESCRIPTION: Building on previous research by the civil society organization Twaweza, the KLI Grant is funding a study that compares the impact of rewarding relative and incremental gains in student learning against rewarding teachers whose students pass a defined threshold, in order to improve service delivery. By continuing to work in this set of schools, the research will provide insights into the “long term” effect of learning in an environment where teachers are paid for performance. The research will generate information about how teacher P4P programs can be best structured to benefit all children. PROGRESS TO DATE: treatment groups, that all modules were Baseline fieldwork was conducted from asked to all relevant respondents, and that February 22 – March 17, 2016. Each field enumerators are not significantly influencing team was composed of two interviewers interviewees responses. and one supervisor; each supervisor was in CHALLENGES AND LESSONS LEARNED: charge of two teams of interviewers (that This year, due to the free education policy is, one supervisor led two teams of two recently implemented by the government interviewers who went to separate schools). of Tanzania, there were some schools that Each set of interviewers spent two days had over 800 students enrolled, which was at each school. Teams were encouraged more than the survey software allowed, to begin with student tests and teacher meaning that only 800 students could be interviews on the first day of a school visit, included in the sample. to ensure that any absent students or teachers could be captured on the second In addition, it was deemed that two day of the visit. Day two was reserved for interviewers had falsified teacher data for remaining interviews that could not be 15 teachers. In these instances, additional completed on day one, along with the phone interviews were conducted and the household and school questionnaires. two portions that had visual components will have to be gathered at endline. Upon receipt of final data, Innovations for Poverty Action (the contracted vendor) did a Otherwise, strong quality assurance quality check to ensure that data was properly processes were in place to ensure proper collected, key indicators are balanced across capture of data. 28 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Vietnam Are School Characteristics and Teaching Practices Reliable Proxies for Learning Gains? (KLI Call #1) GRANT DESCRIPTION: Vietnam has a large amount of data available about learning across the whole system. The KLI Grant is funding research that will use existing data to establish the underlying factors that affect school quality in the country. The results will inform the redesign of the general education teacher performance evaluation system using evidence about what works in Vietnam. The research will both generate knowledge about Vietnam and establish a model that could be adapted by other countries to evaluate which factors impact learning in their unique contexts. PROGRESS TO DATE: CHALLENGES AND LESSONS LEARNED: The team has advanced quite a bit in The challenge is whether or not the most selecting and preparing the Young Lives important predictive variable will be subject survey database for their analyses of “over- to intervention and therefore amenable performance” and “under-performance” to results-based approaches, though this on math and reading. The team also challenge is beyond the team’s control. The consolidated and cleaned the useful data weights/importance of some immutable and re-coded it so that it can be used in aspects of student background, such as the predictive model. Thirdly, they have living in rural communities, is currently developed the predictive analysis software larger than other factors such as teacher and plugged in the data. Currently, the feedback on homework. However, the predictive model correctly classifies 90 team will use a variety of methodologically percent of students by test score based on sound approaches for the investigation and background variables. Finally, the team has continues to believe that this approach will commenced discussions with partners in prove robust and be a means for making Vietnam establishing the best methods for results-based projects based on previously disseminating our results later in 2017. unused data. REACH ANNUAL REPORT 2016 29 Updates on October 2015 KLI Grants The majority of the KLI grants awarded in October 2015 are still not at mid-term stage, thus only progress to date is reflected in the descriptions. Cameroon Results-Based Financing for Improved Education Service Delivery (KLI Call #2) GRANT DESCRIPTION: Building on the success of RBF in the health sector in Cameroon, the government of Cameroon is keen to experiment with RBF as a tool for increasing girl’s enrollment and improving service delivery in two of its most disadvantaged districts. The KLI Grant funds a feasibility study and pre-pilot for a performance-based school grants program in the North and far-North of the country. The lessons learned will inform a two-year pilot, which will be scaled up beyond pilot districts if successful. The pre-pilot being financed by REACH will pave the way to bring the performance-based school grant scheme to scale, potentially reaching over 400 schools. PROGRESS TO DATE: A national team has been set up to action plan triggers a first disbursement conduct the pre pilot with consultants and into the school’s budget. From there, a national NGO. The pre-pilot is based on disbursements are linked to trimestral lessons of implementation experiences of assessments by the partner agency and RBF type interventions in Cameroon and local inspector. Selected indicators have elsewhere in both the education sector and been carefully designed to improve the health sector. school functioning, teacher monitoring, Practically speaking, the pre-pilot is set students’ participation and retention (with up in one disadvantaged subdivision, and a focus on girls and vulnerable students, targets 20 primary public schools. With including IDPs and refugees), respect of the technical support of a national NGO free education, and other good practices hired as partner agency, schools prepare at local level. At the end of the school year, an individual school project and a short another final incentive will reward global action plan. This documentation is discussed improvements of the school. Expected with the active participation of the school incentives could lead to a 20 to 30 percent council “conseil d’école” which is an official increase of actual annual school budgets. representative structure, involving school Specific disbursement mechanisms will management, student-parent associations, be further detailed during the pre-pilot local administration and school community. implementation and included in the The existence of a school project and operational manual of the pre-pilot. 30 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) China Assessment of Teaching Practices for Changes in the Classroom (KLI Call #2) GRANT DESCRIPTION: The Guangdong Department of Education is innovating to improve the impact of in-service training towards greater effectiveness of its teaching force. The KLI Grant funds a pilot intervention to incentivize teacher-training institutions who deliver in-service training that teachers actually use in classrooms to improve student learning outcomes. PROGRESS TO DATE: A desk review was conducted to assess The first wave was made up of the availability of classroom observation representatives from the provincial instruments. On the basis of this review two Department of Education as well as candidate instruments were identified: representatives from all 16 of the project the Stallings and the Classroom Assessment counties (including representatives from Scoring System (CLASS). A pre-pilot was county bureaus of education, teachers and undertaken to assess the viability of these principals from primary and middle schools). instruments in the Guangdong context. In this wave, 14 of 18 trainees passed the The pre-pilot was conducted in two ways. first round of validation testing, and another The first consisted of applying both Stallings two passed on the second round of testing. and CLASS to several classrooms visited One participant was unable to during a project identification mission. take the second round of testing and The second consisted of applying CLASS another took it, but did not pass. to several videotaped classroom sessions The second wave was made up of partner from Guangdong. The review of videotaped institutions largely based in Guangzhou: classroom sessions was summarized in 1. South China Normal University, a brief and the findings discussed with the government. On the basis of these 2. Guangdong Institute of Education, discussions with the government the 3. Guangdong Teachers College of Foreign Classroom Assessment Scoring System Language and Arts, (CLASS) instrument was selected for 4. Guangdong University of Education, further investigation. 5. Guangdong Department of Education Training on the use of CLASS was conducted Of the 16 participants in this second wave, in Guangzhou in two waves: the first wave nine passed during the first round of testing. took place from September 7–10, 2016 and A second round of validation testing was the second wave took place from September not held due to the mid-Autumn Festival 12–14, 2016. The training was conducted holiday. The participants who have yet by Teachstone, the developers of CLASS. to pass are currently receiving a second A total of 34 individuals participated in opportunity to take the test. This is being the training. The training was followed by done using an online format where Chinese validation testing that each participant needs translation voiceover has been provided. to successfully complete in order to receive certification on use of the CLASS tool. REACH ANNUAL REPORT 2016 31 Colombia Results-Based Monitoring System (KLI Call #2) GRANT DESCRIPTION: The Colombian government has ongoing efforts to manage its education system based on results. Since 2014, Colombia has had a “synthetic index of education quality” for all basic education schools, with yearly targets at the school level. However, this index is limited, and more information is needed for policymakers to make well-informed decisions. The KLI Grant will support the development of a results-based monitoring system that covers multiple dimensions of education quality, with a view to targeting fiscal transfers towards improvement in those dimensions. PROGRESS TO DATE: The team has been working closely with the many indicators, which also tend to be Ministry of National Education (MEN), and concentrated on inputs and structural has advised on the selection of indicators factors in the school. The World Bank team based on the following dimensions is currently working to simplify the matrix by selected by the MEN: (i) pedagogy; cutting down the number of indicators, and (ii) teachers; (iii) administration; (iv) focusing them on key outputs, outcomes infrastructure; (v) community engagement; and processes related to quality of service and (vi) student welfare. A preliminary delivery. In addition, the team plans to matrix of indicators has been developed pilot the collection and reporting of select for all six dimensions, but it includes too indicators in 1,000 schools. 32 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Congo-Brazzaville Congo Citizen Voice for Education (KLI Call #2) GRANT DESCRIPTION: It is believed that leakage in the education system in Congo-Brazzaville results in only a small proportion of funds disbursed by the Ministry of Finance arriving at schools. The KLI Grant finances the set-up of an open data system, which allows community members to report on the funds which arrive at their local school. If the system is effective, it could be used to inform an RBF program for school finance. PROGRESS TO DATE: The grant design went through multiple a technology solution that allows collecting iterations, and the team will now work and managing data about satisfaction from closely with BlueSquare rather than Cordaid, students, parents and volunteer teachers though the two organizations often work through mobile phone. The solution should together in the region. BlueSquare is also plan for other forms of data collection a company that provides technology in case there is no phone coverage; and (iii) solutions around RBF (particularly in the a public dashboard to display results from health, education and governance sectors). the surveys. BlueSquare will produce the following: (i) The survey is embedded in a larger WBG an analysis of the education monitoring project that is awaiting government framework and identification of indicators signature, which has been delayed due to to be collected through routine surveys; (ii) elections and post-elections violence. REACH ANNUAL REPORT 2016 33 Democratic Republic of Congo Impact Evaluation of RBF approach in South Kivu (KLI Call #2) GRANT DESCRIPTION: The Government of South Kivu, in partnership with the Dutch NGO Cordaid, has been piloting RBF to improve education service delivery. As the end of the pilot approaches, the KLI Grant funds an impact evaluation to assess whether its RBF approach has generated better education results and whether learning outcomes in primary schools receiving RBF subsidies are higher than in primary schools that do not receive them. A challenge in DRC that is true for several grants is the availability of good data, which is a necessary precondition for RBF. PROGRESS TO DATE: One of the primary challenges in this fragile violence. The fieldwork and data entry have context is the availability of good data, just started, because identifying and training especially since target beneficiaries reside staff and enumerators took longer than in Shabunda 1 and Walungu 1, districts that originally anticipated. have been historically affected by ethnic 34 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) India Improving Teacher Performance through Outcome Linked Incentives (KLI Call #2) GRANT DESCRIPTION: The Government of Bihar is designing a teacher performance program to improve service delivery in its schools using a teacher composite score designed by the Indian civil society organization Pratham. The KLI Grant funds research to: i) test the validity of an existing teacher assessment/composite score to inform this work; ii) if the assessment is valid, design a pilot for this program. PROGRESS TO DATE: The team has developed the design for a state of Bihar and include about 3,000 study that will use three sources of evidence teachers and 55,000 students. to establish validity of test scores. The validity The team has also had discussions with study aims to establish content validity, experts and with the Government of Bihar construct validity and predictive validity. to determine the type of incentive that will In addition, the team deliberated on be given to teachers. The Government has several options for the design of the impact had some apprehension around financial evaluation taking into consideration aspects incentives for teachers and this concern is of external validity and reliability of findings especially sensitive in the context of Bihar while balancing scope and costs towards where teachers are hired at the decentralized implementation. While a larger sample of level by the community and any tinkering schools and teachers would be preferable with their structures lead to drawn out legal in terms of being able to generalize the battles. The team has undertaken a quick findings across the state of Bihar, the cost of field survey to gather principals’ opinions implementing the study in a large number on the types of incentives that will motivate of schools while avoiding spillover effects teachers towards better performance. The and closely monitoring progress would be findings from the survey indicate that the too substantial. Thus, the study will cover Government is most likely to agree to non- about 750 schools across 4 districts in the monetary rewards for teachers. REACH ANNUAL REPORT 2016 35 New Horizons (KLI Call #2) GRANT DESCRIPTION: India’s National Institute of Open Schooling (NIOS) provides second chance education for religious and other minority students who have dropped out of school. The KLI Grant funds research to evaluate the design of performance contracts between the Government of India and service providers, as well as the cost-effectiveness of varied levels of monitoring. PROGRESS TO DATE: The REACH grant is in two parts. The first Thus far, the pilot intervention has revealed activity consists of a qualitative survey that training providers did not focus on to better understand how private sector students passing the exam when financing providers are reacting to the performance was not conditioned on that result. However, based contracts. The study is looking at this is anecdotal and the team hopes to get the performance of education training more detailed information over the next providers before performance based month or so. financing on education outcomes was The biggest obstacle so far has been: included in the contract. The first group of getting the education and training providers students have just completed their exams, hired and convincing the new Secretary and and initial findings will not be ready until Joint Secretary to keep the agreed upon March 2017 or later. performance based financing milestones in The second activity is a randomized impact the terms of reference for the recruitment evaluation of alternative monitoring of training providers. They were nervous instruments. This activity is going slower that good providers would not want to than expected, as the agencies that will participate in a program where their implement the project have not yet been payments were conditional on something awarded their contract due to staff changes they may not be able to control (i.e. in the Ministry. The team expects that these students passing the exam). The team was agencies will begin providing education and able to overcome this concern by finding training to students by April 2017. an education target that they were more comfortable with. 36 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Indonesia Assessing the Impact of Alternative Approaches to Linking Funding to School Performance in Jakarta (KLI Call #1 and #2) GRANT DESCRIPTION: The two activities under this proposal are part of an overarching exercise by the Jakarta government to experiment with different approaches to linking financing with results: a) the introduction of a performance and equity school grant; and b) the introduction of performance contracts with schools. The already implemented performance and equity school grant program looks at whether the announcement of an incentive would be enough to change school behavior and ultimately student performance. The pilot for the performance based contracts goes further by using the national standards to identify areas of weakness at the school level developing school improvement agreements between the school and the district office based on this information. The KLI Grant funds the evaluation of both activities. PROGRESS TO DATE: For the evaluation of the performance and On the performance contracting side of equity grant, the team is still collecting and things, the schools have started using the cleaning data that will allow them to ensure electronic (or web-based) school planning that some of the effects they are finding are forms. For school evaluation, the team not driven by other changes in the system. has conducted the baseline survey for 150 The team also intends to select a small schools and is now doing the analysis. group of schools (20–30) to do in-depth In addition, the school self-evaluation interviews and data collection exercises. tool has been developed and the team The purpose would be to better understand has conducted a “training of trainers” the impact of the incentive in terms of for the self-evaluation too. Subsequently, changes in school activities/organization 100 schools of the 150 will be trained in as well as how schools have changed the utilizing the instrument. way they spend their money (meaning, all resources, and not just the grant amount) in response to the grant. REACH ANNUAL REPORT 2016 37 Morocco Support to Performance Contracting in Education (KLI Call #2) GRANT DESCRIPTION: The KLI grant supports the design of a performance-based contracting model in education between national, regional, local and school-levels in Morocco, with a specific focus on improving the effectiveness and efficiency of mandatory education service delivery (preschool through lower secondary). The REACH grant has helped align actors such as the Ministry of Education, Regional Academies, the Ministry of Finance and international donors, and moved their conversations away from inputs. PROGRESS TO DATE: Performance contracts are being piloted In May, a workshop was held to present in two regions, the mostly urban region of the performance contract framework to Rabat and the rural region of Oriental, with the Ministry of Education (MoE), regional a view to using lessons learned to design academies, Ministry of Finance and and implement a client-led scale up in more international donors. Following that, a regions if the activity is successful. The framework was developed by WBG and grant is also contributing to the preparation MoE experts in line with the National of the Morocco Education Program-for- Performance plan. In June 2016 regional Results, a USD$200 million pipeline WBG academies will submit their diagnostics operation that aims to support governance and draft formal contracts. These drafts reforms in the education sector to increase will be reviewed and further revised based accountability among stakeholders and on feedback received from participating strengthen service delivery along the results- institutions, making the contracting process chain. This results-based financing activity an iterative one. would be the first of its kind in the Middle East & North Africa region. 38 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Update on REACH for Reading In collaboration with the Global Book Proposals for REACH for Reading grants will Alliance, the REACH for Reading Call for be slightly larger than the general KLI calls, Proposals was launched on September as applicants can request up to US$500,000. 8, 2016, International Literacy Day. The Similar to the previous KLI calls, submissions REACH for Reading Call for Proposals seeks will undergo a rigorous selection and to support innovations that get the right interview process. Approval criteria include books to all children at the right time, using relevance to reach for reading’s goals, results-based financing modalities. Activities technical soundness, methodological financed through this call for proposals will rigor, strategic opportunity for long-term vary in scope, scale and duration but will engagement and government and partner primarily be designed to transform the quality commitment. Approximately 10-12 grants and availability of primary grade textbooks are expected to be announced in the and reading materials for underserved Spring of 2017 for interventions that will last languages. During the open call period, anywhere between six months to two years. nearly 250 public and private partners An overview of the REACH for Reading submitted proposals for RBF-based REACH review process and evaluation rubric is for Reading interventions in 64 countries. located in the Annexes. REACH ANNUAL REPORT 2016 39 Spreading the Word: Building Capacity and Confidence in RBF as a Real-World Tool REACH is dedicated to expanding the RBF knowledge given how important yet body of knowledge on RBF in education difficult it is to capture it. and building evidence on what works for a ■■ Connecting teams and topics. This systems approach to education reforms and consists of bringing together actors from investments. To accomplish these goals, different sectors from inside and outside REACH focuses on generating (through KLI of the WBG to share knowledge and to grants), acquiring and sharing knowledge further illuminate issues that are often on what works in RBF. The main knowledge referenced but never teased out, i.e. how management goal is to equip operational are schools like (and unlike) health clinics? staff inside and outside the World Bank with ■■ Building communities of practice (CoPs). the knowledge and tools to implement RBF These informal CoPs focus on specific programs and to inform a broader audience topics for continuous sharing and about RBF and its benefits, limits, and uses. provision of just-in-time support by WBG The REACH knowledge sharing and experts. For example, REACH is heavily organizational strategy for 2016 focuses on: involved in the Education Finance CoP. ■■ Transfer of tacit operational ■■ Creating and compiling tools and knowledge. REACH has a specific focus information. The learning events serve on acquiring, packaging, and sharing tacit as a way to share, but also to extract and 40 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) compile knowledge from a wide range of the only result worth financing?; actors. The key conclusions of each event (ii) Can school grants buy learning?; are shared with the participants and the and (iii) Can fiscal transfers generate broader RBF community with a focus on results? The fourth ‘It’s Debatable’ meets the information being in an operationally ‘Smackdown’ on teacher incentives is friendly, concise format. currently being planned (details to follow). ■■ Feedback loops to ensure continuous ■■ Operational Clinic: This model allows KLI relevance of the activities. REACH is grantee teams to receive inputs from other regularly collecting feedback from the country teams and expert discussants for target groups on the type of support specific operational questions. needed/received to design/improve the knowledge sharing activities accordingly. ■■ Talk Show: This format is a way to bring out diverse views through moderated The Knowledge Sharing and Organizational discussion, with the goal of the guests Learning Plan was developed through a staying on message but also engaging in consultative process with WBG staff and open discussion. partners. The REACH team also sought guidance and support from the WBG’s ■■ Open Space Expert Round Tables: communications team. These events Loosely based on the “open space” typically receive high interest and feature concept, these round tables focus on interactive involvement of participants. sub-topics identified by participants, A focus is placed on embedding ICT to with expert discussants at each table. modernize the pedagogy and maintain a Participants are allowed to move fast pace at the learning events. around freely. The REACH team deeply invests in the ■■ Trial: This will be featured at HD Week in preparation of learning and training events May 2017, where RBF will participate in a in order to have state of the art content courtroom trial, complete with a judge, on hand, and well-informed, dynamic jury, defense attorney and prosecutor. speakers and presenters. Event formats are also designed carefully, with attention to ■■ Brown Bag Lunches (BBLs): These are choosing the best fit for desired learning more traditional learning events hosted outcomes. An important criterion for by the WBG that generally involve one REACH learning activities is whether or presenter, a chair and discussants. not they meet the immediate needs of REACH believes in engaging the audience operational teams. Overall, the goal is to and sustaining their attention as much as create an environment conducive to candid possible. To do this, various tools are utilized, and open sharing. including: (i) audience engagement and Examples of formats used at REACH polling tool to answer survey questions via learning events are: mobile phone; (ii) Ignite presentations ■■ ‘It’s Debatable’: The signature REACH that are time-bound and short; and event using debate as a way to highlight (iv) competition quizzes that get the pros and cons of a new topic. There have audience to interact with one another been three events to date: (i) Is learning and the presenters. REACH ANNUAL REPORT 2016 41 Knowledge Sharing and Learning Events 2016 In 2016, REACH organized nine knowledge sharing and learning activities this year, reaching over 250 participants: 1. DLI TTL Luncheon: Learning from Early Operations Date: February 24, 2016 Description of Luncheon with a survey of World Bank Task Team Leaders (TTLs) using the Event: Disbursement Linked Indicators (DLIs) in their projects to capture their tacit knowledge and learn from their early experiences. Format: A facilitated discussion, using an interactive, online survey to ask provocative questions, each question followed by a discussion for the participants to elaborate on their answers and further explore the topics. Client/Co-host: Global WBG regional teams Main Objective: To capture the tacit knowledge of TTLs who have significant experience using DLIs and share that information with interested teams Background REACH was interested in extracting some lessons / testing some hypotheses Information: about the WBG’s early experiences with DLI operations in Education. In particular, the team was interested in getting more knowledge on how to craft meaningful and realistic DLIs, as they become a greater and growing part of the portfolio Participants: 15 World Bank TTLs with experience on DLIs and Operational Policies and Country Services (OPCS) unit representatives Materials: Note on Lessons Learned 42 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) 2. GPE BBL: “The Promise and Potential of RBF in Education” Date: March 22, 2016 Description of Sharing lessons learned from the WBG’s operational teams with GPE the Event: colleagues. Specifically, presenting Tanzania’s Big Results Now; one of the first Program-for-Results (PforR) operations in Education. Agenda: Short presentations were made by the program manager of REACH on ‘Top 10 Lessons Learned about RBF to date’ and the TTL of Big Results Now, followed by a Q&A. Format: BBL with a presentation including an audience poll and an illustration quiz Main Objective: Prepare the GPE staff to advise clients on RBF Client/Co-host: Global Partnership for Education (GPE) Staff Background Every GPE grant has a variable tranche, which is 30% of financing linked to RBF. Information: The new funding model is relatively recent, and GPE and WBG staff are still working through what it means in implementation. This session was designed to share information with GPE staff on the benefits and challenges with RBF. Audience: 25 GPE staff Materials: Presentation on Lessons Learned thus far, Presentation of Big Results Now REACH ANNUAL REPORT 2016 43 3. KLI BBL: Beyond Test Scores: Measuring Education Quality in Colombia Date: April 7, 2016 Description of Brown Bag Lunch on Colombia’s current system for measuring school the Event: performance and ongoing work in the selection of education quality indicators. Discussion on how to create a new, more results based monitoring system that looks at quality beyond test scores. Agenda: 1. Presentations made by the representative of the Colombian Ministry of Education, followed by the World Bank education team. 2. A World Bank Education Practice manager served as a discussant, reflecting on the presentation, providing examples of similar programs and relevant research findings and addressing the questions and challenges raised by the team. 3. Open discussion. Format: traditional BBL Main Objective: Sharing experiences of identifying the metrics for predicting education quality. Main discussion topics: ■■ What is a rigorous but practical methodology for identifying indicators for school quality, beyond test scores? ■■ How could the results based monitoring system be designed for better targeting? Client: Colombia Education Team/Ministry of National Education (MNE) Co-host: Latin America and Caribbean Practice Manager Background Colombia has a “synthetic index of education quality” for all basic Information: education schools. This index is a single number for each school comprised of three components focusing primarily on test scores. To address some of its limitations, the index is complemented with a monitoring system of dimensions that looks at education quality in broader terms, beyond test scores. A set of indicators linked to each dimension is being created with a focus on key outputs, outcomes and processes related to quality of service delivery rather than mere inputs. Emphasis is put into how to use the system to better target programs in the MNE; how it could be used for performance based transfers at different levels, and how it could be used to engage the community. Participants: 29 World Bank and external participants, mostly from the education sector Materials: Presentation by the MNE 44 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) 4. REACH KLI Operational Clinic on Education Quality Metrics Date: April 12, 2016 Description of A concentrated operational clinic for the Colombian MNE and the REACH the Event: KLI grantee team to get feedback from other country teams and experts for specific operational questions (below in Background information section). Agenda: 1. Short presentation of the Colombia project, focusing on the proposed approach by the team, main challenges, and outstanding questions to be addressed. 2. Open discussion between KLI team and discussants chaired by the education practice manager: All discussants reflecting on the presentation and providing inputs based on their own experience on similar considerations and challenges, and providing feedback on how to address the questions and challenges raised by the KLI team. Format: Operational Clinic Main Objective: Identifying the metrics for predicting education quality beyond test scores in Colombia. Client Colombia Education Team/Ministry of National Education (MNE) (KLI grantee): Background See the Background information in “BBL: Beyond Test Scores: Measuring Information: Education Quality in Colombia” Specifically, the Colombia team was seeking feedback on the following areas: ■■ The matrix content ■■ Dimensions: Are all dimensions included, that should be included? ■■ Selecting indicators. ■■ Piloting the monitoring system ■■ Uses of the system ■■ Piloting of the use of the monitoring system for “policy making” Participants: 20 Internal and external experts providing feedback on the matrices, including the Shanghai, Vietnam and Haiti teams with similar projects. “You can actually improve learning outcomes through teacher incentives. Well designed incentives can actually move the needle a little bit.” Professor Isaac Mbiti from the University of Virginia, KLI Grantee REACH ANNUAL REPORT 2016 45 5. IT’S DEBATABLE: Fiscal Transfers Don’t Generate Results Date: May 4, 2016 Description of This event is part of REACH’s ‘It’s Debatable’ series. It explored whether or not fiscal the Event: transfers can buy results in education. Learning from flagship examples from health and governance (Argentina’s Plan Nacer in the health sector and the Mozambique Public Management PforR led by the Governance GP), the debate and the following discussion aimed to inform the approach that education sector teams should take when designing results-based projects with clients.  In particular, the event explored how education reform initiatives can be better embedded in larger public financial management reform, starting with inter-governmental transfers and performance-based budgeting. One of the key questions posed was whether public financial management mechanisms can be harnessed to ensure more efficiency, equity, or even student performance. Agenda: ■■ Facilitated by the Vice President of HD, Keith Hansen ■■ Opening remarks by Claudia Costin, Senior Director of the Education GP ■■ Overview, using an audience poll ■■ 5 minute ignite presentations by the TTLs of the Argentina and Mozambique programs ■■ Debate between Debbie Wetzel, Senior Director, Governance GP and Teresa Curristine, Senior Economist, IMF ■■ Discussion ■■ Closing arguments by the debaters Format: Debate Co-host: Governance global practice (with support from the HD Vice President’s office) Main Objective: The purpose of this event was to: (i) encourage cross functional and organizational sharing of perspectives on performance-oriented fiscal transfers in the social sectors, and (ii) contribute to the knowledge base on RBF in education by debating what key sector results, if any, might occur if fiscal transfers in education are linked to performance.  The work on this topic continues through the recently established Education Finance Community of Practice (“EduFin CoP”). Background Demand from clients on how to structure national – subnational arrangements for financing Information: service delivery is growing and at the same time, the World Bank is moving toward meeting President Kim’s commitment of doubling RBF in education to US$ 5 billion by 2020. The Education GP therefore expects substantial growth in this area, and is reaching to teams within the Bank to help task teams be better positioned to advice clients. Performance-oriented fiscal transfers in the Health Sector have demonstrated promising outcomes. Groups such as Governance also have rich experience to draw from which could support this work in education where such transfers are more nascent with a limited body of existing evidence to draw from. Drawing on expertise from colleagues in governance, education and health from inside and outside of the World Bank, this event brought together groups who are engaged in RBF and fiscal management work, and fostered knowledge sharing on intergovernmental transfers and binding financing on results. Participants: Participants: 34 mostly education staff (current and former WB staff and external researchers. Additional 30 participants through WebEx. Materials: notes, video, presentations 46 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) 6. RBF: Paradigm Shift, or Latest Fashion? An Interactive Talk Show about RBF in Education Date: June 16, 2016 (Oslo, Norway) Description of In this talk show-style event RBF experts and practitioners debated the the Event: benefits and possibilities, as well as the limitations and concerns related to RBF in an entertaining way, yet sharing substantial information. The ‘host’ of the show led a conversation by a diverse group of guests about what RBF can and cannot achieve, how RBF can strengthen education systems and about the use of RBF interventions, i.e. performance-based school grants and teacher incentives. The panelists also covered RBF risks and how they can be mitigated. The talk show ‘guests’ represented a wide variety of actors working on RBF from NGOs and practitioners to researchers and program designers, as well as many different viewpoints. Format: Talk Show Additionally, audience members shared their opinions, commented and asked questions throughout the session using an interactive online tool, Poll Everywhere. Co-host: Norwegian Ministry of Foreign Affairs, NORAD, and the Norwegian Teachers’ Union. Organized as part of the side event to the GPE Board Meeting in June. Main Objective: To showcase some of the latest work on results-based financing in education, and highlight the benefits and opportunities as well as the challenges and limits of RBF. Background Education is one of the surest means to end poverty, but much of this Information: potential is lost if students do not acquire skills in school that they need to succeed in life. Achieving learning for all will require moving beyond simply financing the inputs that education systems need, to strengthening these systems to deliver results. This, in turn, implies aligning governance, financing rules, incentives, and management practices with the ultimate goal of better learning outcomes, especially for children from the world’s poorest families —for whom access to quality education remains deeply inequitable. There is growing demand from countries for RBF, which is a promising set of tools to help achieve this critical alignment within education systems. But there are also many questions that remain to be answered and addressed if RBF is to be effectively and productively used in education. Audience: 40–50 Nordic civil society organizations, teachers’ union representatives, teachers and government representatives and some GPE Board members. Materials: Video, Note on main takeaways, Note on points presented by the panelists REACH ANNUAL REPORT 2016 47 7. PforR: Scaling up for Sustainable Results Date: September 27, 2016 Description of This second PforR training, organized with OPCS and ESDP, targeted staff the Event: who are getting ready to prepare a PforR. The one-day event helped to highlight the key pillars for developing successful PforRs and discuss how PforR relates to institutional capacity building elements and results based financing approaches. Drawing from four cases (India, Vietnam, Lebanon and Tanzania) the focus was on exploring why and when to use PforR, explain challenges related to it and share key lessons. Agenda: ■■ Overview of PforR: Towards Building Institutions and Financing Results ■■ India PforR on Skills Training for Employability Leverage PPP ■■ Vietnam Enhancing Teacher Education PforR ■■ Tanzania PforR, including preliminary lessons learned from the Mid- Term Review and Process Evaluation for Tanzania Basic Education PforR (BRNEd) ■■ Lebanon PforR providing an example of developing PforR in a complex situation ■■ Simulation where participants got a chance to apply what they learned during the day to convince the government of to agree to use PforR Format: Part I: Short, focused PowerPoint presentations with the overview presentation including polling questions to stimulate audience engagement Part II: Simulation Co-host: Education Staff Development Program (ESDP) and OPCS Main Objective: To enhance knowledge of PforR instruments among Education GP task teams and introduce key concepts that are essential to program design and implementation; and to link PforR theory/principle with education global practices. Background PforR is still a new financing instrument for many TTLs. REACH has made an Information: effort to join forces with the ESDP and OPCS to provide regular offering of this training. Audience: 41 world bank staff members Materials: 5 presentations 48 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) 8. Workshop: Dos and Don’ts on GPE’s Variable Tranche Date: October 13, 2016 Description of An informal breakfast workshop of Bank TTLs to share real-life experiences the Event: with GPE’s new funding model that has an RBF component. This was the first meeting among the World Bank TTLs implementing GPE projects to discuss: 1. How to get most out of the GPE grant variable tranche? 2. How to get started with designing and implementing the RBF component? 3. How to customize the variable part? 4. How to present and explain the variable tranche to government counterparts? 5. What kind of training/support is needed for the TTLs going forward? Agenda: 1. Taking the temperature – a participant poll and facilitated discussion about where we are at in implementing the variable tranche and about its benefits and challenges.  2. Lessons shared by those furthest along in implementing the variable tranche. Format: Discussion facilitated using poll questions. 6 minute ignite presentations by the TTLs with Q&A and open conversation throughout. Co-host: The World Bank GPE focal point Main Objective: To understand the experience of designing and implementing the GPE variable tranche to be able to better support the Bank TTLs in this area. Background Recent experience indicates that projects financed under GPE’s new funding Information: model are proving to be more complex and time-consuming to design and implement than under the prior model. In particular, designing the variable tranche of these projects is leading to delays in preparation, which ultimately affects the submission of applications. Participants: 13 staff, mostly World Bank TTLs implementing or planning on implementing a GPE grant Materials: Compiled Dos and Don’ts extracted from the presentations and conversation “RBF is a tool for learning, it’s a tool for experimentation, and it’s a tool for course correction.” Petra Vergeer, Sr. Health Specialist at the Global Health Financing Facility team at the World Bank REACH ANNUAL REPORT 2016 49 9. How are Schools like Clinics: SP&L, HNP and EDU Joining Forces on RBF Date: November 22, 2016 Description of Cross sectoral knowledge sharing event contributing to the knowledge base the Event: on RBF across HD. Bringing together operational teams from three sectors to share lessons on RBF, identify what works and to what extent the lessons from one sector apply to the others. Sub-topics identified for the groups to discuss: incentives, sustainability, verification, demand side incentives, indicators, and verticalization of the system. Each group discussed the lessons and their applicability within the subtopic. Agenda: ■■ Welcome and introduction by Amit Dar, Director of Strategy and Operations, HD ■■ Introducing the topic and the objective of the session ■■ Presentation on results of the Impact Evaluations on RBF in Health ■■ Creating the working agenda: The audience is given a chance to suggest additional topics and vote for the most relevant ones. ■■ Expert Round Tables ■■ Closing Plenary by Roberta Gatti, Lead Economist, HD The event was promoted as a “Road to HD Week ‘17” linking it to the biggest learning event of the sector and testing the new format’s suitability for it. Format: Open Space Expert Round Tables, testing a new format to the World Bank that encouraged informal sharing among colleagues. Co-host: RBF Health Team Main Objective: Share lessons on RBF between sectors. Capture the tacit knowledge of the operational teams working on RBF, identify key lessons, and discuss to what extent those lessons apply to the other sectors. Background Encouraged by the success of RBF approaches particularly in the health Information: sector, some countries – and country teams – have started asking for guidance on how to implement similar programs in Education. This raised the questions: “how are schools like (and unlike) clinics?” The innovative new format of “Open Space Expert Round Tables” was developed specifically for this event to test whether small group conversations with limited structure and more freedom for people to discuss what was most relevant to them would be a good way to capture the often hard to manage tacit operational knowledge. Audience: Discussants in each group: HNP, education and SP&L expert discussant on each sub-topic Other participants: TTLs and other staff across sectors; 52 participants in total Materials: Summary of the main conclusions from each table (sub-topic) 50 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Knowledge Sharing and 3. HD Week: Learning Events 2017 ■■ Third PforR Training As REACH continues to build on its brand ■■ RBF on Trial for dynamic and highly productive learning ■■ Second Open Space Expert Round events, plans for 2017 are already in motion. Tables event on RBF sub-topics As before, events will focus on capturing, 4. Comparative International Education compiling and sharing new knowledge Society (CIES) Annual Conference. on RBF, with select events open for public This annual conference draws in over participation, either in person or virtually 2500 participants from all over the via WebEx. world. The REACH team will participate 1. Operational clinic on performance- on two panels. The first will feature based school grants between four a frank conversation on the virtues teams. School grants are one of the and vices of RBF with Professor Alec most popular RBF schemes in education. Gershberg at the New School; Petra Potential discussion topics are: design; van Haren at Cordaid; Joseph Holden levels vs. gains; verification; capacity. of Foresight Associates and John 2. ‘It’s Debatable’ (fourth in the series) Patch with PriceWaterhouseCoopers on teacher incentives (co-sponsored (and evaluators of the Girls Education with the Africa Chief Economist Office’s Challenge); former WBG staff Barbara ‘Smackdown’ program). David Evans, of Bruns will serve as chair; discussants will the Africa Chief Economist’s Office, has be Karen Mundy of GPE and Najeeb published extensively on the topic and has Safiq of the University of Pittsburgh. The been featured in previous REACH events. second panel will focus on the REACH for Reading process and grantees. REACH ANNUAL REPORT 2016 51 5. Building Evidence in Education (BE2) has identified experts in the topic who are meeting. The 2017 BE2 meeting provides available for technical support. another opportunity to host an event REACH also has a publicly available cloud- similar to Oslo, with an emphasis on based folder of RBF materials – this includes evidence in RBF, for an external audience. RBF FAQs, various research on RBF and The REACH team will present the a DLI Library that lists sample DLIs from paper, “Financing Results to Strengthen World Bank projects (within the last 10 Systems: The WBG Approach to RBF years), categorized by level of education in Education” and host a panel with and type of indicator. This has proven to be diverse representation from international very useful for Bank TTLs who are designing partners, academia and NGOs.  RBF projects, so that they do not have to invent new DLIs. There are currently over 125 Communications examples, with the library being regularly maintained as new projects are approved. This year REACH has made a concerted effort to ensure that updated information and materials on RBF are made publicly Just-in-Time Support available at: www.worldbank.org/reach. Focus While knowledge sharing and learning are was also given to collaboration with the top priorities for REACH, one of the most World Bank communications and knowledge valuable and impactful services REACH offers management colleagues for their guidance, is support to operational teams seeking to and to identify the best channels and implement RBF. As part of REACH’s stated methods to effectively share knowledge. goal to be “the most task team friendly” This website is updated with information trust fund within the World Bank Group, about calls for proposals (two open REACH has developed strong camaraderie Webinars were hosted to answer questions with multiple teams working with clients that about REACH for Reading); the latest range from fragile states to Upper Middle literature and media on RBF; information Income Countries. REACH’s involvement related to topics of events hosted by REACH is iterative, with support being provided at and other RBF initiatives within the Bank. various stages of the project cycle (concept, REACH has also published several blogs, design, implementation, evaluation). Some both internal and external, to publicize the examples of the customized support that trust fund itself and to offer perspectives REACH has provided are below: on RBF. These resources are available in the Lebanon: Aside from being the recipient News and Events section of the website. of a Country Program Grant, WBG REACH has also guest blogged on other management requested that REACH websites, notably the Education Finance provide intense support in the preparation Commission’s, on how RBF can help of the overall operation in the education achieve the SDGs. sector, with the view of having the project In addition, REACH captures and collects be results-based. This resulted in the knowledge on RBF for Bank staff – this is REACH program manager serving as co- evident in the creation of the Education task team leader, helping to lead the policy Finance Community of Practice. The dialogue, assemble the team, craft the conversation continues in this virtual space results, design the financial engineering amongst staff members and the Community across implementers, and build in the 52 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) requisite technical assistance to ensure that underwent a second iteration. The REACH the national actors are able to carry out team’s original comments on the complexity the activities and achieve the agreed upon and overly ambitious nature of the first results. This implied three trips to Beirut in scorecard were taken into account and it is four months to accelerate the preparation now being simplified. The Sierra Leone team of the project, which represents the first is now aiming to reduce the signal-to-noise time the Government of Lebanon used the ratio of the indicators and is using lessons Program-for-Results lending instrument (in learned from RBF education projects in any sector). Pakistan’s Sindh and Punjab provinces. The Nigeria: In Nigeria, the REACH team served consultative process with REACH will help in an official capacity of peer review for further ground the design in the realities of the operation. The original project design rural Sierra Leone. In addition, REACH has proposed an ambitious model using a form invited the Sierra Leone team to showcase its of “output-based aid” similar to how the performance-based school grant model in a health sector has operated in the country. forthcoming operational clinic. After much deliberation, further discussions Cambodia: Cambodia’s Ministry of Education with the client and the fiduciary team, and has requested Bank support for a Secondary separate one-on-one advisory sessions with Education Improvement Project. The project the REACH team, it was agreed that the will feature disbursement-linked indicators, country context, especially the fiduciary and as such the REACH team was called capacity within the targeted states for the upon to review the proposed indicators, and operation, was not yet ready for embracing advise on their design and selection. such a form of RBF. In the end, the team Cameroon: REACH provided peer review settled on a traditional Investment Project services to the Cameroon team in the Financing approach. This serves as an design of their forthcoming Education important example of the relevance of Sector Reform Project, which plans to use traditional approaches in some country disbursement-linked indicators. In line with contexts, and that RBF is not necessarily the recommendations from the REACH always the best approach. team, the design of the project is now being Liberia: Continuing from the support greatly streamlined to ensure greater success provided last year, REACH facilitated a with RBF. The original project had too many workshop with the Ministry of Education’s DLIs and disparate activities that did not link planning team to discuss options for RBF together to leverage the benefits of RBF as a under the GPE model. Working from the systems strengthening modality. “GPE Do’s and Don’ts” list (see previous Uruguay: REACH participated in a workshop section), REACH guided the task team, exploring the rationale and the options government team, and local education donor for introducing RBF elements in a pipeline group on some important principles, options, education project in Uruguay. It was agreed and lessons, as they move forward in the that a project component would feature DLIs process. In December, REACH will serve in an as a way to introduce the concept with the official peer review capacity for the operation. client, to see if this small pilot could work. Sierra Leone: Also continuing from 2015, The component will include results related REACH’s technical support was requested to student assessment systems and other again as the original school scorecard design key sector policies. REACH ANNUAL REPORT 2016 53 Indonesia: The Indonesian authorities are examples of results frameworks to help seeking to address issues of teacher under- illustrate the possibilities of such an approach performance by introducing incentives and/ to the Ministry of Education in Egypt. The or bonus pay, as part of the overall teacher team will go on mission in mid-December remuneration package. REACH collated 2016 and have requested that REACH be on evidence in this area for the Indonesia team, standby to potentially virtually participate in and put them in contact with the Principal meetings around RBF. Investigators for the REACH-financed KLI Tanzania: The Education Global Practice’s grants that are also exploring teacher first PforR was in Tanzania. The project incentives. This conversation is ongoing, is now at the midterm review mark, two- with the Indonesia designers benefiting years into implementation. REACH was first hand from the experiences of similar requested to review the proposed additional pay-for-performance programs in Kenya financing for the program, to total US$100 and Tanzania. million. In particular, REACH provided Egypt: The Ministry of Planning and some feedback on the design and selection International Cooperation has requested a of the indicators, and on how to set the new education operation in Egypt, and would targets for each proposed DLI. REACH will like to use the PforR lending instrument. As continue to advise the Tanzania team as the such, the Egypt education team contacted development of the targets matures, and as REACH to help bring the team up to speed discussions with the client advance. on the operational implications of PforR, REACH’s experience in supporting teams and to share early lessons from other PforR allows the trust fund to keep a finger on the experiences in education. REACH prepared pulse of all RBF-related activities at the WBG a custom-made presentation for the team and ensures that REACH is well-versed in the to deliver to the client, and gave them with real-life challenges and benefits of RBF. 54 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) REACH: Looking Ahead In brief, REACH’s activities in 2016 were ■■ REACH team: The core REACH team well aligned with the overall trust fund has been consolidated. The team objectives. In 2017, while most of the has acquired both a deep and broad planned activities and knowledge products understanding of RBF in education, and is have already been outlined in above steeped in the available evidence. This is sections, REACH will continue to grow its complemented with skills in logistics and impact through: systems, communications, and training. ■■ Financing: Thus far, and learning from Together, these have resulted in high and the Health Results and Innovation Trust quickly growing demand for Just-in-time Fund experience, REACH has moved support from REACH. quickly to the operational level, financing As the international development community a number of activities in countries makes a push towards the SDGs, even more to begin supporting clients in this attention will be focused on how financing area immediately. As a result, REACH can achieve greater results in education. resources have been committed in their REACH looks forward to bringing more entirety. Future activities will require evidence to the table, and helping to grow additional financing. the RBF agenda. REACH ANNUAL REPORT 2016 55 REACH Gross Balance Sheet 2015-2017 Updated (in US$) Total   2015 2016 2017 Commitments DONOR COMMITMENTS Federal Ministry for Economic Cooperation 2,149,700.00 0 1,074,850.00 1,074,850.00 and Development NORAD 12,302,341.10 7,516,512.53 4,785,828.57 0 USAID 11,700,000.00 4,500,000.00 7,200,000.00 0 Total 26,152,041.10 12,016,512.53 13,060,678.57 1,074,850.00 ALLOCATIONS TOTAL ALLOCATIONS     Administration Fee (2%) 523,040.82 240,330.25 261,213.57 21,497.00 Program Management 2,236,759.28 885,936.46 976,700.36 374,122.46 and Administration* Country Program Grant 8,500,000.00 4,250,000.00 4,250,000.00 0 Knowledge, Learning 13,392,241.00 3,392,241.00 10,000,000.00 1,500,000.00  and Innovation Grants General Calls 4,892,241.00 3,392,241.00 0 1,500,000.00 REACH for Reading 10,000,000.00 0 10,000,000.00 0 Total 26,152,041.10 8,768,507.71 16,987,913.93 1,895,619.46 *The Program Management and Administration fee is calculated based on actual and projected costs. 56 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Results Framework Objective / Indicatorsa Baseline 2015 2016 2017 2018 Actual Planned Planned GOAL OF THE REACH TRUST FUND: To support efforts toward more and better education services, especially to those most excluded, by helping country systems focus more sharply on results. RESULT 1. Country systems and capacity for RBF in education 0 14 16 15 15 strengthenedb 1.1 Financial and technical support to Country Programs and Pilots (see 1 2 2 2 CPG Results Frameworks, Annex IV) 1.2 Financial and technical support to activities for Knowledge, Learning, 19 18 29 29 and Innovation (KLI) 1.3 Just-in-time support to WBG program teams, informing RBF 4 6 7 8 approaches on-the-groundc RESULT 2. Global evidence base for RBF in education is developed and made publicly available 2.1 Total number of KLI grant policy notes on RBF approaches prepared 0 0 9 18 and disseminated 2.2 Total number of impact evaluations 6 6 8 8 2.3 REACH website operating and regularly updated with new 1 1 1 1 knowledged 2.4 Number of RBF learning events 1 2 5 8 (open to the public) 2.5 Number of external blogs/articles 1 3 10 15 published RESULT 3. Approach Paper on RBF in education published and 0 0 1 1 1 disseminated a Data in this table is cumulative b Represents number of countries. Some countries receive support across different streams. c Countries that are not receiving CPG or KLI Grants d Including externally-generated evidence REACH ANNUAL REPORT 2016 57 Annexes REACH for Reading Review Committee Guidelines Evaluating Applications The Review Committee will evaluate applications based on how each applicant contributes to the REACH for Reading goals of funding high quality, robust, cost-effective experiments, designed to transform the literacy space through results based-financing models. This rubric is the core of the REACH for Reading application evaluation, and is designed to guide in assessing the strength of each applicant. The five rating criteria are as follows: ■■ Relevance to REACH for Reading’s goals (30%) ■■ Technical Soundness (35%) ■■ Methodological Rigor (20%) ■■ Strategic opportunity for long-term engagement (15%) ■■ Government and Partner commitment (First Round Review Only) Three panels will review applications: 1. REACH Core Team – Provides an initial screening of all applications received, using a pre-designed rubric 2. REACH for Reading Review Committee – Provides a first round review of all eligible applications, scoring against the same pre-designed rubric. A quorum of individuals from the REACH for Reading Review Committee and the REACH Core Team will also conduct interviews 3. Technical Review Committee (World Bank Education GP management team) – Authorizes all grants (as per the governance arrangements specified in the Administrative Agreement) 58 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Review Process The application submissions are assessed in the following order: 1. November 28 – November 30th (First Round Review): The REACH Core Team screens all applications received using the (evaluation rubric as a guide) according to the eligibility and submission criteria. If it becomes clear before, during or after the review phase that one or more of the eligibility criteria have not been fulfilled, the proposal is declared ineligible and withdrawn from any further examination. Only those applications that meet all four of the following eligibility and submission criteria are referred to the REACH for Reading Review Committee: 1. Proposed intervention reforms one or more aspects of the way books are developed, procured, produced or supplied, using results-based financing 2. A focus on primary grade books for reading instruction and practice in an underserved language is clearly articulated 3. Evidence of an existing reading program and/or reform is provided 4. Applicant is likely to gain government endorsement for the proposed intervention 2. December 1st – December 9th (Second Round Review): REACH for Reading Review Committee review applications that pass the initial screening stage and score against the Evaluation Rubric: 3. December 12th (Virtual Review Meeting): Once all committee members have completed their assigned applications for scoring, a Virtual Review Meeting is convened to discuss the scores awarded. Proposals that are below the interview threshold of 80% (but above 65%) are prioritized for discussion. However, the Review Committee may decide, at their discretion, to discuss proposals with scores above the 80% interview threshold. The Review Committee reaches a consensus and the top 15–20 applicants are referred to the interview process. 4. January 9th – January 20th (Final Round – Interviews): REACH for Reading Review Committee interviews candidates using a standard form that includes proposal specific notes/questions derived from the Virtual Review Meeting discussion. When possible, Review Committee members are assigned to interviews based on expertise. World Bank Task Team Leaders (TTLs) may be invited to participate in interviews of external candidates for which the proposed intervention (country) is aligned with the TTLs current portfolio. The Committee reaches a consensus and applicants are scored: (2) Highly recommend (1) Recommend with contingencies (0) No recommendation. 5. January 23rd – January 31st (Final Round – Recommendation): REACH for Reading Review Committee interview notes and recommendations are compiled and submitted to the committee for a final review before submission to the World Bank Technical Review Committee for endorsement. Letters of acceptance are emailed to candidates starting February 1, 2017. REACH ANNUAL REPORT 2016 59 REACH for Reading: Evaluation Rubric Relevance to REACH for Reading’s Goals (30%) ■■ How does the applicant’s proposed intervention reform one or more aspects of the way books are developed, procured, produced, or distributed? ■■ Does the applicant’s proposed intervention strongly demonstrate the use of results- based financing mechanism to transform book quality, availability, accessibility, affordability and/or use? ■■ To what extent does the applicant’s proposed intervention focus on early grade books in underserved languages? Is there an existing literacy program in place? SCORE CORRESPONDING ASSESSMENT 30–25 ■■ The fit between the proposed intervention and likely impact on content development, access, financing, procurement and/or supply chains is very strong. ■■ The proposed intervention strongly demonstrates the use of results- based financing to transform book quality, availability, accessibility, affordability and/or use. ■■ The proposed intervention is clearly focused on early grade books in underserved languages. 24–17 ■■ The fit between the proposed intervention and likely impact on content development, access, financing, procurement and/or supply chains is demonstrable. ■■ The proposed intervention demonstrates the use of results-based financing to transform book quality, availability, accessibility, affordability and/or use. ■■ The proposed intervention has a strong focus on early grade books in underserved languages. 16–7 ■■ The fit between the proposed intervention and likely impact on content development, access, financing, procurement and/or supply chains is adequate. ■■ The proposed intervention somewhat demonstrates the use of results- based financing to transform book quality, availability, accessibility, affordability and/or use. ■■ The proposed intervention has a focus on early grade books in underserved languages but it is weakly demonstrated. 6–0 ■■ The fit between the proposed intervention and likely impact on content development, access, financing, procurement and/or supply chains is weak or unclear. ■■ It is unclear how the proposed intervention incorporates the use of results-based financing to transform book quality, availability, accessibility, affordability and/or use. ■■ The proposed intervention has no focus on early grade books in underserved languages. 60 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Technical Soundness (35%) ■■ Has the applicant included a clearly defined theory of change? Is the theory of change grounded in evidence? Does it demonstrate how the intervention could lead to improved book development, procurement, production or distribution systems/ supply chains using RBF? ■■ Is the proposed intervention likely to lead to improved book quality in a country or region using one or more underserved languages as a language of instruction? ■■ What, if any, impact will the applicant’s proposed intervention have on strengthening country systems? ■■ Has the applicant clearly demonstrated the capacity required to implement all the proposed activities? SCORE CORRESPONDING ASSESSMENT 35–27 ■■ The intervention is very likely to lead to improved book quality, affordability and accessibility in a country or region using one or more underserved languages as a language of instruction. ■■ It is very likely that the intervention will directly lead to lower unit costs, more efficient supply chains, higher quality and/or more available books for the primary grades. ■■ Very strong demonstration of the use of results based financing to strengthen country systems and transform the literacy space. 26–17 ■■ The intervention is likely to lead to improved book quality, affordability and accessibility in a country or region using one or more underserved languages as a language of instruction. ■■ It is likely that the intervention will directly lead to lower unit costs, more efficient supply chains, higher quality and/or more available books for the primary grades. ■■ Strong demonstration of the use of results based financing to strengthen country systems and transform the literacy space. 16–7 ■■ The intervention is somewhat likely to lead to improved book quality, affordability and accessibility in a country or region using one or more underserved languages as a language of instruction. ■■ It is likely that the intervention will indirectly lead to lower unit costs, more efficient supply chains, higher quality and/or more available books for the primary grades. ■■ Adequate demonstration of the use of results based financing to strengthen country systems and transform the literacy space. 6–0 ■■ The intervention does not clearly address how it will lead to improved book quality, affordability and accessibility in a country or region using one or more underserved languages as a language of instruction. ■■ It is unclear how this intervention will indirectly lead to lower unit costs, more efficient supply chains, higher quality and/or more available books for the primary grades. ■■ Weak or unclear demonstration of the use of results based financing to strengthen country systems and transform the literacy space. REACH ANNUAL REPORT 2016 61 Methodological Rigor (20%) ■■ To what extent has the applicant demonstrated its ability to rigorously evaluate the intervention using quantitative and/or qualitative methods? ■■ How will the applicant’s intervention contribute to or inform the global evidence base? ■■ Has the applicant demonstrated that effective data quality control measures are in place? SCORE CORRESPONDING ASSESSMENT 20–15 ■■ Evaluation methods (including quantitative and qualitative) and used to determine the impact of the intervention are rigorous AND Proposed mechanisms to monitor and evaluate the intervention are rigorous AND highly effective data quality measures are in place. 14–10 ■■ Evaluation methods (including quantitative and qualitative) and used to determine the impact of the intervention are effective AND Proposed mechanisms to monitor and evaluate the intervention are effective AND effective data quality measures are in place. 9–5 ■■ Evaluation methods (including quantitative and qualitative) and used to determine the impact of the intervention are adequate AND Proposed mechanisms to monitor and evaluate the intervention are adequate AND adequate data quality measures are in place. 4–0 ■■ Evaluation methods (including quantitative and qualitative) and used to determine the impact of the intervention are inadequate AND Proposed mechanisms to monitor and evaluate the intervention are inadequate AND inadequate data quality measures are in place. 62 RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Strategic Opportunity for Long-term Engagement (15%) ■■ Does the applicant’s proposed intervention have the potential for cost -effective scale up within country (or other countries/regions) beyond the funding window? ■■ Does the knowledge generated from this intervention have a strong likelihood of informing policy decisions? SCORE CORRESPONDING ASSESSMENT 15–12 ■■ It is very likely that the activity will be scalable and implemented in a cost effective way either in the current geography or elsewhere beyond the funding window AND The value added from the knowledge generated from this activity either in the current geography or elsewhere for informing policy decisions is very high. 11–8 ■■ It is somewhat likely that the activity will be scalable and implemented in a cost effective way either in the current geography or elsewhere beyond the funding window AND The value added from the knowledge generated from this activity either in the current geography or elsewhere for informing policy decisions is significant. 7–5 ■■ It is unlikely that the activity will be scalable or implemented in a cost effective way either in the current geography or elsewhere beyond the funding window AND The value added from the knowledge generated from this activity either in the current geography or elsewhere for informing policy decisions is adequate. 4–0 ■■ It is very unlikely that the activity will be scalable or implemented in a cost effective way either in the current geography or elsewhere beyond the funding window AND The value added from the knowledge generated from this activity either in the current geography or elsewhere for informing policy decisions is inadequate. Government and Partner Commitment (First Round Review Only) ■■ Does the applicant demonstrate commitment and ability to secure government endorsement of the proposed activities? (Required) ■■ To what extent are country teams/local researchers strongly engaged in planning and implementation? Does the applicant demonstrate commitment and ownership from a partner/implementing agency? REACH ANNUAL REPORT 2016 63 Results in Education for All Children (REACH) Education Global Practice World Bank 1818 H Street, NW Washington DC, 20433 USA Website: www.worldbank.org/reach Email: reach@worldbank.org