RESULTS-BASED FINANCING IN EDUCATION: FINANCING RESULTS TO STRENGTHEN SYSTEMS RESULTS-BASED FINANCING IN EDUCATION: FINANCING RESULTS TO STRENGTHEN SYSTEMS Abstract This paper outlines the World Bank Group’s approach to how results-based financing (RBF) can resolve some of the forthcoming constraints faced by client countries to achieving the education Sustainable Development Goal (Goal 4). In sum, the WBG approach uses RBF as an instrument to strengthen education systems, by aligning and incentivizing actors around a set of system results. This stands in contrast to other approaches whereby RBF is first and foremost considered as a way to increase value-for-money, or more generally as a tool for “smarter aid”. The RBF agenda in education takes a wide lens, consisting of knowledge activities, convening services, and financing instruments such as the Program-for-Results lending instrument, and other financial modalities that reward the achievement of independently verified results. The World Bank Group has committed to doubling its current education portfolio of RBF operations to US $5 billion by 2020. CONTENTS ACKNOWLEDGEMENTS 2 ACRONYMS AND ABBREVIATIONS 3 SUMMARY 4 INTRODUCTION 6 What are “Systems”, and Why do they Matter? The Education Finance Crisis that RBF Might Prevent THE RISE OF RBF 11 Operating Definitions of RBF What Have Others Said about RBF? Evolution of RBF in the WBG Education Practice Results First: Neither “Laissez-Faire”, nor “Backdoor” RBF New Conditionality, or Easy Money? WHY WE BELIEVE IN RBF: Four Reasons 17 Flip the Policy Dialogue to Where We Want to be in Five Years Sustain Attention across Crises, Fads, and Changing Governments Align All Actors around Results that Matter Institutionalize Measurement Systems for Lasting Impact SOME FREQUENTLY ASKED QUESTIONS 25 What are the Challenges RBF Can Solve? What are the Limits of RBF? What are the Risks to RBF? HOW: A Systems Approach 29 THE WORLD BANK AND RBF: Implications for the Business Model 34 Rising Demand for RBF Mutual Accountability: “We’re in this Together” Artisanal RBF Knowledge Agenda CONCLUSIONS 38 REFERENCES 39 ANNOTATED BIBLIOGRAPHY: Theories Underpinning RBF, and Early Lessons 42 Early Lessons from RBF Programs Acknowledgements This paper was written by Peter A. Holland and Jessica D. Lee. We are grateful to Luis Benveniste, Practice Manager, World Bank Group, for his guidance, feedback, and support. Helpful comments and feedback were also provided by Melissa Adelman, Juan Baron, Penelope Bender, Fadila Caillaud, Bridget Crumpton, Peter Darvas, David Evans, Deon Filmer, Emily Gustafsson-Wright, Michael Holländer, Sachiko Kataoka, Elizabeth M. King, Ines Kudo, Christoph Kurowski, Toby Linden, Minna Mattero, Jenny Beate Møller, Karen Mundy, Wenna Price, Dhushyanth Raju, Furqan Saleem, William Savedoff, Shobhana Sosale and Atussa Ziai. The REACH trust fund is supported by the governments of Germany, Norway and the United States. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors. They do not necessarily represent the views of the International Bank for Reconstruction and Development/World Bank and its affiliated organizations, or those of the Executive Directors of the World Bank or the governments they represent. THE WORLD BANK GROUP 2 ACRONYMS AND COD...............................................................Cash on Delivery DFID............................ Department for International Development ABBREVIATIONS DLI.............................................Disbursement-Linked Indicator ECD.............................................. Early Childhood Development EGRA........................................ Early Grade Reading Assessment EMIS. ........................ Education Management Information System EU................................................................ European Union DIE............................................. German Development Institute GEC..................................................Girls’ Education Challenge GPE.......................................... Global Partnership for Education GDP..................................................... Gross Domestic Product IDB........................................ Inter-American Development Bank IDA................................... International Development Association IEG..............................................Independent Evaluation Group IPF............................................... Investment Project Financing IVA............................................. Independent Verification Agent MINEDH................. Ministry of Education and Human Development NSP............ National Strategic Plan for Early Childhood Development PASEC...Programme d’analyse des systèmes éducatifs de la Confemen PforR.........................................................Program for Results RBF.....................................................Results-based financing REACH.................................. Results in Education for All Children SERCE............ Second Regional Comparative and Explanatory Study SDI...................................................Service Delivery Indicators SDG.............................................Sustainable Development Goal SABER................... Systems Approach for Better Education Results USAID...............United States Agency for International Development WBG............................................................ World Bank Group WDR.................................................World Development Report FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 3 SUMMARY As demand for and interest in results-based financing (RBF) grows, the World Bank Group (WBG) supports a Results First approach as the optimal way to implement RBF. Results First means identifying the desired results and then working backwards to achieve them. This is in contrast to traditional aid approaches that primarily focus on inputs. While, in theory, all well-designed projects should include a discussion around results, Results First prioritizes that conversation and shifts both country government and development partner attention and effort in a way that traditional financing currently does not. This paper presents four theories of change that showcase how RBF can strengthen education systems for lasting change: THE WORLD BANK GROUP 4 Summary RBF: FOUR THEORIES OF CHANGE Flip the Policy Dialogue to Where Sustain Attention across Crises, We Want to be in Five Years Fads, and Changing Governments Flipping the policy dialogue makes better use of tech- In the hectic, crisis-response world of education min- nical expertise, improving chances of resolving bottle- istries, an instrument for sustained attention to the necks. Rather than marshal through the myriad inputs results that matter most is sorely needed. Political and activities related to education, RBF forces the con- time horizons and competing short-term urgencies versation to start with results, and then gets actors to can often supplant a government’s long-term policy work backwards to identify what is needed to achieve agenda. However, sector plans that have medium and the desired outcome. Once results are identified, the long-term results embedded and financially rewarded corresponding critical pathways to reaching those over time helps keep all eyes on the prize despite the results can be plotted out, identifying binding con- political economy. What is more, allocating financing to straints along the way. This encourages a new way of the achievement of certain results helps authorities sig- thinking, which concentrates on content, not process, nal the importance of them. Coupled with strong mon- and moves away from the traditional, inputs-driven itoring, this signal can in and of itself induce behaviour financing pattern. In RBF, financing is instead attached change. Prioritizing specific results can also provide to those milestones that will solve system bottlenecks, overworked ministries of education with much appre- strengthening the coherence of the systems that will ciated relief, as this prioritization helps whittle down the improve results. activities to be carried out. Align All Actors Around Institutionalize Measurement Results that Matter Systems for Lasting Impact Attaining results most efficiently and sustainably requires Achieving sustained results is impossible without accu- the involvement of multiple actors, notably ministries rate and timely measurement, providing feedback loops of finance. Examples abound of countries that have for course correction. RBF only functions with strong increased funding to their education sectors, without the monitoring systems that link information from frontline corresponding uptick in performance. The inefficiency service-delivery to managers and policy makers who of current spending, and the lack of alignment between are able to make use of the information, course correct, spending and results, are well documented (Hanushek, and re-inform service providers. In many countries, there 2003; others). RBF offers a chance to place momentum is very limited measurement outside of those indicators behind larger public financial management reforms collected through Education Management Informa- that influence spending in sectors like education. For tion Systems (EMIS), which themselves are rarely very example, the introduction of performance-oriented fis- robust. RBF therefore seeks to strengthen existing EMIS cal transfers from ministries of finance to subnational systems, oftentimes through independent verification, entities is one area where countries have shown much which not only fosters a culture of measurement, but interest. These types of transfers can be used to correct bolsters the capacity for system measurement and for vertical or horizontal fiscal gaps (for more informa- monitoring in countries. tion, see section How: A Systems Approach). To achieve these, it is often necessary to galvanize all relevant actors, not just those in education, with the ministry of finance most often playing the role of the heavyweight. FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 5 INTRODUCTION RESULTS-BASED FINANCING captures the essence of how the WBG is implementing its education strategy “Learning for All”. The strategy emphasizes learning (rather than just schooling), since that is what leads to economic growth, development, and poverty reduction. Second, the strategy calls for investing “for all” so as to ensure the equity dimension behind the SDGs and the Bank’s Twin Goals of reducing poverty and boosting shared prosperity. These goals can be achieved through strengthening education systems: elements such as schools, teachers, curricula, materials, as well as policies, accountability mechanisms, resources and financing, that, when working together, result in high quality services for children. Crucially, this implies moving beyond simply providing more inputs, to operating in a way that will bring alignment among system components, thereby fostering greater results. How resources are mobilized and managed to flow to — and through — education systems offers one such pathway for coalescing efforts, providing incentives, bringing attention to results, and making actors accountable when results aren’t achieved. RBF has the potential to transform how educa- in education systems, overcoming challenges RBF can tion systems operate, and the types of results that exclude children and youth from attending foster the they achieve for children the world over. Gen- schools, and prevent learning. right incentive erally speaking, RBF does this by rewarding the structures in education delivery of education outcomes through financial The WBG helps clients invest in their education systems, incentives, upon verification that the agreed-upon systems, in order to deliver more and better ser- overcoming result has been achieved, in a manner that can be vices. This support typically takes the form of challenges that credibly sustained over time. As a financing instru- financial services, technical and advisory ser- exclude children ment, RBF shows much promise for helping clients vices, and convening services. Working within the and youth from enroll the 121 million children still out of primary WBG Education Practice’s Systems Approach for attending schools, and and lower secondary school (UNESCO, 2015), and Better Education Results (SABER) diagnostic and prevent learning. teach the 250 million in school but still unable to benchmarking platform, RBF can serve as a tool to read or write (EFA Global Monitoring Report, 2013– strengthen systems, improving how separate pro- 2014). RBF can foster the right incentive structures cesses align. THE WORLD BANK GROUP 6 INTRODUCTION This Approach Paper therefore outlines the World The operational implications of this view of RBF Bank Group’s approach to Results-based financ- on the WBG are presented, with a priority agenda ing in the education sector. It seeks to define RBF for the coming years. Given the paucity of exist- (building on the existing glossary for the health ing evidence in the education space on the effec- sector), and argue for the use of RBF primarily tiveness of RBF, the Approach Paper outlines its as a systems-strengthening tool, as opposed to limits, discussing the risks (and corresponding a more hands-off, “laissez-faire” version of RBF. mitigation thereof) of results-based approaches. SYSTEM REFORM, “Strengthening education systems means aligning their governance,management BEYOND INPUTS of schools and teachers, financing rules, and incentive mechanisms with the goal of learning for all. This entails reforming relationships of accountability among the various actors and participants in an education system so that these relationships are clear, consistent with functions, measured, monitored, and supported. It also means establishing a clear feedback cycle between financing (including international aid) and results. Because failures of governance and accountability typically have their severest effects on schools serving disadvantaged groups, this system approach promotes educational equity as well as efficiency.” - WBG Education Strategy “Learning for All”, 2011 , and Why do they Matter? What are “Systems” Stronger education systems are how student Fostering the desired outcomes that the systems learning will increase sustainably and contin- seek to produce requires system coherence. uously over the long-term. Taking a “systems Typically, systems are designed to achieve results approach” implies a careful consideration of how in coverage (Pritchett, 2015) as the last 15 years the elements and parameters within a system inter- have been building towards the achievement of act with each other and as a whole, and where bot- Millennium Development Goal. The challenge for tlenecks or binding constraints are impeding the achieving the education goals in the SDGs will be to system’s ability to transform inputs into outcomes reshape system coherence such that they align to (Newman, King, & Abdul-Hamid, 2016). This is produce learning. As a recent DFID study observes, different from a project approach, which takes a “past education sector reforms have not resulted narrower view of one part of the system, or even in the desired speed and scale of improved learn- a sector approach, which typically considers an ing outcomes … One proposed way of overcoming entire level of education, but with little consider- these issues is to move away from a mechanistic ation to how the parts of the system work together. reading of the education sector toward diagnosing FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 7 RESULTS-BASED FINANCING APPROACH (RBF) FOR EDUCATION Lack of System Coherence Traditional aid models focus on financing inputs in an uncoordinated way, creating dissonance between various parts of the education Leads to Low Impact system. Money flows between organizations, actors and inputs without and Inefficiency accountability for results. ACTORS Financial Resources Directors Ministry of Teachers Finance Students Ministry of INPUTS Education Curriculum Results? Materials Provinces / States Teacher Training Testing Education Management Information Systems Schools THE WORLD BANK GROUP 8 INTRODUCTION malfunctions in the system through which outputs This paper largely focuses are translated into outcomes” (Faul, 2016). In addi- on how finance, tion to being at the heart of the WBG Learning for and RBF All strategy, a focus on bringing systems into align- specifically, ment will feature in the forthcoming World Develop- can be used as ment Report 2018 “Realizing the Promise of Educa- a lever to pull tion for Development”. education systems into Financing is one of the four key components to coherence. making systems coherent. Making a system work as a whole requires relationships of accountabil- ity among and between the parts of the system. These relationships have four design components: delegation, finance, information, and motivation (Pritchett, 2015). This paper largely focuses on how finance, and RBF specifically, can be used as a lever to pull education systems into coherence. This approach is further underscored by the findings of the Education Finance Commission’s report for The Learning Generation (2016). The Education Finance Commission’s report calls for a Finance Compact between developing coun- tries and the international community (Education Finance Commission, 2016). RBF encompasses all of the four transformations called for: performance, innovation, inclusion, and finance. Specifically, put- ting results first (performance), and embracing results-based approaches to finance (finance) is at the heart of the WBG approach. The approach also offers a new way of approaching problems (innova- tion), and incentivizing results allows for differenti- ating levels of rewards, in order to favor traditionally disadvantaged groups, such as rural girls or children living with disabilities (inclusion). FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 9 INTRODUCTION The Education Finance Crisis that RBF Might Prevent Over the next 20 years, countries around the world learning opportunities for all”. To do so, the inter- To achieve will face a historic triple challenge: to universal- national community has agreed on 10 targets, the SDGs, ize primary and secondary education services, starting with ensuring that children finish primary even more resources will simultaneously improving schools —in a context and secondary school, having acquired meaning- be required, of increased fiscal pressure. While these tasks ful learning outcomes. Measures of learning at and RBF can will play out differently in each country and region, mid-primary, end-primary, and end lower-second- be a financing they will likely have the characteristics listed below. ary have been approved in principle, with specific solution to indicators still under discussion (United Nations, help crowd in The Sustainable Development Goals1 seek to 2016). These are ambitious, and the SDGs, in gen- funds towards guide global efforts in tackling these first two eral, have lofty goals of “overcoming poverty and development challenges. With regards to universalization, it objectives. protecting the planet” (Development Committee, is estimated that 121 million children are still 2015). To achieve them, the joint Development excluded from primary and lower secondary Committee (made up of multilateral aid organiza- school. Perhaps more worrisome are the more tions) has proposed a “billions to trillions” agenda than 250 million that, despite being enrolled in where even more resources will be required, and school, have not been taught to read or write. Goal identified RBF as a specific financing solution that 4 of the Global Goals aims to “Ensure inclusive and can help crowd in more funds towards develop- equitable quality education and promote lifelong ment objectives. THE TRIPLE UNIVERSALIZATION OF PRIMARY AND SECONDARY EDUCATION CHALLENGE Reaching those still excluded from services, whether girls, ethnic minorities, COUNTRIES WILL children living with disabilities, or the rural poor, will imply higher than average FACE OVER THE NEXT 20 YEARS marginal costs. BETTER SCHOOLS Ensuring that students acquire the basic skills needed for future study, work, and life implies services superior to those typically on offer. Improving education services may incur added costs, though it could ultimately result in cost savings. FISCAL PRESSURE In most countries education is already the most prevalent sector in the budget, consisting of the largest proportion of the civil service, and accounting for the largest share of recurrent government spending. As countries try to achieve more at current levels of education financing, in a context where climate change and other crises increasingly compete for scarce resources and cynicism over what existing financing can achieve (Education Finance Commission, 2016), the pressure for ministries of finance and education to make education financing more impactful and efficient will likely increase correspondingly. 1 For more information, please see United Nations (2016). THE WORLD BANK GROUP 10 THE RISE OF RBF RBF is on the rise, for differing reasons. This section starts with an operating definition for RBF, and then outlines the competing motivations behind RBF, and places the WBG approach in that landscape. It traces a brief history of the movement, from standards- based reforms through to the use of financing as a lever for strengthening systems. Operating Definitions of RBF What is results-based financing? RBF means What is a credible, independent verification? Results can different things to different people. For the WBG, Given the consequences of meeting and missing be outputs, it is an umbrella term referring to any program or targets, verification takes on added importance intermediate outcomes, final intervention that provides rewards upon the credi- under RBF. Typically, verification procedures under outcomes (such ble, independent verification of an achieved result. WBG operations call for “credible, independent” ver- as learning) Rewards can be monetary or non-monetary, and ification procedures, oftentimes including the use or a mix. can be partial (e.g. a bonus on top of a salary) or of existing country systems. Though “independent” whole (e.g. the unit cost under output-based aid). usually implies third party actors, this is not always the case, since full-scale verification by private What are results? For the WBG, results are those agents can often be cost prohibitive. Output-based elements within a results chain that lie beyond the aid projects typically make use of third party actors input stage. They can be outputs, intermediate called independent verification agents (IVA) outcomes, final outcomes (such as learning) or — (Loening & Tineo, 2012). Several lessons of rele- more likely —a mix. Importantly, the dividing line vance to the education sector, particularly from between inputs and outputs may be context-spe- infrastructure operations and health, have been cific, depending on the particular bottleneck that documented (see, for example, Perazzo & Joseph- RBF seeks to resolve. For example, getting books son, 2014) and are being incorporated into training into the hands of children would typically be con- programs on RBF, such as training for the Bank’s sidered an input, and having the child use the book Program-for-Results lending instruments. Exam- would be an output (an outcome would be that ples of such lessons are the importance of internal she could read). However, if it is found that chil- consistency between regulation, management dren cannot read because books are unavailable information systems, and the verification protocols, in their school, then we would consider achieving and the role of independent verifiers as mediators that state (books in the hands of children) as a among actors. result worth incentivizing. FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 11 THE RISE OF RBF What Have Others Said about RBF? The literature has explored the conceptual The Principal-Agent model is also used by Clist basis and reviewed the experience to date with and Verschoor, who focus more on weighing the RBF in education. This section discusses only the costs and benefits of using RBF relative to other main components of that foundation, which are forms of aid. The authors advocate that RBF can be mostly taken from economic theory. There is also successfully used to improve efficiency by “linking a non-exhaustive review of the research around the agent’s costs with the principal’s utility, in a situa- accountability, contracts, incentives, motivation, tion where the agent’s action cannot be observed.” and a look at the limited number of evaluations (Clist & Verschoor, 2014). Another argument is that conducted thus far.2 (For more details on this RBF is unnecessary in the case of full alignment review, please see the Annotated Bibliography). between donor and recipient because it renders performance incentives moot and thus becomes The Principal-Agent model dominates the ana- much more inefficient than a simple transfer (Clist lytical underpinnings of how to think about RBF. & Dercon, 2014). In most high-capacity country According to this model, the “principal” is the donor contexts, this proves to be true, making RBF a less or funder, and the “agent” is the recipient. Perakis powerful financing modality. and Savedoff (2015) describe this theory as “a diver- gence in objectives between principals and agents A consideration when applying RBF is whether which principals “solve” by offering a contract that or not the right behaviours are incentivized. It aligns agents’ incentives with their own”. The agents is important for principals to review the types of can then either embrace the principal’s objective behaviours that are actually rewarded, and not and be rewarded by the principal, or continue to just the ones they intended to reward (Kerr, 1995). pursue their own objective, thereby foregoing the The idea that what is incentivized can be different reward. In “Cash on Delivery: A new Approach to from what is desired is also a concept outlined in Foreign Aid”, Nancy Birdsall and William Savedoff Multitask Theory, one that Clist and Verschoor (2010) bring this model to life with a proposed (2014) also use to analyze RBF. This model indi- approach. Cash on Delivery Aid (COD) positions cates that if an incentive is tied to a specific result, RBF as a way to improve foreign aid by altering such as student completion rates, the agent can the relationship between principal (the donor) and still choose a range of options to achieve this tar- agent (the recipient) to achieve shared goals. Bird- get, regardless of whether or not their choice is the sall and Savedoff promote a “hands-off” approach, “best” choice. For example, an agent could choose indicating that the less a donor interferes in the to prevent drop out or repetition, but that would take design and implementation, the more discretion away the focus from learning gains. Some emerg- and responsibility the agent has, with the emphasis ing evidence from health suggests that this risk on transferring ownership to the recipient. does not actually manifest itself at the level of the service providers (Das, Friedman & Mutasa, 2016). 2 Some of these evaluations include a look at five case studies by DIE: Improving Education Outcomes by Linking Payments to Results: An Assessment of Disbursement-linked Indicators in five Results-based Approaches (2015); DFID: Evaluation of Payment by Results (PBR): Current Approaches, Future Needs (2013); and USAID: Incentives and Accountability in Education: A Review (2014). THE WORLD BANK GROUP 12 THE RISE OF RBF Evolution of RBF in the WBG Education Practice The WBG has pledged to double its results- heavily focus on alignment of education systems based financing in education to US $5 billion by and greater school accountability through mon- 2020. At the World Education Forum in Incheon in itoring and, in many instances, student assess- May of 2015, President Kim announced that this ments (Hamilton, Stecher, & Yuan, 2008). The next promise was part of the WBG commitment to end generation of reforms seek to institutionalize the extreme poverty in the world by 2030 by improv- accountability relationships throughout the sector. ing both the quality and equity of education, so Finance represents a tool to ensure compliance that all children are learning the skills they need to with standards, as well as providing a strong feed- lead more prosperous lives. Fulfilling this pledge back mechanism back to authorities about the will make the WBG the largest actor in the RBF achievement of standards, and incentivizing the space. It will require an intensification of efforts continuous moving of the goalposts such that sys- across the WBG, as well as close partnerships with tems can, ideally, be placed on a virtuous improve- international actors and donors active in this area, ment cycle. particularly the Global Partnership for Education (GPE), which has introduced a results-based vari- The rise of RBF in education follows on the heels able tranche in its new funding model. It will also be of successes in the health sector. Launched in important to learn from the early lessons from other 2007, the Health Results Innovation Trust Fund By making a sectors (health in particular), as well as from other (now known as RBF Health) seeks to improve commitment to development partners such as DFID. maternal and child health throughout the world. double results- The fund started with five pilot programs in 2008, based financing Results-based financing is a natural evolution and today supports 36 RBF programs in more than in education to US $5 billion by of support to a systems approach to education 30 countries. Results fall under six maternal and 2020, the World systems. This trend can be dated back to the child health indicators and include over 15 million Bank Group standards-based reforms of the 1990s, one of women receiving antenatal care and over six mil- will become the the most well-known being No Child Left Behind lion children fully immunized. The fund has used largest actor in the United States. These reforms tended to its $400 million in financing to leverage $2.2 billion in this space. THE WBG HAS These will be further discussed in the section The World Bank and RBF: RESPONDED Implications for the Business Model. Starting in the mid-2000s, the WBG began TO CLIENT to support client RBF programs by linking disbursements of WBG lending DEMAND FOR RBF THROUGH operations to the achievement of specific disbursement-linked indicators. VARIOUS This led to the launch of a new financing instrument in 2012, Program for INSTRUMENTS Results (PforR), which focuses on using country systems and disburses upon the achievement of results. PforR was designed in response to both client demand and WBG staff interest. As of January 2017, there have been 52 PforR operations across all sectors, totaling US $13.0 billion of Bank financing and US $60.3 billion in government programs. FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 13 THE RISE OF RBF in IDA financing. Among the global leaders in this particularly difficult to apply in fragile, low capacity RBF is area are DFID, Germany, Norway and USAID, all of contexts, though it can be used in a way that estab- particularly whom finance RBF initiatives at the WBG. lishes preconditions for future RBF (see the section difficult to apply in Artisanal RBF for more detail). However, an advan- fragile, low While RBF is an emerging financing method that tageous feature of RBF is its ability to delink pay- capacity sparks large interest at the WBG and in the inter- ments from inputs, which creates more discretion contexts, national development sphere, it is not always the for recipients to execute their budgets where the though it can “right” choice. For example, in countries that have funds will have greatest impact (Savedoff, 2016). be used in a large infrastructure needs, i.e., building schools or way that Ultimately, it is up to governments and teams to establishes rehabilitating schools, it may make more sense to choose financing instruments and modalities that preconditions take a traditional financing route. Moreover, RBF is work best in any given context. for future RBF. Results First: Neither “Laissez-Faire”, nor “Backdoor” RBF While many agencies have espoused RBF, the name only (for purposes of accelerating disburse- WBG emphasizes a hands-on approach. We char- ment), while continuing with business as usual. acterize three types of RBF: i) Results First, where- by financiers and clients jointly commit to achiev- Results First implies envisioning how the future ing results, travelling the path from conception will be different, and working backwards to figure to results together. This approach best describes out how to get there. While many well-designed the WBG approach; ii) “Laissez-faire” RBF, where- projects may account for this already, explicitly by donors seek to incentivize improved learning defining the Results First approach helps further outcomes, through any means necessary; and sharpen the benefits of RBF. Historically, conversa- iii) “Backdoor RBF”, which seeks to use RBF in tions under traditional financing start with disparate THREE TYPES RESULTS FIRST OF RBF Financiers and clients jointly commit to achieving results, travelling the path from conception to results together. This approach best describes the WBG approach. LAISSEZ-FAIRE” RBF Donors seek to incentivize improved learning outcomes through any means necessary. “BACKDOOR RBF” Seeks to use RBF in name only (for purposes of accelerating disbursement), while continuing with business as usual. THE WORLD BANK GROUP 14 THE RISE OF RBF inputs (e.g., the Ministry of Education needs to pay itself well to the WBG and its relationship with its In the for teacher training, computers for classrooms and clients. The idea that the WBG can shift the ‘pecu- WBG [RBF] school grants), while RBF immediately starts the niary interests’ of partner governments makes the experience, principals conversation around results. This approach has assumption that the agent (client governments) and agents taken various forms in a number of countries. In does not inherently value the result, at least not are equally Jamaica, the process was termed “critical path- as much as the principal (the WBG). This has not motivated in ways”, with disbursement-linked targets identified borne out in the WBG experience. Rather, prin- the pursuit to remove the blockages that impeded the results cipals and agents are equally motivated in the of results. the country seeks on early childhood develop- pursuit of results, and client ownership and the ment. In Malaysia, they refer to it as Big Fast corresponding political will are preconditions for Results, and in Tanzania, Big Results Now. The Mil- successful RBF. Also, as Perakis and Savedoff lennium Challenge Corporation calls its process (2015) put it, “governments are not unitary actors the “continuum of results” while Andrews, Pritchett with well-defined preferences of direct links and Woolcock (2012) name their model “Prob- between decisions and actions”, rendering the lem-Driven Iterative Adaptation”. These processes principal-agent model difficult to apply. Finally, all have the same elements in common: they put clients on the whole request that WBG accom- results first, and then work backwards on how to pany them with technical assistance along the get there. Along the way, they identify the stum- results chain from inputs to outcomes. bling blocks, and seek to resolve them through incentives. This form of RBF is not concerned with “Backdoor RBF” is a results-based approach the source of financing (whether aid, public invest- whereby disbursement modalities differ, but the ment, or other). rest of the relationship remains traditional. Back- door RBF uses RBF in a faddish way meaning that “Laissez-faire” RBF rests on the notion that tra- though results or DLIs are identified, the Results ditional aid patterns have not generated the Framework doesn’t necessarily change and the sought results. Rather, those best placed to solve rest of the conversation continues to be focused the problems related to generating learning out- on inputs, as usual. This type of RBF is not truly comes are those closest to the context. A common using financing to synchronize goals between example of Laissez-Faire RBF is Cash on Delivery donor and recipient, instead seeking to acceler- Aid, developed by the Center for Global Develop- ate disbursements. A great danger in Backdoor ment, whereby donors are hands-off (rather than RBF is that when results are not achieved, and paternalistic) in their financing of aid programs, nonpayment enforced, both donor and recipient “emphasizing the power of incentives rather than are surprised and frustrated. An example of Back- guidance or interference” (Birdsall & Savedoff, door RBF is when projects are initiated by first 2010). This view implicitly discounts the technical choosing the financing modality (RBF) rather than contribution that donors can bring to the table, as thinking about the desired outcomes and whether well as the technical contribution that central gov- or not RBF can help achieve those. In these cases, ernments can bring to schools. it was only at the time of non-disbursement in the context of non-achievement of results that recipi- In reality, a laissez-faire approach has limited ents become fully cognizant of the implications of value for the WBG and its clients. The Princi- RBF (Coffey, 2016). pal-Agent model that underpins it does not lend FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 15 THE RISE OF RBF New Conditionality, or Easy Money? Results First is therefore not a new form of con- for budget support that is unavailable due to poor The value of ditionality. RBF cannot substitute political will. The macro-economic country conditions. Using RBF in the RBF results that it seeks to generate are those shared this way is unsustainable over time. approach lies and, ideally, co-created, with client partners. It in a new way of does not seek to shift pecuniary interests. Instead, In fact, the degree to which the money itself mat- thinking about achieving it seeks to use financing as a common thread for ters as an incentive is unclear. As will be seen in the results through helping to align parts of the system, and actors, next section, the value of the RBF approach lies in a programs. into coherence (along with the other needed ele- new way of thinking about achieving results through ments of accountability relationships). programs. It seems that this value can be harnessed irrespective of the level at which the financial reward Neither is it easy money. Whether the sources is placed. For instance, in the Jamaica ECD project, of financing are national budgets or international the DLIs were only worth $180,000, and yet that funding resources, ministries of finance and of edu- seemed to be a powerful enough incentive to inten- cation, and donors alike will be under increasing sify efforts. To put this in economic terms, perhaps pressure to demonstrate more impact for financ- instead of having ‘high-powered’ incentives, what ing. In contrast to traditional input-based aid financ- matters more is a strong signal-to-noise ratio, that ing, whereby financing was guaranteed, regardless is, that the indicators are calibrated such that front- of achievement of results, or chronic delays in proj- line actors are able to respond to the incentives. ect implementation, the financial architecture of This implies avoiding indicators that may contain RBF promises more efficient and impactful spend- too much statistical noise, as well as those that are ing. This being said, RBF is not a suitable substitute beyond their control. THE WORLD BANK GROUP 16 WHY WE BELIEVE IN RBF WHY WE BELIEVE IN RBF: FOUR THEORIES OF CHANGE Among the many reasons behind the WBG commitment to a Results First approach, four stand out. First, it flips the policy dialogue, with external partners and internally within ministries, to focus more intentionally on what elements in the education system can be improved upon to generate results. Second, it attracts and retains much needed attention — from policymakers to parents — to the ultimate results that are sought. Third, it serves to galvanize and align important (yet often uninvolved) actors in the pursuit of results, notably ministries of finance. Finally, it instills a culture of measurement, which can eventually be institutionalized through strengthening the country’s systems. FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 17 WHY WE BELIEVE IN RBF Flip the Policy Dialogue to Where We Want to be in Five Years Flipping the policy dialogue makes better use the government. Once identified, the correspond- of technical expertise, improving chances of ing critical pathways to reaching those results resolving bottlenecks. Rather than marshal can be plotted out, identifying binding constraints through the myriad inputs and activities related to along the way. This encourages a new way of education, RBF forces the conversation to focus thinking, which concentrates on content, not pro- on developing a theory of change that starts with cess, and moves away from the traditional, inputs- results, prompting actors to work backwards to driven financing pattern. In RBF, financing is instead identify what is needed to achieve the desired attached to those actions that will solve system outcome. This helps focus the discussion within bottlenecks, strengthening the coherence of the education ministries, and between the WBG and systems that will improve results. clients, on which results are truly priority areas for CASE STUDY 1 FLIPPING In 2014, the Dominican Republic announced its National Pact for . THE POLICY Education Reform (World Bank, 2015) which aims to recruit and train DIALOGUE primary and secondary school teachers; assess student learning in primary IN THE and secondary schools; evaluate early childhood development services. DOMINICAN and help decentralize public school management. REPUBLIC During project preparation, many of these objectives were not part of the conversation, namely because discussions between the WBG team and client took a “business as usual” route and centered on inputs such as building early childhood care centers or secondary schools. However, given the context — in 2006, Dominican students ranked last among test takers in regional student assessments (SERCE, 2006) — there was an obvious need for changes that would lead to In 2006, Dominican improved learning. students ranked last among test takers in To get at the heart of the matter, rather than think narrowly about infrastructure regional student assessments. activities, the ministry, together with the WBG team and international financing partners like the Inter-American Development Bank (IDB) and the European Union (EU), and with civil society actors, looked at the larger picture and embarked on an exercise to identify the results the country wanted to achieve by 2020. Working backwards, they then thought about the results chain and how each policy action would influence the next. At the time, a survey of mathematics teachers had come out from the Dominican Institute of Research and Evaluation which revealed that only THE WORLD BANK GROUP 18 WHY WE BELIEVE IN RBF FLIPPING a minority of teachers (45 percent) could correctly answer 50 percent or more of THE POLICY questions related to the content they were supposed to be teaching. If teachers DIALOGUE themselves could not answer the questions, how could they be expected to properly IN THE teach students? DOMINICAN Instead of fixating on single-issue policies such as teacher training or teacher REPUBLIC recruitment or student assessments, the ministry examined the “system” that was in place to ensure children received a quality education. The sector is governed by a set of institutions with complex relationships: the Ministry of Education and the Ministry of Higher Education, Science, and Technology, which are the two primary governing ministries of the sector, and the Salomé Ureña Teacher Training Institute, Autonomous University of Santo Domingo and National Institute of Education and Training of Teachers, the three largest providers of pre-service training. Prior to the project, there was no alignment and haphazard division of labor among actors. As part of the RBF approach, the Government, with support from the WBG and partners, examined how teachers were recruited and trained, and how student assessments were administered. While it is always important to recruit and train qualified teachers, the question then became, why didn’t this already happen? The greatest binding constraint on recruitment of high quality teachers was that there was no mechanism in place to screen potential candidates. The Government chose to resolve this by introducing a competitive entrance exam, to filter out unqualified candidates. To encourage sustained motivation, the exam was tied to a disbursement of funds under the World Bank loan that accompanies the implementation of the National Pact. When the National Pact planning committee turned to address the issues around measuring learning, it became clear that the 8th grade leaving exam served more as an ex-post “autopsy” of the poor learning levels of students, rather than an ex-ante diagnostic of which students were struggling, and with what parts of the curriculum. If they did not pass, they most likely dropped out, and there were no other assessments in earlier grades to detect learning gaps or diagnose other learning challenges. In other words, there was no data to inform teachers on how to focus their efforts. To change this, a DLI was tied to developing a student assessment in grade 3, and using that data to improve teacher training programs. The Government’s approach to identifying results and bottlenecks allowed for: (i) the alignment of actors and actions around results via DLIs and (ii) a shift in the policy dialogue that enabled various parts of the system to “talk” to one another, rather than piecing together ad-hoc interventions that only addressed parts of the system. FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 19 WHY WE BELIEVE IN RBF Sustain Attention across Crises, Fads, and Changing Governments In the hectic, crisis-response world of education importance of them. For example, the government ministries, an instrument for sustained attention of Lebanon financially rewards schools serving Syr- to the results that matter most is sorely needed. ian refugees who raise average attendance rates, Political time horizons and competing short-term underscoring that the government is serious about urgencies can often supplant a government’s long- schools reaching out to families and making sure term policy agenda. However, sector plans with kids come to school. Coupled with strong monitor- embedded medium and long-term results that ing, this signal can in and of itself induce behaviour are financially rewarded over time helps keep all change. Prioritizing specific results can also pro- eyes on the prize despite the political economy. vide overworked ministries of education with much What is more, allocating financing to the achieve- appreciated relief, as this prioritization helps whit- ment of certain results helps authorities signal the tle down the activities to be carried out. CASE STUDY 2 SUSTAINED Jamaica launched its first National Strategic Plan for Early Childhood ATTENTION Development in 2008, and the second in 2013. The Plan features a number THROUGH RBF of results in education, health, and social protection marking important IN JAMAICA milestones in the implementation of the strategy. The development objectives are to: improve the monitoring of children’s development, the screening of household-level risks affecting such development, and early intervention systems of the Government to promote such development; enhance the quality of early childhood schools and care facilities’; and strengthen early childhood organizations and institutions. The disbursement-linked targets include the increase in the percentage of preschools that have permits to operate, the increase in the number of health centers offering well-child clinics, and the number of additional children covered by the social protection system (World Bank, 2008a). This strategy was This strategy was brought to life by several government agencies recognizing the brought to life by several government need to improve human development outcomes, given the poor school readiness agencies recognizing of Jamaican children. In 2003 the Government took legislative action to revamp the need to improve human development the organization and coordination of the early childhood development (ECD) outcomes, given the sector and established the Early Childhood Commission to oversee and coordinate poor school readiness ECD activities. Both the previous and current governments (at the time) had also of Jamaican children. increased budgetary allocations to the sector, signaling its importance. The ECC THE WORLD BANK GROUP 20 WHY WE BELIEVE IN RBF SUSTAINED led a multi-sector process to define and validate national objectives for ECD and ATTENTION set implementation milestones to keep things on track, which were conditioned on THROUGH RBF disbursements. These milestones range from the development of a child “health IN JAMAICA passport” to ensuring that a majority of ECD institutions have been inspected for quality and national registration. One of the greatest achievements of the National Strategic Plan for Early Childhood Development is that it survived, and is now in its second phase, despite political changeover. Though the plan has bipartisan support, its survival is largely because RBF ensures that targets are being set in real-time, so new governments cannot change policy direction easily. If the model used had been more of a traditional one, the results would have been assessed later down the line, but RBF keeps results at the forefront. Align All Actors around Results that Matter Attaining results most efficiently and sustain- transfers from ministries of finance to subnational ably requires the involvement of multiple actors, entities is one area where countries have shown notably ministries of finance. In other words, much interest. These types of transfers can be “align all the stakeholders with power” (USAID, used to correct for vertical or horizontal fiscal 2014). Examples abound of countries that have gaps3. To achieve these, it is often necessary to increased funding to their education sectors, with- galvanize all relevant actors, not just those in edu- out the corresponding uptick in performance. The cation, with the ministry of finance most often play- inefficiency of current spending, and the lack of ing the role of the heavyweight. Another example alignment between spending and results, are well is how RBF can help bring the Paris Declaration of documented (Hanushek, 2003; others). RBF offers donor harmonization to life, such as in Lebanon, a chance to place momentum behind larger pub- whereby indicators selected for disbursement lic financial management reforms that influence were taken from the multi-agency RACE pro- spending in sectors like education. For example, gram, supported by all donors, including UNICEF, the introduction of performance-oriented fiscal UNHCR, and DFID (World Bank, 2016a). See Case Study 3 on the following page. 3 These transfers will be described in further detail under How: A Systems Approach. FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 21 WHY WE BELIEVE IN RBF CASE STUDY 3 WORKING A Program-for-Results operation (World Bank, 2014) in Mozambique seeks ACROSS to improve the transparency and efficiency of expenditures for the storage, SECTORS distribution and availability of medicines and for more transparent and TO ALIGN accountable management of primary schools. ACTORS IN MOZAMBIQUE This multi-sector project is framed within the Government’s comprehensive public financial management reform. While other areas, such as cash management and bank reconciliation have made progress, there remains much to be done in the daily operation of institutions, including the Ministry of Education and Human Development (MINEDH). In the education sector, the Government began a wholesale system reform starting with the decentralization of financial and management authority to district and schools in order to support efficient, transparent and accountable use of public resources to In the education sector, deliver results. Previously, efforts were partly unsuccessful due to obstacles such as the Government began a wholesale system limited capacity of financial management and monitoring at the district and school reform starting with level and inadequate transparency in budget allocation. To bring about better human the decentralization development outcomes, the Government decided to take a more programmatic of financial and management authority approach and incentivize ministries, provinces, districts and service units to work to district and schools together to achieve results. in order to support efficient, transparent To do this, the Program-for-Results operation conditioned transfers to MINEDH based and accountable use of public resources to on achieving targets such as the introduction of a new spending classification, and the deliver results. transfer of school grants in time for the new school year. These two indicators required coordination and collaboration between the Ministry of Finance and the line Ministry, contributing to align actors around a common objective. In 2016, for the first time, all school grants reached the school on time. Similarly, the Program for Results introduced performance-based incentives with districts to ensure that district offices perform supervision visits with adequate frequency and using the appropriate methodology. RBF applied in this scenario allowed diverse stakeholders to come together and work together, however difficult, to achieve a certain set of results. THE WORLD BANK GROUP 22 WHY WE BELIEVE IN RBF Institutionalize Measurement Systems for Lasting Impact Finally, achieving sustained results is impossi- of those indicators collected through Education ble without accurate and timely measurement, Management Information Systems (EMIS), which providing feedback loops for course correction. themselves are rarely very robust. RBF therefore RBF only functions with strong monitoring systems seeks to strengthen existing EMIS systems, often- that link information from frontline service-deliv- times through independent verification, which not ery to managers and policy makers who are able only fosters a culture of measurement, but bol- to make use of the information, course correct, sters the capacity for system measurement and and re-inform service providers. In many coun- monitoring in countries (ideally, education author- tries, there is very limited measurement outside ities themselves). CASE STUDY 4 INSTITUTION- There have been iterations of projects in the Sindh (World Bank, 2009a) . ALIZING and Punjab (World Bank, 2009b) provinces of Pakistan that introduced . MEASUREMENT large education sector reforms. Sindh is Pakistan’s second largest province AND FEEDBACK and known to be a commercial and industrial hub, but despite its relatively LOOPS IN strong economic position, enrollment rates, especially among rural girls, PAKISTAN remains quite low and public education expenditures accounted for only . 1.5% of GDP in 2008. In Punjab, the most populated province, there were still 38% of out-of-school children in 2009. Both provinces suffered from poor, non-strategic budget management and low learning outcomes. In fact, Pakistan has some of the lowest education levels in the world, with adult literacy at 50 percent in 2005 (Evidence to Policy, 2012). Both provincial governments launched sector-wide reforms to raise the quality of education. Pakistan has some of In Punjab, part of the reforms included subsidies to low-cost private schools on the the lowest education levels in the world, with condition that they improve access to education by waiving tuition fees and meeting adult literacy at 50 specific pass rates on a standardized test. An impact evaluation found that schools percent in 2005. who were going to get cut off from the subsidy always managed to raise scores. While this sounds like a situation ripe for gaming, integrity was maintained through extensive monitoring and randomization (i.e. schools didn’t know what day they would be tested, and there were multiple versions of the test). In addition, there was concentrated effort to ensure that the existing EMIS was used as the primary repository of school data so that policymakers could use it to make more informed decisions. FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 23 WHY WE BELIEVE IN RBF INSTITUTION- Similarly in Sindh, an RBF intervention paid schools more to raise the enrollment ALIZING rates of rural girls. To ensure that the data was accurate, there were school inspections MEASUREMENT and reported information, such as attendance, was cross-verified and checked. Such AND FEEDBACK monitoring and feedback mechanisms were also established in School Management LOOPS IN Committees, with parent and local community representatives who underwent training PAKISTAN to understand their role in the feedback loop. This type of monitoring was crucial to the success of the project, as the Government was committed to strengthening its ability to use data just as in Punjab. In both instances, the incentive to collect, track and use data was able to establish a stronger monitoring and evaluation system that allowed both governments to improve fiscal sustainability and more effectively allocate public education expenditures. THE WORLD BANK GROUP 24 SOME FREQUENTLY ASKED QUESTIONS What are the Challenges RBF Can Solve? The four theories of change outlined in the above onslaught of daily requests pertaining to various In many country section illustrate how RBF can be applied to competing priorities. RBF can cut through the examples, results achieve transformative results. While specific tape, using financing to nudge actors to action. are often not achieved due to cases are mentioned, previous research has called In Lebanon, there are many crises to tackle, the bureaucratic for the need to determine the actual circumstanc- most recent being the influx of Syrian refugees. “red tape” and es under which RBF can “most likely result in be- It is difficult to put order to everything that needs the onslaught of havioural change, leading to changes in outcomes” attention. While Lebanon has some education daily requests (Perrin, 2013). Since formal evaluations of RBF pro- data available, the EMIS could be strengthened pertaining to grams and interventions in education are still limited, to provide more robust and accurate information. various competing it is useful to draw from operational lessons of WBG However, due to the urgency of other issues, EMIS priorities. RBF can cut through projects. This experience indicates that RBF can is often displaced from priority status. Using RBF the tape, using help solve the following challenges, among others. is a good way to ensure that the Government will financing to invest and build up their EMIS so that necessary nudge actors Inefficiency, low quality, inequity. Education sys- data is available to design programs or interven- to action. tems can suffer from a variety of issues, with prob- tions that improve the quality of education for all lems that range from getting underserved chil- children, including refugees. dren in school to ensuring that school grants are disbursed on time to training teachers to be more Institutional challenges. In a similar vein, RBF can effective in the classroom. These are some of the also help designate responsibilities across actors types of challenges that can be solved with RBF. in a more systematic way. Early childhood devel- For example, in Bihar, a province in India that has opment requires stakeholders from health and lagged in economic development, a WBG project education sectors to work together, but it is often is tying disbursement-linked indicators (DLIs) to unclear who will do what, and when. More clarity ensuring certification of unqualified teachers and comes when financing is at play, forcing actors to providing continuous professional development outline which activities are shared and which are for teachers already in service, with the goal of individually owned. In the Jamaica ECD case, there eventually improving education quality. are DLIs for various ministries, from the Ministry of Finance to the Ministry of Planning, that specify Bureaucratic inertia and competing priorities. their relationships to one another and how each is In many country examples, results are often not responsible for the result (even if that responsibility achieved due to bureaucratic “red tape” and the is simply to disburse money on time). FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 25 FREQUENTLY ASKED QUESTIONS What are the Limits of RBF? While RBF can be highly effective in a variety of there must be a mechanism to flow funds between Anchoring settings, it rightly has limitations, as even the stron- agencies; there needs to be some type of data sys- payment to gest proponents of RBF would not call it a panacea tem in place to monitor and evaluate; some base- results that are outside of (Perrin, 2013). Continuing with examples from line indicators should exist so that progress can be the recipient’s operational experience, situations that RBF cannot tracked. However, these preconditions can also be control is both overcome are: established through RBF. In Niger, the foundation unfair and for RBF to be applied in the future is being built by demoralizing. Lack of political will and ownership. As empha- developing an integrated data system that orga- sized in the introduction, in some instances, there nizes SDI, EGRA, PASEC, and household survey is neither the desire nor ability for a government data. This system will eventually provide reliable to condition financing on results, for political indicators on which future financing could be con- reasons or otherwise. There have been many ditioned. Similarly, in Colombia, a results-based countries where RBF has been raised as a viable monitoring system is being designed, using indi- and suitable approach during discussions with cators that can serve as proxies for learning. For WBG teams and clients, and different financing countries with particularly weak systems, inde- approaches were ultimately chosen because pendent verification can come in the form of third RBF had no champion within the government, or party verification, through contracting with private because better alternatives existed. actors. This can be a good short-term solution, but will not resolve the capacity constraints in the long- Lack of capacity and country systems. For RBF to term. Without mitigating for low capacity to imple- work, there are some preconditions that must be ment, monitor and evaluate programs, serious met. This includes country capacity and systems — delays or even project standstill could occur. WHEN THE F For RBF to be successful, donors (and other paying entities) need to be serious IN RBF IS NOT about withholding payment if the agreed results are not achieved. This can be UNDERSTOOD difficult, as there are several examples where donors did not follow through (Clist & Verschoor, 2014). That is why it is crucial that the idea of nonpayment be communicated upfront with recipients, indicating a change in the way things are normally done. If a donor’s funding model is using RBF in some form (be it applied to all of the available funds or just a percentage), then recipients must understand and accept that they may not receive the expected monies if they do not meet agreed targets. In the case of the Girls Education Challenge, though RBF was clearly stated in application documentation, it did not specify how it would look in practice. This led to confusion and frustration among recipient organizations, delaying implementation (Coffey, 2016). THE WORLD BANK GROUP 26 FREQUENTLY ASKED QUESTIONS When results are not within the recipient’s con- funder and recipient so that they work together trol. In RBF schemes it is important to consider to think through the theory of change at the onset what factors the recipient can actually affect in and choose appropriate targets. A WBG-managed order to achieve results (Clist, 2016; Holzapfel & GPE project in Sierra Leone is embarking on just Janus, 2015). Anchoring payment to results that such a consultation to ensure that the front-line are outside of the recipient’s control is both unfair incentives are clear enough to be understood by and demoralizing. This links back to the idea that teachers and parents, and strong enough to evoke RBF requires a consultative process between changes in behaviour. What are the Risks to RBF? In some contexts there is a low capacity to bear It may take time to pitch the results at the right Striking the the inherent risk of a results-based approach. level, between unambitious and stretch targets. balance between, Most RBF approaches imply that governments take Attaching financing to future results demands that cost, effort, feasibility and on greater risk, since activities need to be financed targets, and their values, be estimated. Such esti- ambition is whether results are achieved. This makes it a less mations are imperfect, and their margins of error crucial. To get predictable financing flow, and countries may find vary widely. Some targets may cost very little (e.g., a there, the results themselves in a kind of double jeopardy: if they are policy change) but be very difficult to achieve, and chain must be unable to achieve a result, frustration may ensue, thus, are highly valued. Others may cost a lot (e.g., carefully crafted. and shortly thereafter, greater frustration at not building schools) but not be “worth” very much receiving reimbursement. Although in the aggre- if the country already has a good track record of gate aid flows too are unpredictable (for example, doing such work. In fact, risk-averse recipients and Kharas, 2008), for any given operation, traditional financiers may shy away from estimates that repre- financing often gives governments the assurance sent stretch targets for fear of not disbursing. Alter- that all of the promised funds will eventually be dis- natively, some targets may be achieved regard- bursed, even if much later than originally planned, less of whether or not RBF was used (World Bank, whereas RBF holds no such guarantee. Mitigation 2015). Striking the balance between, cost, effort, options abound, such as cash advances to ease feasibility and ambition is crucial. To get there, the cash flow considerations; flexibility surrounding results chain must be carefully crafted, and the disbursement arrangements, such as partial pay- critical pathways therein can help estimate more outs for partial results; payouts irrespective of when appropriate targets. results are met (whether early or late); and coun- tries can also choose to address inefficiencies Gaming and unintended consequences. A con- that do not require significant financial investment. cern that plagues many about aid, with RBF as a Building domestic support for RBF, through engag- subset, is the opportunity for gaming and unin- ing civil society actors, or having parents, families, tended consequences. To use Goodhart’s law: or community members involved in the verification “once a measure becomes a target, it ceases process, can help sustain political will of govern- to be a good measure.” If financing binds on ments for taking on RBF risk. Ultimately, however, assessment results, then programs run the risk there is a residual risk that is not mitigatable, and so that teachers will just teach to the test, or worse, governments need to evaluate potential rewards school administrators will find other, more malfea- as outweighing potential costs, and have the politi- sant ways to raise test scores. It is arguable that no cal will to commit to this promising approach. form of financing is completely free from the risk FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 27 FREQUENTLY ASKED QUESTIONS THE MULTITASK As mentioned earlier, multitask theory argues that in some instances, RBF DILEMMA will cause the recipient to prioritize certain actions over others to achieve the financed result(s), potentially leading to the neglect of other, important actions (Clist, 2016). This is indeed a real risk. Anecdotally, in WBG projects attention is often heavily focused on DLIs, while other indicators in the Results Framework may receive less scrutiny. However, this risk can be alleviated by ensuring strong, mutual alignment of funder and recipient, as emphasized throughout this paper. of cheating, and with RBF, it is even more import- applicable to RBF operations and traditional ant to set targets towards things that are condu- financing approaches, attaching financing to the cive to learning as a way to incentivize behavioural result indicator adds pressure for getting the indi- change in positive rather than negative ways. cator right. However, RBF does bring more clarity as to whether the desired impact was achieved or There is no guarantee of impact, or that learn- not, due to a heightened focus on measurement. ing will occur. Despite paying out against results, And while RBF may not always achieve learning, it sometimes measures are not far enough along the can pave the way to get there. These intermediate results chain, leaving time for things to go wrong results are just as important if they pave the way for further down the line. Although this risk is equally longer-term, sustainable outcomes. THE WORLD BANK GROUP 28 HOW: A SYSTEMS APPROACH The WBG “Learning for All” Strategy serves as the guiding vision for a Results First approach. Launched in 2011, the strategy seeks to achieve “Learning for All” by promoting country systems strengthening, along with the global evidence base powerful enough to guide those reforms. This system approach requires clarity on the roles of the system actors, including students, providers, and institutions, and the laws, rules, and regulations that connect them. Timely data and information on the performance of each of these helps identify the most pressing needs and institutional gaps, as well as options for filling them — including through the use of financial incentives. The forthcoming WDR 18 “Realizing the Promise payments/disbursements. For RBF Interventions, Systems of Education for Development” also emphasizes there are a number of modalities in use (e.g. perfor- rarely make use the need for a systems view in order to achieve of financing mance-based contracting, results-based budget- features such as learning at scale. Systems are complex entities, ing, bonuses, etc.) arrangements for with technical and political factors that lock some fiscal transfers, countries into low-quality, low-accountability equi- Looking at the global state of education financ- budget execution libriums (World Bank, forthcoming). Systems ing, however, systems rarely make use of these rules, payment financing can be used to break through these equi- conditions, and financing features. Instead, traditional input-based libriums to achieve the result that the education verification schooling policies prevail, and the tendency is to protocols for system seeks to produce: learning. call for an increase in education expenditures, payments/ regardless of whether such increases correspond disbursements. Specifically, results-based financing can be a with better education outcomes (Hanushek, 2003; prime implementation lever for ensuring system de Hoyos, Holland, & Troiano, 2015; others). While coherence. With regards to financing, as discussed in the previous section, features of the financing justifiable in contexts of extreme shortages of architecture can be used to introduce incentives inputs, such as fragile states, whereby some evi- and align actors around common results. There dence exists that suggests that more inputs can are a variety of financing features currently being make a difference (World Bank, 2011), countries used in RBF Programs. These include arrange- that are not outliers on spending are unlikely to ments for fiscal transfers, budget execution rules, see improvements in systems through increased payment conditions, and verification protocols for financing alone. FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 29 RESULTS-BASED FINANCING APPROACH (RBF) FOR EDUCATION System Alignment In RBF Programs, financing (budget execution flows, fiscal transfers, etc) can be used to align the education system. When financing and results using RBF Programs are tied together, accountability relationships are better defined, education data is better captured, and inputs are used more efficiently. ACTORS SABER Early Directors Childhood Development Ministry of Finance Education Teachers Management Information Systems Students Results Financial Resources Student Assessment INPUTS School Ministry of Autonomy and Curriculum Education Accountability School Materials Finance Teacher Teachers Training Provinces Testing / States EMIS SF SA Schools ECD T SA&A THE WORLD BANK GROUP 30 HOW RBF can strengthen systems in two ways: By ministries, there is an opportunity for embedding Federal to using RBF Programs to bring elements of the sys- RBF into elements of overall public financial man- sub-national tem (e.g. SABER domains) into alignment, and agement reform, such as the WBG approach in fiscal transfers can correct by using RBF Interventions to reinforce any given Mozambique (see Case Study 3, p. 22). the imbalance policy domain. An example of an RBF Program caused when would be a country that has federal fiscal transfers Fiscal transfers typically correct for vertical and/or revenue is flow against the achievement of results. An exam- horizontal imbalances. Vertical imbalances occur primarily ple of an RBF Intervention would be to reinforce when revenues like tax revenues accrue dispropor- collected at a single SABER domain such as Teacher Policies tionately to levels of government that do not bear the federal the corresponding fiscal responsibility for service level but the using performance bonuses for teachers.4 Though sub-national the WBG argues that both RBF Programs and provision. For example, this happens in countries governments Interventions can be impactful, results payments where revenue is primarily collected at the federal are mandated — to teachers, schools, firms — will only have the level but the sub-national governments (e.g. prov- with financing intended system-level effect if they can be imple- inces or states) are mandated with financing edu- education mented at large scale (Savedoff, 2016). cation services. In these cases, this imbalance can services. be corrected through federal to sub-national fiscal The evidence of such financial innovation in the transfers. In the instance of horizontal imbalances, education sector is scant. This section therefore sub-national governments may have different abil- takes an exploratory tour of macro-level financing ities to raise funds from their tax bases, leading to features of RBF Programs (e.g., fiscal transfers) inequalities in revenue across states/provinces, dis- and micro-level features of RBF Interventions (e.g., tricts, or municipalities. Fiscal transfers can there- payment conditions for outputs). This is an area fore serve to equalize the ability to finance educa- ripe for innovation, and the next five years will bring tion services across geographic regions.5 much experimentation. Estimating the effects and capturing the lessons will require an intentional Fiscal transfers can also achieve three types of and ex-ante evaluation agenda to accompany results in education. Aside for correcting for imbal- implementation. ances, transfer payments can be used to incentivize subnational entities to achieve technical efficiency, At the macro level, performance-oriented fiscal allocative efficiency and/or student performance. transfers are the first RBF lever that RBF Pro- These transfers entail conditioning payments on grams can use to strengthen systems. Whether improvements in specific indicators in the afore- with regards to the relationship between the Min- mentioned three areas. An example of technical effi- istry of Finance and sectoral ministries, or to guide ciency is improving pass rates in education, whereby fiscal transfers across federal entities, or even with education financing becomes more efficient since it regards to the decentralization agenda, each trans- takes fewer resources to produce the same output. fer of fiscal resources presents an opportunity for Allocative efficiency is where financing can work to aligning actors around the ultimate goal of achiev- correct the inequalities in the quality of serve deliv- ing results. They allow the WBG to leverage its ery, such as providing more financing for improving expertise in public financial management, working the performance of the bottom of the distribution of closely with ministries of finance. Making use of the reading scores. Lastly, student performance refers WBG’s comparative advantage of working across to measures that improve actual learning levels of sectors and closely with education and finance students (Holland & Mattero, 2016). 4 A forthcoming edited volume on RBF Interventions will further explore how RBF can strengthen the various components to systems, all along the policy design, implementation, evaluation cycle. 5 For more information on allocative efficiency and equity in relation to decentralized education financing, please see the background paper to The Learning Generation report by Simão & Zabaleta (2016). FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 31 HOW Other macro level financial features that RBF incentives like bonus pay to personnel in the edu- Programs can use relate to national budgets. cation sector. The evidence base in this area is For instance, budget execution rules can serve to fairly developed (Lazear, 2003; Glewwe et al, 2010; resolve bottlenecks around spending in line min- Fryer et al, 2012; Muralidharan & Sundararaman, istries. In the previously cited case of the Jamaica 2012; Bruns & Luque, 2014). Another example of a Early Childhood Development Project (see Case financial feature, this time from the school finance Study 2, p. 20), for example, there were concerns domain, is the use of verification protocols linked around the release of funds from the Ministry of to quality assurance systems in service deliv- Finance to the line ministries of health and edu- ery. For example, in Haiti, financing to non-pub- cation. In order to help assure that these minis- lic schools will be conditioned on those schools tries had the requisite budgets to achieve the pre- meeting minimum requirements for infrastructure, agreed (and disbursement-linked) targets under materials, teachers and school management prac- the National Strategic Plan for ECD, the budgets tices (World Bank, 2016). Finally, payment condi- lines linked to those results needed to be executed tions embedded in performance-based contracts, at a minimum of 70 percent each year (World Bank, for instance linked to textbook provision, can help 2008b). Otherwise, irrespective of the achieve- resolve bottlenecks in the supply chain of learning ment of the results, the financing under the World materials in schools. This could include withhold- Bank operation would not be released. This design ing as much as 50 percent of the payment under a served to pilot a results-based budgeting approach publishing contract with schools until schools (or to public finance in Jamaica. In 2017, the team will even parents) have independently verified that the assess how much this has helped safeguard the books have been received. sectoral budgets. In sum, these sub-systems (e.g., Teacher Policies, On the RBF Interventions side, the financial fea- EMIS, Student Assessments, and School Finance) tures include bonus pay, verification protocols, rely on one another for maximum performance. and payment conditions, among others. From the How funds flow from various levels can help har- domain of Teacher Policies, a popular example of monize these elements. a financial feature is the provision of performance THE WORLD BANK GROUP 32 RESULTS-BASED FINANCING APPROACH (RBF) FOR EDUCATION System Component In RBF Interventions, the example below shows how financing can be conditioned on specific results (i.e. teacher recruitment, training and Alignment using evaluation) within the Teacher Policies domain. Taking the case of the RBF Interventions Dominican Republic (see case study #1), teacher standards, criteria for recruitment and evaluation were not part of the teacher training cycle. These elements were introduced through DLIs, leading to improved teaching and learning. Teacher Teacher Training Recruitment Performance Entrance SABER Evaluation Exam TE AC H E R S Teacher Training System DLI. Disbursement Linked Indicators Standards Recruitment Pre-service Entrance Training Exam Evaluation Teacher In-service Performance Training Results FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 33 THE WORLD BANK AND RBF: IMPLICATIONS FOR THE BUSINESS MODEL As demand for RBF in education grows, the WBG business model has and will evolve. This section establishes the growing trends, outlines the instruments used to meet this demand, and explores the implications for optimizing the use of the instruments. Rising Demand for RBF Demand for RBF has increased substantially in formula, whereby 30 percent of project financing PforR the last few years. In fact, demand from clients is disbursed against the achievement of high-level supports for doing business differently has been grow- outcomes that focus on equity, efficiency, and national programs, ing steadily across various sectors. Since 2010, learning (Global Partnership for Education, 2015). disburses education has been witness to the same trends loan (or grant) that began in health and infrastructure a decade To respond to this upward trend, the WBG will proceeds before. Operations using RBF modalities financed be called to transform its business model in against the by IBRD, IDA grants, and GPE grants are growing, three important ways to ensure it is fit for pur- achievement culminating with the introduction of the World pose. First, RBF represents a new level of mutual of agreed upon accountability between the WBG and its clients. milestones, Bank’s Program-for-Results (PforR) lending instru- and uses ment. PforR supports national programs, disburses Second, it requires a finer customization to the country systems loan (or grant) proceeds against the achievement local conditions. Rather than having clients adopt for financing of agreed upon milestones, and uses country uniform global Bank procedures, the PforR instru- management systems for financing management and procure- ment relies on country systems in the execution of and procurement. ment. Another results-based financing modality the proceeds of the loan. This has implications for is to embed “disbursement-linked indicators” the WBG’s provision of technical support, and the (DLIs) in traditional Investment Project Financing. building up of national capacity. Finally, given this This approach differs from the PforR instrument transformational period of financing models, there in three important ways: i) it does not require a is an additional onus placed upon the Bank to national program, ii) it still relies on WBG fidu- adequately evaluate and capture the lessons of ciary procedures, and iii) it can be limited to one these new business models, and quickly bring component of a traditional project. Finally, there teams up to speed across regions. This includes is the ‘variable tranche’ under the GPE funding learning from other sectors, such as health. THE WORLD BANK GROUP 34 THE WORLD BANK AND RBF Mutual Accountability: “We’re in this Together” Traditional lending models carry minimal conse- the balance between ambition and realism takes There may be quences for projects that fall short of achieving on even greater importance. some desire to their intended results. This is because external choose “easy” ratings only come at the end of the project, and Co-creating results frameworks increases own- targets, but this is where mutual closing dates are often extended for several years. ership, motivation, and the likelihood of success. accountability Even if a project receives a poor rating, not much Perhaps more importantly, the co-creation that a and the balance can be done once the project is closed. In con- Results First approach calls for, as both Bank and between ambition trast, under RBF, undisbursed loan proceeds due client work together to define results, instills a and realism takes to unachieved results attract immediate attention deeper sense of partnership around projects. Trav- on even greater from ministries of finance and WBG management elling this path together leads to greater ownership, importance. alike. This accelerates the course correction of and can produce an “Ikea effect” (Norton, Mochon projects, pushes the WBG to be a more account- & Ariely, 2012), whereby labor invested in a product able development partner, and intensifies efforts leads to greater valuation of the end results. This (both in terms of technical assistance on the WBG greater valuation represents the aforementioned side and prioritization and action on the client side) precondition of political will that is needed for mak- for achieving results. ing RBF successful. For WBG staff, RBF raises the stakes for setting This section focused on WBG (and GPE) financing, targets. Based on findings from an internal WBG but is generalizable to other sources of financing. report, under traditional lending, there is very little The WBG support for RBF, and its willingness to test stopping teams and clients from overcommitting to the concept (ideally with concessional financing), is results that are unlikely to be achieved. Under RBF, a good way to introduce clients to it. This ‘demon- theories of change are more carefully examined, stration effect’ that the value of doing business and greater empiricism is employed in estimating differently under RBF can help dispel myths or allevi- targets. There may be some desire to choose “easy” ate concerns clients might have of operating in this targets, but this is where mutual accountability and way, and can serve to guide investment strategies. RBF EMPLOYS In preparation of the Support to RACE 2 Project, for example, the Program-for- GREATER Results operation seeks to improve coverage rates and learning levels for both EMPIRICISM IN Syrian refugee and Lebanese children. Since the Government will only receive ESTIMATING payments upon achieving the targets set in the grant and loan agreements, RESULTS TARGETS the targets were calculated using the best available data, and making careful assumptions about expected trends. In order to avoid unintended consequences, the Project allows for targets to be achieved ahead of schedule, but also caps yearly disbursements to ensure that there are funds available in outer years to maintain the incentives in place for continued high levels of enrollments and learning (World Bank, 2016b). FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 35 THE WORLD BANK AND RBF Artisanal RBF Artisanal customization means adapting to cli- on the WBG‘s comparative advantage: technical Connecting ents, contexts, and capacity. As we’ve outlined, assistance on education policy and public man- governance and this vision of RBF for strengthening systems can agement reforms. Simultaneously, in the case of social sectors allows a country be adapted to the entire range of clients that the the PforR instrument, embracing country systems to reap the WBG works with. Clearly the forms that RBF will usually requires strengthening those systems benefits of an take will differ from upper middle income coun- through technical assistance in order to have RBF system tries to fragile states. In part, this could require them operational by the project start date. Finally, beyond the short sequencing of reforms such that the data systems as government teams start working to achieve the period of WBG are first established as preconditions to RBF, as technical milestones that have been identified, supported illustrated by Niger and Colombia (see p. 26). Also, WBG teams are called upon to accompany the cli- operations by embedding the choice of RBF instrument used also depends ent down that critical path toward the results. All results into the on the local context and capacity. In some of this effort has implications for greater costs to national instances, government programs will be in place WBG teams in preparation of and implementation financing and that will allow the WBG to co-finance using existing support to projects. budget systems. country systems through the PforR lending instru- ment. In others, programs of eligible expenditures In addition to TA for education topics, many of will need to be created under the Investment Proj- these RBF Programs require substantial gover- ect Financing instrument. Some other support to nance reforms. Though the governance sector clients may consist of technical advice on public lends itself less to RBF given that specific results financial management reform that embeds RBF are difficult to identify and achieve (Klingebiel, mechanisms, without corresponding financing 2012), that means it is even more imperative to from the WBG. attach quantifiable results to social sectors like education. Connecting the two sectors allows The increased customization in turn requires a country to reap the benefits of an RBF system that the WBG dramatically increase its capac- beyond the short period of WBG supported oper- ity to deliver on technical assistance programs ations by embedding results into the national with clients. RBF efforts require a more intense financing and budget systems. The Education effort from WBG teams at both the design and Practice therefore must work in close partnership implementation stages of projects. First, during with the teams from Governance, as well as the design, the aforementioned co-creation of pro- Macroeconomics and Fiscal Management teams gram results frameworks, and the careful calibra- that are also leading the public sector moderniza- tion of the targets therein, require a substantial tion agenda in many of our client countries. amount of upfront time and resources. This builds THE WORLD BANK GROUP 36 THE WORLD BANK AND RBF Knowledge Agenda There is an urgency for more knowledge on when In addition to impact evaluation evidence from Through the and how to deploy RBF. Given that this global RBF experiences across institutions, learning Results in agenda is at a moment of transition from traditional from the WBG’s operations is ongoing and can Education for All Children to more modern financing modalities, the WBG, be intensified. In 2016, the Independent Evalu- (REACH) initiative, together with partners and clients, has placed ation Group (IEG), carried out a review to assess the WBG has knowledge on RBF at the top of the agenda. This experiences to date and to identify ways in which started to fill the priority-level means that more knowledge, (explicit, the WBG’s PforR instrument could be strength- gaps in this and tacit) must be produced, on all parts of the ened. Since none of the WBG’s PforR operations knowledge policy-making process (from upstream policy dia- have closed, it is too early to draw definite lessons, agenda, and is logue, through program design, implementation, but early insights point to the growing importance funding a variety of RBF activities and evaluation). Through the Results in Education of the instrument. The report’s other conclusions with the hope of for All Children (REACH) initiative, the WBG has also show that DLIs must be well integrated in broadening the started to fill the gaps in this knowledge agenda, Results Frameworks and that government own- evidence base. and is funding a variety of RBF activities with the ership is crucial to implementation success. As hope of broadening the evidence base. In partic- projects begin to close, it will be critical to capture ular, a more granular approach to the evaluation both WBG-specific lessons as well as broader les- agenda is needed such that the next iteration of sons around RBF (IEG, 2016). research focuses on analyzing RBF Programs or Interventions through a country context lens, Regarding tacit knowledge, documenting early rather than just within the theoretical frame (or lessons in what works, and disseminating said pros/cons) of RBF. It would be of great benefit to lessons quickly, will keep clients from repeating governments interested in using RBF to see how it mistakes made by other countries/sectors. In works in low-income countries, and middle-income particular, there is a tremendous amount of oper- countries, and to explore why high-income coun- ational knowledge in the health sector that lends tries don’t seem to adopt it as much. This implies itself very well to both RBF Programs and RBF Inter- learning from experiences beyond those funded by ventions in the education sector. Stronger linkages the WBG (e.g. Girls Education Challenge and the across sectors, with the health team in particular, forthcoming case studies from GPE), and financ- will go a long way in unlocking the flow of knowl- ing evaluations to be conducted by external actors. edge across practices and regions. FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 37 THE WORLD BANK AND RBF CONCLUSIONS RBF will help accelerate the achievement toward SDG 4. As countries face a triple challenge of universalizing coverage, pivoting toward quality, and doing these in a context of likely fiscal contraction, governments will be under even greater pressure to have education financing deliver greater results. This is best achieved through strengthening education systems to provide sustained quality services, and RBF Programs and Interventions offer tools for them to do so. RBF will strengthen systems through four theo- part of the problems that systems face, in some ries of change. First, it shifts the policy dialogue places. Still, as governments embark on edu- away from the laundry lists of day-to-day activities cation reform efforts to rise to the challenges of and inputs that are, to what are the end line results the SDGs, RBF is already changing how we do we most desire. Second, it sustains the attention business in places as diverse as Haiti, Jamaica, on those results, keeping all eyes on the prize, and Pakistan, and Tanzania.6 protecting those long-term priorities from short- term urgencies. Third, it aligns actors not usually The WBG supports a Results First approach involved in the education planning, such as min- as the way forward. This approach is a highly istries of finance, such that they now have a stake effective way to move all stakeholders away from in sectoral performance, and can hold line agents thinking about inputs and goes hand-in-hand with accountable for quick course correction when the theories of change detailed in this paper. results are off-track. Finally, it institutionalizes the Moreover, it prevents Backdoor RBF and promotes measurement of results, and forces feedback loops defining results together, compelling country cli- through the system, that will give system operators ents and the WBG to be more mutually account- the information to course correct, and will launch a able for shared goals. The Results First approach virtuous cycle of stronger monitoring, better infor- differs in that it emphasizes working backwards by mation, and more appropriate responses to prob- focusing on desired outcomes, identifying binding lems and bottlenecks as they arise. constraints, and using financing as a way to unlock those constraints. This is the greatest potential of RBF is not a panacea, but it is already making RBF; to change the way countries think about and a difference in many places. Making better use finance education in an effort to bring us closer to of education financing is only going to resolve learning for all. 6 For up to date information on the results achieved in these programs, see http://projects.worldbank.org/ and search for P155191 (Haiti), P095673 (Jamaica), P125952 (Pakistan), and P147486 (Tanzania). THE WORLD BANK GROUP 38 REFERENCES Andrews, M., Pritchett, L., & Woolcock, M. (2012). de Hoyos, R., Holland, P., & Troiano, S. (2015). Escaping Capability Traps through Problem Driven Iterative Understanding the Trends in Learning Outcomes in Adaptation (PDIA). Harvard University. Cambridge: Center Argentina, 2000 to 2012. Washington DC: World Bank. for International Development. Retrieved from http://elibrary.worldbank.org/doi/ pdf/10.1596/1813-9450-7518 Ariely, D., Gneezy, U., & Mazar, N. (2009). Large Stakes and Big Mistakes. Review of Economic Studies, 451–469. Development Committee. (2015). From Billions to Retrieved from http://people.duke.edu/~dandan/webfiles/ Trillions: Transforming Development Finance. Retrieved PapersUpside/Large%20Stakes%20Big%20Mistakes.pdf from http://siteresources.worldbank.org/DEVCOMMINT/ Documentation/23659446/DC2015-0002(E) Ariely, D., Norton, M., & Mochon, D. (2011). The “IKEA FinancingforDevelopment.pdf Effect”: When Labor Leads to Love. Harvard Business School Working Paper. Retrieved from http://www.hbs.edu/ Duflo, E., Hanna, R., & Ryan, S. (2012). Incentives Work: faculty/Publication%20Files/11-091.pdf Getting Teachers to Come to School∗. American Economic Review, 1241–1278. Retrieved from http://economics.mit. Birdsall, N., & Savedoff, W. (2010). Cash on Delivery: A edu/files/5582 New Approach to Foreign Aid. Washington DC: Center for Global Development. Retrieved from http://www.cgdev. Education for All Global Monitoring Report. (2013–2014). org/publication/9781933286600-cash-delivery-new- Teaching and Learning: Achieving quality for all. Paris: approach-foreign-aid UNESCO. Retrieved from http://unesdoc.unesco.org/ images/0022/002256/225654e.pdf Bruns, B., & Luque, J. (2014). Great Teachers: How to Raise Student Learning in Latin America and the Faul, M. (2016). Scoping Study: Education systems Caribbean. Washington DC: World Bank. Retrieved from diagnostics tools. London: Department for International http://www.worldbank.org/content/dam/Worldbank/ Development (DFID). document/LAC/Great_Teachers-How_to_Raise_Student_ Learning-Barbara-Bruns-Advance%20Edition.pdf Friedman, J., Das, A., & Mutasa, R. (2016). Rewarding Provider Performance to Improve Quality and Coverage Burgess, S., Metcalfe, R., & Sadoff, S. (2016). of Maternal and Child Health Outcomes. Washington DC: Understanding the Response to Financial and Non- World Bank Group. Financial Incentives in Education: Field Experimental Evidence Using High-Stakes Assessments. Bonn: IZA. Fryer, Jr., R., Levitt, S., List, J., & Sadoff, S. (2012). Retrieved from http://ftp.iza.org/dp10284.pdf Enhancing the Efficacy of Teacher Incentives through Loss Aversion: A Field Experiment. NBER Working Paper Series. Clist, P., & Verschoor, A. (2014). The Conceptual Basis Retrieved from http://www.nber.org/papers/w18237.pdf of Payment by Results. Norwich: University of East Anglia. Retrieved from https://www.gov.uk/dfid-research-outputs/ Glewwe, P., Ilias, N., & Kremer, M. (2010). Teacher the-conceptual-basis-of-payment-by-results Incentives. American Economic Journal: Applied Economics, 205–227. Retrieved from https://www. Clist, P. (2016). Payment by Results in Development Aid: povertyactionlab.org/sites/default/files/publications/ All That Glitters Is Not Gold. The World Bank Research Teacher%20Incentives%20%28AEJ%29%202010.pdf Observer, 290–319. Retrieved from http://wbro. oxfordjournals.org/content/31/2/290 Global Partnership for Education. (n.d.). Global Partnership for Education. Retrieved from Funding: Clist, P., & Dercon, S. (2014). 12 Principles http://www.globalpartnership.org/funding for Payment By Results (PbR) in International Development. Norwich: University of East Anglia. Gustafsson-Wright, E., Gardiner, S., & Putcha, V. (2015). Retrieved from https://assets.publishing.service.gov.uk/ The Potential and Limitations of Impact Bonds: Lessons media/57a089d2e5274a27b20002a5/clist-dercon-PbR.pdf from the First Five Years of Experience Worldwide. Washington DC: Brookings Institute. Retrieved from Coffey. (2016). GEC Process Review Report: Evaluation https://www.brookings.edu/wp-content/uploads/2016/07/ Manager Girls’ Education Challenge Fund. Coffey. Impact-Bondsweb.pdf Retrieved from https://www.gov.uk/government/uploads/ system/uploads/attachment_data/file/501596/Process- Review-Report.pdf FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 39 REFERENCES Hamilton, L., Stecher, B., & Yuan, K. (2008). Standards- Klingebiel, S. (2012). Results-Based Aid: New aid Based Reform in the United States: History, Research, approaches, limitations and the application of good and Future Directions. Washington DC: RAND. Retrieved governance. Bonn: German Development Institute. from http://www.rand.org/content/dam/rand/pubs/ Retrieved from https://www.die-gdi.de/uploads/media/ reprints/2009/RAND_RP1384.pdf DP_14.2012.pdf Hanushek, E. (2003). The Failure of Input-based Lazear, E. (2003). Teacher Incentives. Swedish Economic Schooling Policies. The Economic Journal, F64–98. Policy Review, 179–214. Retrieved from http://www. Retrieved from http://onlinelibrary.wiley.com/ uaedreform.org/wp-content/uploads/2013/08/Lazear- doi/10.1111/1468-0297.00099/abstract 2003-SEPR-Teacher-incentives.pdf Hanushek, E. (2013). Financing Schools. In J. Hattie, Leoning, E., & Tineo, L. (2012). Independent Verification & E. Anderman (Eds.), International Guide to Student in Results-Based Financing. Washington DC: World Bank. Achievement, 134–136. New York: Routledge. Retrieved from https://openknowledge.worldbank.org/ Retrieved from http://hanushek.stanford.edu/sites/ handle/10986/16175?show=full default/files/publications/Hanushek%202013%20 IntGuideStudAchieve.pdf Muralidharan, K., & Sundararaman, V. (2009). Teacher Performance Pay: Experimental Evidence from India. Hart, O., & Holmstrom, B. (1986). The Theory of NBER Working Paper Series. Retrieved from http://www. Contracts. Cambridge: MIT, Department of Economics. nber.org/papers/w15323.pdf Retrieved from https://dspace.mit.edu/bitstream/ handle/1721.1/64265/theoryofcontract00hart. Neal, D. (2011). The Design of Performance Pay in pdf?sequence=1 Education. Handbook of the Economics of Education, 495–550. Holland, P., & Mattero, M. (2016, May 4). REACH News & Events. Retrieved Feb 4, 2017, from REACH website: Perakis, R., & Savedoff, W. (2015). Does Results-Based http://pubdocs.worldbank.org/ Aid Change Anything? Pecuniary Interests, Attention, en/969251478722075552/Its-Debatable-Fiscal-Transfers- Accountability and Discretion in Four Case Studies. NOTES.pdf Washington DC: Center for Global Development. Retrieved from http://www.cgdev.org/sites/default/files/ Holmstrom, B., & Milgrom, P. (1991). Multitask CGD-Policy-Paper-52-Perakis-Savedoff-Does-Results- Principal-Agent Analyses: Incentive Contracts, Asset Based-Aid-Change-Anything.pdf Ownership, and Job Design. Journal of Law, Economics, & Organization, 7, 24–52. Retrieved from https://faculty. Perazzo, A., & Josephson, E. (2014). Verification of fuqua.duke.edu/~qc2/BA532/1991%20JLEO%20 Performance in Results-Based Financing Programs: Holmstrom%20Milgrom.pdf The Case of Plan Nacer in Argentina. Washington DC: World Bank. Retrieved from http://documents. Holzapfel, S., & Janus, H. (2015). Improving Education worldbank.org/curated/en/559871468206969542/ Outcomes by Linking Payments to Results. Bonn: German pdf/950830WP0Box390ntina0Final00PUBLIC0.pdf Development Institute. Perrin, B. (2013). Evaluation of Payment by Results Independent Evaluation Group (IEG). (2016). Program (PBR): Current Approaches, Future Needs. London: DFID. for Results: An Early-Stage Assessment of the Process and Retrieved from https://www.gov.uk/government/uploads/ Effects of a New Lending Instrument. Washington, DC: system/uploads/attachment_data/file/213938/payment- World Bank. Retrieved from http://ieg.worldbankgroup. results-current-approaches-future-needs.pdf org/Data/Evaluation/files/program-for-results-full.pdf Pritchett, L. (2015). Creating Education Systems Kerr, S. (1995). On the folly of rewarding A while hoping Coherent for Learning. RISE Working Paper. Retrieved for B. Academy of Management Executive, 9 (1), 7–14. from http://www.riseprogramme.org/sites/www. Retrieved from http://www.supplychain247.com/images/ riseprogramme.org/files/RISE_WP-005_Pritchett.pdf pdfs/on_the_folly.pdf Savedoff, W. (2016). A Global Offer for Learning Kharas, H. (2008). Measuring the cost of aid volatility. (GOL): Based on Experiences with Paying for Results. Washington DC: Wolfensohn Center for Development at Washington DC: Center for Global Development. Brookings. Retrieved from https://www.brookings.edu/ wp-content/uploads/2016/06/07_aid_volatility_kharas.pdf King, E., Abdul-Hamid, H., & Newman, J. (2016). The Quality of Education Systems and Education Outcomes. International Commission on Financing Global Education. THE WORLD BANK GROUP 40 REFERENCES Second Regional Comparative and Explanatory World Bank. (2014). Project Appraisal Document: Study (SERCE). (2006). Second Regional Big Results Now in Education Program. Washington Comparative and Explanatory Study (SERCE). DC: World Bank. Retrieved from http://documents. Retrieved from http://www.unesco.org/new/en/ worldbank.org/curated/en/147121468312011600/ santiago/education/education-assessment-llece/ pdf/845450PJPR0P14010Box385244B00OUO090.pdf second-regional-comparative-and-explanatory-study-serce/ World Bank. (2014). Project Appraisal Document: Simão, M., & Zabaleta, M. (2016). Education Financing Mozambique Public Financial Management for Results in Decentralized Systems — enquiries into the allocative Program. Washington DC: World Bank. Retrieved from efficiency of educational investment and the effects on http://www.projects.worldbank.org/P124615?lang=en other dimensions of quality education policies. Paris: UNESCO International Institute for Educational Planning. World Bank. (2015). Approach Paper: Program for Results (PforR)—IEG Process Evaluation. Washington DC: World The Learning Generation: Investing in education for a Bank. Retrieved from https://ieg.worldbankgroup.org/ changing world. (2016). Data/reports/Approach_Paper_PforR.2a1_0.pdf UNESCO/UNICEF. (2015). Fixing the Broken Promise World Bank. (2015). Project Appraisal Document: of Education for All. Paris: UNESCO. Retrieved from Dominican Republic Support to the National Education https://www.unicef.org/education/files/allinschool.org_ Pact. Washington DC: World Bank. Retrieved wp-content_uploads_2015_01_Fixing-the-Broken-Promise- from http://documents.worldbank.org/curated/ of-Education-For-All-full-report.pdf en/858741468189574332/pdf/PAD850-PAD-P146831- R2015-0187-1-Box393201B-OUO-9.pdf United Nations. (2016). Sustainable Develompent Goals. Retrieved from Sustainable Development World Bank. (2015). Project Appraisal Document: Knowledge Platform: Enhancing Teacher Effectiveness in Bihar. Washington DC: https://sustainabledevelopment.un.org/?menu=1300 World Bank. Retrieved from http://documents.worldbank. org/curated/en/184631468000251240/pdf/92972-PAD- USAID. (2014). Incentives and Accountability in P132665-IDA-R2015-0096-1-Box391421B-OUO-9.pdf Education: A Review. Washington DC: USAID. Retrieved from https://www.eddataglobal.org/reading/index. World Bank. (2015). Project Appraisal Document: Lebanon cfm?fuseaction=pubDetail&ID=722 Support to Reaching All Children with Education (RACE 2) Program-for-Results. Washington DC: World Bank. World Bank. (2008). Project Appraisal Document: Retrieved from http://documents.worldbank.org/curated/ Jamaica Early Childhood Development. Washington en/980641475200856910/pdf/Lebaon-RACE2-PforR- DC: World Bank. Retrieved from http://documents. Board-Package-PAD-WB-9-5-16-09072016.pdf worldbank.org/curated/en/970381468038930751/ pdf/421920PAD0P0951e0only10R20081007811.pdf World Bank. (2016). Project Appraisal Document: Haiti Education for All Project II. Washington DC: World Bank. World Bank. (2009). Project Appraisal Document: Retrieved from http://projects.worldbank.org/P124134/ Pakistan Punjab Education Sector. Washington education-all-project-support-second-phase-education-all- DC: World Bank. Retrieved from http://documents. program?lang=en&tab=overview worldbank.org/curated/en/582481468325217093/ pdf/475910PAD0P102101Official0Use0Only1.pdf World Bank. (2009). Project Appraisal Document: Pakistan Sindh Education Sector. Washington DC:   World Bank. Retrieved from http://documents. worldbank.org/curated/en/235111468073731706/ pdf/476420PAD0P107101Offciial0Use0Only1.pdf World Bank. (2011). Learning for All: Investing in People’s Knowledge and Skills to Promote Development. Washington DC: World Bank. Retrieved from http://documents.worldbank. org/curated/en/685531468337836407/ pdf/644870WP0Learn00Box0361538B0PUBLIC0.pdf FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 41 ANNOTATED BIBLIOGRAPHY: THEORIES UNDERPINNING RBF, AND EARLY LESSONS This annotated bibliography further delves into the theoretical underpinnings on which RBF rests, according to various viewpoints. It also summarizes some of the early operational lessons of RBF initiatives in the education sector. Building on the Principal- Agent foundation presented earlier in this paper, this section discusses contract theory, and research around accountability, incentives, and motivation. Contract Theory offers a conceptual premise for in some instances, the higher the pay, the worse RBF. In “Theory of Contracts”, Hart and Holmstrom the performance (especially when the incentive suggest that a “Pareto optimal contract proceeds triggers greater self-consciousness, or, causes the by maximizing one party’s expected utility subject actor to “choke under pressure.”). This supports to the other party (or parties) receiving a mini- the notion that the relationship between incen- mum expected utility level.” This is the idea that tives and performance is nonlinear. Like employ- both sides expect the other to do their fair share, ees, aid recipients must be open to being incen- and negotiate thusly to ensure that that actually tivized, motivated to achieve the result, and held happens. In later work, Holmstrom and Milgrom accountable for achieving it. But getting to that (1991) examine performance pay using the Princi- point is no easy feat. pal-Agent model. While their work focuses primar- ily on employment, their most pertinent conclu- Neal looks at teacher performance pay systems sions for this paper’s purpose are: (i) it is difficult designed to induce more effort. A key finding to observe and, subsequently, reward effort, espe- from his research shows many systems designed cially when there is a combination of individual and for accountability and performance pay use test group effort and (ii) if results are not measurable scores (one indicator) to produce information for and if the set of potential actions to reach the result both rewards (or punishment) for teachers and as too complex, the agent will unlikely be motivated to a measure of student learning (two things). The attain the result(s). author concludes that this does not work. While Neal found that there was some evidence to indi- The other main areas of research that the WBG cate that performance pay increased teacher Approach draws on are those relating to account- effort, schemes for bonuses based on objective or ability, incentives, and motivation. Much of the subjective criteria were still vulnerable to manipula- research comes from organizational develop- tion and did not necessarily correlate to improved ment, but can also be applied to RBF. In Ariely et measures on student assessments. Neal argues al’s (2009) work, the authors note that increased that a missing, but necessary part is the sustainabil- motivation doesn’t always lead to increased ity of measurement — that performance pay plans performance. Their analysis also reveals that incen- must be based on continuous assessments that tives often backfire when applied to complex tasks cover a consistent curriculum but vary in content that require high cognitive reasoning or creativity, and format (Neal, 2011) THE WORLD BANK GROUP 42 ANNOTATED BIBLIOGRAPHY Early Lessons from RBF Programs There have also been a number of recent reviews Both DFID and the German Development Insti- of experiences, though the literature is nascent. tute (DIE) also produced evaluations on RBF. Evaluations have been conducted by USAID, DIE DFID’s working paper concentrates on providing and DFID. An evaluation commissioned by USAID guidance for future RBF evaluations whereas DIE looks at 17 studies on financing incentives and examines the selection, design, and use of indica- teacher accountability; 36 studies on incentives tors, using a case study approach exploring five aimed at parents and students; and seven studies RBF approaches. on school report cards and school accountabil- ity. The authors’ conclusion on teacher account- The DFID evaluation is framed in respect to ability is that the evidence for RBF is mixed, with their view of RBF as a financing model that can RCTs (Muralidharan and Sundararaman, 2011; get “value for money of expenditures in devel- Muralidharan, 2012), showing the strongest argu- opment aid”. The primary recommendations ments for it. In those cases, teacher effort is gener- that come from the evaluation are: (i) the need ally increased around subject areas being tested to identify under what circumstances RBF could or in areas directly tied to a bonus. The review also work (like most of the evaluations to date); (ii) the tries to distinguish between incentives as a function need for more mixed methods approaches; and of inputs or outputs. In many of the studies, teacher (iii) to apply more theory-based evaluations that performance pay was based on student test scores, will examine causal processes. Ultimately, the whereas in others, such as Duflo et al (2012), the evaluation does not fully endorse RBF, citing the incentive was focused on an input, teacher atten- limited evidence of its effectiveness and the likeli- dance, which proved to be successful in reducing hood of unintended, negative consequences. absenteeism without affecting teacher effort in the classroom. The DIE evaluation takes a more granular approach and seeks to answer the question On the demand-side, much of the existing lit- “What are good indicators and how can they erature confirms that incentives can positively be selected?” The authors offer a classification impact attendance or enrollment rates, but gen- and typology of DLIs that are not all outcome level erally do not impact learning outcomes. More ones. They also offer criteria to assess the quality of interestingly, it does seem that students can be DLIs, which include: (i) focus on results; (ii) control; financially (and non-financially) incentivized to (iii) financial incentives; (iv) measurability and verifi- learn, e.g. paying students to complete math ability; and (v) unintended consequences. Their con- assignments or rewarding better test scores with clusions most relevant for this paper are that DLIs high-value trips. It also appears that these types of should as much as possible rely on outcome-indica- rewards do not diminish intrinsic motivation and tors, that they should reward incremental improve- their effects can be sustained over time. Of course, ments, and that they should put more emphasis on such schemes must be well-targeted and carefully leverage effects than on value for money. designed and implemented. As for school report cards and school accountability, the research The largest RBF Program in Education is the shows that scorecards can be a good accountabil- Girls’ Education Challenge (GEC), which offers ity mechanism especially when involving parents lessons of importance to the WBG approach. and community members in the process (the idea The GEC is a DFID initiative that aims to ensure a of shared ownership over the product). quality education for one million marginalized girls. FINANCING RESULTS TO STRENGTHEN SYSTEMS: THE WBG APPROACH TO RBF IN EDUCATION 43 ANNOTATED BIBLIOGRAPHY Coffey (2016) undertook an independent pro- cess evaluation at midline stage to inform future funding and to share lessons learned. One of the main findings is that RBF can drive coherence and consistency across projects (in line with the WBG emphasis on systems), as they all aim to achieve the same goal (which, for the GEC, is very explicitly stated). Another conclusion is that RBF must be well understood by both sides upfront, otherwise agents (grant recipients) do not have the requisite information about the types of risks they are taking on, and principals may keep “changing the rules”. As outlined above, much of the literature around RBF frames it within economic theory. While there are good lessons to learn from the Principal-Agent and Multitask models, they are not always suffi- cient to understand how RBF changes, or doesn’t change behaviours in real-life. The Principal-Agent model assumes that principals (donors) and agents (recipients) have differing goals and that the primary way to shift agents’ actions is through incentives, while the Multitask model does not always indicate how “distracted” an agent will be from the pursuit of the desired result. Ultimately, it is important to take into account the diverse moti- vations and behaviours of all stakeholders, and to be cognizant of country contexts. THE WORLD BANK GROUP 44 RESULTS-BASED FINANCING IN EDUCATION: FINANCING RESULTS TO STRENGTHEN SYSTEMS RESULTS-BASED FINANCING IN EDUCATION: FINANCING RESULTS TO STRENGTHEN SYSTEMS RESULTS IN EDUCATION FOR ALL CHILDREN (REACH) Education Global Practice World Bank 1818 H Street, NW / Washington DC, 20433 / USA worldbank.org/reach / reach@worldbank.org