103170 Operationalizing the Science of Delivery Agenda to Enhance Development Results Maria Gonzalez Asis Michael Woolcock Operationalizing the Science of Delivery Agenda to Enhance Development Results Maria Gonzalez Asis Michael Woolcock October 2015 © 2015 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW, Washington, DC 20433 Telephone: 202-473-1000; Internet: www.worldbank.org Some rights reserved 1 2 3 4 18 17 16 15 This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Nothing herein shall constitute or be considered to be a limitation upon or waiver of the privileges and immunities of The World Bank, all of which are specifically reserved. Rights and Permissions This work is available under the Creative Commons Attribution 3.0 IGO license (CC BY 3.0 IGO) http://creativecommons.org/licenses/by/3.0/igo. Under the Creative Commons Attribution license, you are free to copy, distribute, transmit, and adapt this work, including for commercial purposes, under the ­following conditions: Attribution—Please cite the work as follows: Gonzalez Asis, Maria, and Michael Woolcock. 2015. “Operationalizing the Science of Delivery Agenda to Enhance Development Results.” World Bank, Washington, DC. License: Creative Commons Attribution CC BY 3.0 IGO Translations—If you create a translation of this work, please add the following disclaimer along with the attribution: This translation was not created by The World Bank and should not be considered content or error in this an official World Bank translation. The World Bank shall not be liable for any ­ translation. Adaptations—If you create an adaptation of this work, please add the following disclaimer along with the attribution: This is an adaptation of an original work by The World Bank. Views and opinions expressed in the adaptation are the sole responsibility of the author or authors of the adaptation and are not endorsed by The World Bank. Third-party content—The World Bank does not necessarily own each component of the content contained within the work. The World Bank therefore does not warrant that the use of any third- party-owned individual component or part contained in the work will not infringe on the rights of those third parties. The risk of claims resulting from such infringement rests solely with you. If you wish to re-use a component of the work, it is your responsibility to determine whether permission is needed for that re-use and to obtain permission from the copyright owner. Examples of components can include, but are not limited to, tables, figures, or images. All queries on rights and licenses should be addressed to the Publishing and Knowledge Division, The World Bank, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@ worldbank.org. Cover photo: gabydesign/Shutterstock. Used with the permission of gabydesign/Shutterstock; further permission required for reuse. ­ Cover design: Bill Pragluski, Critical Stages, LLC Contents Abstract iv Operationalizing the Science of Delivery Agenda to Enhance Development Results 1 What Is the Problem? 2 Five Core Principles of the Science of Delivery 5 Principle I: Relentless Focus on Citizen Outcomes 6 Principle II: Multidimensional Response 9 Principle III: Evidence to Achieve Results 11 Principle IV: Leadership for Change 17 Principle V: Adaptive Implementation 19 Toward Adaptive Implementation at the World Bank 21 Conclusion25 Notes27 References29 Box 1 The Global Delivery Initiative in Action 25 Operationalizing the Science of Delivery Agenda iii Abstract The clear development gains achieved in recent decades should not deflect attention from the scale and type of challenges that remain. The strategies largely responsible for these initial gains have been technical reforms promot- ing economic growth and logistical systems supplying basic inputs. Today, strategies are needed that focus on enhancing the quality of ­implementation— for example, ensuring learning and not just building schools and enrolling students. This concern now spans numerous domains of professional practice (especially health) and has entered World Bank discussions framed as the “science of delivery.” At the World Bank, the Global Delivery Initiative (GDI) is an operational manifestation and extension of these ideas. To date, the GDI has prepared a number of different case studies across numerous sectors on ways in which innovative teams solve particular problems during project implementation. On the basis of the initial case studies, the authors outline five key principles of how high-quality implementation occurs and invite others to add to this growing storehouse of knowledge. Specifically, task teams are encouraged to develop “live” case studies by and for their staff, documenting how, in real time, implementation challenges are being met. Projects must “learn” more rapidly and systematically how to solve the myriad range of complex implementation challenges they inevitably encounter, since most of these ­ (by definition) cannot be anticipated ex ante. Delivery challenges of this kind will only intensify in the coming years as citizens demand effective responses to ever-more complex—and contentious—policy domains, such as justice, regulation, and taxation. iv Operationalizing the Science of Delivery Agenda Operationalizing the Science of Delivery Agenda to Enhance Development Results The seminal World Development Report 2004: Making Services Work for Poor People issued a compelling call for greater accountability to improve the quality of service delivery for the poor. Considerable gains have been made ­ in the intervening decade, but despite concerted efforts to promote policy reforms, introduce technical solutions, and allocate additional resources to support these initiatives, far too many poor people still lack access to basic services, let alone services that meet international standards. There are dire costs from this failure to deliver effective services. While under-five mortal- ity rates have been cut dramatically since 1970, in 2012, some 6.6 million children (or 28,000 per day) died before reaching their fifth birthday.1 One ­ way to save these children’s lives is through the provision of clean water and basic ­sanitation, but these services remain unmet for roughly 2.5 billion peo- ple worldwide.2 Similarly, while primary enrollment rates have vastly improved around the world, more than 60 million children remained out of school in 2010, 75 percent of these were in South Asia and Sub-Saharan Africa,3 and “completion rates, especially for girls, remain a major concern.”4 Moreover, children in developing countries continue to lag significantly behind their developed-nation peers in their achievement on standardized international tests (such as the PISA, which measures learning outcomes); some 90 percent of children from lower-middle-income developing coun- tries score below 400 on the PISA, a level which indicates that basic math learning goals have not been met.5 Approximately 775 million people in the world remain illiterate.6 These enduring realities show that there are clearly major gaps separating carefully articulated policies and their realization in hoped-for results. This gap is particularly frustrating because, on the surface, the goals set by development agencies (and developing countries) are often deceptively simple. As a major development organization, the World Bank knows, gener- ­ ally, where it wants to be—it has established the twin goals of ending extreme poverty and boosting shared prosperity. But how do we get there? What is ­ missing? What does it take to make a real difference in the lives of people who still experience the effects of a deficit in access to and provision of quality service delivery? Will the strategies that delivered the initial rounds of The authors would like to thank Jacob Bathanti for his valuable contributions, research, and editorial assistance. Thanks also to our colleagues in the Global Delivery Initiative and elsewhere in the World Bank for their numerous thoughtful conversa- tions with us on the themes and ideas of this paper. Operationalizing the Science of Delivery Agenda 1 development success suffice for advancing them still further or in contexts that are especially difficult (such as fragile states)? In recent years, a number of organizations and groups have sounded the call for change. The emerging agendas share some features while diverging on others, and different articulations of the agendas may emphasize different aspects. Nonetheless, the agendas converge on three important points: taking local contexts and political incentives seriously; building the capability of agencies (teams) to implement increasingly complex and contentious tasks at scale; and solving concrete problems and adapting solutions in real time to emergent successes and failures (as opposed to crafting elaborate plans up front and measuring success by faithful adherence to it). This basic framing is similar to initiatives emerging in other professional fields where high-quality implementation is critical to success, most notably health care. Some have called these initiatives an attempt to build a “science of delivery” (Barber 2015) and it is this language that has launched a broader conversation within the World Bank (and elsewhere) on how to improve implementation. This paper attempts to lay out a framework for operationalizing the emerg- ing science of delivery. It identifies five delivery principles that practitioners often use to achieve greater impact in their interventions, to make more informed decisions, and to get more consistent results on the ground. This paper identifies examples of these principles in the literature on develop- ment, drawing on both theoretical and applied work while identifying specific  examples where the principles have been put into practice in a ­ ­development activity. In so doing, this paper seeks to demonstrate that a range of development organizations, including the World Bank, are taking concrete steps to increase their emphasis on delivery as a way of “doing development differently” and thereby doing it better.7 What Is the Problem? A key reason why these gaps endure between policy aspiration and ­ performance reality is that the bureaucratic systems for delivery—and the corresponding incentives they generate—are very well suited to address certain types of problems and very poorly suited to address others. In short, ­ they excel at addressing logistical and technical problems, that is, those that can be standardized (reduced to a best practice) or resolved by “10 smart people” (whether to raise/lower interest rates), but they struggle to respond to adaptive problems, that is, those that require ongoing human interaction (teaching or health care) and discretionary responses to inherently a­ mbiguous, context-specific problems (such as reconciling competing land ownership claims, ensuring law enforcement, or providing agricultural extension s ­ ervices in remote areas).8 As the development process unfolds, there has been a shift in the relative salience of these two types of problems as the implementation challenges being confronted become more complex and contentious. At the start, the 2 Operationalizing the Science of Delivery Agenda core challenge is providing basic inputs—raising the funds, constructing the schools, training the teachers, and printing the textbooks. Thereafter, however, the challenge is how to combine the inputs—how to ensure that ­ teachers are present each day, are attending to the different learning needs of each child, and are actively engaging with (and responding appropriately to) parents and community members.9 For present purposes, the policy-­ performance gap is endemic in development because, in the face of a clear delivery failure, our bureaucratic imperatives compel us to presume that the core underlying problem is technical and thus can (indeed should) be addressed by a technical solution, such as better design, more funding, improved training (that is, capacity building), Sometimes these concerns certainly are the problem, but we—development practitioners generally and the World Bank in particular—critically lack coherent actionable mecha- nisms for distinguishing between technical and adaptive problems during implementation, and the corresponding strategies for anticipating and responding seriously to adaptive problems as and when they occur. Building reliable systems that can respond to these implementation challenges should, we argue, stand atop the 21st century development agenda. A world in which “1.3 billion people still survive each day on less than the price of a daily newspaper”10 will not be transformed into a world of dignity, justice, and opportunity for all unless and until such ­ implementation systems can be applied, in their own way, everywhere. A Way Forward Development organizations and governments have invested substantial resources in increasingly sophisticated technical solutions—such as new infrastructure, health programs, budgetary reforms, and access to clean and affordable energy. But investments per se do not always have their intended impact on people’s lives. Too often, implementation processes are disrupted, and otherwise sound technical solutions do not deliver the returns that policy makers want and citizens expect. Capacity shortfalls, misaligned ­ incentives, electoral cycles, and fragmentation between national, state, and local authorities are just some of the many nontechnical problems or deliv- ery challenges that practitioners face. Today’s development challenges and the new Sustainable Development Goals (SDGs) require that the proposed agenda be more ambitious; the agenda needs to go beyond technical solu- tions and standalone projects to incorporate a better understanding of delivery.11 The international community is setting the stage for international coopera- tion to achieve the SDGs by proposing a shift in thinking—from a narrow focus on Overseas Development Assistance to “investments of all kinds: public and private, national and global, in both capital and capacity.”12 Since ­ the SDGs will only be achieved through international cooperation that sup- ports the understanding of complex implementation processes, development organizations such as the World Bank should aim to collaborate and cultivate learning and knowledge sharing beginning with implementation. Among Operationalizing the Science of Delivery Agenda 3 development practitioners, there is a dire need to deepen our understanding of delivery insights that can help us to learn, adapt, and scale up. A key part of the challenge is that, as an organization, the World Bank often acts under certain assumptions and imperatives, without carefully defining the problems and solutions. This approach can lead to goals and expectations that are improperly aligned with those of clients and beneficiaries that, in turn, leads to a waste of resources as we struggle to achieve those improperly aligned goals. Our efforts are further hampered when we fail to adapt these goals and the procedures to adjust and achieve them. We lack a systematic cumulative understanding of what works and what does not work, where, why, and especially how; indeed, the need to build a systematic cumulative body of knowledge on “the how” is urgent. While practitioners on the ground often have this knowledge, large development agencies such as the World Bank lack a systematic process to allow them to capture this information and integrate it in real time into their practice. As important as large-scale reviews and evaluations are, too often the “lessons” emerging from them arrive far too late to be of practical benefit or to serve as an instrument of iterative organiza- tional learning.13 This gap in our knowledge of what works and how to deliver generates a corresponding gap in the delivery of better outcomes to the poor. Concurrently, the World Bank is trying to accelerate and deepen develop- ment impact in poor countries and especially in the lives of the most marginal groups in those countries. The Bank is doing so in a world where countries increasingly face complex challenges—such as job creation, social inclusion, justice, and environmental sustainability. These challenges require bold, cross-sectoral, multistakeholder solutions and need to take into account the private sector’s contributions to investment and job creation, as well as the many capable development partners around the world. To realize their devel- opment objectives, countries need both coherent national strategies and high-capability delivery mechanisms connecting national level policymakers to mid-level officials to frontline implementers. Moving coherently in this direction requires greater everyday collaboration in pursuit of context-­specific solutions to context-specific problems.14 What is needed to bridge the policy-performance gaps and accelerate development impact for the poor? We need a comprehensive approach that combines technical expertise with on-the-ground delivery know-how. We need to systematically collect and understand insights from practitioners in the field. We need to harness this knowledge to develop a systematic understanding of what works and what doesn’t, applying it to deliver needed services to the poor, thereby accelerating progress towards realizing the World Bank’s twin goals of ending extreme poverty and boosting shared prosperity. We need to reject assumptions of linearity in development trajecto- ries and move instead toward a greater understanding of complex systems, the networks and evolutionary (often endogenous) dynamics that undergird these systems, and the way that local dynamics and local solutions factor into these complex systems. We need to embrace innovative ideas across sectors, constantly encouraging experimentation and scaling up the most promising innovations to reach more people, with more impact, more consistently.15 4 Operationalizing the Science of Delivery Agenda Five Core Principles of the Science of Delivery A set of shared characteristics can be distilled by observing successful and unsuccessful projects and what effective delivery efforts have in common. This emerging body of knowledge on how effective implementation occurs is needed to more consistently deliver solutions that both work for people every time (rather than appearing to be “hit or miss”) and are tailored to making greater gains in citizens’ well-being. Ideally, the cultivation and use of such knowledge—within projects, ministries, and implementing agencies—should not depend on outside expertise but be routinized into everyday practice, thereby creating a culture in which meeting high standards and seeking constant improvement is normal. ­ Outlined below are the common characteristics for delivering solutions for citizens. These characteristics are not new, as they draw on ideas expressed over many decades and the hard-won lessons of practitioners in the field. What is new is their incorporation into a systematic, cumulative body of knowledge. The goal is not to arrive at a universal set of authoritative, ­top-down, step-by-step, programmatic prescriptions, but rather guiding ­ principles—the sensibilities, mental guideposts, or (more popularly) professional “mindsets”16— that can be adopted and adapted in different contexts and s ­ectors. Such principles enable teams to constantly and thoroughly interrogate their ­ assumptions, to ask the right questions, to more accurately identify and prioritize problems, and to move iteratively toward more effective and ­ context-specific delivery solutions for everyone, especially the poor. These ­ principles are most likely to realize SDG 17, which focuses on the character- istics of successful partnerships for implementing and achieving the SDGs. These five core principles are as follows: 1. Relentless focus on citizen outcomes: Defining the goal as attaining measurable but realistic (as determined by recent performance trajecto- ries) gains in citizens’ well-being; identifying the nature of the delivery problem associated with achieving these gains from both the provider’s and (especially) the user’s perspective; understanding the factors shaping citizens’ demands and expectations and the various contextual issues that influence delivery outcomes. 2. Multidimensional response: Facilitating multi-stakeholder coalitions and multi-sectoral perspectives to identify and prioritize problems and (possible) solutions; convening various development partners and build- ing on their respective comparative advantages. 3. Evidence to achieve results: Using the best available evidence to identify problems and solutions; developing local evidence to understand vari- ations in implementation quality (across time, place and social groups) and using it to refine solutions; collecting evidence of results throughout the project cycle to identify weak links; contributing to the global body of knowledge with the evidence collected. 4. Leadership for change: Understanding local political economies and drivers (as well as resisters) of change; identifying the incentives that moti- vate behaviors and integrating these findings into the both the design and Operationalizing the Science of Delivery Agenda 5 implementation of delivery solutions; forging durable social compacts between leaders and constituencies to diligently support change through difficult times. 5. Adaptive implementation: Developing an adaptive implementation strategy that allows for iterative experimentation feedback loops and mid-course correction; building a committed team with the right skills, experience, and institutional memory; maintaining the capacity to reflect on actions and their results. These principles are not discrete, disjointed elements that operate in isolation. Rather, they are organic and interconnected and as such should complement and enable one another. For example, Principle 3 is indispensable to realizing the other principles, as follows: rigorously conceived and diligently collected evidence is crucial to defining the right problems (Principle 1), to under- standing local political economies and building consensus for change (Principle 4), and to monitoring our own progress and adapting to achieve our goals (Principle 5). Similarly, leadership for change (Principle 4) is natu- rally multisectoral (Principle 2) and is impossible to achieve without careful identification of the problem faced by beneficiaries (Principle 1). Other principles are embedded across multiple approaches, lending them a degree of commonality; for example, the citizen engagement principle, currently being mainstreamed across the Bank, is important for focusing on ­ citizen outcomes (Principle 1) or the SDGs because it enables us to under- stand which problems citizens are prioritizing. It also provides important sources of evidence for results, coming directly from the beneficiaries (Principle 3), and it provides crucial information and impetus to adapt during program implementation (Principle 5). As such, these different facets of successful delivery work in concert to enable better delivery solutions and ­ outcomes. The relationships between them will also vary depending on the context of a particular intervention. The remainder of this paper discusses each of the core principles in greater detail, with reference to related literature, selected projects that illustrate these principles in action, and tools that have drawn on these principles to achieve results. It concludes with some notes on the implications of getting delivery right to accelerate progress toward the World Bank’s twin goals and increas- ing the Bank’s impact more generally. Principle I: Relentless Focus on Citizen Outcomes In the past, most World Bank projects were designed to focus on generating outputs—or products—rather than the ultimate outcome of producing posi- tive impacts on people (that is, healthy mothers or children who can read at grade level).17 As a result, many projects have not consistently achieved the ultimate goal of helping the intended beneficiaries. To counteract this ­ tendency, it is crucial to put people and positive outcomes for people at the center of our projects. Before embarking on an initiative we must carefully interrogate our desired outcomes, ensuring that we are “solving the right 6 Operationalizing the Science of Delivery Agenda problem”18 and that we understand who benefits from our interventions (and  who doesn’t) and why. Doing so requires using both qualitative and quantitative methods to ­ rigorously measure the outcomes that actually make people’s lives better and tracking the processes by which different outcomes are experienced. A relentless focus on citizen outcomes means that a project’s success is ­ ultimately not measured by the amount of funds disbursed, or by completing the stages in a project cycle, or by the ratings produced by the Independent Evaluation Group, but rather it is gauged by whether or not it achieves measurable improvement in the well-being of its intended beneficiaries. ­ This  focus enables practitioners to establish clear priorities, based on what will improve outcomes for the citizens in question, and to align a common vision around these priorities. This approach also allows for flexibility, ­moving beyond precise processes or preconceived methods and helping to create spaces within which solutions can be tailored to the problem at hand. This process requires broad considerations of the project context to identify  the right problem, based on a thorough understanding of citizens’ ­ demands and the local context. “The right problems” are those that are locally-nominated and prioritized, rather than those identified from the ­ ­outside.19 At the same time, solutions to most complex development c ­ hallenges cannot be simply imported from different contexts (for the evidence-based implications of this principle, see section III, below). Replicating practices across contexts is highly problematic.20 Citizen engagement plays an impor- tant role in this process; “listening upstream” in program design enables us to make sure that interventions suit the needs and preferences of potential ­beneficiaries.21 Finally, a relentless focus on citizen outcomes requires looking beyond the bounds of the project itself and considering other factors that influence citizen outcomes. For example, a Bank project that focused on ensuring safety on Argentine highways took on a locally nominated problem with considerable concrete impacts. Road crash fatalities represent the leading killer of Argentine youth, topping 10,000 deaths per year to reach the public health stature of an epidemic. In this case, as in others, traditional Bank activities, such as putting ­ together a package to finance infrastructure improvements, was not an appro- priate or sufficient solution for the problem. This challenge required the Bank and its Argentine partners to go beyond an output framework to focus on outcomes (which were literally life and death). To address this problem, a single lead agency was created to focus efforts on decreasing crash fatalities and attacked the problem on numerous fronts: project leaders leveraged diverse sources of data to find solutions to the prob- lem; they cultivated ties with numerous other agencies to ensure implementa- tion and with civic groups to undertake an education campaign; they identified particularly dangerous highway corridors on which to concentrate safety efforts. The expertise gained through this experience is now being dissemi- nated through a Regional Observatory on Highway Safety. Recent works have labeled the correct identification of problems and the diagnosis of the underlying cause of these problems as indispensable first steps Operationalizing the Science of Delivery Agenda 7 in designing solutions that work for people. Datta and Mullainathan (2012), for example, emphasize the role of behavioral approaches in helping to diag- nose underlying problems and define the scope of the problems—understand- ing people’s motivations and their reasons for doing things shows us how to target their real needs.22 Ramalingam (2013) urges decision makers to consider complexity in designing solutions and to take into account both the problems that are defined and the solutions that are emergent at the local level.23 While these recent works differ in their approaches or in the general toolkit that they apply to solving problems, they agree that rigorous thinking about the sources of the problems and the rigorous definition of what the solution may be—with extensive reference to the ways that poor people define and attempt to solve their problems—are indispensable when attempting to create better outcomes for  citizens. Moreover, they suggest the importance of understanding the complex dynamics in play in the local development arena and of seeking to ­ enable and nurture locally-driven solutions to locally-­defined problems, rather than imposing problems or solutions on the people that we seek to help. A 2013 Note from Monitor Deloitte, which champions the use of innova- tive methods grounded in behavioral insights, also identifies the initial pre- condition for attempting a solution as “[understanding] the nature of the underlying challenge, the population you are trying to target, and [defining] the desired outcome you are trying to achieve.”24 It is this deep understanding of the problem that then enables what another Deloitte white paper describes as “disciplined focus on strategic priorities.”25 These approaches have recently gained additional currency at the World Bank with the publication of the World Development Report 2015: Mind, Society, and Behavior (WDR 2015). With its focus on mental and behavioral factors, WDR 2015 sheds considerable light on the issues of problem definition and challenging assumptions, taking into account behavioral approaches to design better solutions. This process applies to the analysis of the preferences and decision-making processes of beneficiaries, but does not exempt develop- ment professionals from continuously analyzing their own assumptions. WDR 2015 notes that “[b]ecause development practitioners often have preconceived notions about a problem and its potential solutions, they may believe that they know what should be done without having made their assumptions explicit and without having diagnosed the actual problem and its causes.”26 WDR 2015 notes that experience in the field shows that even seasoned implementers should take this into account, especially because of the degree to which context matters and is highly variable from place to place. For ­ example, in a water program in Kenya, cited by the WDR 2015, the key “­problem” changed repeatedly: from clean water at the source, to providing people with tablets to sterilize the water, to making sure that they added the tablets. Focusing on a particular problem may necessitate repeatedly redefin- ing the problem as new information and evidence on what affects beneficia- ries comes to light. This redefinition is a potent argument both for adaptation during implementation and for careful diagnosis of problems during the design phase of an intervention. One such diagnostic exercise that was under- taken in Zambia, appears deceptively simple: before providing “mama kits” 8 Operationalizing the Science of Delivery Agenda to pregnant women, a series of semi-structured interviews were used to deter- mine what items the women wanted in the kits. The provision of attractive and useful kits provided an incentive to give birth in a hospital, resulting in an increase of 44 percent in the number of women who gave birth in hospital, which was the larger aim of the intervention. Principle II: Multidimensional Response A single actor can rarely solve a complex problem alone; in devising solutions to complex problems, a single method or discipline is rarely sufficient in ­isolation.27 Rather, responses to complex issues need to incorporate diverse perspectives into their analysis, while securing buy-in from a wide range of stakeholders to devise sustainable durable solutions. An example of this multistakeholder, multisectoral, and multidisciplinary approach to solving a highly complex problem is the Bank’s approach to coun- tering the threat posed by the H5N1 avian flu. Pandemics do not respect national or sectoral boundaries; especially in the case of H5N1, they involve a wide range of actors, from customs agencies to hospitals and the rest of the health sector, from aviation authorities to the animal sector, police, and all levels of political office. No single country or agency could address this threat. To counter the profoundly damaging effects of uncoordinated, low-impact, single-sector, or single-perspective response, the Bank assembled the widest possible range of stakeholders to build a shared framework for preventing a human pandemic at its animal source. The Bank leveraged its convening power to bring together the human and animal health sectors, civilian and military authorities at the national and local levels, the United Nations, and donors. This process ensured coordinated implementation globally and in individual countries, bringing a wide variety of perspectives to bear on the problem and ensuring that all stakeholders’ needs were met. This crucial coordination yielded significant shared benefits for all by countering the threat posed by the H5N1 avian flu. Development practitioners have long recognized the importance of work- ing closely with local stakeholders to get results and build capacity. A long- standing literature also emphasizes the importance of facilitating change by engaging with multiple agents in a process of dynamic learning and capacity building.28 Such group-based engagement helps build space for change by establishing legitimacy, authority, and the ability to move from one equilib- rium (the status quo) to another (the desired outcome).29 A key aspect of such change processes is the identification of both the bind- ing constraints of the desired reforms in each context and the entry points and reforms that can potentially have a major impact in the country or region. Through a collective, deliberative approach, a robust dialogue based on shared experiences facilitates innovation by expanding the choices available to stakeholders.30 Solutions to complex problems should incorporate diverse perspectives to identify problems and solutions; indeed, they often require buy-in from a Operationalizing the Science of Delivery Agenda 9 wide variety of stakeholders to succeed. Without having local champions at the table, an understanding of the players who stand to lose, or who has an interest in proposed projects or interventions, the proposed solutions will not succeed in delivering the hoped-for outcomes. To amplify impact, it is essen- tial to engage all stakeholders to develop solutions, supporting reform- minded stakeholders while minimizing the possible negative effects of those opposed to change. Once stakeholders are on board, it is essential to continue to listen to them and take their needs into consideration, nurturing and solidifying a broad coalition, and bringing partners together around shared priorities. A dynamic, catalytic, and transformational approach to change must inspire and galva- nize groups of practitioners to action. Exposure to comparable experiences and exchanges of lessons learned may inspire these practitioners, which in turn translates into learning and strategies for action.31 The Action Learning Program (ALP) is one such mechanism for ­inspiration.32 The ALP is built to support reform programs. It combines a conceptual framework with analytical rigor on how to conduct sustainable ­ reforms; the ALP takes into account the tools, mechanisms, and specific actions that can be implemented using a mix of technology. The ALP’s main objectives are as follows: 1. Facilitating the prioritization of reform areas through joint design and implementation of specific diagnostic tools (that is, in-depth, country-​ specific surveys) that are key to building local capacity and promoting partnerships to monitor governance while generating new information. 2. Creating a space for exchanging knowledge and experiences to build on the political initiative of a first set of 4 to 5 participating countries is recommended, but the methodology can also be used at the national and ­ local level. Knowledge sharing can serve as a conduit for creating interest in reforms or deepening existing political will. 3. Promoting and facilitating coalition building among key actors through participatory linked activities. Information and communication technol- ogies (ICTs) are crucial in forging these coalitions, as they enhance access to information, allow people to make better judgments and informed choices, and reduce time and cost constraints of links among practitioners. Forging coalitions requires inspiration, loyalty, time, and trust. 4. Strengthening national capacity to formulate and implement reforms. 5. Fostering regional consensus and strengthening the capacity of key actors to support sustainable development in the region. Successful multistakeholder approaches can be scaled up. In such cases, scaling up is not just about expanding these programs into other countries, ­ but it is also about building endogenous problem-solving processes and community engagement, and working with networks that can follow the ­ ­ processes with multiplying effects. These engagements can have multiple spinoffs as well, leading groups to identify additional problems that need to be solved, mobilizing attention and resources, and putting new issues on the development agenda. In this way, all these initiatives promote a broader set of 10 Operationalizing the Science of Delivery Agenda opportunities in development for donors to support. These locally-owned opportunities are endogenously identified, and thus are highly likely to be implemented. “Hackathons,” which have increasingly drawn interest over recent years, bring together a wide variety of actors to collaboratively develop ideas and form new partnerships; as such, they are another useful tool for achieving multistakeholder coordination and for bringing multiple perspectives and disciplines together to solve problems. The new connections formed during such events have the potential to bring actors together around innovative approaches, catalyzing new initiatives and ways of thinking to achieve impact. Andrews, Pritchett, and Woolcock (2013) note the importance of sustain- ing broad enough political support, or at least consultation, and the resulting legitimacy that this confers to ensuring the sustainability of reforms for implementation, arguing that “change primarily takes root when it involves broad sets of agents engaged together in designing and implementing locally relevant solutions to locally perceived problems.” The authors go on to empha- size the significance of a body that plays a convening and connecting role, noting the importance of bridging the gap between those with power and those with ideas. Mohan and Dinneen (2013) apply a similar logic.33 They argue that “[t]o effectively design behavioral insight solutions, it is important to involve a diverse group of stakeholders,” acknowledging that broad support is crucial to ensure the correct policy response. Analysts for McKinsey and Company concur, stating that stakeholders in an intervention, from members of the political opposition to the public at large, “need to be part of the action from beginning to end” to develop solutions and enable the process they refer to as “Delivery 2.0.”34 An illustrative example of this multisectoral and multistakeholder approach, and its particular relevance for achieving impact at scale, is a proj- ect in Andhra Pradesh in which the state government, with support from the Bank, mobilized over 11 million women into self-help groups. Often acting in conjunction with their umbrella federations, the self-help groups increased ­ their access to savings, food consumption, school enrolment, and assets by working across sectors and with many different actors and institutions. Ryan and Ali (2013) claim that successful leaders work with other organi- zations and individuals, and that this is particularly relevant to solve complex problems that defy siloed approaches. Instead, they emphasize forging con- nections across institutional boundaries to enable ideas and expertise to be shared and to build shared ownership of initiatives. Principle III: Evidence to Achieve Results In designing projects, knowledge of past solutions can often be reasonably extrapolated to the question at hand.35 By starting with and learning from a survey of known facts and evidence from others’ experience, we can save time and resources, while making sure that we are centered on the needs of our Operationalizing the Science of Delivery Agenda 11 beneficiaries. Within the Bank, however, as in many institutions, the relevant evidence is not always brought to bear in the right way on the problem that the project is intended to solve. There is always the hazard of “cherry-picking” evidence to fit our preferred intervention; indeed, significant studies have found that across the development and aid landscape, “there is far more policy-­ based evidence than evidence-based policy.”36 The temptation to ignore ­ certain kinds of evidence due to methodological bias is equally dangerous. A strong commitment to systematically gathering the best available evidence from across the spectrum of disciplines and methodological ­ approaches, and to using that evidence to tailor approaches to local context, is crucial to ensuring that development interventions succeed. Gathering infor- mation is not sufficient, of course; to bring about gains in citizens’ well-being, interventions must be based on evidence of what the beneficiaries need and on evidence of what will meet that need. Evidence must then be integrated into our decision-making framework. What Kind of Evidence? Policy-makers increasingly accept that policy and program design must be based on evidence, and that the ability to properly define problems and solu- tions rests on a solid evidentiary base.37 But what constitutes evidence, or the “right kind” of evidence? Answering this question is vital to move toward an approach of evidence-based interventions. One school of thought holds that randomized evaluations represent the “gold standard” for producing rigorous evidence, and randomized con- trolled trials (RCTs) are increasingly touted by development scholars. RCTs are an experimental method that attempt to isolate and measure the effect of a given intervention. RCTs randomly assign a treatment or intervention (e.g., receiving water purification tablets) to one group of people, while also measuring results for a control group of participants who are as similar as possible to the project beneficiaries with the sole exception of not having received the tablets (or other intervention). RCTs then measure the impact on those who were targeted for the treatment or intervention, as compared to the control group. The RCT approach makes important contributions to understanding what works and what doesn’t, but it is not a magic bullet for development, and use- ful evidence is not restricted to RCTs (nor would proponents of this method claim it to be so). Like all knowledge, RCT results are bounded by the place and context in which they are undertaken (Pritchett and Sandefur 2015). As development practitioners, we face many heterogeneous problems, in tre- mendously diverse contexts, and undertake a wide variety of heterogeneous interventions. In view of this heterogeneity, certain types of evidence may be only be “rigorous” in narrow, context-specific bounds; the common prac- tice of transposing evidentiary findings from one context to another, on the grounds that an intervention’s success in one place will ensure its success in another, is problematic no matter the strength of an original intervention’s or experiment’s methodology.38 Similar logic applies to questions pertaining to 12 Operationalizing the Science of Delivery Agenda the wisdom of scaling up given interventions on the basis of positive results from initial pilots. Previously, one of us (Woolcock) has argued for a more contextualized and fitted understanding of rigor, in which “a truly rigorous evaluation is [construed as] one that deploys the full arsenal of social sciences methods as part of a strategy focused on achieving…an optimal match” between method(s) and the problem to which a given intervention responds.39 Without abandoning randomization per se, this school of thought calls for “specification of alternatives, rigorous counterfactuals, and increased real- time measurement to learn about project efficacy.”40 In an environment of heterogeneous contexts and interventions, it is vital that the problem itself “generates the methodological response, not the other way around, just as the available policy/project ‘solutions’ should not determine which prob- lems are addressed.”41 In light of the heterogeneity of problems, solutions, and contexts with which development concerns itself, an inclusive conception of useful evi- dence is needed, which acknowledges the importance of context, draws from diverse sources and spans multiple methods and disciplines. Useful evidence may be qualitative or quantitative; it will often be local, highly contextual, and will facilitate study of change over time; it may come from single studies or systematic reviews of many studies, expert opinions or ethnographies, personal experience, or “big data,” or case studies, or RCTs. Some measures ­ may need to be further refined to be properly used; for example, measures of participation, citizen voice, and “how people feel” are difficult to capture and are currently not being captured well.42 When beneficiaries’ preferences are not well understood, it may lead to problematic disjuncture between these preferences and interventions. In the health sector, for example, this gap leads to a waste of perhaps as much as 40 percent of expenditures when allocation of resources is misaligned with beneficiaries’ needs.43 Moreover, different sources of evidence should not be isolated from one another. Exchange of knowledge and experiences among practitioners is vital, but this type of learning is only reinforced when practitioners also have access to diverse types of “systematic and rigorous analysis” on what works and does not, using these different kinds of analysis to strengthen each other.44 Using Evidence for Results Merely arriving at an ample understanding of useful evidence, however, is not enough; we must bring it to bear on the complex process of designing and implementing interventions. We need to start a project with a deep under- standing of the characteristics, needs, and preferences of the actors who will be affected, and respond throughout implementation to emerging evidence that may suggest needed adjustments. A commitment to using high-quality evidence has at least two components – it means collecting evidence and then putting it to work to inform decisions. Evidence for results comes into play at several points in the project cycle: in the lead up to the project, when past evaluations, newly commissioned studies, and other evidence can help shape Operationalizing the Science of Delivery Agenda 13 the intervention design and enable proper problem identification; during implementation, when evidence being generated can enable adjustments and flexibility; and after final evaluations, when it adds to the global body of knowledge. It is vital that the best available evidence be used in developing our projects, starting with identifying the problem. This process means taking into account the existing body of knowledge about the case to be addressed and drawing on similar cases for insight; it also means conducting formative research to help establish the proper solution and to determine what interventions are well-suited to the local context. But the collection and deployment of high-quality evidence does not end there. We must collect various forms of evidence—from data on relevant ­performance indicators to the assessment of seasoned practitioners and infor- mation on the perceptions of beneficiaries—throughout the project cycle. Indeed, the collection of evidence on what works and what doesn’t work needs to be an everyday practice on the ground.45 Constant collection of information can allow for informed adjustments, including mid-course when necessary; ideally, it will entail a shift from regarding “monitoring data” as an instrument of compliance to a tool for active learning. Putting useful high-quality evidence to work offers considerable potential for adaptation, allowing “that lessons learned in the first year or two of operation might lead to program refinements, improved program implementation, and improved client outcomes.”46 Pritchett, Samji, and Hammer connect this notion to the idea of “structured experiential learning,” in which an experiential compo- nent is added to traditional monitoring and evaluation (M&E), allowing for real-time learning and “searching” for iterative improvements and innova- tions to increase an intervention’s impact.47 Citizen engagement and participation is another important source of evidence. As World Bank President Jim Kim has noted, citizen feedback ­ “mid-stream” is vital in enabling mid-course correction, while citizen feed- back at the end of the project cycle is indispensable for understanding what worked, what didn’t, and for whom.48 In other words, this information can be used to monitor both progress and results. Mechanisms to engage citizen feedback and integrate this feedback into project implementation, such as the E-ISR+ (External Implementation Status and Results Report Plus) and asso- ciated ICT programs to augment this approach, have been deployed in a number of different countries. The practice of gathering citizen feedback on projects through E-ISR+ has shown promising results, with projects being adapted and improved as a result of citizen inputs; this approach illustrates the importance of using evidence derived from citizen feedback to improve the effectiveness of interventions.49 At the end of a project, and/or at an evaluation point, the evidence col- lected, including the experiences of the people who put this project into action, adds to the global body of knowledge and informing future projects and interventions. When the project is under consideration for scaling up, scaling down, or continuation, having rigorous multidisciplinary evidence on hand is crucial. The real-time learning that has taken place during the project 14 Operationalizing the Science of Delivery Agenda may be useful, in conjunction with a well-specified theory, to predict the project results in changed conditions (of scale, place, time, etc.).50 ­ All of this is perhaps easier said than done, but it is possible. Evidence col- lection and evaluation need to be strongly embedded in the overall project framework, with teams and partners firmly committed to using it (i.e., not putting studies “on the shelf ”). Evaluations should be focused firmly on achieving results for people and embedded in feedback loops, allowing teams to change direction when necessary.51 When evidence shows negative results for a project, we need to be genuinely prepared to change course when feasi- ble.52 And given that evidence is never the sole determinant of how or whether interventions are implemented or scaled, the framework in which the use of evidence is embedded needs to be transparent, inclusive, and fair.53 We must ­ context, pay attention to the social and political contexts, making sure that the “ the networks, and the knowledge are all in alignment.” Finally, the evidence 54 gathered and the results of various interventions should be broadly shared, allowing for learning across the Bank and even beyond its confines. As a corollary to this, research and evidence will need to be presented to ­ ­ policy makers and practitioners in useful forms.55 Additional Literature, Examples from the Field, and Relevant Tools An increasing focus on the timely use of evidence is clearly becoming de rigueur in government circles,56 though many government actors continue to work to refine this approach. It is of no small significance, for example, that in 2013 the U.S. Office of Management and Budget issued a directive for federal agencies to more aggressively use evidence to determine which programs were effective (and therefore deserved continued resource allocation) and which were not. To take another example, Nigeria’s Ministry of Health has committed to expanding support for evidence-based health interventions, with Muhammad Ali Pate, the Minister of State for Health between 2011 and 2013, writing that rigorous and relevant data “are an absolute prerequisite of effective delivery.”57 This proposition is applicable from the national level, as in the examples above, to the local level (see Metz et al. [2015], which lays out an example of an evidence-driven implementation process for a child welfare program in a single North Carolina county). It also runs the gamut of meth- odological approaches, ranging from the use of “big data”58 to ethnographies or focus groups.59 In the Bank, the Development Impact Evaluation Initiative (DIME) works to build structures of iterative operational learning that put evidence to work “by (i) sharing existing evidence at the design stage to introduce potential solutions; (ii) experimentally test[ing] project modalities to improve results; and (iii) validating project’s overall development hypothesis.”60 Projects in many different sectors have developed innovative methods to collect evidence, and monitor results, such as using smartphones to provide credible information outcomes in real time, incorporating features such as geotagging and photographs to make results credible, and using user-friendly Operationalizing the Science of Delivery Agenda 15 maps to present data. These relatively low-cost and flexible tools have a broad variety of applications, as they lend themselves to various instruments and applications that can be adapted and tailored to the task at hand.61 The process can be replicated at large scale and at periodic intervals to address the gaps in monitoring of the quality of the service. An example of global evidence is the joint monitoring program for water supply and sanitation with WHO and UNICEF that reports on the status of water and sanitation, and supports countries in their efforts at the global level. Another is SIASAR, a platform designed to monitor the development and performance of rural water and sanitation. The SIASAR system generates performance indicators that are aggregated at several geographic levels. ­ It  produces summary reports that detail the performance of communities, infrastructure systems, and service providers, then collects and uploads data using a specialized app for Android smartphones. Similarly, a cash-for-work program in Sierra Leone used smartphones to generate information that improved the delivery of services. Monetary payments were carried out via mobile transfer, which, besides being faster and ­ reducing leakage, also allowed citizens to monitor service provision, give feed- back in cases where payments were not delivered to standard, and p ­ rovide feedback on what services they used (and those they did not). In this way, using a relatively low-cost ICT intervention to gather evidence from the ground up, improved the program’s ability to meet the needs of its beneficia- ries and will allow the program’s methodology to be piloted in other sectors.62 Other examples illustrate the ways in which data-gathering projects them- selves must be informed by gathering certain types of evidence. A World Bank project in Nigeria needed to create an ICT mechanism to improve com- munication between decision makers and project beneficiaries, enhancing social accountability. Many ICT applications fail, however, going unused because their design fails to account for local context and user patterns. To avoid falling into this trap, project leaders undertook exhaustive efforts to gather information on the ways that local people use communications tech- nology, including an extensive ethnographic study that resulted in an ICT solution that beneficiaries would actually use. While using evidence to achieve results is vital at the project level, in other contexts the system to gather and apply evidence may need to be put in place at the country level, building long-term capacity to use evidence for results. In Indonesia, for example, partners in the government faced considerable ­ challenges in producing rigorous evidence and applying it to poverty reduc- tion programs. As a result, a World Bank program currently supports the gov- ernment of Indonesia to improve the process of evidence-based policy making, providing assistance at each stage of the project cycle, and contribut- ing to a shift towards evidence-based policy. As a result, Indonesia has created a targeting system to direct benefits and services to the country’s poor, while supporting innovative social assistance and cash transfer programs. Another relevant tool for getting from evidence to results is the BOOST data tool, a database that uses detailed budgetary data to assess the allocation, implementation, and effectiveness of public expenditures. Armed with this 16 Operationalizing the Science of Delivery Agenda high level of detail on national and subnational spending, decision makers can perform the analysis needed to improve the quality and efficiency of public service delivery. ­ Principle IV: Leadership for Change Development is fundamentally a process of change. Yet too many change interventions sought by development practitioners fail or underperform because of inadequate attention to managing “the complex process of change that inevitably accompanies any intervention.”63 Initiatives may lose support as time goes on or they may fail to win support, undone by resistance from interests deeply invested in the status quo. In addition to this course of ­ “initiative decay,” Andrews, McConnell, and Wescott (2010) cite studies showing that only 40 percent of major corporate change initiatives produce positive change and about one-third actually make the situation worse. They conclude that “change is difficult everywhere, and answers about how to do change cannot be assumed as intuitive or well known.”64 To address this problem, the leadership principle for change stresses using a thorough understanding of the political economy of a particular context to overcome political resistance and build support for the right interventions. In this sense, leadership is not the action of a single person, but rather is a process of bringing together coalitions to support certain innovations or interventions—that is, necessary changes in the prevailing way of doing things—and to confront political obstacles.65 Moreover, the leadership for change process will often need to bring many stakeholders to the table who do not agree with one another or who may be opposed to changes in the status quo. For this, the leadership for change principle encompasses at least three main elements, as follows. •• First, it is necessary to understand and take into account the political economy, power relations, and political dynamics of the location where we are attempting to effect change. Vital to this process is an understand- ing of the key players, including beneficiaries and actors who are likely to  be obstructive, and the way they are likely to respond to particular initiatives. •• Second, it is important to understand and align incentives, so that all parties have similar goals, both at the institutional level and often at the ­ individual level. •• Third, stakeholders must be convened and brought together within this incentive framework, with the Bank playing a convening and mobilizing role to build and sustain “political and developmental coalitions.”66 Change begins with a careful definition of the problem to be solved and with a careful definition of the population that one is trying to reach. Long-term change depends on these initial careful steps, aiming for measurable effects on citizen well-being and proceeding with the ultimate goal in mind.67 Only when the local context and the intricacies of the political economy landscape Operationalizing the Science of Delivery Agenda 17 are understood, does it become possible to engage productively with other actors and project beneficiaries. Andrews, Pritchett, and Woolcock (2013) highlight the example of textbook procurement, in which vendors, teachers, and other agents all have their own interests, as well as their own patrons or connections. It is crucial to understand who stands to gain or lose in a given scenario and how actors relate to one another when designing projects and engaging stakeholders. It is also necessary to understand what actors are likely to press for change in the context of reforms for improving delivery and which are likely to uphold the status quo.68 In Bangladesh, for example, the International Finance Corporation (IFC) confronted a complex problem in the textile industry that required consider- able leveraging of incentive structures to promote what its project leader referred to as “leadership by all.” While Bangladesh’s textile industry is an important driver of the national economy, it is also a heavy consumer of water and a major polluter of water bodies around it. Suboptimal use of chemicals and inadequate effluent treatment are contaminating surface waters, nega- tively affecting the lives of nearly 12 million people. Adopting the stance that in transforming a sector, relationships—among governments, communities, and the private sector—must also be transformed, World Bank Group (WBG) leadership brought together multiple stakeholders to change incentives and the behavior of diverse actors so as to promote a change agenda. In practice, this meant that the IFC used its convening power to bring together many actors, working to link buyers, factories, financial institutions, sector associa- tions, local communities, and local governments for coordinated action on water sustainability and using financing, technical assistance, and the prom- ise of unified action as part of a partnership of 200 factories to powerfully incentivize changes in behavior. To assist in the formation of this partnership for transformation, it was imperative to understand the incentives of the actors, as well as the social and political landscape in which the incentives were embedded. A survey was developed to assess levels of trust among actors, determining stakeholders’ levels of awareness of issues and of solutions, perceptions of various stake- holders about their role and that of others in addressing the issues, and the attitude of stakeholders towards each other. Trust levels were then tracked over time, providing information on the evolution of different actors’ behav- ior, attitudes, and interactions throughout the program. A focus on relationships—among poor people, states, service providers, and the World Bank—and how to build accountability into these relation- ships was at the heart of the WDR 2004. Additional research has continued to support the centrality of the fundamentally political issue of accountability relationships for service delivery.69 This research illuminates a central chal- lenge in service delivery: better outcomes cannot happen without under- standing the local political context and power dynamics, and shaping incentives so that decision makers and service providers are accountable to the poor.70 A recent report from the Independent Evaluation Group found, for example, that successful projects “probed more deeply into the local context, used local expertise…and formulated actionable recommendations ­ 18 Operationalizing the Science of Delivery Agenda to fit local administrative and political economy constraints.”71 Information and evidence is of course crucial to this effort, both for understanding the landscape and for disseminating information to help reshape accountability relationships and shape behaviors.72 In addition, in the art of leadership for change, individual incentives ­matter. More attention has been focused on the importance of understanding indi- vidual and group behaviors to affect development outcomes, including in the WDR 2015 (World Bank 2014). Individual behavioral levers can be applied to beneficiaries and interlocutors alike to bring about more effective solutions.73 Behavioral economics offers important insights on how to nudge or guide individuals toward choices that improve their individual development out- comes or, in the case of service providers, improve service delivery directly. Often the interventions involved are quite modest, as a Center for Global Development policy paper found, such as reminding project beneficiaries of the availability of a service or small incentive payments linked to attendance at clinics or schools for beneficiaries and providers alike. Framing, messaging, and providing information are also key; sometimes project beneficiaries need to be reminded of the benefits of participating in certain programs or be convinced of their benefits, and often the information provided needs to be precisely targeted to change behavior.74 One example of putting individual incentives around behavior in action is SaniFOAM (focus, opportunity, ability and motivation). This intervention gathered evidence on bottlenecks ­ to individual behavior changes to develop an effective sanitation marketing program. Principle V: Adaptive Implementation In the course of project implementation, teams will almost certainly encoun- ter challenges and unexpected obstacles.75 If teams do not step back, reflect, learn and adapt, they risk persisting with interventions or strategies that are not well-fitted to the situation that they face. Therefore, teams must be ready and willing to adapt mid-course, to experiment and scale up what works, and to iterate and integrate feedback into implementation. Together with careful planning and the elaboration of a clearly articulated theory of change, the incorporation of “rapid feedback loops” into an endeavor is crucial, as is using these for “learning in response to ongoing challenges.”76 The framework ­ outlined here draws on the structured experiential learning approach in that teams collecting data use that information in real time with “direct feedback into the decision loops.”77 This approach often means bringing the right people to work on a problem. Experienced practitioners bring important insights gained from years of hard-won experience, as well as the institutional memory needed to bring these insights to address implementation and delivery problems. Experienced practitioners must also maintain the capacity to reflect on actions, stepping back to assess the results achieved and consider what worked and what did not. Reflection and action are linked in what Donald Schon calls “a dialogue Operationalizing the Science of Delivery Agenda 19 of thinking and doing through which I become more skillful.”78 In Schon’s thinking, this is an integral part of the manner that practitioners learn about their own actions, the results of those actions, and the implication of these actions for future actions. As teams iterate in the course of a project, this focused reflection on whether a particular intervention delivered better citi- zen outcomes allows teams to scale up the things that work, while adjusting strategies that do not deliver the hoped-for outcomes. With the goal of making a dent in the 2.5 billion people worldwide without access to improved sanitation, the Global Scaling Up Rural Sanitation pro- gram aptly illustrates this principle. The project was launched in three coun- tries, which served as a sort of at-scale pilot or a set of learning laboratories from which the program developed its theory of change for rural sanitation. After this pilot phase concluded, the project then made the necessary adjust- ments and scaled up to a further 10 countries; to date, it has provided some 22 million people in 13 countries with improved sanitation. What enabled successful scale up and impressive impacts in this project? The use of pilots in the initial “learning laboratory” countries provided ­crucial knowledge about what worked and what didn’t. This information was then disseminated through a global network, allowing team members to reflect and analyze the results of their actions. Team leaders were able to learn from these initial lessons in real time, allowing for quick adaptation. An iterative and adaptive approach was also hardwired into the program, giving task team leaders both the freedom and the mandate to apply lessons learned in their countries or areas of responsibility, while also adapting and correcting course as they scaled up and collecting their own evidence locally to target effective behavior changes and interventions. Another Bank project currently in very early conceptual stages is the ­PAL-Nigeria project, which explicitly builds adaptation and learning during implementation into project design. This project highlights initial chal- lenges that may be faced by teams attempting to undertake such an approach. The project design states that “…there is little solid experience of what works and what does not. This notwithstanding, there is demand across task teams to find nimble, context appropriate strategies for generating data around projects in a way that will provide timely feedback in the short term.” The project will work to increase the impact of specific projects and work streams in the country by giving teams the information they need to “adapt on-the-go and correct, as needed, mid-stream if impact is not seen.” In addition, the project will seek to contribute to the knowledge base for Nigeria as a whole, adding what works and what doesn’t. In this sense, it also exhibits (as do many projects) a number of the principles of the science of delivery; not only is adaptive implementation baked into the project, but it also helps to generate deeply contextual evidence and explicitly promises to improve our knowledge of political economy frameworks (particularly relevant to Principle IV).79 The concept of a feedback loop is a key feature of this approach and now appears repeatedly in the relevant literature. Andrews, Pritchett, and Woolcock (2013) propose “tight feedback loops that facilitate rapid experiential 20 Operationalizing the Science of Delivery Agenda learning.” Kohli (2013) concurs, arguing that agencies “should be using data to refresh their strategies in real time.” While implementation is ongoing, the data that this process produces is analyzed, performance is reviewed, and strategy refined in response, enabling the quick responses and tight loops referenced above. Citizen engagement can be a key part of this feedback ­ loop. As indicated above, feedback from program beneficiaries can provide important inputs for learning and course correction.80 Also crucial for adaptive implementation are small-scale pilot initiatives, with successful efforts subsequently being scaled up. Ryan and Ali (2013) advocate trying new programs initially on core constituents, then scaling up to broader populations if the intervention is successful, in a process they call “rapid prototyping.” This approach provides early feedback and reduces costs; if prototypes do not deliver results, they can be phased out “before significant time and resources are expended.” The Malaysian government’s premier implementation group, PEMANDU (Performance Management and Delivery Unit), operates on similar principles. Andrews, Pritchett, and Woolcock (2013) note that this approach is unlike  “traditional monitoring and evaluation mechanisms that focus on compliance…and allow ‘lessons’ only at the end of a project.” It is also differ- ent from field experiments and other testing mechanisms. Rather, it is about trying a real intervention in a real context. The goal is to enable learning in the process of developing solutions, allowing teams to adjust their methods in response to the feedback generated by the delivery process. Even successful projects may need to confront failure and change direction to ensure that they ultimately reach a positive impact. For example, the Afghanistan Rural Enterprise Development Project, which focused on facili- tating entrepreneurship and access to finance in Afghanistan, was faced with compelling evidence that its initial approach was not working. Project leaders led an overhaul, changing the approaches, hiring local experts to assist with facilitation, and creating special teams to reach nomadic populations. This process provides an excellent example of the importance of changing or abandoning approaches that are not delivering, with the aim of finding implementation strategies that will work for beneficiaries. ­ Toward Adaptive Implementation at the World Bank Since the late 1990s, the WBG has attempted to wrestle with the imperative to adjust its programming to changing circumstances and to adapt to new ­information generated in the course of implementation. For example, in 1997, the WBG introduced two new instruments for investment lending: Learning and Innovation Loans (LILs) and Adaptable Program Loans (APLs). Both instruments were intended to “contribute to a results-oriented culture with a strong focus on learning,” with task team leaders learning from project experiences to refine implementation details.81 The APL was intended for ­ long-term ­multi-operation engagements, often in environments where the Operationalizing the Science of Delivery Agenda 21 success of a loan or project depended on significant institutional and behav- ioral change. These instruments encouraged learning-by-doing and sustained continuous partnership with clients. As such, APLs are built in multiple phases over a period of up to 15 years and set “defined performance mile- stones (or “­triggers”) for moving from one phase to the next, while adapting the implementation details in the course of the program.”82 The LIL was intended to be more limited in scope and to “permit piloting and innovation over a relatively short timeframe,” with the option to test new strategies.83 Particularly in h ­igh-risk environments and/or when using unproven approaches, the LIL could allow for testing followed by scale up for programs with positive results and discontinuation or adaptation of less successful operations. The results of these instruments were mixed. While both instruments received considerable support from clients and staff, reviews in 2005 found that neither had lived up to expectations. The LIL was judged “insufficiently flexible and overly demanding in its requirements,” characteristics which indicated that this instrument was not sufficiently differentiated from the Specific Investment Loan.84 At the same time, the monetary and preparation time cap on the instrument created a variety of disincentives to use it. This cap also created a hobbling paradox as follows: while strong M&E was central to the LIL objective, “many staff found it challenging to develop strong M&E mechanisms, especially given the limited time and resources available.”85 Indeed, after FY00, the number of LILs sharply decreased for several years, with none being approved in FY05. The APL similarly suffered from insufficient flexibility and demanding requirements for approval, particularly in light of (not entirely correct) assumptions that approvals would be streamlined for these instruments, particularly following the first phase. While partners appreciated the implica- ­ tion of this long-term engagement, decisions about adaptation were not always sufficiently transparent; the 2005 review highlights the importance of “budget allocations for follow-on phases and decisions to continue or discon- tinue the APL” and recommended a more inclusive process of discussion around adaptations.86 Experiences with these adaptable instruments seem to indicate both staff appetite for increased flexibility and adaptability and a certain degree of difficulty in putting this into practice. For example, staff noted in the review ­ that additional guidance on the process for preparing and implementing APLs was needed, in a sense, the “how-to” of adaptability. Moreover, it appeared that insufficient resources were allocated for the monitoring neces- sary to gauge success and enable adaptations, particularly in the especially “risky” LILs. This mixed assessment of these flexible and adaptive instruments indicates that in the mid-2000s, the Bank was still searching for the right policies to enable adaptive implementation. ­ The Bank undertook significant investment lending reform starting in 2009. Approved in April 2013, the new Operational Portal and Bank Procedures and BP documents saw significant changes for both the APL and the LIL. The APL was integrated into a unified policy framework, acknowledging that some 22 Operationalizing the Science of Delivery Agenda programs would need sustained multiphase support. The triggers, however, were seen as “unwieldy” and were eliminated.87 In place of setting triggers, the approval of successive projects in a series would take into account the perfor- mance of the preceding phase or project. The LILs were eliminated entirely, as demand had slackened and the numerical cap on this instrument was seen as too restrictive.88 The imperative to be flexible and adaptive when implement- ing has remained, however, with consolidation and streamlining of policies potentially serving this end. As part of this reform, the Bank’s process for project restructuring has been significantly simplified. This process has removed the need to parse fine dis- tinctions between three different types of restructuring. Board-level restruc- turing occurs in three clear instances, as follows: when the project development objective is changing; when a new safeguard policy is activated; and when the safeguards category rises to level (A), which denotes a high level of risk. All other restructuring is carried out on the basis of the country director’s approval and signature. In either case, a simple restructuring paper is used; this is supplemented by a memorandum of the president in cases where the case is sent to the Board. This process allows for more flexibility and adaptive implementation. The policy has been recently approved, so it is premature to make judgments on its effects. However, the policy has the potential to allow for greater flexibility and adaptive implementation. Other Bank experiences and publications also speak to the importance of adaptability at the Bank. The World Development Report 2015: Mind, Society, and Behavior concludes by arguing for the importance of adaptive imple- mentation. Because a particular intervention may work differently in differ- ent contexts and because it can be difficult to know beforehand what features will “matter,” the WDR 2015 advocates for a greater “tolerance for failure” and for promoting a culture of experimentation and empirical agility. The WDR 2015 concludes that “In a complex and iterative process…prob- lems may need to be redefined and rediagnosed, and multiple interventions may need to be piloted simultaneously—some of which will fail—before an effective intervention can be designed.”89 For example, one intervention profiled in the WDR 2015 (and discussed above)—to improve access to clean water in Kenya—went through multiple waves of experimentation and refinement of the results to arrive at the most effective approach. The implementers repeatedly experimented with ­different interventions. With each iteration, approaches that were ineffective were revealed and the underlying assumptions of the implementers had to be ­ modified. With those assumptions shifting, the problem to be solved was refined, leading in turn to adaptation in the intervention methods; this iterative approach led eventually to implementers identifying the most ­ ­effective approaches to reliably providing people with clean water. In another case, in Zambia, beneficiaries of a bednet distribution program were divided into random groups and assigned an interval time for caseworkers to check in and verify that the beneficiaries had hung the nets. This approach led to implementers being able to quickly identify the optimal timing for check-ins. Operationalizing the Science of Delivery Agenda 23 Charting a New Path: The Global Delivery Initiative As noted in the introduction, the World Bank is trying to accelerate and deepen development impact in the lives of the poor. It is doing so in a world in which countries increasingly face complex multisectoral challenges, such as climate change, gender equality, job creation, and social inclusion. These challenges require bold, cross-sectoral, multistakeholder solutions and ­cannot be solved with a “business as usual” approach. For example, a recent paper on  the post-2015 country development diagnostics, which applied the framework to a pilot study of Uganda, stated unequivocally that “business as ­ usual clearly is insufficient to achieve the global SDG ambitions.”90 Countries need comprehensive development solutions. Effective interventions require quick and agile responses to problems that emerge during implementation. The ­ development community needs to adapt by better understanding and  responding to contextual dynamics throughout the implementation cycle. Because this delivery know-how is institutionally and geographically dispersed, partnership with other organizations is vitally important for enhanced learning and better delivery. The traditional development paradigm captures and produces knowledge when it is too late to apply lessons to live projects. We need ways to develop and challenge our hypotheses while we execute, to be courageous enough to question our assumptions. We need to encourage course-correction during implementation when necessary, without stigmatizing or discouraging the admission that an intervention is not working. We need a much more robust learning-by-doing approach to solve difficult challenges with (rather than for) clients and want to help make the evidence-based and adaptive delivery approach business as usual across the WBG. In April 2015, the Global Delivery Initiative was launched. The GDI is a joint effort supported by multiple partners to create a collective and cumula- tive evidence base of delivery know-how to arm practitioners to deliver inter- ventions that result in transformational change. The GDI supports the science of delivery by building on the experience of its partners; connecting perspec- tives, people, and organizations across sectors and regions; and ensuring that staff and clients have the knowledge they need for effective implementation. Leveraging delivery know-how and expertise could help to support the effective implementation of the SDGs by bringing partners together. The GDI ­ seeks to accomplish this overarching mission through a variety of modalities such as: producing delivery case studies; creating a Global Delivery Library; convening partners to facilitate sharing of experiences and lessons learned on delivery; providing support to practitioners in member organizations as needed; training prospective case writers; and identifying common delivery challenges to provide support to practitioners. The GDI could achieve transformational impacts in a more consistent manner by working in partnership to integrate technical expertise with on-the-ground delivery know-how. Systematically identifying, classifying, understanding, and solving the most common delivery challenges provides practitioners with additional means to achieve scale up interventions and 24 Operationalizing the Science of Delivery Agenda Box 1  The Global Delivery Initiative in Action The examples below illustrate the impact that GDI is already having in the WBG; collaboration through substantive cases has been used to change perspectives, analyze delivery challenges, and improve results, as follows: 1. In close collaboration with the Nigeria Country Management Unit the Science of Delivery team developed three case studies that informed the next phases of the projects and helped to improve the quality of operations for the whole country team. 2. In Mexico, a case on delivering services to indigenous populations through the certification of bilingual staff was key to engaging the government to overcome its reluctance to renew the project and to ensuring project continuity. The case was an important instrument to encour- age further discussions; the project is now being scaled up. reach the SDGs. Rather than ad hoc learning from individual experiences, the GDI offers an opportunity to more systematically deploy insights on imple- mentation and delivery know-how to benefit teams across the World Bank, as well as the WBG as a whole. The GDI could add value to the Bank by offering a platform for partners to connect and share experiences and expertise. The GDI seeks to enable Bank staff to be more effective and to build a better understanding of delivery among the Bank’s collaborators. These connections create a powerful multiplier effect by giving all practitioners—not just Bank staff—access to practical know-how, helping practitioners to make more informed decisions, and providing practical support to their important work. Through a strong collaboration with the Middle East and North Africa (MENA) region, four cases are helping to raise the profile of delivery c ­ hallenges in the health and education sectors. Conclusion Using the broad science of delivery principles outlined above, a delivery focus offers a crucial opportunity for the WBG to amplify its impact and accelerate progress toward the twin goals of ending extreme poverty and boosting shared prosperity. Delivery has often been the missing bridge from “strong, coherent development policies and programs” to results for citizens.91 By sys- tematically collecting the tacit knowledge of experienced practitioners and using this systematic approach to draw the lessons that will enable us to deliver better outcomes to the poor, the World Bank can multiply the impact of these expert problem solvers, both inside and outside the Bank.92 This process does not mean taking a linear, step-by-step approach to bring a ­ particular solution to a defined problem. Rather, this approach represents a ­ way of thinking and asking the right questions about development. This method involves being attuned to local processes and landscapes, including locally-driven changes that we should nurture and enable to effect change. Operationalizing the Science of Delivery Agenda 25 The Bank is well-placed to lead this effort, but not by itself. A joint effort supported by multiple partners to create a collective and cumulative evidence base of delivery knowledge can help practitioners make more informed deci- sions and produce consistent results on the ground. The GDI supports the science of delivery for doing development differently, by building on the experience of its partners, connecting perspectives, people, and organizations across sectors and regions. With decades of implementation knowledge, the Bank is a hub for experienced practitioners. These practitioners have the accumulated delivery know-how that needs to be collected, systematized into an understanding of what works and what doesn’t, and brought to bear on stubborn delivery problems. The Bank also plays a distinctive role as a trusted interlocutor and partner for many actors around the world. One of the key points made by numerous scholars and practitioners is the importance of convening and connecting actors to develop solutions and generate buy-in for these solutions. Moreover, because important knowledge lies outside the Bank’s walls as well as within its staff network, this convening power matters for generating and collecting knowledge. With the Bank’s status as a widely-trusted partner, it is well-­ positioned to play this role. A decade beyond the WDR 2004, much work remains to be done. Many of the insights contained in the report, such as the focus on poor people participating to incentivize service providers’ behavior, are still very much ­ applicable today, including the following prescient passage: Innovating with service delivery arrangements will not be enough. Societies should learn from their innovations by systematically evaluating and dissem- inating information about what works and what doesn’t. Only then can the innovations be scaled up to improve the lives of poor people around the world. The recent MENA report on understanding subnational variation in ­service delivery quality as a basis for guiding improvement demonstrates the utility of such an approach.93 The science of delivery will take time to create and will never truly be com- plete94 since development challenges constantly evolve, settings become more diverse, and tasks to be undertaken become complex and contentious. Still, an effective science of delivery approach can be outlined. Following Bank President Jim Kim, it “would ensure that all schools learn from the schools where children learn best. It would mean that people don’t just receive health- care but actually become healthier, while high-performing hospitals and clin- ics serve as models and resources for practitioners elsewhere.”95 This science (and the “art” within it) would cultivate vibrant communities, including those facilitated by the GDI among “implementers across countries and regions, in all development sectors. These communities would enable joint problem solving and would continuously link local action to global evidence.”96 ­ There is still a need to construct a systematic body of knowledge about how to do adaptive implementation, what works and what doesn’t; to work rigor- ously to understand the complex nonlinear dynamics that characterize the development landscape; to disseminate this information; and to use this 26 Operationalizing the Science of Delivery Agenda understanding to bring about delivery solutions. This agenda points the way to achieve a greater impact for the Bank; indeed, by getting delivery right, the Bank may be able to catalyze an impact as great as any in its history. Notes 1. See http://www.un.org/millenniumgoals/pdf/Goal_4_fs.pdf. 2. See http://www.worldbank.org/en/topic/sanitation. 3. World Bank (2013), 42. 4. Ramalingam (2013), 12, citing the Global Monitoring Report 2011. 5. Bruns, Filmer, and Patrinos (2011). Only 22 percent of their peers from OECD countries score at this level. See also Pritchett (2013). 6. According to the CIA Factbook, 75 percent of these people are concentrated in 10 developing countries: Bangladesh, Brazil, China, the Democratic Republic of the Congo, Egypt, Ethiopia, India, Indonesia, Nigeria, and Pakistan. 7. To more effectively respond to these challenges, a wide range of development prac- titioners and scholars have recently sought to articulate a set of core principles for “doing development differently” (see http://doingdevelopmentdifferently.com/). 8. The basic distinction between technical and adaptive problems comes from Heifetz (1994); a more detailed set of distinctions is outlined in Pritchett and Woolcock (2004) and Pritchett, Woolcock, and Andrews (2013). 9. To use a different metaphor, consider baking a cake. To bake a cake, it is abso- lutely essential to have the correct type and amounts of high-quality ingredients. Assembling all of these ingredients at a given time, place, and cost is mostly a logistical exercise and can be achieved by faithfully following the recipe. As we all know from experience, however, it is another thing entirely to know how to com- bine the ingredients; no matter how diligently novice and experienced chefs fol- low the recipe’s instructions, the latter’s cake is likely to be vastly superior (because they are deploying all sorts of skills that can’t readily be codified—for example, they know when and how to make mid-course corrections or improvements). In development, the core implementation problem is that we think most of our “bad baking” challenges stem from assembling inadequate or low-quality ingredi- ents or from a flawed recipe (such as a policy) when, increasingly, the challenge is one of nurturing entire systems of excellent chefs. 10. Kim (2013a). 11. Please note that the authors use the following definitions of development and delivery challenges. Development challenges are defined as the pressing issues that countries face, which, if addressed at scale, would significantly contribute to end- ing poverty and enhancing the quality of life of their citizens. Delivery challenges are defined as the nontechnical problems that hinder development interventions and that prevent countries and staff from translating technical solutions into results on the ground. 12. World Bank, “Post-2015 Development Agenda,” http://www.worldbank.org​ /­mdgs/post2015.html. 13. This was a central conclusion that IEG reached in its recent major review of the World Bank as a “learning organization”. 14. Adapted from “Management Note: Monitoring and Evaluation Framework for Implementation of the Strategy and Change Process,” World Bank (2014). 15. Jim Y. Kim, “Remarks,” April 10, 2014, “Making Innovation Policy Work.” 16. See, for example, Ramalingam (2013). 17. An additional problematic tendency is the instinct to see a project’s end goal as effectively lying within the realm of project design, rather than project implemen- tation (see Woolcock [2013]). Operationalizing the Science of Delivery Agenda 27 8. Spradlin (2012). 1 19. Andrews, Pritchett and Woolcock (2013). 20. Woolcock (February 18, 2014). See also Ramalingam (2013), on what he refers to as “best-practicitis”; PAL-Nigeria Concept Note (2014), which documents current evolution, including within the Bank itself, away from this syndrome and toward “best-fit” approaches. 21. Kim (2013b). See also Gigler et al. (2014). 22. Datta and Mullainathan (2012), 3. 23. Ramalingam (2013). 24. Mohan and Dinneen (2013), 1. 25. Ryan and Ali (2013), 12. 26. World Bank (2014), 193. 27. Page (2007). 28. Argyris (1990); Kanter, Stein, and Jick (1992); Kotter (1995); Rondinelli (1993). 29. Andrews (2008); Andrews, McConnell, and Wescott (2010). 30. Gonzalez de Asis (2012). 31. Ibid. 32. This paragraph and the following description of ALP objectives and techniques draws heavily on Gonzalez de Asis (2012). 33. On this point, see also World Bank (2014). 34. Daly and Singham (2013); see also Gittell (2003) on measuring teamwork. 35. Woolcock (2009). 36. Ramalingan (2013), 27. 37. Mohan and Dinneen (2013); Kohli (2013) 38. Pritchett (2014). 39. Woolcock (2009). 40. Pritchett, Samji, and Hammer (2013). 41. Woolcock (2009). 42. Mulley (2014), remarks given at a Scaling-up Water Supply and Sanitation Delivery conference; Ramalingam (2013). 43. Mulley (2014). 44. Keefer, P., et al. (2012). 45. Jim Y. Kim, Remarks, JPAL@Ten (December 7, 2013). 46. Epstein and Klerman (2012), cited in Samji (2014), which also provides a useful short discussion of these ideas and how they connect other work on learning and adapting. 47. Pritchett, Samji, and Hammer (2013). 48. Kim (2013b). 49. Gigler et al. (2014), 3–6. 50. See Woolcock (2009), on the importance of applying theory to evidence to predict or interpret results. 51. See Soto (2010). 52. Lanthorn and Chattopadhyay (2014a). 53. Lanthorn and Chattopadhyay (2014b). 54. Ramalingam (2013), p. 10. 55. Jim Y. Kim, “Remarks,” JPAL@Ten (December 7, 2013). 56. Kohli (2013). 57. Pate (2013). 58. See Meier (2013), which describes the use of large datasets to build better ­crisis-mapping software. 59. Mohan and Dinneen (2013). 60. Pritchett, Samji, and Hammer (2013), 35. 61. Martin and Rosas (2014a). 62. Martin and Rosas (2014a); Martin and Rosas (2014b); this material was also discussed during a lunch at the World Bank on (May 8, 2014). ­ 28 Operationalizing the Science of Delivery Agenda 3. J. E. Campos, Foreword to Andrews, McConnell, and Wescott (2010). 6 64. Andrews, McConnell, and Wescott (2010), 4. 65. Leftwich (2010). See also www.buildingstatecapacity.com, (January 29, 2014), in which Matt Andrews argues, on the basis of his research, that “leadership is about multi-agent groups and single-agent autocrats.” 66. Leftwich (2010). 67. NESTA (2014). 68. Andrews, Pritchett, and Woolcock, (2013). 69. See, for example, WDR 2014 background papers; Commins, (2007); Wild and Foresti, (2013); Duflo, Dupas, and Kremer (2012). 70. See Devarajan (2013); Ramalingam (2013); Brixi, Lust and Woolcock (2015). 71. Independent Evaluation Group (2013), xxviii. 72. See Devarajan (2013); WDR 2014. 73. Mohan and Dineen (2013). 74. Datta and Mullainathan (2012). 75. Pritchett, Samji, and Hammer (2013), 36; see also Schon (1983). 76. Pritchett, Samji, and Hammer (2013), p. 1. 77. Ibid, abstract. 78. Schon (1983), 31. 79. PAL-Nigeria Concept Note (2014). 80. See Kim (2013b); Gigler et al. (2014). 81. World Bank (2005), 1. 82. Ibid, 15–16. 83. Ibid, 1. 84. Ibid, 37. 85. Ibid, 37. 86. Ibid, 39. 87. World Bank (2012), 9. 88. Ibid, 10. 89. World Bank (2014), 193. 90. Gable, Lofgren, and Osorio-Rodarte (2015). 91. Kim (2012). 92. Ibid. 93. See Brixi, Lust and Woolcock (2015). 94. Kim (2013a). 95. Ibid. 96. Ibid. References Andrews, M. 2008. “Effective Political Engagement.” In Governance Reform Under Real World Conditions, ed. Sina Odugbemi and Thomas Jacobson. Washington, DC: World Bank. Andrews, M., J. McConnell, and C. Wescott. 2010. Development as Leadership-led Change. Washington, DC: World Bank. Andrews, M., L. Pritchett, and M. Woolcock. 2013. “Escaping Capability Traps through Problem-Driven Iterative Adaptation (PDIA).” World Development 51 (11): 234–44. Argyris, C. 1990. Overcoming Organizational Defenses: Facilitating Organizational Learning. Boston: Allyn and Bacon. Operationalizing the Science of Delivery Agenda 29 Barber, M. 2015. How to Run a Government (So That Citizens Benefit and Taxpayers Don’t Go Crazy). London: Penguin. Brixi, H., E. Lust, and M. Woolcock. 2015. Trust, Voice and Incentives: Learning from Local Success Stories in Service Delivery in the Middle East and North Africa. Washington, DC: World Bank. Bruns, B., D. Filmer, and H. Patrinos. 2011. Making Schools Work: New Evidence on Accountability Reforms. Washington, DC: World Bank. Datta, S., and S. Mullainathan. 2012. Behavioral Design: A New Approach to Development Policy. Washington, DC: Center for Global Development. Duflo, E., P. Dupas, and M. Kremer. 2012. “School Governance, Teacher Incentives, and Pupil-Teacher Ratios: Experimental Evidence from Kenyan Primary Schools.” NBER Working Paper 17939, National Bureau for Economic Research, Cambridge, MA. Epstein, D., and Klerman, J. A. “When is a Program Ready for Rigorous Impact Evaluation? The Role of a Falsifiable Logic Model.” Evaluation Review 36 (5): 375–401. Gable, S., H. Lofgren, and I. Osorio-Rodarte. 2015. “Country Development Diagnostics Post-2015.” http://pubdocs.worldbank.org/pubdocs/publicdoc/2015​ /1/­970021422471798633/country-development-diagnostics-post-2015.pdf. Gigler, B., Custer, S., Bailur, S., Dodds, E., Asad, S., and Gagieva, E. 2014. Closing the  Feedback Loop: Can Technology Amplify Citizen Voices. Washington, DC: World Bank. Gittell, J.H. 2003. The Southwest Airlines Way. McGraw-Hill Audio. Gonzalez de Asis, M. 2012. “A Practical Approach to Mobilizing Agents and Facilitating Change.” PREM Note Number 162. Washington, DC: World Bank. Heifetz, R. 1994. Leadership Without Easy Answers. Cambridge, MA: Harvard University Press. IEG (Independent Evaluation Group). 2013. “Overview: Knowledge-Based Country Programs.” Washington, DC: World Bank. International Bank for Reconstruction and Development/World Bank. 2011. Global Monitoring Report 2011. Improving the Odds of Achieving the MDGs: Heterogeneity, Gaps, and Challenges. Washington, DC: World Bank and The International Monetary Fund. Kanter, R. M., B. A. Stein, and T. D. Jick. 1992. The Challenge of Organizational Change. New York: Free Press. Kotter, J. P. 1995. “Leading Change: Why Transformation Efforts Fail.” Harvard Business Review 73 (2): 59–67. Keefer, P., G. Mansuri, V. Rao, and M. Woolcock. 2012. “How Can the Knowledge Bank Make Development More Effective?” Let’s Talk Development (blog). http://blogs.worldbank.org/developmenttalk/how-can-the​-knowledge-bank​ -make-development-more-effective. Kim, J. Y. 2012. “Delivering on Development: Harnessing Knowledge to Build Prosperity and End Poverty.” Speech at World Knowledge Forum, Seoul, Korea, October 8. 30 Operationalizing the Science of Delivery Agenda ———. 2013a. “Toward a Global Science of Delivery.” In The Art and Science of Delivery, McKinsey and Company. http://voices.mckinseyonsociety.com​ /­toward​ -a-global-science-of-delivery/. ———. 2013b. Jim Y. Kim, “Remarks, Keynote.” Speech at Citizen Voices: Global Conference on Citizen Engagement for Enhanced Development Results, (March 18, 2013). http://www.worldbank.org/en/news/speech/2013/03/18/citizen-voices​ -global-conference-on-citizen-engagement-enhanced-development-results. ———. 2013c. “Within Our Grasp: A World Free of Poverty.” Speech at Georgetown University, Washington, DC, April 2. ———. 2014. “Making Innovation Policy Work.” Speech. Washington, DC. Kohli, J. 2013. “From Data to Impact: Harnessing Performance Data to Transform Government Effectiveness.” Monitor Deloitte. Lanthorn, H., and S. Chattopadhyay. 2014a. “Have Evidence, Will… um, erm?” (1 of 2). www.hlanthorn.com. ———. 2014b. “Have Evidence, Will… um, erm?” (2 of 2). www.hlanthorn.com. Leftwich, A. 2010. “Beyond Institutions: Rethinking the Role of Leaders, Elites and Coalitions in the Institutional Formation of Developmental States and Strategies.” Forum for Development Studies 37 (1): 93–111. Martin, A., and N. Rosas. 2014a. “Using Smartphones to Improve Delivery of Safety Nets Programs in Sierra Leone.” World Bank, Washington, DC. ———. 2014b. “Improving Program Efficiency and Reducing Leakages through Community Involvement and e-Solutions in Sierra Leone.” World Bank, Washington, DC. McKinsey and Company. 2013. The Art and Science of Delivery. McKinsey and Company. http://voices.mckinseyonsociety.com/the-art-and-science-of-delivery/. Meier, P. “Crisis Maps.” In The Art and Science of Delivery, McKinsey and Company. Metz, A., L. Bartley, H. Ball, D. Wilson, S. Naoom, and P. Redmond. 2015. “Active  Implementation Frameworks for Successful Service Delivery: Catawba  County Child Wellbeing Project.” Research on Social Work Practice 25 (4): 415–422. Mohan, A., and A. Dinneen. 2013. “Deploying Behavioral Insights.” Monitor Deloitte. Page, S. 2007. The Difference: How the Power of Diversity Creates Better Groups, Firms, Schools, and Societies. Princeton, New Jersey: Princeton University Press. Pate, M. A. 2013. “A health care vision for Nigeria.” In The Art and Science of Delivery, McKinsey and Company. Pritchett, L. 2013. The Rebirth of Education: Schooling Ain’t Learning. Washington, DC: Center for Global Development. ———. 2014. “Rigorous Evidence Isn’t.” Building State Capability (website), February 20. http://buildingstatecapability.com/2014/02/20/rigorous-evidence-isnt/. Pritchett, L., S. Samji, and J. Hammer. 2013. “It’s All About MeE: Using Structured Experiential Learning to Crawl the Design Space.” Working Paper No. 322, Center for Global Development, Washington, DC. Operationalizing the Science of Delivery Agenda 31 Pritchett, L., and J. Sandefur. 2015. “Learning from Experiments When Context Matters.” American Economic Review: Papers and Proceedings 105 (5): 471–75. Pritchett, L., and M. Woolcock. 2004. “Solutions When the Solution is the Problem: Arraying the Disarray in Development.” World Development 32 (2): 191–212. Pritchett, L., M. Woolcock, and M. Andrews. 2013. “Looking Like a State: Techniques of Persistent Failure in State Capability for Implementation.” Journal of Development Studies 49 (1): 1–18. Ramalingam, B. 2013. Aid on the Edge of Chaos: Rethinking International Cooperation in a Complex World. Oxford United Kingdom: Oxford University Press. Rondinelli, D. 1993. Development Projects as Policy Experiments. New York: Routledge. Rubio Soto, G. M. 2010. “Integrating Evaluation into Decision Making: The Mexico Experience.” Presentation. http://siteresources​.worldbank​.org​/EXTHDOFFICE​ /Resources​ / 5485726-1256762343506​ / 6518748​ - 1275681412812​ / 7130285​ -1275681​441795​/9.Case_Mexico_Rubio_ENG.pdf. Ryan, K., and A. Ali. 2013. “The New Government Leader: Mobilizing Agile Public Leadership in Disruptive Times.” Deloitte University Press. Samji, S. 2014. “Rigorous evaluation, but to what end?” Building State Capability (website), March 28. http://buildingstatecapability.com/2014/03/28/rigorous​ -evaluation​-but-to-what-end/. Schon, D. 1983. The Reflective Practitioner. New York: Basic Books. Spradlin, D. 2012. “Are You Solving the Right Problem?” Harvard Business Review. http://hbr.org/2012/09/are-you-solving-the-right-problem/ar/1/. Wild, L., and M. Foresti. 2013. Working with the politics: How to improve public services for the poor. Briefing 83, Overseas Development Institute. http://www​ ­ .odi.org/sites/odi.org.uk/files/odi-assets/publications-opinion-files/8587.pdf. Woolcock, M. 2009. “Toward a Contextual Approach to Understanding Project Impact and Trajectories.” Journal of Development Effectiveness 1 (1): 1–14. ———. 2013. “Using Case Studies to Explore the External Validity of Complex Development Interventions.” Evaluation 19 (3): 229–248. World Bank. 2004. World Development Report 2004: Making Services Work for Poor People. Washington, DC: World Bank. ———. 2005. Fourth Review of Adaptable Lending. Washington, DC: World Bank. ———. 2012. “Investment Lending Reform: Modernizing and Consolidating Opera­ tional Policies and Procedures.” World Bank, Washington, DC. ———. 2013. Atlas of Global Development (Fourth Edition). Washington, DC: World Bank. ———. 2014. World Development Report 2015: Mind, Society and Behavior. Washington, DC: World Bank. 32 Operationalizing the Science of Delivery Agenda SKU K8526