IEG WO R K ING PA PE R 2016 / N O.3 Introducing a Framework for Evaluating Service Delivery in Sector Evaluations: Urban Transport, Water and Sanitation, and Nutrition IEG Working Paper 2016/No 3 Introducing a Framework for Evaluating Service Delivery in Sector Evaluations: Urban Transport, Water and Sanitation, and Nutrition Susan A. Caceres, Robert Yin, Anthony Tyrrell, Judy Hahn Gaubatz, and Sian Williams © 2016 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e- mail: pubrights@worldbank.org. ii Contents Abbreviations and Acronyms ................................................................................................................. v Acknowledgments ................................................................................................................................. vi 1. Introduction ........................................................................................................................................ 1 IEG’s Strategic Engagement Areas and Purpose of Working Paper.................................................. 1 Service Delivery: Definition .............................................................................................................. 1 Organization of Working Paper ......................................................................................................... 4 2. Methodology to Develop Theory-Based Framework ......................................................................... 4 3. Framework to Evaluate Service Delivery........................................................................................... 6 Enabling Conditions ...................................................................................................................... 8 Inputs ............................................................................................................................................. 8 Service Delivery Implementation .................................................................................................. 9 Service Outputs and Service Outcomes ....................................................................................... 11 Service Outputs............................................................................................................................ 11 Service Outcomes ........................................................................................................................ 11 Lessons Learned about Service Delivery Model ......................................................................... 12 4. Integrating Service Delivery in Sector Evaluations ......................................................................... 12 Analytic Protocol ............................................................................................................................. 15 5. Early Test of Analytic Protocol ........................................................................................................ 15 6. Conclusion and Next Steps............................................................................................................... 16 References ............................................................................................................................................ 18 Appendix A. Elements of Service Delivery from Literature to Develop Working Paper’s Definition 23 Appendix B. Additional Literature Reviewed ...................................................................................... 27 Appendix C. Developing Sector Concepts: Textbooks, Articles, and Websites Examined ................. 29 Appendix D. List of Bank Staff Interviewed........................................................................................ 32 Appendix E. List of Projects Reviewed ............................................................................................... 33 Appendix F. Analytical Protocol .......................................................................................................... 36 Appendix G. Early Version of Analytical Protocol .............................................................................. 42 Appendix H. Glossary of Terms to Assist Users of the Analytical Protocol ....................................... 47 Tables Table 1.1. Elements of Service Delivery Definition .............................................................................. 2 iii Table 2.2. Projects Reviewed................................................................................................................. 5 Figures Figure 1-1.1. Citizen Accountability as a Major Service Delivery Principle ........................................ 3 Figure 3-1 Framework for Evaluating Service Delivery........................................................................ 7 iv Abbreviations and Acronyms IEG Independent Evaluation Group IFPRI International Food Policy Research Institute OECD Organisation for Economic Co-operation and Development PPAR Project Performance Assessment Review SEAs Strategic Engagement Areas UNDP United Nations Development Programme WDR World Development Report v Acknowledgments This work is the result of the contributions of the entire team. Sian Williams reviewed studies and textbooks to gain insights into sectors and service delivery. This work helped to develop the definition of service delivery and key features. Aline Dukuze went beyond providing valuable administrative support and assisted with the search process. Judy Hahn Gaubatz reviewed project documents and took the lead in developing and refining the analytical protocol, based on valuable insights and guidance from Robert Yin. Anthony Tyrrell interviewed World Bank Group staff prior to the preparation of the concept note to gather perceptions of service delivery, which informed the scope of the framework. He also shared an early draft of the framework with a group of Bank Group staff. The feedback received during these meetings helped to sharpen the final version. Valuable feedback was provided by each of the sector evaluation teams of the Independent Evaluation Group (IEG) in refining the analytical protocol and framework. Robert Yin was more than an advisor to this work; he contributed to every facet of the work and ensured methodological rigor. The development of the framework was the collective work of the entire team. Maria MacDicken’s design capabilities improved the visual communication of concepts in these figures. While the working paper was predominantly written by Susan Caceres (task team leader), valuable inputs were provided by the entire team. Finally, feedback received from IEG’s Strategic Engagement Area champions—Midori Makino and Marie Gaarder—throughout the process helped sharpen the framework, analytical protocol, and working paper. vi 1. Introduction IEG’s Strategic Engagement Areas and Purpose of Working Paper 1.1 The Independent Evaluation group (IEG) has organized its evaluation work program (FY17–19) into three Strategic Engagement Areas (SEAs): Sustained Service Delivery for the Poor, Inclusive Growth, and Environmental Sustainability. These SEAs were implemented to generate deeper insights across evaluations for dialoguing with the World Bank Group and influencing its work. These areas cover a large share of the Bank Group’s efforts that are directed toward sustainable development outcomes. 1.2 This paper supports one SEA—Sustained Service Delivery for the Poor. The paper is one of two (the second one addresses behavior change) that will develop frameworks and tools to be employed to supplement IEG evaluations in urban transport, water and sanitation, and nutrition (IEG Work program 2016).1 The outputs of the two papers are complementary, but they are being prepared in parallel. While it is recognized that aspects of behavior change relate to service delivery, other synergies will be explored after the analytical protocols are applied across the respective sector evaluations. 1.3 This paper serves three purposes. First, it defines service delivery and its key concepts and features as well as provides examples of how the World Bank Group has supported service delivery. The paper does not attempt to describe or track all the initiatives the Bank Group is doing in service delivery. Second, the paper serves a practical purpose within IEG in developing a framework and analytical protocol to evaluate service delivery as part of three ongoing sector evaluations in urban transport, water and sanitation, and nutrition. Finally, the paper discusses how to integrate service delivery when designing and implementing other sector evaluations. 1.4 The primary audience for this paper is staff within IEG. While the content may also be beneficial to evaluators in other agencies and staff within the Bank Group, it is not targeted to address the operational needs of their staffs. Service Delivery: Definition 1.5 The Bank Group and international development actors are giving growing recognition to a focus on service delivery. Service delivery consists of a series of highly localized actions by agents in public agencies or private enterprises to provide needed goods and services to citizen beneficiaries “in a way that meets their expectations” (Kim 2012). This definition draws on literature from a variety of disciplines (i.e., economics, behavioral economics, social sciences, and engineering) to provide a broader perspective to service delivery, rather than any discipline-centric view.2 1.6 At the same time, not all of what the Bank supports should be considered service delivery. The Bank has expanded the sectors beyond those initially included within Making Services Work for the Poor (World Bank 2003a)—health, education, water, sanitation, and electricity access—to include social protection, information, transport, financial services, and credit markets. In this spirit, a service delivery perspective may not be appropriate for 1 investment climate, macroeconomic development or health financing, as those are not directly targeted at goods or services. Similarly, a service delivery perspective may not be appropriate in evaluating the Bank’s administrative interventions, such as results-based budgeting or public sector management reforms. 1.7 To understand service delivery more fully, elaboration of the initial definition is needed (see table 1.1). Service delivery has a life-cycle span (i.e., plan, design, operate, maintain, and monitor) to sustain services such as nutrition or transport or goods such as electricity or water to citizen beneficiaries (Kim 2012; Lockwood and Smits 2011). Getting (or delivering) these services or goods requires access by citizen beneficiaries, and the goods or services should meet the expectations of the beneficiaries, which implies taking into account their needs and demands (Kim 2012; Hodge 2007; Fiszbein, Ringold, and Rogers 2011). Accountability between policy makers, service managers, front-line service providers, and citizen beneficiaries is required (Caseley 2006; Fiszbein, Ringold, and Rogers 2011) (see figure 1.1). Multiple arrangements (or models) can provide the goods or services, such as centralized or decentralized government provision or contracting (Lamothe, Lamothe, and Feicok 2008; DCF 2009; Alexander and Hearld 2012); private provision; public-private partnership provision; and citizen-directed provision. All of these arrangements are context sensitive and rooted within human-made systems. (See appendix A for discussion of statements contained in the literature used to develop the definition.) Table 1.1. Elements of Service Delivery Definition Element in Definition Explanation Delivery involves a series of highly Life-cycle span localized actions by agents Context-sensitive actions Agents—policy makers, service managers, and front-line providers Delivery by public agencies or Arrangements or models: private enterprises Centralized or decentralized government provision Hybrid of centralized and decentralized government provision Central or decentral government contracting Hybrid of centralized and decentralized government contracting Private sector provision public-private partnership provision Citizen-directed provision Other innovative provision Delivery provides goods or services Delivery: getting goods and services, which requires access to citizen beneficiaries Goods: includes infrastructure such as electricity, water, roads Services—education, health, nutrition, conditional cash transfer, transportation Delivery of good or service meet Implies satisfaction of need or demand and moves beyond citizen expectation achievement of output Requires accountability in delivering the service or good 2 1.8 Citizen beneficiaries are central for service delivery, as illustrated in figure 1.1. This means citizens’ voice and understanding their needs and demands, as well as the delivery experience, are important to create services that are accountable to all citizens, especially the poor. Making Services Work for the Poor (World Bank 2003) identified accountability (across and between citizens, government, and providers) as the critical condition for services to benefit the poor. Figure 1.1 builds from the triangle relationship with features stressed in recent service delivery studies expanding the view, but also stressing accountability as the central principle of service delivery. Figure 1-1.1. Citizen Accountability as a Major Service Delivery Principle 1.9 Service delivery needs responsiveness by service providers, managers, and policy makers to be answerable (i.e., information or decision), to provide enforcement (i.e., service standards), and to make relevant organizational changes (i.e., delivery of services) (Caseley 2006; Asis and Woolcock 2015) to make services “people-centred” (UNDP 2013). The Bank Group is increasingly supporting technological development and innovation, with particular reference to integrated digital solutions to promote and support a more citizen-centric approach. The application of new technologies can, for example, enable wider, more frequent, more tailored, and less expensive information sharing, can support transactional exchange between government and citizens (and businesses), and can (through connected services) encourage and facilitate citizen input and feedback on an ongoing and real time basis. These tools are used to provide feedback about the service (or good) so that learning and adjustments can be made (Asis and Woolcock 2015). While figure 1.1 uses arrows to represent the feedback loop. This process is occurring in real time so solutions can be generated during implementation. 1.10 Contextual conditions interact with all of the agents involved in service delivery— beneficiaries, front-line providers, service managers, and policy makers (see figure 1.1). Differences in how systems are organized—whether governance structures are centralized or decentralized—internal conditions such as the availability of resources, intergovernmental transfers, consistent financial policies, and desired cost recovery impact the delivery of goods and services (Camm and Stecher 2010; Wild and Harris 2012; Hecock 2006; Cleary 2007; 3 World Bank 2003; Smets 2015). Likewise, cultural conditions such as religious, linguistic, gender, and ethnic differences can impede beneficiaries’ willingness or capacity to participate in a service or create a barrier between them and the service. Organization of Working Paper 1.11 The remainder of this paper is organized into five chapters, including the methodology used to develop a theory-based framework; service delivery concepts in the framework as described through examples from literature and project documents; integration of service delivery in sector evaluations; and early testing of the working paper’s analytical protocol. Finally, the paper discusses how service delivery can be integrated into IEG’s sector evaluations. The paper concludes with next steps for this work and plans within IEG’s Strategic Engagement Area of Sustained Service Delivery for the Poor. 2. Methodology to Develop Theory-Based Framework 2.1 Evaluations over the past several decades have increasingly attended to specifying a theory of change about the workings of and outcomes from a course of events such as occurring in an intervention ( Chen 1990, 2015; Chen and Rossi 1992; Yin 1992; Funnell and Rogers 2011). The resulting theory of change model (or program logic model) reflects how an evaluation team thinks change will occur (Knowlton and Phillips 2013, 9; Burke, 2014). An alternative term, results chains, appears in IEG and other development evaluations, and is frequently used in place of the term program logic models (Bamberger, Rugh, and Mabry 2012, 620.) A logic model’s sequence of inputs-activities-outputs-outcomes-impacts may be considered a hypothesized set of events and can serve both design and analytic functions (Yin 2014, 155–163). 2.2 To develop the theoretical basis for this framework (i.e., results chain), key service delivery concepts and features were identified and derived from: (i) project documents; (ii) literature; (iii) sector- specific concepts; and (iv) interviews with Bank staff across Global Practices and other Vice Presidential Units formally involved in analyzing or conceptualizing service delivery processes. 2.3 Project documents. Project documents (i.e., project appraisal documents, project completion results reports, and IEG Project Performance Assessment Reports [PPARs]) from 34 projects across the three sectors were reviewed. The sample of projects was drawn from the sector portfolio identified by each IEG evaluation team 3 and reflected a range of subsectors and regions as shown in table 2.2. (See appendix E for list of projects reviewed.) Within each project, components that supported service delivery were reviewed, while those unrelated to service delivery or outside the sector were excluded. 4 Table 2.2. Projects Reviewed Water and Nutrition Urban Transport Sanitation AFR, EAP, AFR, EAP, LAC, AFR, EAP, ECA, Regions covered LAC, SAR MNA, SAR LAC, MNA, SAR Dates of project period FY20–14 FY98–15 FY97–14 dates covered Number of projects 12 10 12 Mass transit (i.e., bus Rural and urban water and railway) roads supply Subsectors n/a Nonmotorized (i.e., Rural sanitation bicycles) Sewerage Drainage Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. 2.4 Review of service delivery literature. Literature was identified through key word searches of electronic databases inside and outside the Bank. The process emphasized selection of empirical research studies rather than purely conceptual or theoretical pieces. Thus, priority studies included Bank Group and non-Bank Group research that evaluated service delivery, particularly those within the three sectors (see appendix B). 2.5 Identification of sector-specific concepts. Textbooks and articles were reviewed (see appendix C) to gain an understanding of subsectors, key issues, and unique aspects within each sector that service delivery had to consider. 2.6 Interviews with Bank staff and Feedback from IEG sector evaluation teams and others. Key staff involved in the World Bank’s Public Services Delivery Global Solutions Group, Global Delivery Initiative as well as those working in Global Practices or evaluating service delivery were interviewed. The initial interviews informed the scope of the framework and provided an understanding of the various views within the Bank Group of service delivery. Follow-up interviews served the purpose of refining the framework and analytical protocol. (Appendix D lists the Bank staff interviewed.) Perceptions were gathered from front-line providers, service managers, and policy makers during an April 2016 IEG field PPAR mission to Senegal. 2.7 The literature and interview data were reviewed to identify key service delivery features or concepts. These concepts where embedded within the phases of a system— planning, designing, operating, maintaining, and monitoring—and put within a results chain. Project data and studies were used to guide the application of how the key feature applied to respective sectors. Existing logic models (Bua, Paina, and Kiracho 2015; Wenene, Steen, and Rutgers 2015; Mehndiratta et al. 2014; World Bank 2012; Smets 2015; Glavey et al. 2015; Caseley 2006; Rafil, Lddi, and Hasan 2012; Hodge 2007; Sridhar 2008; IFPRI 2015; Milat et al. 2012; Horton et al. 2010; Martin 2015)4 were drawn on to specify service outputs and outcomes within the framework. Likewise, insights were gleaned from studies related to 5 measuring service delivery activities (OECD 2012; AusAid 2009; DCF 2009). A single framework was developed to provide common data for cross-sectoral comparisons. 2.8 Then, the concepts in the framework were refined with more specificity for the analytical protocol. The protocol was piloted with project documents several times. (See appendix G for an early version.) After each pilot, revisions were made to the protocol to develop clearer guidance questions or pull out concepts with multiple questions. Feedback was provided by Bank and IEG sector evaluation teams and incorporated into the final version of the analytical protocol (see appendix F). 3. Framework to Evaluate Service Delivery 3.1 The framework (see figure 3.1) has two main features. It is about service delivery, but more important, it is a framework for evaluating service delivery rather than just operationalizing it. The dual features will assist and support existing and future IEG sector evaluations to address service delivery issues as part of their assessments of sector evaluations. To minimize disruptions to the flow of the sector evaluations, the service delivery issues are meant to supplement any given sector evaluation and to be embedded within its preexisting framework such as its design matrix. As well, the framework can stand alone, if the sole purpose is evaluating service delivery. 3.2 The framework, and analytical protocol, have four elements:  enabling conditions,  inputs,  service delivery implementation, and  service outputs and outcomes. The subsequent portion of the paper integrates findings from the literature and Bank Group operations to discuss the framework’s elements, with particular emphasis on their main components, which are noted in bold text in figure 3.1. 6 Figure 3-1 Framework for Evaluating Service Delivery 7 ENABLING CONDITIONS 3.3 Services operate within the broader government context that enables (or constrains) delivery. Many of these external conditions are beyond the control of a particular service activity such as the extent of patronage, corruption, or issues of political economy. Mcloughlin and Batley (2012) propose viewing service provision as a two-way process “in which services are formed by, and formative of, state-society relations and processes of state building,” and placing service provision in a broad question of control—“that is, how and by whom is control exercised over which services are delivered to whom.” 3.4 For this reason, the framework contains the assessment of enabling conditions such as political economy analysis, leadership, and development of budget, policy, regulatory, legal, capacity, or data systems (see figure 3.1). These are precursor conditions to support the actual service delivery that are outside the immediate point of delivery between the provider and the citizen beneficiary. Within the sample of project reviewed, these aspects are typically supported by the Bank Group as well as capacity development for decentralized levels of government. Political economy factors are determinants of performance, as political and incentive problems are likely to retard improvements in service delivery (Wild et al. 2012). IEG’s work at both micro and macros levels consistently highlights the importance of understanding the political economy with a view to sustainable success.5 3.5 The analytical protocol (see appendix F) includes questions to identify the type of upstream support provided (e.g., budget, regulation, capacity, and supply chain production) These questions are designed to detect the specific enabling conditions supported (or not supported) by the Bank Group. They can then be examined in relation to service delivery model implementation, as these conditions affect how services are delivered and often explain differences in service failures (Mcloughlin and Batley 2012). INPUTS 3.6 The framework (and analytical protocol) emphasizes the preparation of actions specific to the design of service delivery.6 Some of the elements (i.e., identification of beneficiaries, needs analysis, and establishment of mechanisms for accountability) are meant to detect whether service delivery is sensitive to the needs and demands of citizen beneficiaries, which is consistent with Bank Group policies. The Bank Group strategy (World Bank 2013b) identifies citizen beneficiaries as key agents in the realization of the twin goals of reducing poverty and boosting shared prosperity. The strategy urges respective Bank Group institutions to engage more effectively with citizen beneficiaries, including the poor, to gain insights into the results valued by ordinary people and to integrate citizen voice in development programs as an accelerator for achieving results. The Bank Group social inclusion strategy (World Bank 2013a) envisages a more deeply integrated form of citizen beneficiary engagement (e.g., in policy formation, service planning and design, and monitoring) as well as an assessment of how well the more immediate arrangements are part of the wider political and economic systems with a view to ensuring sustainability. Systems of Cities: Harnessing Urbanization for Growth and Poverty Alleviation (World Bank 2009) urges increased emphasis on demand-side governance, including civil society participation in budgeting and investment planning as well as increasing the voice of citizens. 8 3.7 Serving poor citizen beneficiaries is a core concern for the Bank Group. However, the poor are not a homogenous group. All of the other demographic attributes (e.g., gender, age, disability, and ethnicity) are also present among the poor. The diversity also requires efforts to understand citizens’ demands (Wild et al. 2015) and to gather and respond to beneficiary feedback. Understanding their desires and constraints to ensure accessibility and affordability heightens the likely utilization and derived benefit and sustainability of the service. Simply collecting feedback is not enough. The complete delivery system needs to include active involvement of citizen beneficiaries for application of the feedback received. Recognition of the heterogeneity of citizen beneficiaries, including the extent and severity of their poverty, strongly suggests the need to collect disaggregated data (AusAid 2009), yet IEG evaluations typically find this feature missing within project monitoring data (IEG 2012). For this reason, the protocol examines whether disaggregated data are provided to document the presence or deficiency of these types of data. 3.8 Needs analysis also applies to the needs of front-line service providers and managers as well as the assessment of the existing service delivery model. Was the capacity of front- line providers and managers considered during planning and design of service delivery arrangements? Was there an examination of the existing service delivery model with a view to determine which entity could deliver the service efficiently, including the optimal role for government, and at which level. Yet, the existing modality is what is typically utilized, particularly when the service requires highly specific assets (i.e., associated with services such as electricity or water) that present a barrier to entry (Lamothe, Lamothe, and Feiock 2008). Consistent with the finding from this study, most appraisal documents reviewed did not provide a reason for selecting a specific model and selected the existing one. 3.9 Other elements in the framework examine what has been done to optimize service delivery arrangements—end-to-end implementation planning, establishing service standards, designing both operating and maintaining aspects, and establishing mechanisms for monitoring and accountability. 3.10 Contextual conditions need to be an explicit part of the design in delivery systems. Identifying key contextual conditions that are likely to impinge directly on service delivery (such as inadvertent gender differences between service providers and service beneficiaries) will require explicit attention during planning and design to put in place appropriate service delivery arrangements to meet the multifaceted needs of heterogeneous citizen beneficiaries- including the poor. The analytical protocol includes questions to document the contextual conditions that were explicitly considered. SERVICE DELIVERY IMPLEMENTATION 3.11 The model of delivering services is a central part of implementation in the Framework. While Making Services Work for the Poor (World Bank 2003) advanced the concept of “eight sizes fits all,” this framework builds on that foundation established but contains others found in the literature and supported in Bank Group projects (i.e., central provision or contracting, decentral provision or contracting, hybrid between central and decentral provision or contracting, private-public contract, private sector, citizen-directed 9 provision such as voucher or community driven development), as well as leaves the possibility of advancing other innovative provisions. 3.12 Any model can deliver water and sanitation services (or urban transport or nutrition), but how effectively or sustainably it does depends largely on contextual and enabling conditions (as well as accountability relationships). For example, political will was lacking to implement the tariff structure and key design elements within the Lebanon Ba’albeck Water and Wastewater Project, thus the water utility was not financially sustainable. On the other hand, the presence of other enabling conditions (such as leadership, behavior change, and evidence) supported the success of the community-led sanitation program where 25 million rural people in Indonesia were provided access to improved sanitation (Glavey et al. 2015). “Every environment has its own distinct social and political characteristics, and solutions have to be either drawn from, or adapted to them” (Brixi, Lust, and Woolcock 2015, vi). For this reason, the protocol examines whether the distinct social and political conditions were understood when designing and implementing the model. 3.13 Cost recovery and subsidies are major concerns in water and transport. As well, in cases where services are provided to small populations or remote from other points of service delivery, an understanding of the additional costs to operate and maintain the service is needed (Kumara 2013). The analytical protocol contains questions related to these aspects. 3.14 Accountability must receive attention during implementation. The Bank Group’s discourse has expanded to take on broad connections among the modality of delivery, the front-line service providers and managers, and the broader governance system, all with a view to ensuring citizen-focused, citizen-informed, sustainable services. Accountability relationships are complex affairs (Caseley 2006) between citizen beneficiaries, front-line providers, service managers, and policy makers (see figure 1.1). The interdependent relationships have been critical to sustained improvements with respect to new connection of services and resolution of complaints (Caseley 2006). Systematic citizen engagement (Wild et al. 2015; Asis and Woolcock 2015; Brinkerhoff and Wetterburg 2013), as opposed to excessive or erratic engagement (Caseley 2006) through feedback loops with all agents, is needed (Wild et al. 2015; Milat, Bauman, and Redman 2015; Pérez-Escamilla et al. 2012). 3.15 Accountability affects how front-line providers and mangers behave and how citizens experience quality and efficiency of services and goods (Fiszbein, Ringold, and Rogers 2011) and how the model is implemented. Thus, the protocol contains questions to determine the presence of accountability mechanisms—such as report cards, complaint resolution, or other ways—and at which phase of service delivery and what parties are involved, as well as use of a feedback loop as knowledge and evidence to adapt implementation are required (Kim 2012; Asis and Woolcock 2015). All of the previously noted characteristics can be examined in relation to enabling conditions to provide a better understanding of how the model worked in that particular context. 10 SERVICE OUTPUTS AND SERVICE OUTCOMES 3.16 The final section of the framework contains service outputs and outcomes. They can be viewed in relation to: (i) the service delivery activity, and (ii) the use of the service (Fiszbein, Ringold, and Rogers 2011). SERVICE OUTPUTS 3.17 The points of contact between the service provider, manager, and citizen beneficiary may be experienced differently across different sectors, and in particular the three IEG sectors of Urban Transportation, Water and Sanitation, and Nutrition. Consider for example nutrition, where a citizen’s interactions with a community nutrition worker (front-line service provider) is individualized and nuanced. On the other hand, contacts with front-line providers in water or transport are more limited. Despite these differences, it is important to understand how citizens experience the delivery of the goods or services. This suggests the need for evaluating process conditions related to delivery (Asis and Woolcock 2015) such as provider performance and the presence of service monitoring or quality control. 3.18 Examples of service monitoring in reviewed projects include, the Colombia Integrated Mass Transit Systems, which finances user surveys to assess whether mass transit users rate the new rapid bus system better than the traditional public transportation system. Lima Urban Transport develops user scorecards to measure public transport performance. The Ghana Small Towns Water Supply and Sanitation Project brings beneficiaries and implementers (from each town) together on a regular basis to evaluate effectiveness and sustainability of activities undertaken as well as to provide feedback for improving processes. SERVICE OUTCOMES 3.19 Understanding the delivery experience in relation to the quality of the service suggests the need for benchmarks for quality (i.e., efficacy in delivering nutrition, regular and clean water supply, and reliable and safe transport) (Amin, Das, and Goldstein 2008; Kim 2012; Fiszbein, Ringold, and Rogers 2011; Asis and Woolcock 2015; World Bank 2012; Glavey et al. 2015; Milat et al. 2012) Measures of beneficiary satisfaction should be interpreted cautiously, since beneficiaries’ expectations may be low (and thus report high level of satisfaction) and beneficiaries may not have the information providers have (Fiszbein, Ringold, and Rogers 2011). 3.20 Likewise, an important aspect of service delivery is a focus on outcomes (World Bank 2003; Brinkerhoff and Wetterburg 2013)—what is achieved from using the goods or services. Simply accessing nutrition services is not enough. The service must be delivered in a way that helps to make pregnant and lactating mothers and children healthier. This requires movement from measurement of outputs to an accountability for outcomes (Amin, Das, and Goldstein 2008). For this reason, the framework and the analytical protocol include general measures related to reliability, utilization, sustainability, and sector-specific beneficiary outcomes. It is understood that not all Bank Group projects collect outcome data for a variety of reasons. Implementation time frame as well as existing monitoring and evaluation capacity may constrain these aspects. 11 LESSONS LEARNED ABOUT SERVICE DELIVERY MODEL 3.38 Technical or logistical solutions have not been sufficient to address delivery challenges (Kim 2012; Asis and Woolcock 2015). Knowledge and learning are also required. For this reason, the framework connects implementation and service delivery outputs and outcomes through a feedback loop to develop lessons learned about the service delivery model. Asis and Woolcock (2015) suggest the need for “adaptive implementation with a process of developing solutions, allowing teams to adjust their methods in response to the feedback generated in the delivery process.” In other words, understanding whether the model is achieving the expected results and understanding whether the model is meeting citizen’s expectations (both questions that are part of the analytical protocol). The model should be re-examined to know whether the arrangement is “solving concrete problems and developing solutions” (Asis and Woolcock 2015). 3.39 Across the span of three nutrition projects in Senegal, changes in the service delivery model and responsibilities (across level of governments and nongovernmental organizations) occurred initially to be consistent with the government’s decentralization efforts, and from the broader focus of good governance at each level. According to a mission to Senegal, Subsequent changes occurred through a process of learning of lessons in nutrition services because a learning culture was established, and there was accountability for the collection and use of data. 3.40 The complete framework (through the guidance questions in the analytical protocol) will provide deeper findings related to planning and implementation than typically collected within IEG evaluations. The evidence resulting from framework relate to: (i) citizen beneficiaries (i.e., assessment of who is being served and how they are (or not) engaging within the cycle of service delivery); (ii) contextual conditions (e.g., which conditions receive explicit attention during planning); (iii) feedback loops to improve implementation (e.g., whether feedback is collected, how it is used; and whether there are adaptations to service delivery); and (iv) service delivery models and outcomes (to provide a deeper understanding of how the model worked in that particular context to add to our understanding of how to deliver goods and services). 4. Integrating Service Delivery in Sector Evaluations 4.1 The prevailing literature on designing and doing program evaluations suggests that the framework for evaluating service delivery can be helpful at four conceptual stages within an IEG evaluation: (i) in dealing with its evaluation questions; (ii) in (further) defining the criteria for assessing the relevance, effectiveness, and efficiency of the Bank Group’s efforts; (iii) in specifying and collecting the data about the events in a results chain; and (iv) in addressing contextual conditions (Scriven 1967, 1991; Mark, Henry, and Julnes 2000; Rossi, Lipsey, and Freeman 2003; Stufflebeam and Shinkfiled. 2007; Patton 2008; 2015; Stufflebeam and Coryn 2014; Newcomer, Hatry, and Wholey 2015). Each of these activities draws on all aspects of the framework and analytical protocol for evaluating service delivery and correspond with the conceptual stages of evaluation so that this can be embedded within a sector evaluation. 12 4.2 Evaluation Questions. First, all evaluations typically start with a set of evaluation questions. Commonly, these questions deal with how and why interventions took place the way they did and with what results (e.g., Markiewicz and Patrick 2016, 93–119). At a minimum, an initial service delivery question would ask about the extent to which the Bank Group portfolio being evaluated included service delivery projects because many Bank Group interventions may strengthen and build the capacity of institutions rather than support services to citizen beneficiaries. 4.3 In relation to service delivery, the evaluation questions could then seek to define the beneficiaries of the services being provided, with such illustrative sub-questions as (Pawson 2006): Who benefits from the water or bus system? How do they benefit? Why? Who does not benefit and why? Such questions would recognize the criticality for the service delivery projects in identifying and understanding the needs and demands of citizen beneficiaries. Table 4.1 illustrates how service delivery concepts can be included within the evaluation question. Table 4.1. Reframing Sector Evaluation Questions to Include Service Delivery Original Evaluation Question Evaluation Question to Include Service Delivery To what extent has the World Bank To what extent has the Bank Group supported client Group supported client countries for countries for provision of affordable water and provision of affordable water and sanitation services through: reflecting consumer sanitation services through: reflecting demand and willingness to pay; targeted subsidies; consumer demand and willingness to and the extent of any remaining underserved service pay; targeted subsidies? users? To what extent has the Bank Group supported client To what extent has the Bank Group countries to ensure focus on vulnerable groups supported client countries to ensure including the poor, women, and children, focus on vulnerable groups including particularly girls, including participation by them or the poor, women, and children, their representatives in the design and particularly girls? implementation of water and sanitation services? 4.4 Criteria for Relevance, Efficiency, and Effectiveness. The framework and questions in the analytical protocol align with the core evaluation criteria—relevance, effectiveness, and efficiency. Typically, IEG evaluations assess relevance in relation to Bank Group operations and government’s plans, policies, and strategies. One central question to consider is what is the comparative advantage of the Bank Group-in terms of efficiency of delivery; scale and resources, capacities and expertise, presence in the country, taking into account the landscape of institutions supporting the service in the country? Given the centrality of citizen beneficiaries to service delivery, relevance must also validate demand and push a deeper level of analysis to establish if, for example, in designing and introducing a service, citizen beneficiaries (including the poor) have been consulted and their needs analyzed. And, if relevant preferences expressed—based, for example, on identity (e.g., ethnic, gender), or other concerns—are taken into account in service design and implementation. In other words, 13 the relevance criterion can be adapted by asking: What efforts have been made to tailor, or otherwise fit, service design and roll-out to the expressed need or demand of citizens (including the poor)?7 As one example, the design of a new bus system did appropriately seek to locate bus stops in neighborhoods dominated by poorer residents to serve them better. However, the residents were later found not to make good use of the completed bus system because they needed to reach the locations of their daily jobs. In retrospect, the designers had not queried the residents about their transportation patterns, and the bus routes did not end at the needed feeder connections (IEG 2015). 4.5 Having established relevance and built relevant contingencies and preferences into planned design and roll-out, the next step is to establish the optimal role for government (if any) in delivering the service. Whatever the role of government, the guiding principle with reference to the efficiency criterion is, clearly, the identification of the most productive arrangement for delivery. The issue of cost-recovery also needs to be taken into account, as relevant, under the efficiency criterion. Cost-recovery may also have a profound effect on longer-term sustainability. 4.6 The effectiveness criterion also can be augmented by a service delivery perspective. Besides covering the conventional effectiveness measures, evaluations can collect better data about the beneficiaries being served (especially their economic profiles) as well as being more sensitive to estimating the beneficiaries that might still be underserved, or assessing service utilization trends. 4.7 Results Chains. Although IEG sector evaluations will have developed their own results chain as part of their evaluation framework, it can be enhanced to include service delivery. For example, the results chains, besides needing to cover issues related to service beneficiaries, also may need to clarify the presumed course of events with regard to service providers, covering both the role of the specific service delivery organizations charged with implementing the service being evaluated as well as the front-line service providers participating in the delivery system. 4.8 Contextual conditions. In principle, a complete program logic model should not only be directed at endogenous conditions, but also at exogenous conditions, such as contextual conditions. The contextual conditions can be far ranging, including political, economic, institutional, historical, and cultural matters (Bamberger, Rugh, and Mabry 2012, 23, 28–29; Patton 2015, 9). In practice, while most evaluations acknowledge the importance of contextual conditions, the evaluations do not specify them in any great detail. Such a practice occurs mainly because too many contextual conditions appear to be potentially relevant, and no single evaluation will be able to cover all these conditions. Nevertheless, some explicit attention to contextual conditions should be part of an IEG sector evaluation. 4.9 The overabundance of contextual conditions may inhibit their inclusion in evaluations, despite the near-universal recognition that “context matters” ( Stufflebeam and Shinkfield 2007, 57–80; Patton 2015, 9). One approach for counteracting the multitude of contextual conditions is to adopt a proximal-distal perspective and try to focus on the most proximal contextual conditions first. For instance, evaluations should cover the gender and other potential socioeconomic differences between service beneficiaries and service 14 providers. Other proximal conditions could account for cultural or religious beliefs affecting beneficiaries’ use or perception of the services being delivered. Analytic Protocol 4.10 The previously described entry points into the evaluation represents a departure from previous IEG sector evaluations as the three upcoming sector evaluations will embed a service delivery perspective within their work. This will require service delivery issues contained in the analytical protocol to be embedded within the data collection for the country case studies and project portfolios. The data generated will permit addressing the following questions:  What service delivery features lead to sustainable services? How does this vary across context?  Has the service delivery being evaluated been targeted to reach specific groups of users? If so, how?  How was evidence used (i.e., identification of problems and solutions or implementation understanding and refinement) to ensure that the Bank Group’s service delivery initiatives focused on outcomes for citizens?  How does the Bank Group promote and support service delivery initiatives? What are potential points of entry for the Bank Group? 5. Early Test of Analytic Protocol 5.1 The early testing of this working paper’s analytical protocol8 has revealed data gaps in assessing service delivery. Aspects, such as assessment of the needs and demands of citizen beneficiaries and capacity assessment of front-line providers, are not uniformly covered across the three sectors in project appraisal documents. As well, outputs typically do not extended into the domain of front-line service providers. 5.2 In the Nutrition sector, need assessments were conducted in nearly all reviewed projects (11 out of 12) by way of nutritional screening, growth monitoring, or participatory health needs assessment. While lack of demand from users is explicitly acknowledged as a barrier in half of the projects, the majority of reviewed projects included activities to generate demand, such as information campaigns. However, in the reviewed projects in the Water sector, only two conducted analyses of willingness to pay among target users, while none of the reviewed projects in included a needs assessment of particular groups within the population. 5.3 In the Nutrition and Urban Transport sectors, no projects conducted formal capacity assessment of front-line providers. Yet, in the Nutrition sector, all projects included training to service providers in project design (for example, training to community nutrition volunteers). In the Urban Transport sector, one project discussed the capacity weaknesses of the front-line providers (bus drivers) as a barrier to service delivery, but there was no capacity assessment conducted. In the Water sector, as front-line providers and managers were often government entities, capacity assessments were more frequently conducted. 15 5.4 In the Nutrition sector, most project designs (8 of 12) included “real-time” monitoring and sharing of program results with beneficiaries (regular meetings with community leaders to discuss ongoing results, scorecards). However, only one of these projects also included explicit process of assessing results and using that experience to make adjustments to the interventions (i.e., feedback loop). Similarly, in the Water and Sanitation sector, only one project included an explicit feedback loop. There were none in the Urban Transport sector. 5.5 Within the 34 projects reviewed, most do not report data on improved service quality, affordability of services, increased utilization of services, or increased efficiency. Data on access are provided more often. In the Nutrition sector, most projects (10 out of 12) reported at closing on changes in access or coverage of services. In the Urban Transport and Water and Sanitation sectors, less than half of projects provided these data. There are many explanations why projects do not collect these types of data (i.e., capacity constraints, methodological issues, or implementation time), but insights gained can be useful to promote measurement of improvements in well-being for citizen beneficiaries. - Project completion reports offer minimal analysis of the contextual conditions that impacted—negatively or positively—service delivery outcomes, suggesting that these documents will not provide answers to some of the questions in the protocol. One purpose of this exercise will be to identify gaps within the Bank’s support of service delivery with the intent of informing and enhancing future support in client countries. Overall, the tentative results from the analytic protocol highlight how past practices within the Bank Group may not align with its stated policy aspirations. 6. Conclusion and Next Steps 6.1 The analytical protocol was piloted to explore the potential for data collection from project documents in the upcoming sector evaluations. Further use of the protocol will lead to refinements in the type of data collected and highlight the key areas of analysis. In essence, the framework for evaluating service delivery and the analytical protocol should be viewed as a living enterprises, adapted with each subsequent use by IEG, rather than a one-time-only effort. For example, one planned enhancement is connecting and synergizing the service delivery work with the complementary, but parallel efforts within IEG in relation to a parallel work on evaluating behavior change, as it is recognized that aspects of behavior change relate to service delivery (e.g., citizen beneficiaries and service providers). The common framework and collection of consistent data across sectoral evaluations will eventually enable IEG to produce a chapeau report that draws on and contrasts findings across the individual sector and service evaluations. The chapeau report will describe the features of service delivery most commonly supported and compare their relative efficacy around reaching the poor, supporting quality services, building sustainability, engaging citizens, and obtaining sector outcomes. 1 In IEG’s work program (FY17–19), the nutrition evaluation has been replaced with essential health services. 16 2 For instance, in the name of service delivery, economists may be said to focus mainly on incentives, accountability mechanisms, and broader fiscal issues; behavior economics use psychological insights in human behavior understand economic decision-making; social scientists on social systems arrangements; and engineers on physical systems arrangements and procedures. 3 In the case of the nutrition portfolio, this was identified by IEG’s evaluation of the Bank’s support to early childhood development (IEG 2015). 4 These studies evaluated service delivery. 5 Sustainable is meant as the maintenance of the perceived benefit of investment projects (including convenience, time savings, livelihoods, or health improvements) following the ending of the active period of implementation (Lockwood and Smits 2011) or the extent to which the policies, institutional and regulatory framework, sector management capacity, and financial arrangements remain in place to ensure that infrastructure is operated and maintained, enabling sustained provision of infrastructure and services over the long term (IEG 2013). 6 The framework also includes the typical system inputs such as human resources, financial resources, and technology. 7 The need to validate demand at this level is itself underpinned by a principle that the citizen matters, not only from a political perspective in terms of inclusion, voice, and so on, but also from a more functional perspective based on the proposition that demonstrating an understanding of would-be consumers in the design and roll-out of a service, will result in improved outcomes. These assumptions are in line with World Bank Group strategy, the new emphasis on citizen engagement, and the thinking that underpins the science of delivery concept and ongoing activities, such as the Global Delivery Initiative, within the Bank Group. 8 The early test of the protocol was based on an earlier version of the protocol (see appendix G). 17 References Alexander, J.A., and L.R. Hearld, 2012. “Methods and Metrics Challenges of Delivery System Research.” Implementation Science 7:15. Amin, S., J. Das, and M. Goldstein, eds. 2008. Are You Being Served: New Tools for Measuring Service Delivery. Washington, DC. World Bank. Asis, M.G., and M. Woolcock. 2015. Operationalizing the Science of Delivery Agenda to Enhance Development Results. Washington, DC: World Bank. AusAID (Australian Agency for International Development). 2009. Service Delivery for the Poor: Lessons from Recent Evaluations of Australian Aid. Canberra: AusAID. Bamberger, Michael, Jim Rugh, and Linda Mabry. 2012. Real World Evaluation: Working under Budget, Time, Data, and Political Constraints. Thousand Oaks, Calif. Sage. Brinkerhoff, D.W., and A. Wetterburg. 2013. “Performance-Based Public Management Reforms: Experience and Emerging Lessons from Service Delivery Improvement in Indonesia.” International Review of Administrative Sciences 79 (3): 433–457. Brixi, H., E. Lust, and M. Woolcock. 2015. Trust, Voice, and Incentives: Learning from Local Success Stories in Service Delivery in the Middle East and North Africa . Washington, DC: World Bank Group. Bua, J ., L. Paina, and E.E. Kiracho. 2015. “Lessons Learnt during the Process of Setup and Implementation of the Voucher Scheme in Eastern Uganda: A Mixed Methods Study.” Implementation Science 10:108. Burke, W.W. 2014. Organizational Change: Theory and Practice. Thousand Oaks, Calif.: Sage. Camm, F.A., and B.M. Stecher. 2010. Analyzing the Operation of Performance-Based Accountability Systems for Public Services. Caseley, J. 2006. “Multiple Accountability Relationships and Improved Service in Delivery Performance in Hyderabad City, Southern India.” International Review of Administrative Sciences 72 (4): 531–546. Chen, Huey. 1990. Theory-Driven Evaluations. Newbury Park, Calif.: Sage. Chen, H., and P.H. Rossi, eds. 1992. Using Theory to Improve Program and Policy Evaluations. New York: Greenwood Press. Chen, Huey T. 2015. Practical Program Evaluation: Theory-Driven Evaluation and the Integrated Evaluation Perspective. Thousand Oaks, Calif.: Sage. Cleary, M. R. 2007. “Electoral Competition, Participation, and Government Responsiveness in Mexico. ” American Journal of Political Science 51 (2): 283–299. DCF (Decentralization Support Facility). 2009. Alternative Mechanisms of Service Delivery. Legal and Regulatory Framework Review. Volume 1. www.dsfindonesia.org. Fiszbein, A., D. Ringold, F. Halsey Rogers. 2011. “Making Services Work. Indicators, Assessments, and Benchmarking of the Quality and Governance of Public Service Delivery in the Human Development Sector.” Policy Research Working Paper 5690, World Bank, Washington, DC. Funnell, S.C., and P.J. Rogers. 2011. Purposeful Program Theory: Effective Use of Theories of Change and Logic Models. San Francisco: Jossey-Bass. Glavey, S. et al. 2015. How to Scale Up Rural Sanitation Service Delivery in Indonesia . Case Study. World Bank: Global Delivery Initiative. Hecock, R. D. 2006. “Electoral Competition, Globalization, and Sub-national Education Spending in Mexico.” American Journal of Political Science 50 (4): 950–961. Hodge, G. 2007. Regulatory Frameworks for Urban Services. Background Discussion Paper, OECD. http://www.oecd.org/gov/regulatory-policy/39218313.pdf. 18 Horton S., M. Shekar, C. McDonald, A. Mahal, and J.K. Brooks. 2010. Scaling Up Nutrition: What Will It Cost? Washington, DC: World Bank Group. IEG (Independent Evaluation Group). 2012. Youth Employment Programs: An Evaluation of World Bank and IFC Support. Washington, DC: World Bank. ———. 2013. Improving Institutional Capability and Financial Viability to Sustain Transport: An Evaluation of World Bank Group Support since 2002. Washington, DC: World Bank. ———. 2015. World Bank Support to Early Childhood Development. Washington, DC: World Bank. ———. 2016. “Proposed Strategic Engagement Area (SEA) for Objective 1 of FY16 –18 Work Program: Sustained Service Delivery for the Poor.” In Work Program and Budget (FY16) and Indicative Plan (FY17–18). Washington, DC: World Bank. IFPRI (International Food Policy Research Institute). 2015. Global Nutrition Report 2015: Actions and Accountability to Advance Nutrition and Sustainable Development. Washington, DC: IFPRI. http://dx.doi.org/10.2499/9780896298835. Kim, J. K. 2012. “Delivering on Development: Harnessing Knowledge to Build Prosperity and End Poverty.” World Bank Group President Jim Yong Kim, Keynote Speech to World Knowledge Forum, Seoul, Republic of Korea, October 9, 2012. http://www.worldbank.org/en/news/speech/2012/10/08/delivering-development-harnessing- knowledge-build-prosperity-end-poverty. Knowlton, L.W., and C.C. Phillips. 2013. The Logic Model Guidebook: Better Strategies for Great Results. Thousand Oaks, Calif. Sage. Kumara, H.S. 2013. “Revisit the Debate on Issues of Metropolitan Governance and Service Delivery: A Trajectory of Efficient Service Delivery Model for Water Supply in Bangalore, India. ” Environment and Urbanization Asia 4 (1): 203–220. Lamothe S, M. Lamothe, and R.C. Feiock. 2008. “Examining Local Government Service Delivery Arrangements over Time.” Urban Affairs Review 44 (1): 27–56. Lockwood, H., and S. Smits. 2011. Supporting Rural Water Supply. Moving toward a Service Delivery Approach. Practical Action Publishing: Rugby, UK. Mark, Melvin M., Gary T. Henry, and George Julnes. 2000. Evaluation: An Integrated Framework for Understanding, Guiding, and Improving Policies and Programs. San Francisco: Jossey-Bass. Markiewicz, Anne, and Ian Patrick. 2016. Developing Monitoring and Evaluation Frameworks. Thousand Oaks, Calif.: Sage. Martin, L.L. 2015. Purchase of Service Contracting Versus Government Service Delivery: The Views of State Human Service Administrators. New York: Columbia University. Mcloughlin, Claire, and Richard Batley. 2012. “The Effects of Sector Characteristics on Accountability Relationships in Service Delivery (August 30, 2012).” Overseas Development Institute Working Paper 350. http://ssrn.com/abstract=2209074. Mehndiratta, S., et al. 2014. Targeted Subsidies in Public Transport: Combining Affordability with Financial Sustainability. Washington, DC: World Bank. Milat, A..J, L. King, A. Bauman, and S. Redman. 2012. “The Concept of Scalability: Increasing the Scale and Potential Adoption of Health Promotion Interventions into Policy and Practice.” Health Promot Int. 28 (3): 285–98. Milat, A.J., A. Bauman, and S. Redman. 2015. “Narrative Review of Models and Success Factors for Scaling Up Public Health Interventions.” Implementation Science 10:113. Newcomer, Kathryn E., Harry P. Hatry, and Joseph S. Wholey, eds. 2015. Handbook of Practical Program Evaluation. Hoboken, NJ: Wiley. 19 OECD (Organisation for Economic Co-operation and Development). 2012. Integrated Services and Housing Consultation. November 8–9, 2012. Social Policy Division. www.oecd.org/social/integratedservices.htm. Patton, Michael Quinn. 2008. Utilization-Focused Evaluation. Thousand Oaks, Calif: Sage. ———. 2015. Qualitative Research and Evaluation Methods. Thousand Oaks, Calif.: Sage. Pawson, Ray. 2006. Evidence-Based Policy: A Realist Perspective. London: Sage. Pérez-Escamilla, R., L. Curry, D. Minhas, L. Taylor, and E. Bradley. 2012. “Scaling Up of Breastfeeding Promotion Programs in Low- and Middle-Income Countries: The ‘Breastfeeding Gear’ Model.” Adv Nutr. 3 (6): 790–800. Rafi1, M. M., S.H. Lodi, and N.M.Hasan. 2012. “Corruption in Public Infrastructure Service and Delivery: The Karachi Case Study.” Public Works Management & Policy 17 (4): 370–387. Rossi, Peter H., MARK w. Lipsey, and Howard E. Freeman. 2003. Evaluation: A Systematic Approach. Thousand Oaks, Calif.: Sage. Scriven, Michael. 1967. “The Methodology of Evaluation.” In Perspectives of Curriculum Evaluation, AERA Monograph Series on Curriculum Evaluation, No. 1. Chicago: Rand McNally. ———. 1991. Beyond formative and summative evaluation. In M. W. McLaughlin & D.C. Phillips (Eds.), Evaluation and education: At quarter century (pp. 19-64). Chicago: University of Chicago Press. Smets, Susanna. 2015. Turning Finance into Services for the Future: A Regional Synthesis of the Service Delivery Assessments for Water Supply and Sanitation in East Asia and the Pacific . Washington, DC: World Bank Group. http://documents.worldbank.org/curated/en/2015/03/24476645/east-asia-pacific- service-delivery-assessments-water-supply-sanitation. Sridhar, D. 2008. “Hungry for Change: The World Bank in India.” South Asia Research 28 (2): 147–168. Stufflebeam, Daniel L., and Anthony J. Shinkfield. 2007. Evaluation Theory, Models, and Applications. San Francisco: Jossey-Bass. Stufflebeam, Daniel L., and Chris L. Coryn. 2014. Evaluation Theory, Models, and Applications. San Francisco: Jossey-Bass. UNDP (United Nations Development Programme). 2013. National Workshop on “Good Local Governance and People Centered Services—What Can We Learn from Best Practices in the Region? ” New York: UNDP. Wenene, M.T., T. Steen, and M.R. Rutgers. 2015. “Civil Servants’ Perspectives on the Role of citizens in public service delivery in Uganda.” International Review of Administrative Sciences 0 (0) 1–21. Wild, L., David Booth, Clare Cummings, Marta Foresti, and Joseph Wales. 2015. Adapting Development: Improving Services to the Poor. London: Overseas Development Institute. Wild, L., and D. Harris. 2012. The Political Economy of Community Scorecards in Malawi. London: Overseas Development Institute. Wild, L, M. King, V. Chambers, and D. Harris. 2012. Common Constraints and Incentive Problems in Service Delivery. London: Overseas Development Institute. Wild, L. 2015. “Doing Development Differently: Politically Smart, Adaptive Approaches to Address Governance and Accountability.” Powerpoint presentation at the World Bank, November 17, 2015, Washington, DC. http://www.thegpsa.org/sa/event/bbl-doing-development-differently-politically- smart-and-adaptive-approaches-address-governance. World Bank. 2003a. World Development Report 2004: Making Services Work for Poor People. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/5986. 20 ———. 2003b. Water Supply and Sanitation Sector: Business Strategy . Washington, DC: World Bank. http://documents.worldbank.org/curated/en/2003/07/14134265/water-supply-sanitation-sector- business-strategy ———. 2009. Systems of Cities: Harnessing Urbanization for Growth and Poverty Alleviation. Washington, DC: World Bank. ———. 2012. Linking Service Delivery Processes and Outcomes in Rural Sanitation: Findings from 56 Districts in India. Global Scaling Up Sanitation Project Report 76714, World Bank, Washington, DC. ———. 2013a. Inclusion Matters: The Foundation for Shared Prosperity. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/16195. ———. 2013b. World Bank Goals: End Extreme Poverty and Promote Shared Prosperity. Washington, DC: World Bank. Yin, Robert K. 1992. “The Role of Theory in Doing Case Study Research and Evaluations.” In Using Theory to Improve Program and Policy Evaluations, edited by H. Chen and P.H. Rossi, 97–114. New York: Greenwood Press. Yin, Robert K. 2014. Case Study Research: Design and Methods. Thousand Oaks, Calif.: Sage. 21 Appendix A. Elements of Service Delivery from Literature to Develop Working Paper’s Definition Statement from Source Implication Source “Delivery” is an elegant word for Expectations implies Kim (2012) getting goods and services to people in satisfaction of need or a way that meets their expectations. demand—going beyond output and outcome. Getting implies the achievement of coverage, reach, and access. Delivery priorities in development Encompasses services (such as Kim (2012) include material infrastructure like human development) or roads, power grids and water systems. infrastructure (such as They also include services like electricity, water, and roads). education, health care and social protection. To improve development outcomes, we need to tackle delivery challenges head-on. Access is critical to the definition of There is no delivery without OECD (2008) delivery of a service. access; if the service isn’t accessible then it isn’t delivered. The model for service delivery (2004 Access to quality services that World Bank (2003) World Bank Development Report) is improve outcomes (e.g., live entry point to understand whether the decent lives and participate in poor have access to the quantity and economic opportunities). quality of services that helps them live decent lives and participate in economic opportunities. The Service Delivery Approach (SDA) Life cycle of service in the case Lockwood and Smits is a conceptual approach taken at of water supply would consist (2011, 19) sector level emphasizing the entire lifeof both engineering or cycle of a service. construction elements and software required to provide some level of access to water. A service delivery model describes the For sustainability of the service, http://www.waterservi service to be provided under the model, consider all aspects to operate cesthatlast.org/resourc as well as the infrastructure and the and maintain, including the es/concepts_tools/serv management model or institutional infrastructure. ice_delivery_models arrangements for the service provider to operate and maintain the infrastructure in order to provide the service. Delivery is defined “at the point of Focuses attention on what Fiszbein, Ringold, and contact between provider and client.” actually occurs at the point of Rogers (2011, 11); contact and the experience of World Bank (2015, delivery—process, engagement, 33) interaction—and the 23 APPENDIX A Statement from Source Implication Source benchmarks for quality of service. Features of delivery: (1) problem- Service delivery is embedded in Kim (2012) solving and context-sensitive; (2) complex, human-made systems complex systems; (3) multi- where solutions are not solely disciplinary; (4) knowledge is technical or logistical. interactive evolving. Delivery value-chain. Individual activities in the Kim et al. (2010, delivery cycle contribute to the 184); World Bank value, but only in relation to (2015, 20) other activities in the cycle. They are interdependent: who performs the activity; skill required; where the activity reaches the beneficiary; what supplies are needed; what information is captured and communicated; how activities link to previous and subsequent activities; and how beneficiaries are engaged. To deliver value, vertical or stand- In resource-poor settings, value Kim, Farmer, and alone projects must be integrated and is increased through the shared Porter (2013); Kim et share delivery infrastructure so that delivery infrastructure, al. (2010, 185) personnel and facilities are used wisely providing coordination and and economies of scale reaped. integration at the facility providing the service. …[A]lignment of delivery with local Importance of local context Kim, Farmer, and context by incorporating knowledge of with delivery. Porter (2013) both barriers to good outcomes. Effective care delivery helps break the System needs to be focused on Kim, Farmer, and cycle of poverty and disease. We will equitable outcomes. Porter (2013) not end extreme poverty without sustained investments in health-care delivery. Principles of service delivery: (1) Service delivery is focused on Asis and Woolcock citizen outcomes; (2) multi- outcomes for citizen well-being (2015); Kim et al. dimensional; (3) evidence; (4) and reliant on evidence and (2010, 184) leadership of change; and (5) adaptive learning in the process to allow implementation. teams to adjust in response to feedback generated in the delivery process. Interaction and engagement with beneficiaries. Engagement—a reciprocal relationship Active citizen engagement and Caseley (2006, 537) (not necessarily constant or equal) accountability needs to be part operating between two actors, whereby of service delivery. demands for improved service delivery performance are articulated through 24 APPENDIX A Statement from Source Implication Source formal, accessible, and transparent accountability mechanisms. Transparent and accessible citizen- Top-down demands for Caseley (2006) based accountability mechanism have improved services was not the potential to contribute to effective in making changes in organizational change and sustained front-line providers improvements in service delivery performance, but it took performance in public sector service triangulating between citizens, providers. It was the interdependent senior managers, and front-line accountability dynamic operating workers through formal between citizens, senior managers, and accountability mechanisms. front-line workers, which was the key to sustained improvements in the provision of complaints and new connections services. From the perspective of service Accountability mechanisms Fiszbein, Ringold, and delivery, governance can be need to be part of service Rogers (2011, 3) understood as the set of incentives and delivery. They affect how front- accountabilities that affect the way line providers and managers provider organizations, their managers, behave and how citizens and staffs behave, as well as the quality experience quality and and efficiency with which they deliver efficiency of the service. services. References Asis, M.G., and M. Woolcock. 2015. Operationalizing the Science of Delivery Agenda to Enhance Development Results. Washington, DC: World Bank. Caseley, J. 2006. “Multiple Accountability Relationships and Improved Service in Delivery Performance in Hyderabad City, Southern India.” International Review of Administrative Sciences 72 (4): 531–546. Fiszbein, A., D. Ringold, and F. Halsey Rogers. 2011. “Making Services Work: Indicators, Assessments, and Benchmarking of the Quality and Governance of Public Service Delivery in the Human Development Sector.” Policy Research Working Paper 5690, World Bank, Washington, DC. Kim, J.K. 2012. “Delivering on Development: Harnessing Knowledge to Build Prosperity and End Poverty.” World Bank Group President Jim Yong Kim, Keynote Speech to World Knowledge Forum, Seoul, Republic of Korea, October 9, 2012. http://www.worldbank.org/en/news/speech/2012/10/08/delivering-development-harnessing- knowledge-build-prosperity-end-poverty Kim, J.Y., P Farmer, and M.E. Porter. 2013. “Redefining Global Health-Care Delivery.” The Lancet 382 (9897): 1060–1069. Kim, J.Y., J. Rhatigan, S.H. Jain, R. Weintraubb,and M.E. Porter. 2010. “From a declaration of Values to the Creation of Value in Global Health: A Report from Harvard University’s Global Health Delivery Project.” Global Public Health 5 (2): 181–188. Lockwood, H., and S. Smits. 2011 Supporting Rural Water Supply. Moving towards a Service Delivery Approach. Rugby, UK: Practical Action Publishing. OECD (Organisation for Economic Co-operation and Development). 2008. “Service Delivery in Fragile Situations: Key Concepts, Findings, and Lessons.” Journal on Development 9 (3). 25 APPENDIX A World Bank. World Bank. 2003. World Development Report 2004: Making Services Work for Poor People. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/5986. ——. 2015. Rapid Assessments and Action Plans to Improve Delivery (RAAP-ID) in Subnational Governments. October 2015 World Bank, Governance Global Practice, Latin America and Caribbean. 26 Appendix B. Additional Literature Reviewed Abrams, L., I. Palmer, and T. Hart. 1998. Sustainability Management Guidelines. Pretoria: Department of Water Affairs and Forestry. Adama, O. 2012. “Urban Governance and Spatial Inequality in Service Delivery: A Case Study of Solid Waste Management in Abuja, Nigeria.” Waste Management & Research 30 (9): 991–998. Agarwal, R. 2008. “Public Transportation and Customer Satisfaction: The Case of Indian Railways.” Global Business Review 9 (2): 257–272. Ayres, I., and J. Braithwaite. 1992. Responsive Regulation: Transcending the Deregulation Debate. New York: Oxford University Press. Bjorkman, M., and J. Svensson. 2009. “Power to the People: Evidence from a Randomized Field Experiment of a Community-Based Monitoring Project in Uganda.” Quarterly Journal of Economics 124 (2): 735– 769. Black, J. 2002. “Critical Reflections on Regulation.” Australian Journal of Legal Philosophy 27: 1–36. Blankenship, J. 2014. “Using Results from Coverage Assessment Surveys to Improve Program Operation. ” Presentation at the Micronutrient Forum, June 2–6, 2014, Addis Ababa, Ethiopia. Choudhury, N. 2007. Irrigation Service Delivery in Canal Systems: A Study Of Eight Canal Systems in India. International Journal of Rural Management 3 (1): 127–148. Dahl-Ostergaard, Tom, et al. 2005. Lessons Learned on the Use of Power and Drivers of Change Analyses in Development Cooperation. Sussex: DFID. Davis, J. 2004. “Corruption in Public Service Delivery: Experience from South Asia’s Water and Sanitation Sector.” World Development 32 (1): 53–71. Delisle, H, O. Receveur, V. Agueh,, and C. Nishida. 2013. “Pilot Project of the Nutrition-Friendly School Initiative (NFSI) in Ouagadougou, Burkina Faso and Cotonou, Benin, in West Africa .” Global Health Promotion 20 (1): 39-49. Freiberg, A. 2006. The Tools of Government and Regulatory Design, presentation to the Australasian Compliance Institute, Sydney, 26 April referenced in Hodge 2007 below. Goel, R.K., and Rich, D.P. (1989). On the economic incentives for taking bribes. Public Choice, 61, 3, 269-275. Hodge, G.A. 2000. Privatisation: An International Review of Performance , Monograph published by Perseus Books - Westview Press, USA, 312 pp. Hyden, G. 2013 Culture, administration, and reform in Africa . International Journal of Public Administration 36(13): 922–931. PPP in Infrastructure Resource Center for Contracts, Laws and Regulation (PPPIRC) www.worldbank.org/ppp. Japan International Cooperation Agency. 2008. The study on water supply and sewerage system in Karachi in the Islamic Republic of Pakistan (Draft Final Report, Vol. 2). Tokyo, Japan. Joshi, A. 2013. “Do They Work? Assessing the Impact of Transparency and Accountability Initiatives in Service Delivery.” Development Policy Review 31 (S1): S29–S48. Kaufmann, D., A. Kraay, and M. Mastruzzi. 2006. Measuring governance using cross-country perceptions data. In S. Rose-Ackerman (Eds.), International handbook on the economics corruption (pp. 52–104). Cheltenham, UK: Edward Elgar. Kenny, C. 2007. Infrastructure governance and corruption: where next?. (Policy Research Working Paper No. 4331). World Bank, Washington, DC. Kim, J.Y., J. Rhatigan, S.H. Jain, R. Weintraubb, and M.E. Porter. 2010. From a declaration of values to the creation of value in global health: A report from Harvard University’s Global Health Delivery Project. Global Public Health. Vol. 5, No.2, March 2010, 181–188, Routledge, UK. 27 Kim, J.Y., P. Farmer, and M.E. Porter. 2013. Redefining global health-care delivery. www.thelancet.com Published online May 20, 2013 http://dx.doi.org/10.1016/S0140-6736(13)61047-8. Leisher, S.H., and S. Nachuk. Making Services Work for the Poor: A Synthesis of Nine Case Studies. Accessed at http://www.researchgate.net/publication/228818186. Mansuri, G., and V. Rao. 2013. Localizing Development: Does Participation Work? Washington, DC: World Bank. Martin, G. 2014. Powerpoint Linking Spending and Outcomes. The World Bank Group. DC. Minogue, M. 2006. Apples and Oranges: Comparing International Experiences in Regulatory Reform, in Regulatory Governance in Developing Countries, edited by Martin Minogue and Ledivina Carino, CRC Series on Competition, Regulation and Development, Edward Elgar, pp61–81. OECD (Organisation for Economic Co-operation and Development). 2008. “Service Delivery in Fragile Situations: Key Concepts, Findings, and Lessons.” Journal on Development 9 (3). Smith, J.J., and T.A. Gihring. 2006. Financing Transit Systems Through Value Capture: An Annotated Bibliography. American Journal of Economics and Sociology, Volume 65, Issue 3. July 2006, p. 751. Smoke, P. 2015. Accountability and service delivery in decentralizing environments: Understanding context and strategically advancing reform in Accountability and service delivery in decentralizing environments A Governance Practitioner’s Notebook: Alternative ideas and approached. OECD pp 219-232. Water Services that last. http://www.waterservicesthatlast.org/resources/concepts_tools/service_delivery_models World Bank. 2013. World Bank Group Strategy. Washington DC. 28 Appendix C. Developing Sector Concepts: Textbooks, Articles, and Websites Examined Alexeeva, V. Urban Transport Evaluation: FY2002–2011 Draft Issue Note. Note draws on other studies: “Investing in Sustainable Urban Transport: the GEF experience.” Global Environment Facility (GEF) Publication; Lima Transport Project (FY04), Implementation Completion Report; Sustainable Transport and Air Quality For Santiago (GEF) Project (FY04), Implementation Completion Report. Bhandari N, Kabir AKM, Salam MA. Mainstreaming nutrition into maternal and child health programmes: scaling up of exclusive breastfeeding. Matern Child Nutr. 2008;4(s1):5 –23. Bhutta, Z. A., J. K. Das, A. Rizvi, M. F. Gaffey, N. Walker, S. Horton, P. Webb, et al. 2013. “Evidence-Based Interventions for Improvement of Maternal and Child Nutrition: What Can Be Done and at What Cost?” The Lancet 382 (9890): 452–477. Bridging the Gap—a multistakeholder partnership to promote sustainable transport in the international climate debate (www.transport2020.org). Bruun, Eric Christian, 2014. Better Public Transit Systems: Analyzing Investments and Performance. New York, Publisher. Cairncross, S. and Feachem R. 1993. Environmental Health Engineering in the Tropics. An Introductory Text. 2nd ed. John Wiley & Sons. Note: 1st edition 1983 was consulted. Chichester, U.K.: John Wiley and Sons Ltd, New York, Publisher. Ceder, A. 2016. Public Transit Planning and Operation. Modeling, Practice and Behavior. Second Edition. CRC Press, Taylor and Francis Group: London. Chakravorty, S (1996) A measurement of spatial disparity: the case of income inequality. Urban Studies 33(9): 1671–1686. DFID (U.K. Department for International Development). 2002 Transport's Role in Achieving the Millennium Development Goals. Produced by the DFID Transport Resource Centre. Ganoulis, J. 2009. Risk analysis of water pollution. 2nd ed. Chichester, U.K.: John Wiley & Sons. Griffiths M., Dickin K., and Favin M. 1996. Promoting the Growth of Children: What Works Rationale and Guidance for Programs. The World Bank Group. DC. Horton, S., Shekar M., McDonald C., Mahal A., and Brooks J. K. 2010. Scaling up Nutrition: What will it cost? The World Bank Group. DC. Publisher. ICLEI’s EcoMobility Initiative (http://www.ecomobility.org/). IEG (Independent Evaluation Group). 2010. An Evaluation of World Bank Support, 1997–2007. Volume 1 Water and Development. IEG Study Series. ———. 2010. Gender and Development: An Evaluation of Evaluation of World Bank Support, 2002–2008. IEG Study Series. Washington, DC: World Bank. W9. ———. 2013 Improving Institutional Capability and Financial Viability to Sustain Transport. ———. 2015. Project Performance Assessment Report. Republic of Peru. Lima Transport Project. (IBRD- 72090 TF-52877 TF-52856). December 17, 2015. https://ieg.worldbankgroup.org/webpage/what-lima- metropolitano-transport-project-can-teach-us-about-delivering-sustainable-urban. International Food Policy Research Institute. 2015. Global Nutrition Report 2015: Actions and Accountability to Advance Nutrition and Sustainable Development. Washington, DC: IFPRI.. http://dx.doi.org/10.2499/9780896298835. Institute for Transportation and Development Policy (ITDP) www.itdp.org. International Journal of Sustainable Development and Planning. http://www.witpress.com/elibrary. 29 APPENDIX C Janssens JC and others. 2011. The Affermage-Lease Contract in Water Supply and Sanitation. Incentive Structures and New Developments. PPP Insights Vol.1 Issue 2. Mason, J. B., S. Sanders, P. Musgrove, Soekirman, and R. Galloway. 2006. “Community Health and Nutrition Programs.” In Disease Control Priorities in Developing Countries (2nd Edition), ed. Dean T. Jamison, et al., 1053–74. NewYork: Oxford University Press. Mehndiratta, S and others 2014. Targeted Subsidies in Public Transport: Combining Affordability with Financial Sustainability. World Bank. See also: http://blogs.worldbank.org/transport/will-you-take- me-1000-pesos-making-sure-public-transport-subsidies-really-target-poor. Neville L, O’Hara B, Milat AJ. Computer -tailored nutrition interventions targeting adults: a systematic review. Health Educ Res. 2009; 24(4):699–720. Pearson BL, Ljungqvist B. REACH: an effective catalyst for scaling up priority nutrition interventions at the country level. Food Nutr Bull. 2011; 32(Supplement 2):115S –27S. Ratnawali, 2010. Supplementary Nutrition to Women and Children: A Situational Analysis of Anganwadis in Tribal Areas of Gujarat, Social Change 40(3) 319–343 © CSD 2010 SAGE Publications. Rosenbloom, S. 1990. Developing a Comprehensive Service Strategy to Meet a Range of Suburban Travel Needs. Office of Technical Assistance and Safety, Urban Mass Transportation Administration. U.S. Department of Transportation: Washington, DC. Ruel M and Alderman H (2013) Nutrition-sensitive interventions and pro-grammes: how can they help to accelerate progress in improving maternal and child nutrition? Lancet, 382, pp. 536-551. Schlosser, C. Effective Strategies for Sustainable Urban Transport: Policy Options for National and Local Governments, UN HABITAT. See presentation at http://mirror.unhabitat.org/downloads/docs/8858_98181_1_UNHa%20overview%207oct10.pdf. Sussman, J. 2000. Introduction to Transport systems. Artech House: Boston, USA. The HUB, a capacity-building platform by Embarq India/WRI. The Partnership on Sustainable Low Carbon Transport (SloCaT) (www.slocat.net). The Urban Electric Mobility Vehicles Initiative (UEMI) (http://unhabitat.org/action-platform-onurban-electric- mobility-initiative-uemi/). UN HABITAT www.unhabitat.org. UN–HABITAT (The United Nations Human Settlements Programme). 2013. Global Report on Human Settlements 2013: Planning and Design for Sustainable Urban Mobility. Accessed at http://unhabitat.org/books/planning-and-design-for-sustainable-urban-mobility-global-report-on- human-settlements-2013. UN-HABITAT PAPER ON TRANSPORT & MOBILITY, HABITAT III ISSUE PAPERS, 19 —TRANSPORT AND MOBILITY, New York, 31 May 2015. UNICEF (United Nations Children's Fund). 2008. Handbook on Water Quality.2008. New York: UNICEF. United Nations Standing Committee on Nutrition, 2015. SCN News. Nutrition and the Post-2015 Development Agenda. Seizing the Opportunity. No.41 2015 ISSN 1564-3743. United Nations World Water Assessment Programme (WWAP). 2015. The United Nations. van Hollen, Cecilia (2003) Birth on the Threshold: Childbirth and Modernity in South India.Berkeley: University of California Press. Victora CG, Barros FC, Assunção MC, Restrepo-Méndez MC, Matijasevich A, Martorell R. Scaling up maternal nutrition programs to improve birth outcomes: a review of implementation issues. Food Nutr Bull. 2012;33(2):S6–26. World Bank. 2006. Repositioning Nutrition as Central to Development: A Strategy for Large-Scale Action. Directions in Development Series. Washington DC: World Bank. 30 APPENDIX C World Water Development Report 2015: Water for a Sustainable World. Paris, UNESCO. World Water Development Report 2015: Water for a Sustainable World. Paris, UNESCO. Worsly, T 2007 Promotion and communication. In: Lawrence M, Worsley T (eds). Public health nutrition Berkshire, UK: Open University Press and Mc-Graw-Hill. (cited in Delisle and others 2013). 31 Appendix D. List of Bank Staff Interviewed Prior to the preparation of the concept note, semistructured interviews were carried out based on three questions: describe how service delivery is understood in your sector and describe efforts to include service delivery perspective in projects, programs, and strategies; describe any interaction you are aware of between your sector and other sectors; and how useful is it to explore the conceptualization of service delivery framework across diverse sectors. After the initial framework and analytical tool were developed, semi-structured interviews with a subset of those initially met and were carried out on the following questions: What is your reaction to how we propose to evaluate service delivery? Did we miss anything? 1. Adam Wagstaff, Research Manager, DECHD 2. Cem Dener, Lead Governance Specialist, GGOGS (FMIS, Technology) 3. Dominick Revell de Waal, Senior Economist GWASP 4. Hana Brixi, Program Leader, MNA Strategic Co-operation Department 5. Harry Patrinos, Practice Manager Education, GED02 (EAP) 6. Helene Granvoinnet, Lead Social Development Specialist, GovGP 7. Kathy Lindert (Global Lead, SPL Delivery Systems) 8. Leslie Elder, Senior Nutrition Specialist, HNP 9. Maria Gonzales de Asis 10. Maria Gonzales de Asis, Lead Operations Officer, GGHVP 11. Michael Woolcock, Lead Social Development Specialist, DECBI 12. Mike Toman, Research Manager DECEE 13. Paul Prettitore, Senior Public Sector Specialist GG017 (MENA) 14. Verena Maria Fritz, Senior Public Sector Specialist GG025 (FCS) 15. Yasuhiko Matsuda, Senior Public Service Specialist, GSPDR 32 Appendix E. List of Projects Reviewed NUTRITION Bangladesh: National Nutrition P050751 Area-based Community Nutrition National-level Nutrition China: Poor Rural Communities Development P071094 Basic Health (CDD) Guatemala: Maternal-Infant Health and Nutrition P077756 Strengthening Basic package of health services Extending nutrition services to most vulnerable Supervising nutrition activities Maldives: Integrated Human Development P078523 Enhancing Support for Nutrition promotion Honduras: Nutrition and Social Protection P082242 Consolidation and Expansion of Nutrition Afghanistan: Strengthening Health Activities for Rural Poor P112466 Strengthening Basic Package of Health Services Laos: Community Nutrition P114863 Stimulating Behavior Change and Mobilize Mutual Support to Improve Nutrition Stimulating Demand for Key Maternal and Child health Senegal: Nutrition Enhancement P070541 Community-based Nutrition and Growth Promotion Senegal: Nutrition Enhancement II P097181 Community-based Nutrition Senegal: Rapid Response Child-focused Social Cash Transfer and Nutrition P115938 Security Community-based Nutrition Social Cash transfers Mauritania: Health and Nutrition Support P094278 Enhance and Expand Community-based Communications for Improved Nutrition Ethiopia: Nutrition P106228 Supporting Service Delivery Micronutrient Interventions 33 APPENDIX E URBAN TRANSPORT Colombia: Integrated Mass Transit Systems P082466 Bus Rapid Transit Development Argentina: Basic Municipal Services P060484 Municipal Infrastructure China: Guangzhou City Center Transport P003614 Inner Ring Road Traffic mgmt and safety Public transport Road Maintenance Chile: Sustainable Transport and Air Quality for Santiago P073985 Promotion of Bicycle use Congo: Emergency Infrastructure Rehabilitation P074006 Primary and Secondary infrastructure Rehabilitation Railways Rehabilitation Jordan: Cultural Heritage Tourism and Urban Development P081823 Historic City Centers and Urban Infrastructure Improvement Brazil: Bahia Poor Urban Areas Integrated Development P081436 Urban Infrastructure Delivery Rwanda: Transport Sector Development P079414 Paved Road Rehabilitation Peru: Lima Urban Transport P035740 Mobility Improvements Bangladesh: Dhaka Urban Transport P009524 Infrastructure Improvements WATER AND SANITATION Lebanon: Ba'albeck Water and Wastewater P074042 Improving and Increasing Coverage of Water Supply Improving and Increasing Coverage of Wastewater collection Philippines: Manila Third Sewerage P079661 Sewage Management Septage Management Pakistan: Punjab Municipal Services Improvement P083929 Support for Municipal Administrations Hungary: Municipal Wastewater P008497 Budapest Dunaujvaros Ghana: Small Towns Water Supply and Sanitation P084015 Community Subprojects St. Lucia: Disaster Management II P086469 Physical Prevention and Mitigation Works Afghanistan: Urban Water Sector P087860 Kabul Water Supply Brazil: Recife Urban Development P089013 Integrated Urban Territorial Development Sudan: Rural Water Supply and Sanitation P100835 Community water 34 APPENDIX E Community sanitation Bosnia and Herz: Urban Infrastructure and Service Delivery P083353 Investments for Improved Efficiency Philippines: Water Districts Development P004576 Sewerage, Sanitation and Drainage Ghana: Second Community Water and Sanitation P050616 Community subprojects 35 Appendix F. Analytical Protocol Guidance Questions for each item in the Framework (figure 3.1) Response I. BASIC PROJECT INFORMATION I. A. Country Country I. B. Region AFR, EAP, ECA, LAC, MNA, SAR I. C. Country Type FCS- yes/no MIC- yes/no LIC- yes/no I. D. Project Project ID I. E. Project Approval Approval Year I. F. Project Closing Closing Year I. G. Financing Institution IBRD, IDA, IFC I. H. Sector UT, WSS, Nutrition I. I. Services Relevant to SD Bus, Rail, NMT, Road, Water, Sanitation, Nutrition I. J. Components Unrelated to SD explain I. K. SD Screening Question: Project has nothing to do with SD (If yes, yes skip remaining questions) II. ENABLING CONDITION II. A. Was Political Economy Analysis included? yes/no II. A.1. If included, describe findings in relation to service delivery or explain the model of service delivery? Central, decentral, or II.A.2. If yes, for which level? both II.B. Was Leadership Development included? yes/no II. B.1. If included, describe how it is related to service delivery or the explain model of service delivery? Central, decentral, or II.B.2. If yes, for which level? both II. C. Was Policy Development included? yes/no II.C.1. If yes, describe its relation to service delivery or the model of explain service delivery? Central, decentral, or II.C.2. If yes, for which level? both II. D. Was Capacity Development included? yes/no II.D.1. If yes, describe its relation to service delivery or the model of explain service delivery? Central, decentral, or II.D.2. If yes, for which level? both II.E. Was Budgeting included? yes/no II.E.1. If yes, describe its relation to service delivery or the model of explain service delivery? Central, decentral, or II.E.2. If yes, for which level? both II.F. Was Regulatory and Legal Development included? yes/no II.F.1. If yes, describe its relation to service delivery or the model of explain service delivery? Central, decentral, or II.F.2. If yes, for which level? both 36 APPENDIX F II.G. Was Data systems included? yes/no II.G.1. If yes, describe its relation to service delivery or the model of explain service delivery? Central, decentral, or II.G.2. If yes, for which level? both II.H. Was Supply chain (i.e., production of goods and materials to be yes/no used in service delivery) included? explain II.H.1. If yes, describe its relation to service delivery or the model of Central, decentral, or service delivery? both II.H.2. If yes, for which level? II.I. Was Public Financial Management included? yes/no II.I. 1. If yes, describe its relation to service delivery or the model of explain service delivery? Central, decentral, or II.I .2. If yes, for which level? both II.J. Was country procurement system chain included? yes/no II.J.1. If yes, describe its relation to service delivery or the model of explain service delivery? Central, decentral, or II.J. 2. If yes, for which level? both II.K. Was any other enabling condition included? If yes, specify. yes/no II.K.1. If yes, describe its relation to service delivery or the model of explain service delivery? Central, decentral, or II.K.2. If yes, for which level? both II.I. Did the PAD describe contextual conditions that impinge directly yes/no on service delivery (such as inadvertent gender differences between yes/no service providers and service beneficiaries)? If yes, explain II.I.i. If yes, did design, planning, or implementation explicitly take account? II.I.1a. How? III. SERVICE DELIVERY INPUTS: III.A. Did PAD describe Funding for capital, operation, and capital- yes/no maintenance? operation- yes/no maintenance- yes/no III.B. Did PAD discuss Service Providers and Managers? How? yes/no explain III.B.1. Was training provided for Service providers and managers yes/no III.B.2. If yes, what was focus of training explain III.C Was Technology included? yes/no III.D Was design of service delivery supported? yes/no III.D.1 Identification of beneficiaries- were citizen beneficiaries yes/no specified. project beneficiaries, III.D.1.a. What was the basis for determining beneficiaries. sector beneficiaries, III.D.1.b. Are beneficiaries discussed by groups? geographic area, unclear III.D.1.c. Which groups of beneficiaries are described? how defined, specify III.D.1.d. Did the appraisal document describe barriers to the service for other beneficiaries? Yes/No (income, gender, III.D.1.e. Which barriers were reported? minority or marginalized III.D.1.f. Did planning take barriers into account? If so, how? group, age, specify other yes/no Affordability, accessibility, knowledge, 37 APPENDIX F gender, culture, other yes/no explain IIII.D.2.Was there an analysis of needs? yes/no III.D.2.a. Was the analysis conducted in relation to service provider or yes/no explain manager needs or capacity? Describe what was identified. yes/no explain III.D.2.b. Was the analysis conducted in relation to citizen beneficiaries Yes no- which needs or expectations? Describe what was identified. group(high income III.D.2.c. Was the analysis done for particular groups of beneficiaries? quintiles, low income Which ones? quintile, gender, III.D.2.d. Was there analysis of the existing SD model and its minority or marginalized suitability? group, age, specify other) yes/no III.D.3. Was there end to end implementation planning? yes/no III.D.4. Were service standards established? yes/no III.D.5.Was there provision for operation and maintenance? yes/no III. D.6. Was a service monitoring and improvement system developed? yes/no III.D.6.a. Who monitors services central government, decentral government, beneficiary, community, third-party, front -ine provider, manager, other (specify) IV. SD IMPLEMENTATION Central gov provision, IV.A. Which model was selected? central gov with IV.A.1.Were beneficiaries voice collected in relation to selection of SD contracting, decentral model? (skip to selected model) gov provision, decentral gov contracting, hybrid between central and decentral provision, hybrid between central and decentral finance with contracting, Private sector, PPP, citizen directed, other innovative 38 APPENDIX F yes, no, no information provided IV.B. Central government provision, why was this model selected? explain or no information provided IV.C. Central govt finance with contracting, why was this model explain or no selected information provided IV.C.1 Contract with whom? NGO, Community, IV.C.2 Type of contract? Private, Other- specify IV.C.3 What are government roles? Performance, results, input-based, other Select role(s)- Finance, regulate, monitor, operate, maintain, other IV.D. Decentralized govt provision, why was this model selected? explain or no information provided IV.E. Decentralized govt financing with contracting, why was this explain or no model selected? information provided IV.E.1. Contract with whom NGO, Community, IV.E.2. Type of contract? Private, Other- specify IV.E.3. What are government roles? Performance, results, input-based, other IV.F. Hybrid between central and decentral govt finance with explain or no contracting, why was this model selected information provided IV.F.1 Contract with whom? NGO, Community, IV.F.2 Type of contract? Private, Other- specify IV.F. 3. What are government roles at each level? Performance, results, input-based, other Select role(s) for central- Finance, regulate, monitor, operate, maintain, other Select role(s) for decentral- Finance, regulate, monitor, operate, maintain, other IV.G. Hybrid between Central and decentral govt provision, why was explain or no this model selected? information provided IV. G1. What are government roles at each level? Select Central role(s)- Finance, regulate, monitor, operate, maintain, other Select Decentral role(s)- Finance, regulate, 39 APPENDIX F monitor, operate, maintain, other IV.H. Private sector provision, why was this model selected? describe or no IV.H1. Which type of operating license for the service? information provided Type( operating license from government; provisional operating license from government, subject to meeting conditions for full license, without license because absence of government regulatory body; or unlicensed IV.I. Public-private partnerships, why was this model selected? describe IV.I.1. What is contract type? franchise, affermage, IV.I.2. What are roles between private and government for design, lease, Performance, build, operate, and maintain ( if new)? If existing (operate and results, input-based, maintain) other IV.I.3. Are there rules in how the government selects and manages the build, design, operate, PPP? maintain- specify which for government and private yes/no IV.J. Citizen-directed provision, why was this model selected? describe IV.J1. Did citizen design services such as CDD? yes/no IV.J.2. Did citizen influence market such as voucher? If yes, explain yes/no explain IV. K. Other innovative provision- (specify), why was this model selected? yes/no If yes, describe model. no information provided or explain IV. L. Is there cost recovery or subsidy mechanism? If yes, describe yes, no, describe IV. M. Is there a feedback loop? yes/no IV.M.1. Was feedback used to adapt implementation If yes, explain. yes/no If yes, explain V. SERVICE OUTPUTS V. A. Were outputs tracked in relation to service provider performance? yes/no If yes, specify If yes, specify outputs 40 APPENDIX F V. B. Is there an accountability mechanism? yes, no V.B.1. Which mechanism? Report cards, complaint, V. B.2. Which group is involved? Ombudsman, open V. B.3. How does this mechanism hold service managers and providers data/information other accountable? (specify) (policy maker, manager, front-line provider, citizen beneficiary explain V. C. Was there a mechanism to control service quality? yes/no V.D . Are other service outputs tracked? yes/no If yes, specify VI. SERVICE OUTCOME VI. A. Which service outcomes were tracked? coverage- yes/no quality Yes/no reliability yes/no affordability Yes/no satisfaction yes/no sustainability yes/no VI.B. Did the project collect disaggregated data? How are data yes/no disaggregated? If yes, income, gender, marginalized group, geographical area Are beneficiary outcomes tracked? Which ones? yes/no If yes, specify Are trend data tracked? yes/no VII. LESSON LEARNED ABOUT SERVICE DELIVERY MODEL VII. A. Is the SD model achieving the expected results? yes/no explain. VII. B. Is the SD model meeting citizens' expectations? yes/no/no information VII.C. Did enabling conditions or inputs impact implementation of SD yes/no/no information If model? If yes, how? yes, explain VII.D. Are lessons evident from SD model as implemented? If yes, yes/no. If so, explain summarize. VII.E. Are changes to the SD model being considered? yes/no/no information. If yes, explain Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LAC = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. 41 Appendix G. Early Version of Analytical Protocol Country Project Service Delivery Component Project Approval Date Project Closing Date I. SERVICE DEFINITION (SECTOR-WIDE LEVEL) A. Sector/ Sub-sector Nutrition (subsector) UT (subsector) WSS (subsector) B. Services being delivered (List services) C. Level of Support Direct To beneficiaries (individual users or communities) To front-line providers Indirect To managers Of which, non-state managers (NGOs, private sector) Of which, state managers (central govt, decentralized govt, semi-autonomous public entity) D. Service Delivery Arrangement Central govt Central govt financing with contracting Central govt provision Of which, capital/investment costs (infrastructure) Of which, operational costs Local govt Local govt financing with contracting Local govt provision Of which, capital/investment costs (infrastructure) Of which, operational costs Clients Client power - contracts Client power - self monitoring providers Client power - community control, vouchers Client power - imitate market F. Service Area National 42 APPENDIX G Regional Municipal Municipal region Community/ neighborhood Other Point of Use Facility Of which, networked to other facilities Via technology or mobile Local system Of which, roads Of which, water supply Of which, sanitation Other D. Service Fee (paid by) Government Users Other E. Service Fee (financing mechanism) Cost recovery (full or partial) Tariffs (e.g., integrated tariff systems, polluter pays) Subsidy User fees Concession Carbon bonds Mobility fund Other No cost recovery mechanism Govt funded (inc Bank through project lending) Donor funded (not inc Bank loan, but other development partners) K. Service Reaching Poor Users Intended to reach the poor How are “poor” defined? II. PROJECT DESIGN & IMPLEMENTATION A. Prior analytic work Assessment of users User needs (inc lack of knowledge) User demand/ voice Assessment of providers/managers Implementation capacity Assessment of state Institutional capacity 43 APPENDIX G Identification of barriers to service delivery (source: PAD section 2, Annex on sector background or social assessment) Heterogeneity of users (current and expected travel patterns, moving from individualized motor transport) Accessibility (point of use, all users including disabled) Affordability (defining poor, how much to subsidize, comparing alternatives, assess willingness to pay) Efficiency Gender equality Safety Environmental implications Other B. Service Delivery Arrangements Implementation Model Central government PMU Mainstreamed into Ministry Decentralized government Frontline professionals Private organization Community Public/private partnerships Other Characteristics of Access to Service Facilities Densely populated group of users Walking distance Geographic barriers Other C. Key Design Features (and comments on whether it was implemented as planned) User targeting mechanism Formal mechanism to identify beneficiaries Of which, mechanism to reach the poor Activities to generate demand among users Support for Quality of services Providers’ training/certification standards User education/training/orientation in using service State-of-the-art technology Technical standards Other Innovations in service delivery (e.g., technology) Accountability State Measures to increase accountability of state Providers Measures to increase accountability of providers to the state 44 APPENDIX G Measures to increase accountability of providers to beneficiaries Citizens Measures to increase participation Participatory design (tailor services to demand) Participatory implementation/ mgmt Participatory monitoring/feedback Of which, includes incentives for citizens to monitor Measures to increase voice Monitoring and Evaluation Source of monitoring Government Project implementation unit Third party Frontline professionals Community User Other Evaluation mechanism Regular monitoring (e.g., MIS) Learning cycle (learning/providing feedback/adjusting delivery model) Surveys Planned impact evaluation Other Monitoring characteristics Easy to monitor Done in real time Assessing fidelity Includes plan for dissemination/sharing Other III. PROJECT PERFORMANCE/OUTCOMES A. Outputs Facilities Supplies Personnel Other B. Service Delivery Outcomes Among the project population Total # direct beneficiaries Access/ Coverage Quality Efficiency Affordability Other Outcomes among the poor (if any) 45 APPENDIX G Contextual factors that impacted service delivery (source: ICR section 2) External (beyond providers' control) Political commitment at high level of govt Politlcal transitions due to elections/coups Adequate budgetary support Supportive policy/regulatory environment Bureaucratic environment Functioning decentralization Overall economic growth Other Internal Institutional capacity (govt) Institutional capacity (direct service provider) Partnerships Multisectoral coordination Management/supervision arrangements Learning cycle Other C. Institutional impact (including accountability) Positive impacts Negative impacts D. Monitoring and evaluation (source: ICR) Design (e.g., choice of indicators) Implementation (e.g., quality of data, quality of monitoring arrangements, conducting of evaluations) Utilization (.e.g use of data, dissemination of results) E. Sustainability (scale and financial sustainability) Pilot activity Special project with external support With mostly external support and limited host country support With heavy host country support Routinized government activity Of which, at full scale Of which, not at full scale 46 Appendix H. Glossary of Terms to Assist Users of the Analytical Protocol Accountability: A reciprocal relationship operating between two actors whereby demand for and supply of services are articulated and satisfied (Caseley 2006, 537; World Bank 2003, 48). Accountability can involve several interdependent pairs of actors, linking users of services, policy makers, and service providers (OECD 2008). Accountability mechanism: The agreement setting out the terms and expectations (both supply and demand) for service delivery (Caseley 2006). Between service provider and service user, the agreement might take the form of a formal contract, or it might be a less formal compact such as an understanding that the service being delivered is a public good. Between policy maker and service provider, the agreement might take the form of a service level agreement. Adaptive implementation: A structured approach to learning from experience in implementing service delivery and making adaptations and “mid-course corrections” (Asis and Woolcock 2015). Beneficiaries: Those persons who benefit from services delivered through improvement to their well being, and those persons who have the right to benefit also but who are excluded by factors such as geographical location, marginalization of population group, disability, gender, or other dimensions affecting their access to services (Asis and Woolcock 2015). Feedback loop: A structure for obtaining and using data to inform ongoing service delivery on a continuous basis (Asis and Woolcock 2015). Front-line service provider: The person providing the service at the point of contact with the service user or beneficiary. The provider also describes the public, private, nonprofit, and private-for-profit entities that provide the services. Service manager: The person or agency with responsibility for managing the delivery of the service. This responsibility may include all aspects such as quality assurance, supply, and accessibility of the service, or may be limited to a specific aspect of the service being delivered. Service monitoring: A process used to measure both supply- and demand-side results of service delivery performance (Brinkerhoff and Wetterburg. 2013). The process involves policy makers, providers, and beneficiaries. Service quality: An output of the performance of service delivery that has been measured against standards established for clarifying expectations for the service being delivered (Brinkerhoff and Wetterburg. 2013). Supply chain: An interlocking system that includes all the organizations and processes involved in each stage of getting services to people. The processes include both the supply processes involved, such as planning, sourcing, making, and supplying; and the demand 47 processes, such as obtaining feedback for services. Service delivery is one stage in the supply chain (World Bank 2012). Sustainability: The endurance of the systems and processes involved in service delivery. This includes the systems and processes for managing the resources available now and for generations to come, adapting to emerging needs, ensuring social inclusion, and adopting fiscally responsible policies (World Bank 2015). References Asis, M.G., and M. Woolcock. 2015. Operationalizing the Science of Delivery Agenda to Enhance Development Results. Washington, DC: World Bank. Brinkerhoff, D.W., and A. Wetterburg. 2013. “Performance-Based Public Management Reforms: Experience and Emerging Lessons from Service Delivery Improvement in Indonesia. ” International Review of Administrative Sciences 79 (3): 433–457. Caseley, J. 2006. “Multiple Accountability Relationships and Improved Service in Delivery Performance in Hyderabad City, Southern India.” International Review of Administrative Sciences 72 (4): 531–546. OECD (Organisation for Economic Co-operation and Development). 2008. “Service Delivery in Fragile Situations: Key Concepts, Findings, and Lessons.” Journal on Development 9 (3). World Bank. 2003. World Development Report 2004: Making Services Work for Poor People. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/5986. ———. 2012. Getting to Green. A Sourcebook of Pollution Management Policy Tools for Growth and Competitiveness. Washington, DC: World Bank. ———. 2015. 2015 Sustainability Review. Washington, DC: World Bank. https://openknowledge.worldbank.org/handle/10986/22723 48