Approach Paper The World Bank Group Outcome Orientation at the Country Level: What Influences Outcome Management and Measurement Practices across Country Engagements? February 10, 2019 Highlights • The World Bank Group’s success rests on its ability to help its clients overcome the complex development challenges they face. This demands a strong outcome orientation at the country level, defined as the organization’s ability to generate feedback on what works, what does not, and why; use this feedback to engage clients and adapt country programs and portfolios; and ultimately bolster its contribution to country development outcomes. • The country engagement cycle is the framework that organizes how the Bank Group manages and measures its contribution to country development outcomes. Building from the Independent Evaluation Group’s 2017 evaluation on World Bank country engagement, this evaluation will focus on 52 countries that have gone through at least three steps of the engagement cycle between fiscal years 2014 and 2020. • The evaluation will focus on four key dimensions of outcome orientation: (i) the outcome-focus of Country Partnership Frameworks and their results frameworks; (ii) the monitoring, evaluation, and learning activities that take place throughout the country engagement cycle; (iii) adaptive management practices at the country portfolio level; and (iv) client engagements on outcome measurement and management. • The evaluation’s objective is not to assess the extent to which the Bank Group achieves country-level outcomes. The Independent Evaluation Group has several other instruments to answer this question. However, this evaluation will provide insight on how country engagements are linked to country-level outcomes, including the Sustainable Development Goals and how country teams set out to assess the Bank Group’s contribution to these outcomes. The evaluation will identify what factors promote and hinder a strong country outcome orientation, with a specific focus on incentives and behaviors. 1 Report to the Board from the Committee on Development Effectiveness The Committee on Development Effectiveness met to consider the document entitled Approach Paper: The World Bank Group Outcome Orientation at the Country Level. The committee welcomed the approach paper, noting that the topic and its links to other issues such as risk, the Sustainable Development Goals (SDGs) agenda or the World Bank Group strategy for fragility, conflict and violence made the discussion relevant and timely. Members were pleased to learn that the evaluation would be implemented in synergy with other Independent Evaluation Group (IEG) effectiveness workstream activities aimed to support the World Bank Group's efforts to strengthen how it measures and manages its contribution to development outcomes at the country level, including the IEG-management collaborative work on the reform of the Management Action Record. Management explained how the evaluation would complement its initiatives to assess and ensure the Bank Group takes a robust approach to country’s development outcome, namely (i) putting outcome orientation at the core of design and implementation of country programs and operations across the different stages of the country engagement cycle; (ii) enhancing staff learning from evidence to build stronger feedback loops both in operations and country programs, including a reform of management follow-up of IEG’s recommendations; (iii) strengthening the skills of operational teams, including enhancing the approach to assessing and managing risks in programs and projects; (iv) strengthening the outcome orientation through enhanced corporate approach to capturing and reporting results, such as the corporate scorecards and the International Development Association results measurement system; and (v) International Finance Corporation’s country orientation at the project level supported by diagnostics (Country Private Sector Diagnostics and sector deep-dives) and strategy. While acknowledging management’s explanations, members asked management to elaborate on the agenda to increase the outcome focus in Bank Group’s instruments, policies, incentives and related key deliverables, asking about specific actions, expected results and timeline. Members also inquired about how this exercise would inform the pilot approach proposed by management in the July 2019 Executive Directors’ Seminar. Members indicated support for a broader discussion and identification of specific issues and definition of plan and timeline to support their further engagements on these issues. The Committee appreciated IEG’s commitment to revise the approach paper to clarify the focus of the evaluation on the four key dimensions of outcome orientation, that is, (i) outcome focus of the Country Partnership Framework and its results framework; (ii) monitoring, evaluation, and learning activities that go throughout the engagement cycle; 2 (iii) adaptive management at the portfolio level using this outcome information; and (iv) clients' engagement throughout this process. IEG also indicated the revised approach paper would provide insights on how country engagements are linked to country-level outcomes, including the Sustainable Development Goalss; and how country teams set out to assess the Bank Group’s contribution to these outcomes, with a specific focus on incentives and behaviors. 3 1. Background and Context 1.1 In 2005, the World Bank Group rolled out results-based management at the country level; today, it seeks to build on this experience to strengthen its outcome orientation. The country is the Bank Group’s key unit of engagement, and through a package of public and private sector interventions and knowledge work, the Bank Group sets out to contribute to meeting its clients’ specific development needs. Since 2005, country programs have contained results frameworks, along with monitoring and self-assessment processes to track and report achievements. The country engagement model adopted in 2014 and its underlying cycle of diagnostic, planning, and monitoring and evaluation phases is meant to establish a Bank Group–wide, evidence-based, and selective approach to answering country needs. Bank Group management, Board of Executive Directors, and country teams are eager to find ways to strengthen how they manage and measure their contribution to country development outcomes. 1.2 The current country-level results measurement and management framework is meant to serve multiple needs at once, with potential tensions between various intended uses. The country engagement cycle aims to provide relevant and timely information on the Bank Group’s performance and progress toward the desired development outcomes for multiple purposes: external accountability for results to the Board and clients; internal portfolio management by Country Management Units; and learning, innovation, and adaptation by country teams. However, country teams often mention four tensions that weigh on the outcome orientation of country engagements. The extent to which these issues influence the outcome-oriented behaviors of country teams will be investigated in the evaluation: • Measuring attributable results versus assessing contribution to country development outcomes: Making progress on country development outcomes is, by nature, a collective endeavor. Bank Group programs typically provide a fraction of the total resources devoted to development endeavors. Therefore, tension exists between reporting on results that are reasonably attributable to Bank Group interventions and capturing effects on long-term development changes, fueling a long-term debate on the topic (for example, Levine and Savedoff 2006; Honig 2019). This evaluation will seek to inform this debate by providing examples of what could be entailed by a more nuanced understanding of attributing results to Bank Group interventions. The evaluation will present examples of the Bank Group’s effects at the country level, including through scale-up, demonstration and catalytic effects, and transformation of the Bank Group’s piloted solutions into governments’ programs. 4 • Country engagement cycles versus outcome management cycles: The life cycles of projects and country programs do not often correspond. The Country Partnership Framework (CPF) represents a forward-looking strategic exercise that lays out potential interventions to support government priorities over a five- to seven-year period. However, when establishing results frameworks for new CPFs, Bank Group teams mainly draw from the expected results of the operations approved during the previous CPF cycle, which is not necessarily aligned with new objectives. Likewise, when measuring results at the end of a country engagement cycle, many interventions underlying the Bank Group’s country engagement have not yet been fully implemented, let alone yielded measurable results. Furthermore, country priorities can shift within the engagement cycle. These time lags create tensions and trade-offs between the frequency of measurement and the quality, reliability, and usefulness of information on outcomes. The evaluation will illuminate the complementarity of metrics with other types of evidence of results and determine whether current practices strike the right balance. • External reporting versus internal managerial and learning needs: Measuring and managing for results for outside reporting prioritizes the production of comparable, quantifiable, and aggregable results. It puts a premium on tracking progress toward corporate priorities, often crowding out program or context- specific monitoring and evaluation efforts. It may also create a compliance mind- set, which is at odds with an outcome-oriented mind-set (for example, Buntaine, Park, and Buch 2017; Honig 2019; World Bank 2016a; Dahler-Larsen 2011). • Bank Group outcome orientation versus clients’ outcome orientation: Using client countries’ data and systems to support client countries’ capacity to manage national results frameworks and associated statistical systems—and avoid creating parallel frameworks—is key to fostering clients’ outcome orientation. However, the weak data and evaluation capacity of some country clients continues to hinder the outcome orientation of country programs, and the Bank Group and other partners tend to set up their own results management and measurement systems (World Bank 2017d; Vähämäki and Verger 2019; Zwart 2017). 1.3 Generally, these challenges are shared by other development partners, some of which are experimenting with alternative outcome-orientation approaches. The Organisation for Economic Co-operation and Development recently reviewed evaluations of the institutionalization of the “Results Agenda” in key bilateral and multilateral development agencies (Vähämäki and Verger 2019). The review shows issues on both the supply and the demand side of results information. On the supply 5 side, challenges include lack of attention to unintended outcomes, an intervention- centric perspective on results that fosters biases, and limited data and evidence to understand causal contribution. On the demand side, issues include the short-term thinking of decision-makers and institutional inertia to act on the evidence of what works, what does not, and why. Several international organizations and governments are experimenting with alternatives to conventional results-based management, changing structures, policies, and practices with the intention of enabling adaptive management for improved development outcomes (Ramalingan, Wild, and Bufardi 2019). This evaluation will set out to review and draw lessons from other development partners’ experiences with outcome measurement and management at the country level. 2. Objectives and Intended Users 2.1 This learning-focused evaluation aims to help Bank Group management and country teams strengthen the country-level outcome orientation of the Bank Group. Bank Group management and the Board are keen to enhance the focus on country results and foster a stronger and more realistic culture of results in the institution. This evaluation is part of the Independent Evaluation Group (IEG) workstream designed to help the Bank Group implement its objective of enhancing its accountability for results, informed risk-taking, and organizational learning. The evaluation will thus be implemented in tandem with other IEG activities that contribute to this cross-cutting workstream, including a revamped approach to the Results and Performance of the World Bank Group 2020 , a rethink of the Country Program Evaluation Methodologies, a review of IEG’s validation of self-evaluations, an evaluation of World Bank’s engagements in fragile and conflict-affected countries, and a reform of the Management Action Record. 2.2 The evaluation has three primary objectives. The main focus is to learn from different approaches and extents of outcome orientation in various operational contexts. The evaluation is not intended to assess whether the Bank Group achieves outcomes at the country level. Specifically, on the basis of a systematic review and assessment of existing country level practices and experiences, the evaluation will • Identify markers of strong country outcome orientation and their explanatory factors. • Identify challenges and tensions that may be hindering country outcome orientation. • Provide Bank Group management with options for strengthening outcome orientation. 6 2.3 The evaluation targets four groups of stakeholders within the Bank Group. First, the evaluation is intended to help World Bank Country Management Units, Country Leadership Teams, Regional Development Effectiveness units, and International Finance Corporation (IFC) regional and country teams promote a strong outcome-oriented behavior. These are the primary learning partners targeted by the evaluation. The evaluation will seek to promote learning across teams and suggest ideas on how to deal with some of the challenges and tensions that hinder country-level outcome measurement and management practices. Second, the evaluation also intends to serve staff in Global Practices and IFC industry teams because they are key contributors to country programs and to the measurement of results. Third, the evaluation is intended to inform Bank Group management of the challenges that may be hindering the outcome orientation of country engagement and of the opportunities for its further strengthening. The evaluation team will consult closely with Operations Policy and Country Services (OPCS) results and accountability teams and country economics teams; IFC Country Economics and Engagement and Corporate Strategy teams; and Multilateral Investment Guarantee Agency Economics and Sustainability Group. Fourth, the evaluation is intended to inform the Board of the possible trade-offs faced by Bank Group management and staff when trying to meet the outcome information needs of various stakeholders. It will also provide information on the capabilities required for a stronger outcome orientation at the country level. 3. Preliminary Conceptual Framework 3.1 The preliminary conceptual framework underpinning this evaluation design is composed of three components. Its objective is to capture relevant elements at play in determining the outcome orientation of the Bank Group’s country engagement and serve as a backbone for the evaluation design. It will be refined throughout the evaluation process. As illustrated in figure 3.1, the framework consists of three main components: • An explanation of what it means to promote outcome-oriented behaviors. • A succinct description of the evaluand (the process and activities by which the Bank Group embeds its outcome orientation in country engagements). • An initial set of underlying conditions known to influence the strength of outcome orientation of organizations. 3.2 The evaluation’s working definition of outcome orientation is an organization’s ability to generate feedback on what works, what does not, and why; use this feedback to make decisions, adapt country portfolios and programs, and engage clients; and ultimately bolster its contribution to country development outcomes. Feedback means 7 multiple sources of measurement and evidence on progress toward country development outcomes and on shortcomings and ways to address them. This evaluation focuses specifically on country-level portfolios, which are groupings of discrete lending and private sector investment operations, analytical and advisory services, and convening and policy dialogue activities linked by an overarching, time-bound strategy. A more elaborate definition is presented in appendix C. 3.3 The evaluation will focus on four key dimensions of outcome orientation. First, the evaluation will examine how country teams design selective and outcome-focused CPFs and results frameworks, including how they articulate links with country goals, including the Sustainable Development Goals. Second, the monitoring, evaluation, and learning activities that take place throughout the country engagement cycle will be reviewed, including how teams set out to capture the Bank Group’s contribution to country outcomes within and across cycles. Third, the evaluation will inquire about the use of evidence to guide adaptive management practices at the country portfolio level. Fourth, the evaluation will investigate client engagements on outcome measurement and management. A more elaborate definition is presented in appendix C. 3.4 The country engagement cycle is the backbone of the Bank Group’s results measurement and management at the country level. The four stages of the cycle logically organize how the Bank Group identifies, articulates, tracks, assesses, and learns from its contribution to country development outcomes—in other words, its outcome orientation. These four stages are also meant to formalize key moments of interaction between the Bank Group and country clients. They are intended to work as follows: • The Systematic Country Diagnostic (SCD) identifies the key constraints that impede progress on specific country development goals. It is informed by a Country Private Sector Diagnostic led by IFC. It also identifies the mains policies or reforms that would contribute the most to addressing these constraints and achieving the country development objectives, independent of whether the Bank Group will address them or not (World Bank 2014). The extent to which the SCD informs the CPF is outside the scope of this evaluation because it was the topic of IEG’s recent country engagement evaluation (World Bank 2017b). • The CPF delineates the specific intended contribution of the World Bank to the country development goals. The country program considers the findings of the SCD, the Bank Group comparative advantage and resources, country priorities and demands, and activities of other development partners (World Bank 2014). IFC country strategies are also meant to inform the CPF. Three parts of the CPF are of particular relevance to this evaluation: 8 o The results chains for each outcome area that links the constraints identified in the SCD to the World Bank’s proposed package of interventions is supposed to be a fundamental part of the CPF. It is supposed to explain how ongoing and planned activities work together synergistically and in complementarity with the interventions of government, the private sector, and other development partners to contribute to specific development outcomes. o The choice of specific instruments that make up the country program, such as Program-for-Results or investment project financing with disbursement- linked indicators specifically designed to encourage an outcome orientation also come into play. o The results matrix that accompanies the CPF defines select outcome indicators to track progress and measure the attainment of CPF objectives. • The Performance and Learning Review (PLR) is meant to be used periodically during a given cycle to take stock of progress achieved, identify any changes in country priorities or conditions that have implications for the World Bank’s operations, and update the CPF if necessary. The PLR exercise is intended as an important mechanism for adaptive management and course correction. The revised results frameworks will be the basis for the end-of-cycle self-assessment and validation (World Bank 2014). Other processes are also intended to allow for adaptation at the country level, such as the Country Portfolio and Performance Reviews (CPPRs) for the World Bank and the annual review of IFC strategies and business plans for IFC. • The Completion and Learning Review (CLR) is the self-evaluation tool used at the end of a country engagement cycle to assess the extent to which the objectives set out in the CPF were reached and draw lessons to inform the next cycle. A rating scale is used to synthesize the World Bank’s performance. The CLR is reviewed, and ratings are validated by IEG. CLR ratings are aggregated to feed into corporate reporting processes. 3.5 Understanding whether a country program has a strong outcome orientation requires going beyond the framework documents. Although the CPF, PLR, and CLR constitute the formal reporting framework, outcome measurement and adaptive portfolio management practices take place through an array of processes at the program, sector, and intervention level. The evaluation will carefully balance what is articulated in the framework documents with the practice of country and sector teams—for instance, by reviewing how CPF progress is monitored, how programs are adjusted or 9 restructured through CPPRs, and how outcomes are measured and managed within selected sectors of activities that are particularly prominent for the achievement of CPF objectives. As described in the next section, the evaluation methodology relies as much on in-depth interviewing and focus groups as it does on structured document reviews to elicit tacit knowledge and behaviors. 3.6 The evaluation will review built-in assumptions related to the timeliness, robustness, and relevance of data and evidence at key stages of the engagement cycle. There are several assumptions built into the country engagement cycle; whether or not they materialize can weigh on the outcome orientation of the country engagement. This comprises issues of data availability, use of country data systems, evaluation methodologies, and skills. Another set of assumptions relates to the composition of portfolios, notably the extent to which various interventions are expected to relate to one another, and whether the intended outcomes are measurable and the processes of change traceable. The measurement challenges and unpredictability of change processes may be higher for some portfolios than for others. The evaluation will carefully select cases to study these contextual combinations. 10 Figure 3.1. Preliminary Conceptual Framework Source: Independent Evaluation Group. 11 3.7 Organizational conditions also influence the depth of the outcome orientation of country engagements. The literature on managing for development results has rich evidence about the organizational characteristics that are associated with well- functioning evaluation systems (for example, OECD and World Bank 2014, and OECD 2019; World Bank 2016a and 2017b; ADB 2012; Dahler-Larsen 2011; Leeuw and Furubo 2008; and Hojlund 2014). Three sets of characteristics are known to influence the outcome orientation of organizations. First, individual motivation and incentives matter significantly. In international development, agents’ intrinsic motivations to achieve results are critical and must be reinforced by extrinsic incentives, such as “carrots, sticks, and sermons” (Bemelmans-Videx, Rist, and Vedung 1998; Levine and Savedoff 2006). The level of autonomy of agents to use outcome information and have the decision power to adapt interventions and portfolios is also essential (Honig 2019; Ramalingam, Wild, and Buffarid 2019). This evaluation will pay close attention to the issue of management and staff incentives for outcome measurement and management. Second, teams’ capabilities, including staff skills, methods, knowledge, and learning support, as well as relationships and networks are also central tenets of outcome-oriented behaviors (Vähämäki and Verger 2019; OECD 2019; Porter and Hawkins 2019). Third, management practices and their underlying set of rules, requirements, and span of control must also play an enabling role to support and sustain outcome-oriented behaviors. This evaluation will study in detail how these various organizational characteristics on the Bank Group and the country client sides intersect to encourage or hinder a strong outcome orientation as part of the country engagement model. 4. Evaluation Questions and Scope 4.1 The evaluation questions and scope delimitation were informed by three main sources of information: • Eight IEG reports that relate to this evaluation topic and the rich evidence that fed these evaluations, including longitudinal data on project and country strategies performance, hundreds of interviews, focus group discussions, and surveys of World Bank staff and management. • Recent discussions between the Bank Group’s management, the Board, and IEG on the Bank Group’s outcome orientation, along with consultations with several executive directors, country directors, and experts in OPCS and IFC results and country economics teams. • A review of the literature and expert consultations on results-based management, Managing for Development Results, and evaluation systems in development agencies. 12 Evaluation Questions 4.2 The main question motivating this evaluation is, “What influences outcome management and measurement practices across Bank Group country engagements?” Two main lines of inquiry underly this: 1. What factors explain differences in outcome management and measurement practices across Bank Group country engagements, and what can be learned from these differences? 2. What are the opportunities for strengthening the outcome orientation of Bank Group country engagements, and what behaviors, capacities and resources would be necessary? 4.3 Appendix A presents a detailed evaluation matrix with subquestions. 4.4 Four main priors are shaping the evaluation’s scope, questions, and methodological approach. These are summarized as follows: • The evaluation takes as a given that a strong outcome orientation is desirable for the Bank Group’s country engagements. This assumption is in line with the Bank Group management and Board’s vision for an effective Bank Group. Establishing whether greater outcome orientation improves development effectiveness is beyond the scope of this evaluation. However, the evaluation will be informed by the rich literature on the challenges and pitfalls of different results-based management or adaptive management practices in different contexts. • The country engagement cycle provides a framework to anchor the Bank Group’s improved outcome orientation. Several IEG evaluations have confirmed that the four stages of the country engagement model provide a robust framework to anchor the World Bank’s contribution to country development outcomes (World Bank 2017b, 2015d, and 2018a). This evaluation will focus on drawing lessons from the current implementation of this model. • There are good reasons to suspect that there is variation in the outcome orientation of country programs, and that analyzing this will provide learning. The Bank Group is an organization with a great deal of informal learning and tacit knowledge sharing (World Bank 2014a). In such an organization, adaptive management and outcome orientation can operate below the radar, working to meet procedural obligations through existing formal accountability mechanisms while enabling adaptive learning and outcome orientation through a mix of informal and formal approaches (Ramalingan, Wild, and Buffarid 2019). 13 • Tensions between multiple intended uses of the outcome measurement and reporting system influence the outcome orientation of country engagements. Results measurement and reporting frameworks and processes are intended to serve as a governance tool for World Bank clients, Country Management Units, Regions, and Global Practices, as well as donors and the Board. At the same time, they are also supposed to serve as critical performance management and learning tools for World Bank operational teams and clients’ implementation units. The evaluation will pay close attention to potential trade-offs. Evaluation Scope 4.5 Three parameters determine the scope of this evaluation: country perspectives, system boundaries, and timing: • First, the evaluation will take a one Bank Group–approach in analyzing country engagements. Greater emphasis will be placed on the World Bank given that its business model has been country focused for a longer time, and considering its relative portfolio size and its convening role with client government throughout the country engagement cycle. That said, the evaluation will also cover IFC’s emergent country model, including its recent practice of developing IFC country strategy and business plans at the country level. The evaluation will consider how Multilateral Investment Guarantee Agency fits in the one Bank Group– approach to outcome orientation at the country level. Studying the differences in outcome reporting across the institutions at the country level will also shed some light on challenges and the opportunities to strengthen the Bank Group’s outcome orientation. However, the evaluation scope does not include an institution-specific review of outcome measurement and management. • Second, the evaluation’s unit of analysis is the country engagement cycle as the results setting, assessing, and reporting cycle of reference. The evaluation will focus on the CPF-PLR-CLR nexus and the related IFC country-level strategic products. The evaluation will also focus on the notion of within-country, sector- level portfolios as a package of support that includes various types of country- based Bank Group interventions (analytical and advisory work, policy dialogue, country-level convening activities, capacity building, investments, guarantees, and lending). In seeking to understand factors that enable or obstruct adaptive management within a CPF cycle, the evaluation team will select a small number of sectors—particularly those that were key to the CPF intended outcomes and for which there was a multiplicity of contributing activities—to examine how Bank Group teams have practiced adaptive management and how they have measured outcomes. The evaluation will present examples of the Bank Group’s 14 effects at the country level, including through scale-up, capacity-building that trigger systems-change, and adoption of Bank Group’s model and solution into governments’ programs and practices. It will also consider whether specific instruments have been leveraged in specific country cycles to foster the outcome orientation of clients. However, a review of outcome orientation at the project- level is outside the scope of this evaluation. • Third, the evaluation focuses on fiscal years (FY)14–20 and considers recent and real-time developments in the outcome orientation of the Bank Group. This period corresponds with the introduction of the new country engagement model. 5. Approach and Methodology 5.1 The evaluation design is based on principles from case-based evaluation approaches. As specified previously, the unit of analysis of this evaluation is the Bank Group country engagement cycle. The evaluation team assembled a database of 52 countries that went through at least three steps of the engagement cycle between FY14 and FY20. After a preliminary review of country and portfolio data, an initial typology of country engagements was created (see appendix B). It will be refined throughout the evaluation process. Based on this, countries were selected for in-depth inquiry across a number of factors thought to influence outcome orientation. Twenty countries will be studied through a light case study approach consisting of desk reviews and a limited number of interviews with Bank Group teams. Six countries will be studied in-depth, including with a country visit and interviews with partners and clients. Factors for consideration include the following: • Country characteristics: notably, country capacity (Country Policy and Institutional Assessment score, Worldwide Governance Indicators, Open Data Barometer, Statistical Capacity Indicator, and so on); fragility status; aid dependency and significance of the Bank Group in the country’s budget; and income group • Bank Group country engagement and portfolio characteristics: notably, portfolio composition (portfolios dominated by advisory services and analytics and reimbursable advisory services or IFC advisory services versus portfolios with a majority of development policy financing, versus portfolios with a mix of instruments, as well as size of IFC portfolio) and country engagement experience (for example, number of cycles completed and Bank performance ratings) • Bank Group country teams’ characteristics: including size and composition of country teams 15 5.2 The evaluation will undertake three types of data collection and analysis as shown in figure 5.1: (i) structured document review of country engagement cycles, (ii) key informant interviews and facilitated group discussions, and (iii) structured literature review and expert consultations. The evaluation design matrix in appendix A summarizes the design. The evaluation seeks to generate findings that cut across country engagements, while grasping the differences across types of portfolios and country contexts. To do this well, we rely on the typology of country portfolios and contexts and have selected enough cases to enable comparison within categories of portfolios and to identify any challenges or opportunities that cut across country engagements. Figure 5.1. Evaluation Design Source: Independent Evaluation Group. 5.3 The evaluation focuses on the 52 countries with at least three steps of the new country engagement cycle completed between FY14 and FY20. To date, no country has completed the full SCD-CPF-PLR-CLR cycle within the new model. Consequently, the evaluation will consider two groups of countries: (i) countries that have completed the first part of the cycle (SCD-CPF-PLR) within the new model and (ii) countries that have 16 completed the second part of the cycle (PLR-CLR-CPF) within the new model. Figures 5.2 and 5.3 provide the breakdown of this sample by Region and type of engagement. The full list of countries and the emerging typology is presented in appendix B. Figure 5.2. Countries with Three Steps or More Completed in Engagement Cycle FY14– 20 by Region Africa 5 6 East Asia and Pacific 2 2 1 Europe and Central Asia 5 11 Latin America and the Caribbean 7 4 1 Middle East and North Africa 2 1 South Asia 3 2 SCD-CPF-PLR PLR-CLR-SCD-CPF PLR-CLR-SCD-CPF-PLR Source: Independent Evaluation Group, based on Operations Policy and Country Services data. Note: CLR = Completion and Learning Review; CPF = Country Partnership Framework; PLR = Performance and Learning Review; SCD = Systematic Country Diagnostic. Figure 5.3. Countries with Three Steps or More Completed in Engagement Cycle FY14– 20 by Engagement Type Source: Independent Evaluation Group, based on Operations Policy and Country Services data. Note: CLR = Completion and Learning Review; CPF = Country Partnership Framework; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; PLR = Performance and Learning Review; SCD = Systematic Country Diagnostic. 5.4 A structured document review of country engagement cycles will guide the analysis. The objective of the structured document review is to assemble trace evidence of outcome measurement and management in country engagements and identify the distribution of practices across countries. The evaluation team has assembled a database of key documents that pertain to the country engagement cycles in the period spanning 17 FY14–20. The list of documents comprises: available IFC private sector diagnostics and IFC country strategies; CPFs (specifically their results chain and results frameworks); available PLRs and CLRs; CPPRs and IFC business reviews; data from the Systematic Operations Risk-Rating Tool for the CPFs; and any other documents of an evaluative nature. Part of the structured review will consist of classifying results chains, indicators, and evaluative evidence, and adaptation to country portfolios. In addition, in a select number of country cases, the evaluation team will study key decision moments that pertain to the management of country engagement results, such as minutes of Regional Operation Committee meetings. In addition, the team will conduct a sector-level analysis, examining sectors that are key to the achievement of the CPF outcomes. Selection criteria are outlined in appendix B. To identify markers of strong outcome orientation, the evaluation team will use guidance documents developed by OPCS and Development Effectiveness Units based in Regions as a starting point. 5.5 Semistructured interviews and facilitated group discussions: Given that much of the outcome-oriented behaviors and decisions of country teams are not captured in formal documents, the evaluation methods will focus on eliciting the tacit knowledge of teams. Through surveys, interviews, and facilitated group conversation, the evaluation will identify and amplify informal practices. Among the universe of 52 countries, a sample of countries will be selected to conduct interviews with country teams. This sample will cover the full typology of country engagements and include at least four countries per type to allow for comparison of approaches. In each category, the evaluation will identify a “typical case” to conduct more in-depth inquiry with a country visit, which will include interviews with country clients. Client interviews will be conducted to identify what value the Bank Group’s interventions bring to clients beyond financial support and whether this value is captured well by current results frameworks. The client interviews will also touch on the extent to which the Bank Group uses and supports clients’ existing results and evaluation systems. 5.6 Identification of alternative outcome management and measurement practices: The evaluation team will review the gray and academic literature and consult with experts on outcome orientation within and beyond development organizations. The objective is to identify how various industries have approached the challenges of promoting outcome-oriented behaviors in various organizational contexts. The core streams of literature may include policy innovation, administrative science, public service delivery, government labs, and deliverology. The evaluation team will also explore bodies of literature that grapple with outcome-oriented behaviors, incentives, and motivations outside of public service delivery, such as medicine and business administration. 18 5.7 The potential limitations of this evaluation relate to two main challenges. First, issues with the construct validity and reliability of findings may arise. As laid out in paragraph 4.4, the formal results reporting system does not fully reflect the existing results management practices of country teams. Thus, gauging the outcome orientation of country engagements goes beyond reviewing results frameworks, monitoring data, and performance reports. It requires tapping into the tacit knowledge and experience of World Bank staff and management and key stakeholders among clients and partners. The evaluation team will experiment with various interview and dialogue techniques, including “whole system in the room” workshops. A second challenge relates to the transferability of the lessons emerging from the evaluation finding. The evaluation team will need to create meaningful typologies. 6. Engagement, Communication, and Dissemination 6.1 The evaluation team sets out to engage meaningfully and regularly with a range of key stakeholders within the Bank Group through three mechanisms: • First, the evaluation process is designed as a collaborative engagement: As laid out in section 5, the evaluation design has several features that rely on participatory principles. For instance, the evaluation team will work closely with country teams from a select number of countries to capture the factors that influence their outcome orientation. Focus group discussions, structured brainstorming sessions, and design workshops will be organized. The evaluation team will also engage with key internal stakeholders who have both power and interest in strengthening the outcome orientation of Bank Group country engagements, notably, Development Effectiveness Units within Regions, OPCS and IFC country economics teams, the Poverty Global Practice team in charge of SCDs, task team leaders of CPFs, PLRs, and CLRs. In addition to strengthening the validity of the evaluation findings, the objective of these engagement modalities is twofold: creating ownership in the evaluation and fostering process use opportunities. • Second, frequent communication with Bank Group management and Board will underpin the evaluation process: The IEG work program emphasizes that strengthening the Bank Group’s outcome orientation requires close collaboration between Bank Group management and IEG. This evaluation is one of the instruments of this collaboration. Notably, this evaluation will be one of the vehicles for technical exchanges and creating shared understanding on what it means for an organization to be outcome oriented and what it takes to achieve this objective. The evaluation team will provide key inputs to these consultations that will be convened at the director or vice president level. The evaluation will 19 be presented to the Board in September 2020. Bilateral communication and briefing to Board members will take place before this. • Third, the evaluation dissemination is intended as a platform for convening a reflection on the results agenda with partners: In the era of the Sustainable Development Goals, many other international organizations and bilateral development agencies are looking for new ways of measuring and managing their contribution to development outcomes. Many are looking to the World Bank to lead on this agenda. Lessons emerging from this study will be shared with key partners at relevant forums. 7. Team, Budget and Timeline 7.1 The team members for the evaluation are April Connely, Belen Barbeito, Eduardo Fernandez-Maldonado, Estelle Raimondo (task team leader), Gaby Loibl, Juan- Carlos Mendes, Stephen Hutton, and Xiaoxiao Peng. The work will be conducted under the guidance and quality assurance of Jos Vaessen (methods advisor), Galina Sotirova (manager), Oscar Calvo-Gonzalez (director), and Alison Evans (Director-General, Evaluation). 7.2 The external reviewers for this evaluation are Dr. Ruth Levine, internationally recognized leader in the area of impact evaluation, development finance, and organizational learning and currently a policy fellow at the Center for Global Development, past deputy administrator at the U.S. Agency for International Development, and program director at Hewlett Foundation, with deep expertise in designing and evaluating projects at the World Bank; Professor. Frans Leeuw, Professor of Law, Public Policy, and Social Science Research at Maastricht University and an international expert in monitoring and evaluation systems; and Dr. Ben Ramalingam, international expert on adaptive management in international development, executive director of the Global Learning Alliance on Adaptive Management, and senior fellow at the Overseas Development Institute. 7.3 The estimated budget for delivering this evaluation is $474,000, including $30,000 for dissemination in FY21. A 10,000-word report will be e-submitted at the end of August 2020 and presented to the Board in September 2020, according to the process and timeline outlined in table 7.1. 20 Table 7.1. Evaluation Timeline Process Proposed Date Initiation 09/01/19 IEG one-stop review of draft approach paper chaired by 10/10/19 Alison Evans (DGE) Send draft approach paper to World Bank management 11/04/19 Receipt of management comments 12/06/19 Final approach paper to World Bank technical counterpart 12/23/19 and management Approach paper e-submission 12/27/19 IEG one-stop review of draft report chaired by Alison Evans 06/15/20 (DGE) Send draft report to World Bank management 07/06/20 Expected receipt of comments from World Bank 08/01/20 management Final report e-submission 08/31/20 21 References ADB (Asian Development Bank, Independent Evaluation). 2012. Managing for Development Results Relevance, Responsiveness, and Results Orientation. Mandaluyong City, the Philippines: ADB. Andrews, Matt. 2018. “Public Policy Failure: How Often? and What is Failure, Anyway? A Study of World Bank Project Performance.” CID Working Paper 344, Center for International Development at Harvard University, Cambridge, MA. Bemelmans-Videc, Marie-Louise, Ray C. Rist, and Evert Vedung. 1998. Carrots, Sticks and Sermons, Policy Instruments and Their Evaluation. New Brunswick, N.J., U.S.A. : Transaction Publishers Birdsall, Nancy, and Devesh Kapur. 2016. “The Hardest Job in the World: Five Crucial Tasks for the New President of the World Bank.” In Rescuing the World Bank: A CGD Working Group Report and Selected Essays, edited by Nancy Birdsall, 3–65. Washington, DC: Center for Global Development. Buffardi, Anne L., Paige Mason, Claire Hutchings, and Samuel Sharp. 2019. “Monitoring and Learning for Country-Level Portfolio Decision-Making and Adaptation.” Briefing Note (May), Overseas Development Institute, London. Buntaine, Mark T., Bradley C. Parks, and Benjamin P. Buch. 2017. “Aiming at the Wrong Targets: The Domestic Consequences of International Efforts to Build Institutions.” International Studies Quarterly 61 (2): 471–88. https://doi.org/10.1093/isq/sqx013. Busan Partnership for Effective Development Co‐Operation. 2011. Busan High Level Forum on Aid Effectiveness: Proceedings. Busan, Republic of Korea. Dahler-Larsen, P. 2013. The Evaluation Society. Stanford, CA: Stanford University Press DFID (U.K. Department for International Development). 2009. “Evaluation of The Implementation of the Paris Declaration Thematic Study of Support to Statistical Capacity Building Synthesis Report.” Synthesis Report. London: DFID. Gaarder, M., and U. Bartsh. 2014. “Who Cares about Development Outcomes? Market Failures and the Role of the Evaluation Function.” Journal of Development Effectiveness 6 (4): 361–77. García López, Roberto, and Mauricio García Moreno. 2011. Managing for Development Results: Progress and Challenges in Latin America and the Caribbean. Washington, DC: Inter- American Development Bank. Global Partnership for Effective Development Cooperation. 2016. Nairobi Outcome Document. Nairobi, Kenya. 22 Honig, Dan. 2019. “When Reporting Undermines Performance: The Costs of Politically Constrained Organizational Autonomy in Foreign Aid Implementation.” International Organization 73 (1): 171–201. Højlund, S. 2014. “Evaluation Use in Evaluation Systems: The Case of the European Commission.” Evaluation 20 (4), 428–46. https://doi.org/10.1177/1356389014550562 Leeuw, F. L., and Furubo, J.-E. 2008. “Evaluation Systems: What Are They and Why Study Them?” Evaluation 14 (2): 157–69. https://doi.org/10.1177/1356389007087537. Levine, R., and W. D. Savedoff. 2006. “The Evaluation Agenda.” In Rescuing the World Bank: A CGD Working Group Report and Selected Essays, edited by Nancy Birdsall and Devesh Kapur, 183–194. Washington, DC: Center for Global Development. OECD (Organisation for Economic Co-operation and Development). 2016. Evaluation Systems in Development Co-operation: 2016 Review. Paris: OECD. https://doi.org/10.1787/9789264262065-en. ———. 2019. “Managing for Sustainable Development Results: Concept and Preliminary Draft Guiding Principles.” Discussion Paper for the OECD/DAC Results Community Workshop, April 4–5, 2019, Paris. http://www.oecd.org/dac/results- development/docs/Draft_Guiding_principles_RBM_Mar_19.pdf OECD and World Bank. 2014. Measuring and Managing Results in Development Cooperation. A Review of Challenges and Practices among DAC Members and Observers. Paris: OECD. Porter, S., and P. Hawkins, P. 2019. “Achieving Sustainability through Sustainable Organizational Evaluation Systems.” New Directions for Evaluation 162: 87–101. Ramalingam, B., L. Wild, and A. Buffarid. 2019. Making Adaptive Rigour Work: Principles and Practices for Strengthening Monitoring, Evaluation and Learning for Adaptive Management. Briefing Note, ODI, London. Vähämäki, J., M. Schmidt, and J. Molander. 2011. Results Based Management in Development Cooperation. Stockholm: Riksbankens Jubileumsfond. Vähämäki, J., and C. Verger. 2019. “Learning from Results-Based Management Evaluations and Reviews.” OECD Development Co-operation Working Papers 53, OECD Publishing, Paris. https://doi.org/10.1787/3fda0081-en. World Bank. 2009. Emerging good practice in managing for development results (English) . http://documents.worldbank.org/curated/en/909681468339685140/Emerging-good- practice-in-managing-for-development-results ———. 2012. World Bank Group Impact Evaluations Relevance and Effectiveness. Independent Evaluation Group. 23 ———. 2013. Assessing the Monitoring and Evaluation Systems of IFC and MIGA Biennial Report on Operations Evaluation. Independent Evaluation Group. ———. 2014. Learning and Results in World Bank Operations: How the World Bank Learns. Independent Evaluation Group. ———. 2014b. World Bank Group Directive: Country Engagement. http://siteresources.worldbank.org/OPSMANUAL/112526- 1124459412562/23587333/DirectiveCountryEngagement(July2014).pdf ———. 2016a. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Independent Evaluation Group. ———. 2016b. Program-for-Results: An Early-Stage Assessment of the Process and Effects of a New Lending Instrument. Independent Evaluation Group. ———. 2017a. The International Finance Corporation’s Approach to Engaging Clients for Increased Development Impact. Independent Evaluation Group. ———. 2017b. World Bank Group Country Engagement: An Early-Stage Assessment of Systematic Country Diagnostic and Country Partnership Framework Process and Implementation . Independent Evaluation Group. ———. 2017c. World Bank Group Joint Projects: A Review of Two Decades of Experience. Lessons and Implications from Evaluation. Independent Evaluation Group. ———. 2017d. Results and Performance of the World Bank Group 2016. Independent Evaluation Group. World Bank: Washington, DC. ———. 2018a. Enhancing Self-evaluation in the World Bank Group Towards Achieving Better Performance through Monitoring and Evaluation (M&E) and Results Measurement. An IEG Retrospective Review. ———. 2018b. IBRD/IDA/IFC/MIGA Guidance Country Engagement. World Bank: Washington, DC. ———. 2018c. Report to Governors at 2018 Spring Meetings Sustainable Financing for Sustainable Development: World Bank Group Capital Package Proposal. World Bank: Washington, DC. ———. 2018d. Results and Performance of the World Bank Group 2017. Independent Evaluation Group. ———. 2018e. “World Bank/IMF Spring Meetings 2018: Development Committee Com muniqué.” Press Release, April 21. ———. 2019. “Assurance Review of the Use of the Bank’s Corporate Scorecard.” GIA Report IBRD FY19–08 July 16, World Bank, Washington, DC. 24 ———. 2019. “Development Results in the World Bank Group.” Executive Directors’ Seminar. World Bank Group, July 29, Washington, DC. ———. 2019. Bank Directive Program-for-Results Financing. Bank Access to Information Policy Designation. World Bank: Washington, DC. ———. 2019. CLR: Assessing Results and How to Use Lessons Learned–The India CLR. World Bank: Washington, DC. ———. 2019. “A Look into the New World Bank Group and World Bank Corporate Scorecards. ” Corporate Secretariat Technical Briefing, Wednesday, July 31, 2019, World Bank, Washington, DC. ———. 2019. Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity. Independent Evaluation Group. ———. 2019. Including a Lens on Fragile, Conflict and Violent Situations in CPFs. World Bank: Washington, DC. ———. 2019. Learning and Results in World Bank Operations: Toward a New Learning Strategy. Evaluation 2. Independent Evaluation Group. ———. 2019. World Bank Group and World Bank Corporate Scorecards. World Bank: Washington, DC. ———. 2019. The IDA 19 Results Measurement System. World Bank: Washington, DC. ———. 2019. The World Bank Group’s Results and Performance 2018. World Bank: Washington, DC. ———. 2019. IDA19 Second Replenishment Meeting Addis Ababa, Ethiopia, June 18 –20, 2019: Co- Chairs Summary. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/418181566241256305/IDA19-Second- Replenishment-Meeting-Addis-Ababa-Ethiopia-June-18-20-2019-Co-Chairs-Summary. ———. 2019. Program-for-Results: Proposal to Remove the Cap on Commitment Authority. Washington, DC: World Bank. http://documents.worldbank.org/curated/en/188011557406617235/Program-for-Results- Proposal-to-Remove-the-Cap-on-Commitment-Authority ———. 2019. “Preparing a CPF: Practical Advice to Make This a Successful Experience.” Peter Siegenthaler, Economic Adviser, OPSPQCPF Academy, Washington, DC, World Bank. Zwart, R. 2017. "Strengthening the Results Chain: Synthesis of Case Studies of Results-Based Management by Providers.” OECD Development Policy Paper 7, Organisation for Economic Co-operation and Development, Paris. Publishing. https://doi.org/10.1787/544032a1-en. 25 Appendix A. Evaluation Design Matrix Table A.1. Evaluation Design Matrix Data Collection and Analysis Key Questions Methods Sampling and Selection Limitations Overarching evaluation question: What influences outcome management and measurement practices across World Bank Group country engagements? EQ1. What factors explain differences in outcome management and measurement practices across Bank Group country engagements, and what can be learned from these differences? 1a. What are the patterns of outcome Structured document review For the universe of country engagement Formal results orientation exhibited by Bank Group applying a template for coding key cycles that started after FY13, the team documents only partially country engagements? elements of the country will review key documents (CPF results capture the reality of engagement outcome orientation chain and results frameworks, IFC results measurement and strategies and business plans, PLR, CLR if management practices available, Country Portfolio and Performance underlying country Reviews, annual reviews of IFC strategies, engagement cycles and so on). For a purposive sample of country engagement cycles, the team will also review trace evidence of key decision moments (for example, Concept Note reviews, and Regional Operations Committee and Board minutes) 1b. What challenges, trade-offs, Structured document review See above Biases that are inherent and incentives do Bank Group Interviews, survey, and workshops Purposive selection of country in interviews (such as teams currently face that hinder with World Bank staff and engagement cycle cases for which in- recall bias, social a strong outcome orientation at management depth interviewing will be conducted desirability bias, and so the country level? Interviews with country client with key informants based on on) will need to be counterparts stakeholder mapping carefully managed. Process tracing techniques to assess the probative value of 26 Data Collection and Analysis Key Questions Methods Sampling and Selection Limitations various interview evidence will be applied 1c. What conditions (behavioral and Structured document review See above The biases laid out above are organizational) are exhibited when Interviews, survey, and workshops with See above less likely in a scenario of country engagements reach a strong World Bank Group staff and good outcome orientation outcome orientation? management Interviews with country client counterparts EQ2. What are the opportunities for strengthening the outcome orientation of Bank Group country engagements, and what behaviors, capacities, and resources would be necessary? 2a. What specific approaches and Structured literature review Snowballing sampling of core publications Identifying and bounding methodologies seem promising to Expert consultations outcome orientation in various industries the right type of literature overcome the context-specific Experts in evaluation and results-based challenges faced by Bank Group management will be selected based on teams and clients? network sampling techniques, with the goal of maximizing heterogeneity of views 2b. Under what circumstances might Structured feedback gathering with Sample of World Bank and IFC teams, country Feasibility of feedback these approaches be applicable or internal stakeholders economics and results teams, Development gathering will depend in part replicable? Effectiveness units in Regions, other key on the willingness of country stakeholders, IFC sector and country teams to participate economists, IFC Regional and sector directors Note: CLR = Completion and Learning Review; CPF = Country Partnership Framework; CPPR = Country Portfolio and Performance Review; IFC = International Finance Corporation; PLR = Performance and Learning Review. 27 Appendix B. Case Selection There will be two types of cases in the evaluation: 1. Light case study: conducted on up to 20 countries (to reach at least 3–4 cases per type of countries in the overall typology). These will comprise a desk-based analysis and interviews with 3–5 key informants from the World Bank and International Finance Corporation (IFC) country teams in person or on the phone. 2. In-depth case study: conducted on up to six countries (one per main type of country). These will comprise a desk-based analysis, interviews with a wider range of key informants within the Bank Group, and a country mission to consult with clients and partners. Table B.3 provides a list of key informants that will be consulted for each type of case. Country selection approach: The evaluation team is creating a typology of countries that overlay two broad sets of characteristics: 1. World Bank and IFC portfolio compositions; 2. Country type (International Bank for Reconstruction and Development; fragile and conflict-affected situation [FCS], and International Development Association non-FCS) and other country characteristics that relate to a country capacity and readiness for outcome measurement and management. The emerging typology is presented in table B.1 with the selection of case studies. Table B.2 also recaps by Region. Other considerations for country selection: • Tacit knowledge of key stakeholders that have pointed the evaluation team toward countries that have been particularly outcome oriented • Countries that have an IFC strategy and a progress review exercise • Balancing between the two groups of country cycles • Oversampling for fragile and conflict-affected situations because this is part of the IEG workstream, and there was a specific request from Bank Group management to pay specific attention to the need and challenges of outcome management and measurement in fragile contexts 28 • Practical considerations: Maximizing value for money of country visits, the team will target hubs whenever possible and use synergies with other ongoing IEG evaluations (for example, the Chad Country Program Evaluation) Sector analysis within the country portfolio: For all in-depth case studies and for light case studies of countries with particularly large portfolios, a sector-level analysis will be conducted to examine how portfolios of interventions within sectors that are critical to the achievement of Country Partnership Framework outcomes are managed and how their contribution to outcomes are measured. The selection will be based on the following criteria: • Sector is identified as critical to achieving Country Partnership Framework outcomes • Sector portfolio contains multiple and a variety of Bank Group interventions • Balance of various types of sectors across countries to reach relative representativeness 29 Table B.1. Countries Organized by World Bank Group Portfolio Composition SCD-CPF-PLR PLR-CLR-SCD-CPF PLR-CLR-SCD-CPF-PLR IFC Investment Country World Bank IFC investment IFC investment IFC advisory Type Lending Projects dominated and advisory only No IFC projects IBRD IPF only Uruguay Bolivia, non-FCS Bosnia and Argentina, Herzegovina, North El Salvador Macedonia Belarus IPF and DPF Armenia Bosnia and Albania Herzegovina Indonesia, Serbia, Sri Lanka, Tunisia Croatia, Georgia, India Peru, Vietnam DPF significant Guatemala, Colombia, Poland, Egypt, Arab Rep., Romania Panama Morocco, Paraguay No projects Bulgaria IDA non- IPF only Nicaragua, Lesotho Maldives FCS Tajikistan, Zambia IPF and DPF Cameroon, Niger Honduras, Uzbekistan Benin, Burkina Faso, Guinea, Ghanaa, Kyrgyz Republic, Moldova, Nepal, Senegala 30 IFC Investment Country World Bank IFC investment IFC investment IFC advisory Type Lending Projects dominated and advisory only No IFC projects IDA FCS IPF only Papua New Guinea IPF and DPF Afghanistan, Chad Côte d’Ivoire, Haiti Kosovo, Myanmar, Solomon Islands DPF significant Burundi Note: IBRD-IDA lending group is based on the list of economies listed by the World Bank Group in June 2019. Country fragile and conflict-affected situation (FCS) status is based on the World Bank Group’s FY19 list of fragile situations. IPF and DPF: Among IPF and DPF projects, IPF projects count greater than 70 percent and less than 100 percent. DPF significant: Among IPF and DPF projects, DPF projects count 30 percent or more. IFC investment dominated: Among IFC projects, investment projects count 90 percent or more. IFC investment and advisory: Among IFC projects, investment projects count greater than 0 percent and less than 90 percent. IFC advisory only: No IFC investment projects. IDA lending group is based on the World Bank Group’s list of economies in June 2019. CLR = Completion and Learning Review; CPF = Country Partnership Framework; DPF = development policy financing; FCS = fragile and conflict-affected situation; IBRD = International Bank for Reconstruction and Development; IDA = International Development Association; IFC = International Finance Corporation; IPF = investment project financing; PLR = Performance and Learning Review; SCD = Systematic Country Diagnostic. a. Ghana and Senegal will have their CPF discussed by the Board of Executive Directors in the third quarter of FY20 and are included to correct for sample bias in the Africa Region. 31 Table B.2. Country Cases by Region Region SCD-CPF-PLR PLR-CLR-SCD-CPF PLR-CLR-SCD-CPF-PLR Africa Botswana Benin Cameroon Burkina Faso Chad Burundi Côte d’Ivoire Guinea Ghanaa Lesotho Niger Senegala Zambia East Asia and Pacific Indonesia Papua New Guinea Vietnam Myanmar Solomon Islands Europe and Central Asia Albania Armenia Bosnia and Herzegovina Belarus Bulgaria Croatia Serbia Georgia Uzbekistan Kosovo Kyrgyz Republic Moldova North Macedonia Poland Romania Tajikistan Latin America and the Bolivia Argentina Peru Caribbean Colombia Guatemala El Salvador Nicaragua Haiti Paraguay Honduras Panama Uruguay Middle East and North Egypt, Arab Rep. Morocco Africa Tunisia South Asia Afghanistan India Maldives Nepal Sri Lanka Note: CLR = Completion and Learning Reviewl CPF = Country Partnership Framework; PLR = Performance and Learning Review; SCD = System Country Diagnostic. Country names that are underlined correspond to the sample of light case studies and country names that are in bold correspond to in-depth case studies. a. Ghana and Senegal will have their CPF discussed by the Board of Executive Directors in the third quarter of FY20 and are included to correct for sample bias in the Africa Region. 32 Table B.3. Key Informants In-Depth Case Studies Light Case Studies Informant Necessary Optional Necessary World Bank country team Country manager ✓ ✓ Country director ✓ Country Partnership Framework task team ✓ ✓ leader Performance and Learning Review or ✓ ✓ Completion and Learning Review task team leader Country program coordinator ✓ Country operation officer ✓ ✓ Country economist ✓ Program leaders ✓ Regional development effectiveness unit ✓ IFC IFC country manager ✓ ✓ IFC country officer ✓ ✓ IFC investment officer for any sectors with ✓ large exposure IFC Regional director ✓ IFC Regional industry directors for any ✓ sectors with large exposure IFC manager covering Region in Country ✓ Economics Unit Investment officers and operations officer ✓ within industry that are key contributors to IFC country strategies and Country Partnership Framework outcomes Government Formal counterpart (for example, head of ✓ international financial institution division in ministry of finance, ministry of planning) Other relevant staff for finance and planning ✓ ministries Head of project management office (when a ✓ single project management office manages all World Bank projects) 33 In-Depth Case Studies Light Case Studies Informant Necessary Optional Necessary National statistical office ✓ Government evaluation functions as exist ✓ Office of planning in line ministries for any sectors with large World Bank engagements Coordinating ministries that might run monitoring and evaluation for multiple ministries Delivery units or pilot programs within ministries that might have monitoring and evaluation functions Office of the president subunit for priority initiatives Government by results office, sometimes in vice presidential offices Others Regional director of strategy and operations ✓ Practice manager and lead technical ✓ specialist(s) for any Global Practices with very large engagements Executive director or adviser for relevant ✓ constituency Highest level in-country official for ✓ development partners who have significant country presence (especially if operating in same sectors or with harmonized budget support or other programs) Civil society organizations in charge of ✓ monitoring and assessing Country Partnership Framework progress in countries where this mechanism exists Note: IFC = International Finance Corporation. 34 Appendix C. Outcome Orientation Definition and Dimensions What Does It Mean for an Organization to Have a Strong Outcome Orientation? An outcome-oriented organization promotes a shared view among staff, management, shareholders, and clients that success means • Overcoming complex and dynamic development challenges. It involves catalyzing behavioral, institutional, and social changes that take place over the medium to long term, usually as the result of investments in organizational capacity for key development stakeholders (such as national governments, civil society, the private sector) or directly for the target population. Responding to complex development challenges demands contextual learning, which requires…. • Adopting monitoring, evaluation, and learning practices, a set of approaches, tools, staff capacities, and attributes, as well as relationships and incentives that are mobilized to systematically gather data and evidence on the organization ’s contribution to development outcomes and generate feedback on what is working, what is not working, why, and how can it be fixed. • Using feedback on outcomes to adapt programs and portfolios. This requires a set of incentives, processes, and structures that supports and encourages staff and clients to apply evaluative thinking and collective decision-making to manage and adapt interventions, portfolios, and strategies. This has implication for how to engage with clients and partners… • Ensuring outcome-oriented client engagements, which means supporting client countries’ capacity to monitor, evaluate, and manage change in their key development outcome areas. Prioritizing development and use of clients’ results frameworks, data, and evaluation systems, working with partners to avoid creating parallel systems that undermine this objective, and provide spaces to co-create and use outcome information. Source: Independent Evaluation Group, based on review of literature on adaptive management and management for development results. 35 36 Figure C.1. Outcome Orientation Practices 37