Management Action Record Reform IEG’s Validation Report © 2020 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington, DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org ATTRIBUTION Please cite the report as: World Bank. 2020. Management Action Record Reform: IEG’s Validation Report. Independent Evaluation Group. Washington, DC: World Bank. COVER PHOTO Adapted from shutterstock/inter reality This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The bound- aries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. RIGHTS AND PERMISSIONS The material in this work is subject to copyright. Because The World Bank encourages dissem- ination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e-mail: pubrights@worldbank.org. Management Action Record Reform IEG’s Validation Report August 31, 2020 Contents Abbreviations ..................................................................................................................................................... v Acknowledgments ........................................................................................................................................... vi Overview ............................................................................................................................................................vii Report to the Board from the Committee on Development Effectiveness ............................... ix 1. Context for the Management Action Record Reform ....................................................................1 2. Redesigning the Management Action Record System ..................................................................3 3. The Transition to a New Management Action Record System...................................................7 MAR Reporting in FY20.............................................................................................................................. 7 Implementation Progress on FY15–19 Evaluation Recommendations........................................ 7 Transitioning the Recommendations .................................................................................................... 7 Disagreements on Specific Transitions ............................................................................................... 13 Lessons on Process.................................................................................................................................... 15 4. Enhancing the Value Added of IEG Recommendations ............................................................. 17 Insights on Evaluation Topics................................................................................................................. 17 Insights on Recommendation Types ................................................................................................... 18 5. Conclusions ................................................................................................................................................. 23 References ........................................................................................................................................................ 25 Boxes Box 3.1. Examples of Applying Criteria for Deciding Which Recommendations to Retire ........8 Box 3.2. Postevaluation Follow-up to the Capital Markets Evaluation .............................................9 Box 3.3. Lessons on Process ......................................................................................................................... 16 Box 4.1. Engagement in Tough Places ...................................................................................................... 18 Box 4.2. Lessons for IEG on Recommendations ................................................................................... 22 Figure Figure 4.1. Focus of Independent Evaluation Group Recommendations ...................................... 19 iii Contents Tables Table 2.1. Overview of Proposed Management Action Record Reforms ........................................4 Table 3.1. IEG Evaluations Covered by the Management Action Record Reform and Transition Proposals ....................................................................................................................................... 10 iv Abbreviations AIMM Anticipated Impact Measurement and Monitoring CODE Committee on Development Effectiveness FY fiscal year IEG Independent Evaluation Group IFC International Finance Corporation MAR Management Action Record PforR Program-for-Results All dollar amounts are US dollars unless otherwise indicated. v Acknowledgments This report was led by Rasmus Heltberg under the supervision of Sophie Sirtaine and Alison Evans. Luis Alvaro Sanchez was a member of the core team. Many Independent Evaluation Group evaluation team leaders were consulted and made valuable contributions. Zorayda Opana provided administrative support. vi Overview The Management Action Record (MAR) is a key element of the World Bank Group’s accountability framework. The MAR supports accountability in the follow-up of Independent Evaluation Group (IEG) evaluation recommendations by enabling meaningful tracking, dialogue, and self-assessment of Bank Group management’s implementation of IEG recommendations. This document summarizes recent MAR reforms and validates management’s self- assessment. The MAR reforms aim to address key shortcomings in the previous MAR system, including tedious processes, limited engagement of senior management and Board members, and weak evidence of recommendation implementation. The MAR reform aims to address these shortcomings through better tracking, dialogue, and reporting on progress toward the outcomes sought by IEG recommendations. The MAR reform resulted in agreement between IEG and management for the following: • Future IEG recommendations will be fewer and more strategically focused. • Management will clearly state their agreements and disagreements with the recommendations. • Management will report annually on progress toward the recommendations’ intended outcomes. • The existing stock of IEG recommendations from FY15–19 will be transitioned to the new MAR system as follows: 6 evaluations from FY16–19 will be fully transitioned into the new MAR system; 4 evaluations will be partially transitioned, meaning that some recommendations will be retired and some recommendations will be reported; 10 evaluations from FY16–18 will be fully retired; and 6 FY15 evaluations have completed their standard reporting period and will be retired. • Management will also report in the MAR on some cross-cutting thematic areas in which multiple evaluations recommended action, including on monitoring and evaluation, data, and knowledge management. • For two evaluations, IEG and management did not reach agreement, and the Committee on Development Effectiveness’s advice is sought on these disagreements. IEG did not concur with management’s proposal to retire all the recommendations from Growing the Rural Nonfarm Economy to Alleviate Poverty vii Overview and four of the recommendations from Toward a Clean World for All: An IEG Evaluation of the World Bank Group’s Support to Pollution Management primarily because of insufficient evidence to validate management’s stated reasons for stopping reporting. The MAR reform also resulted in insights and lessons on how to enhance the value added of IEG recommendations, including evaluating topics of growing importance and early implementation progress with new approaches and instruments, challenging perceived views, taking strategic perspectives, and making “big-picture” recommendations geared to inform key decision points. viii Report to the Board from the Committee on Development Effectiveness The Committee on Development Effectiveness met to consider the Independent Evaluation Group (IEG) report on World Bank Group management, Implementing IEG Recommendations FY15–19, and IEG’s validation report, Management Action Record Reform. The committee welcomed the World Bank Group management report and IEG’s validation report. Members commended IEG and management for their constructive engagement and the progress made to strengthen the impact of IEG's evaluations by improving the Management Action Record (MAR) system. Members supported the new approach, noting it will enable a better alignment of IEG’s evaluations to the Bank Group’s corporate commitments and institutional priorities and will reinforce the committee’s oversight by ensuring implementation of more focused recommendations. They welcomed the new format and encouraged management to continue clustering evaluations thematically in their self-assessment. Members supported management and IEG’s views in favor of disclosing the management and validation reports. Members acknowledged that the new approach will facilitate more-articulate reporting from management on how it is moving toward achieving the desired outcomes from implementing IEG recommendations and, ultimately, enhance the institution’s development effectiveness. However, some members raised concern about the difficulties of monitoring and following-up on progress with the discontinuation of Management Action Plans. Management explained that, under the new system, management’s responses to IEG evaluations will focus on providing a path for achieving the outcomes rather than establishing activity-based plans with modest relevance to achieving the desired outcomes. Management highlighted that the annual MAR report will help engage members of the committee on management’s progress and the path it is taking to achieving the desired outcomes. Members noted the potential for improving learning and feedback loops through the new MAR system. They acknowledged the challenges of capturing the changes that have taken place and with what effect, given the evidence base available. They highlighted the importance of assessing the effectiveness of the MAR system and to course-correct on the MAR system’s implementation when appropriate. Members agreed with Bank Group management and IEG’s proposal on how to review and deal with the backlog of FY15–19 IEG recommendations. Regarding the recommendations pertaining to the Growing the Rural Nonfarm Economy to Alleviate ix Report to the Board from the Committee on Development Effectiveness Poverty and Toward a Clean World for All evaluations, members appreciated management’s commitment to continuing to reporting on these recommendations. x 1. Context for the Management Action Record Reform The Management Action Record (MAR) plays an important role in the relationships among the Independent Evaluation Group (IEG), management, and the Board of Executive Directors. As such, the MAR system is a key element of the World Bank Group’s accountability framework. Specifically, the MAR’s purpose is to support accountability in the follow-up of IEG evaluation recommendations by enabling meaningful tracking, dialogue, and self-assessment of Bank Group management’s implementation of IEG recommendations. The Bank Group’s Board of Executive Directors, IEG, and management decided in fiscal year (FY)19 to reform the MAR system because it was not working well. Implementation of the action plans that management made in response to IEG recommendations was slow and incomplete, action plans were delayed, and Board members were not kept well informed. IEG and management collaborated during FY20 to reform the MAR system and review the pre-FY20 recommendations. The MAR reform aims to enhance the use value of IEG recommendations, support management’s implementation of the recommendations, and improve the type of information to the Board on evaluation follow-up. These changes are to contribute to the broader purpose of increasing IEG's value added to the Bank Group’s development effectiveness.1 The MAR reform process resulted in agreement between IEG and management on the nature of future IEG recommendations, how to use the MAR process to report on management actions, and how to transition most pre-FY20 evaluations into the new MAR. IEG and management disagreed on how to transition two specific evaluations’ recommendations. This document is IEG’s summary to the Committee on Development Effectiveness (CODE) of the MAR reform and validation of management’s draft self-assessment report (World Bank 2020). The document describes the MAR reform, reviews management’s report on the same process, summarizes areas of agreement and disagreement between IEG and management, seeks CODE’s guidance on the disagreements, and captures insights and lessons from the MAR reform process. This report marks the transition to the new MAR system. It is written for an internal audience; therefore, it assumes some prior knowledge of IEG’s role. Readers unfamiliar with IEG can find background information on IEG’s website.2 The report follows a hybrid structure because it covers both the MAR reform process and the first year of implementing new MAR reporting. The structure of the report is as follows: chapter 2 outlines the proposed future MAR system and explains how its design 1 Chapter 1 Context for the Management Action Record addresses shortcomings in the previous MAR system, chapter 3 outlines the transition to a new MAR system, chapter 4 captures insights and lessons, and chapter 5 offers conclusions. In the future, IEG plans to produce MAR validation reports with more focus on recommendation implementation and without this document’s focus on the MAR reform process. 2 2. Redesigning the Management Action Record System The impetus for the MAR reform was that the system was not working well. As reported in the 2018 Results and Performance Report, after four years, action plans were implemented for only 52 percent of IEG recommendations (World Bank 2019d). There was weak evidence of recommendation implementation and little engagement by senior management, particularly beyond document approval. The Board also had insufficient information to understand what had been implemented, what had not been implemented, and why. There were also significant delays in issuing action plans in response to IEG recommendations, which reduced the Bank Group’s opportunity to make full use of IEG evaluations. The previous MAR system was perceived as tedious. The MAR system was based on action plans with many individual action items listed for each recommendation. Management found the action plans inflexible and unable to allow for changes in context over time. IEG staff found the action plans unambitious in relation to the recommendations and that they were often derived from elements already in management’s work program. CODE did not see or discuss the action plans. Producing and rating the annual updates, or progress reports, on action plans for 25 to 30 evaluations with outstanding recommendations was perceived as tedious. The process involved both management (World Bank, International Finance Corporation [IFC], and Multilateral Investment Guarantee Agency, as applicable) and IEG rating action plan’s progress. Engagement focused on ratings disputes and cases of insufficient evidence on implementation progress. Resolving ratings disagreements and responding to requests for additional evidence was time consuming. Limited engagement by senior management and weak evidence on recommendation implementation have been long-standing weaknesses in the MAR system. Because management does not have mechanisms to gather evidence on the outcomes from management’s implementation of IEG recommendations, updates have focused on discrete actions, not outcomes. Evidence on action plan implementation progress was often weak or anecdotal. The process did not help management, IEG, or the Board understand outcomes from implementing the action plans. The system lacked evidence and incentives for understanding postevaluation outcomes, and instead focused attention on ratings for action plan implementation. Senior management engaged little with IEG, and the work was delegated. This must be understood in a context in which action plans provided a shallow or unambitious response to the recommendations. At times, this may have been because management or part of management did not fully 3 Chapter 2 Redesigning the Management Action Record System buy into a recommendation or felt it was too difficult to implement, as indicated in management’s self-assessment. The MAR reform aims to address the shortcomings of the previous system (see table 2.1). The new MAR process is expected to enable strategic dialogue among IEG, management, and CODE. Management’s annual self-assessments are expected to provide consolidated, strategic, and thoughtful reporting on progress implementing recommendations. Management’s reporting is expected to focus more on outcomes than in the past. Senior levels of management are expected to have awareness, ownership, and accountability for the implementation of its response to IEG evaluations. Management is expected to respond to IEG evaluations with candor and to provide candid MAR self-assessments. Table 2.1. Overview of Proposed Management Action Record Reforms Prereform MAR System New MAR System (starting FY21) Purpose MAR was a tool to provide a The new MAR’s purpose is to support complete, rated account of action accountability in the follow-up of IEG plan implementation. evaluation recommendations by enabling meaningful tracking, dialogue, and self- assessment of World Bank Group management’s implementation of IEG recommendations. IEG evaluations IEG evaluations contained an IEG evaluations will contain fewer average of five recommendations recommendations, which will align more closely with the Bank Group’s strategic linked to the evaluation’s findings. directions and have clear potential to Some recommendations focused on enhance the Bank Group’s development very specific actions, and some effectiveness. contained subrecommendations or Recommendations will be outcome multiple parts. oriented and will usually not have subrecommendations. Management responses Management indicated their Management’s responses will clearly agreement or disagreement with agree or disagree with each each recommendation; partial recommendation. Partial or total agreements were discouraged. disagreements will be stated clearly. However, management’s agreements were sometimes rather cautious with many caveats. Scope of management’s Updates focused on itemized action Action plans will be discontinued. annual MAR reporting plans for the recommendations that Reporting will focus on broad progress management agreed with. In IEG’s toward outcomes recommended by IEG view, action plans were often evaluations (still for those unambitious and not fully aligned recommendations management agreed with the recommendations’ with). intended outcomes. 4 Chapter 2 Redesigning the Management Action Record System Prereform MAR System New MAR System (starting FY21) Additional cross-cutting reporting will cover thematic areas where multiple evaluations have recommended action. Process Annual action plan updates Annual self-assessments will be more produced extensive process-related consolidated, strategically focused on the content, largely among technical- recommendations’ intended outcomes, level staff. and engage higher levels of management. Ratings There were annual ratings for action The new MAR process will not have plan implementation (one rating ratings. per recommendation). Duration The reporting period was up to five The reporting period will not change. years. Reporting took place over four years, starting one fiscal year after a report was discussed at CODE. There was an option to shorten the reporting period. Board of Executive Annual reporting to the Board of Annual reporting to CODE will be via Directors postevaluation Executive Directors was via a management’s self-assessment and IEG’s engagement chapter in the Results and validation of it, with information to the full Performance of the World Bank Board as part of the Results and Group report, based mainly on IEG’s Performance discussion. ratings of action plan implementation progress. Source: Independent Evaluation Group. Note: CODE = Committee on Development Effectiveness; FY = fiscal year; IEG = Independent Evaluation Group; MAR = Management Action Record. Future IEG recommendations will be fewer and aimed at actions with a clear potential to enhance the Bank Group’s development effectiveness. Recommendations will be aligned with the Bank Group’s overall strategic directions. Recommendations will clarify the outcomes sought and will not have subrecommendations. Management’s responses to IEG evaluations will contain a clear statement of agreement or disagreement. Management retains the right to partially agree with IEG recommendations, although IEG discourages the practice. In principle, most of this applied in the previous MAR system. Therefore, IEG evaluations need to focus more judiciously on a few essential recommendations and note other potential areas of improvement as suggestions. Management will not prepare action plans but will instead report against the outcomes sought by the recommendations. Outcome-oriented reporting is intended to enable a focus on management’s progress toward outcomes sought by the evaluation’s recommendations rather than many individual action plan items. It also enables adaptation over time, allowing for new ways to implement recommendations as contexts change. The default reporting period will be four years, starting the fiscal year after CODE discusses the evaluation, with an option to shorten the reporting period 5 Chapter 2 Redesigning the Management Action Record System based on mutual agreement and after presenting a rationale to CODE as part of the annual self-assessment. IEG and management considered rating implementation progress but decided against it. Ratings are not necessary to achieve the MAR system’s purpose of enabling meaningful tracking, dialogue, and reporting on recommendation implementation. It could also be difficult for IEG to rate recommendation implementation on an objective basis. Most rating systems in international development rate against indicators and targets. The future MAR will not contain indicators and targets given the move to report on broad outcomes rather than action plans. Also, many recommendations do not have clear end points, which makes it difficult to assess when they have been fully implemented. For example, there is no end point to the citizen engagement evaluation’s recommendation that “the World Bank should aim to achieve greater depth and quality in citizen engagement.” Management and IEG will share their annual MAR self-assessments and validation reports with CODE for discussion. The full Board will subsequently be informed. Aspects of the MAR system are likely to continue to evolve. For example, the role of CODE and the Board could expand. IEG could increase its efforts to engage with Board members and management on evaluation follow-up by, for example, conducting thematic deep-dive studies and workshops. It would be useful for management’s and IEG’s reporting to CODE and the Board to continue increasing the focus on outcomes that started this year. The purpose would be improved reporting, dialogue, and understanding of the progress implementing IEG recommendations. Better availability of routine evidence could be useful to this end, and management may want to consider improving how it collects evidence on actions taken in response to IEG evaluations and corporate commitments. Management may seek additional opportunities to discuss progress on IEG recommendation implementation at senior management level and report on this to the Board periodically. 6 3. The Transition to a New Management Action Record System MAR Reporting in FY20 MAR reporting for FY20 covers implementation progress for all recommendations made in IEG’s FY15–19 evaluations, a review of these evaluation recommendations to determine which to stop reporting on and which to transition into the new MAR system, and lessons from the review that can help define the parameters of the future MAR system. Implementation Progress on FY15–19 Evaluation Recommendations This year marked a transition to a more outcome-oriented approach to postevaluation follow-up. Management provided implementation updates on individual evaluations and the summary self-assessment shared with CODE. In addition to reporting progress on individual action items, the implementation updates contained holistic and forward- looking discussions under sections headed “Intermediate Outcomes” and “Looking Ahead: Management Proposals.” Meetings between IEG and management focused on progress toward outcomes sought by evaluation recommendations and not on ratings disputes as in the past. These holistic features and discussions would be useful to continue. However, the implementation updates of some individual evaluations’ recommendations had cursory or partial descriptions of management’s actions. In some cases, IEG discovered through meetings with technical counterparts or other means that implementation was more advanced than described in the updates. Ultimately, evidence gaps constrained the ability to identify the outcomes from implementing IEG recommendations. There is room for management to better track its postevaluation follow-up. In recognition of the evidence gap, future MAR validation reports could include in-depth discussions of how management has followed up on select evaluations. Transitioning the Recommendations Transitioning to the new MAR system requires a decision about how to treat the existing recommendations. IEG and management agreed to transition only those recommendations that meet the criteria of the new MAR system. Other recommendations will be “retired,” meaning that management will stop reporting on them. 7 Chapter 3 The Transition to a New Management Action Record System IEG and management also agreed on the criteria to use for deciding when to retire recommendations. The process involved implementation updates on each of the recommendations of the 28 evaluations covered by the MAR reform that IEG commented on, combined with meetings as needed. The agreed criteria for deciding which recommendations to retire included continued strategic relevance for the Bank Group, management’s continued agreement with the recommendations, whether the recommendation depended on client demand or other factors outside Bank Group management’s control, whether the recommendation had already been substantially implemented or there had been meaningful progress toward the outcomes sought, whether a planned IEG evaluation would reconsider the themes covered in the recommendation, and whether other reporting by management would cover the topic, so Board attention via the MAR was no longer required. Box 3.1 offers examples of the criteria. Box 3.1. Examples of Applying Criteria for Deciding Which Recommendations to Retire Recommendations that had been substantially implemented were retired. For example, the World Bank realigned its staff in fiscal year 2020 in response to the recommendation from the Independent Evaluation Group’s knowledge flow and collaboration evaluation to “link the GPs [Global Practices] and Regions better to improve coordination and enhance responsiveness to clients” (World Bank 2019c, xvii). This action justified this recommendation’s retirement. The fiscal year 2020 World Bank Group’s fragility, conflict, and violence strategy moved many of the recommendations in IEG’s evaluation of Bank Group support in situations involving conflict- induced displacement forward. Management will report periodically to the Board of Executive Directors on the fragility, conflict, and violence strategy’s implementation. These reasons motivated the World Bank’s proposal to retire most of that evaluation’s recommendations. However, the International Finance Corporation proposes to continue reporting on a recommendation that pertains to the Bank Group and is of strategic importance to the International Finance Corporation, namely to catalyze private sector solutions to promote resilience of displaced and host communities. Source: World Bank 2019c, 2019f. Some potential criteria for retiring recommendations that were considered but not agreed to by IEG and not adopted include whether the recommendation had monitorable targets, whether the evaluation had a learning-oriented nature, and whether the recommendation was of “ongoing nature,” meaning it was not a discrete reform or did not have a specific end point. IEG intends to continue making recommendations in evaluations that are primarily learning-oriented and in areas where its evidence shows the Bank Group can do more or do better. 8 Chapter 3 The Transition to a New Management Action Record System Recommendations that were of ongoing nature led to more discussion. For example, management considered (but decided against) retiring the following recommendation: “The World Bank Group should integrate capital market development into its activities across different support areas” (World Bank 2016f, xxvi). This recommendation does not have targets or an end point. It is of an ongoing nature in the sense that different departments that support aspects of capital market development will always need to coordinate and integrate their respective business lines. Even so, it is a perfectly appropriate recommendation that, in IEG’s view, could benefit from the accountability associated with continued MAR reporting, as discussed in box 3.2. Box 3.2. Postevaluation Follow-up to the Capital Markets Evaluation The evaluation on capital markets recommended that the World Bank Group integrate capital market development into its activities across different areas of support (recommendation 1), that it enhance the use of the Financial Sector Assessment Program to underpin the design of capital markets interventions (recommendation 2), that it strengthen knowledge management and develop a frontier global knowledge program (recommendation 3), and that it review funding sources available and their impact on program design and sustainability (recommendation 4). Management reports that this evaluation created much traction. It led to the setup of a joint International Finance Corporation–World Bank Capital Markets initiative, including the Joint Capital Market Program (J-CAP) pilot program. J-CAP has stimulated collaboration across business lines and departments. The Independent Evaluation Group agrees that the J-CAP program appears promising and is a meaningful response to some of the recommendations. This progress is evident in the seven J- CAP pilot countries. However, the Independent Evaluation Group also understands that there remains a tendency toward a fragmented approach to capital market development outside the J- CAP pilot countries. This is because there are multiple departments and business lines involved in capital market development. Outside J-CAP, progress on other parts of the Bank Group’s capital markets work appears more limited, including coinvestment platforms for supporting sovereigns; the development of customized instruments for clients, such as vertical funds or minibond markets; and progress toward more stable funding and more cohesive approaches. The process concluded with IEG, the International Finance Corporation, and the World Bank agreeing to continue MAR reporting for this evaluation. MAR reporting will aim to support continued progress beyond J-CAP’s initial pilot countries, counter the tendency to fragmentation, and support full implementation of the recommendations. Source: World Bank 2016f. Note: MAR = Management Action Record The process concluded with the following proposals (see table 3.1): • Six evaluations from FY16–19 will be fully transitioned into the new MAR system. 9 Chapter 3 The Transition to a New Management Action Record System • Four evaluations will be partially transitioned, meaning that some recommendations will be retired and some recommendations will be reported on, with IEG’s concurrence. • Management will report on some cross-cutting thematic areas where multiple evaluations recommended action, including monitoring and evaluation (M&E) and results frameworks and data and knowledge management. This reporting will cover actions related to some otherwise retired evaluations. • Six FY15 evaluations will be retired because they have completed their standard reporting period. • Ten evaluations from FY16–18 will be fully retired, with IEG’s concurrence. For two evaluations, IEG and management did not reach agreement. Management proposed to retire all recommendations in the rural nonfarm economy evaluation and to retire four recommendations pertaining to the World Bank in the pollution evaluation but continue to report on one recommendation that pertained to IFC (World Bank 2017a, 2017f). IEG did not concur with these two proposals; the resulting disagreements are explained subsequently. Table 3.1. IEG Evaluations Covered by the Management Action Record Reform and Transition Proposals No. Topic Fiscal Management’s IEG’s Justification Year Proposal Position 1 Early childhood development FY15 Retire Concur Due to retire; good progress on key actions 2 Electricity access FY15 Retire Concur Due to retire; good progress on key actions 3 Investment climate reform FY15 Retire Concur Due to retire; good progress on key actions 4 Learning and results FY15 Retire, but Concur Due to retire; some progress continue reporting on key actions; continued on learning thematic reporting on learning 5 Poverty focus of country FY15 Retire, but Concur Due to retire; good progress programs continue reporting on key actions; continued on results thematic reporting on M&E frameworks and M&E 6 Financial inclusion FY15 Retire Concur Due to retire; good progress on key actions 10 Chapter 3 The Transition to a New Management Action Record System No. Topic Fiscal Management’s IEG’s Justification Year Proposal Position 7 Self-evaluation systems FY16 Retire Concur Implementation largely complete; continued thematic reporting on results frameworks and M&E 8 Program-for-Results FY16 Retire Concur Implementation largely complete 9 Industry competitiveness and FY16 Retire Concur Good progress on many jobs actions; continued monitoring and reporting under International Development Association 10 Systematic Country FY17 Retire, but Concur Implementation largely Diagnostic and Country continue reporting complete; continued thematic Partnership Frameworks on results reporting on M&E frameworks and M&E 11 World Bank Group FY17 Retire Concur Implementation complete, engagement in situations of recommendations fragility, conflict, and violence incorporated in FCV strategy, continued reporting on FCV strategy 12 Capital markets FY17 Continue reporting Concur Implementation ongoing 13 Higher education FY17 Retire Concur Implementation largely complete 14 Urban transport FY17 Retire Concur Implementation largely complete 15 Data for development FY17 Retire, but Concur Good progress on many continue reporting actions; continued reporting on data and under the data and knowledge knowledge management management cross-cutting theme 16 Shared prosperity FY17 Retire, but Concur Implementation largely continue reporting complete; continued on results reporting under the M&E frameworks and cross-cutting theme M&E 17 Rural nonfarm economy FY17 Retire Disagree Management claims that implementation is largely complete; IEG concurs with retiring two recommendations but does not see a clear justification for, and does not concur with, retiring the other four recommendations 11 Chapter 3 The Transition to a New Management Action Record System No. Topic Fiscal Management’s IEG’s Justification Year Proposal Position 18 Pollution FY18 Retire for World Disagree; World Bank management Bank, continue IEG does claims that its key actions are one not see a complete; IEG does not see a recommendation clear justification for retiring to IFC justification for retiring 19 Water supply and sanitation FY18 Retire Concur Implementation largely complete 20 Trade facilitation FY18 Continue reporting Concur Under implementation 21 Forced displacement FY18 Retire for World Concur Integrated into FCV strategy, Bank, IFC to with continued reporting on continue reporting FCV strategy; reporting to on the continue on one recommendation recommendation to the World Bank Group to catalyze private sector solutions to promote resilience of displaced and host communities 22 Carbon finance FY18 Continue partial Concur Coordination-oriented reporting for the recommendations under World Bank, retire implementation; others no for IFC and MIGA longer relevant after carbon markets’ collapse 23 Citizen engagement FY18 Continue reporting Concur Under implementation 24 Regional integration FY18 Continue reporting Concur Under implementation; on proposal for consolidation of recommendations two recommendations for 1 and 3–5 for the reporting purposes is sensible World Bank; retire for IFC 25 IFC client engagement FY18 Continue reporting Concur Under implementation 26 Health services FY19 Continue reporting Concur Under implementation; IFC for the World will report as part of its other Bank, retire for IFC updates to CODE 27 Knowledge flow and FY19 Continue reporting Concur Some key actions fully collaboration implemented, some disagreed, and some under implementation 28 Creating markets FY19 Continue reporting Concur Under implementation Source: Independent Evaluation Group. Note: CODE = Committee on Development Effectiveness; FCV = fragility, conflict, and violence; FY = fiscal year; IEG = Independent Evaluation Group; IFC = International Finance Corporation; M&E = monitoring and evaluation; MAR = Management Action Review; MIGA = Multilateral Investment Guarantee Agency. 12 Chapter 3 The Transition to a New Management Action Record System Disagreements on Specific Transitions In two instances, management proposed to stop reporting on recommendation implementation; however, IEG sees value in continued reporting and does not see merit in management’s stated reasons for stopping reporting. The pollution evaluation made five recommendations (World Bank 2017f): that the World Bank do more to develop client-country pollution measurement and monitoring systems (recommendation 1), strengthen its country advisory services and analytics on pollution to help countries prioritize their pollution concerns and to help target countries more strategically (recommendation 2), intensify efforts to scale-up and recalibrate its efforts in pollution management (recommendation 3), and leverage the climate change portfolio to combat pollution (recommendation 4). It also recommended that IFC strengthen support to help clients comply with performance standards on pollution (recommendation 5). IFC and IEG agree to continue reporting on the recommendation pertaining to IFC. According to management, implementation “has achieved a critical threshold,” with many changes in the Bank Group’s overall approach. For example, the new Environmental and Social Framework integrates pollution concerns more systematically, and the World Bank has new trust funds, staff training, and a global solutions group, and has undertaken some analytical and financing support for pollution control. Management also emphasizes that some outcomes proposed by the recommendations depend on client decisions and other factors beyond management’s control and that some of the evaluation’s recommendations have no clear end point or criteria for success. IEG does not see a justification for retiring the first four recommendations. It finds the implementation update for this evaluation to be lacking in specific evidence on implementation progress. In IEG’s interpretation, implementation is ongoing but is not close to being complete. The Environment, Natural Resources, and Blue Economy Global Practice, which is implementing the recommendations pertaining to the World Bank, has data it could use for reporting that was not used in this year’s update. For example, the Practice Group has data on how many projects target pollution and on its use of trust funds and analytical work to create clients’ awareness and interest in combating pollution, all of which are relevant for reporting on implementation progress in future MARs. IEG proposes that management report on progress under a consolidated recommendation, which could be reworded as “increase efforts to assist countries to be more aware of the most serious pollution issues and how to monitor and address these.” 13 Chapter 3 The Transition to a New Management Action Record System The rural nonfarm economy evaluation made recommendations that sought to improve how different types of Bank Group support for the rural nonfarm economy come together to address rural poverty (World Bank 2017a). The evaluation’s recommendations envisaged (i) a more informed and coordinated approach to rural nonfarm issues in the Bank Group, (ii) better designed interventions that track sustainable income or employment effects for the rural poor, (iii) enhanced diagnostics and data collection to inform a more strategic approach to the rural nonfarm economy, and (iv) gender-disaggregated M&E. According to management, the World Bank and IFC recognize that the rural nonfarm economy remains an important avenue for rural poverty reduction. Management notes that the cross-cutting nature of this evaluation’s topic made it difficult to implement its recommendations. That said, management also points to some implemented and ongoing actions, particularly in diagnostics, knowledge management, some country- level rural nonfarm economy activities, better ex ante assessment through screening of agriculture operations at the concept stage and in IFC’s Anticipated Impact Measurement and Monitoring (AIMM) system, routine gender screening, and an increase in gender-tagged operations, impact evaluations, and surveys. IEG can only partially concur with management’s proposal to retire all six recommendations and views implementation as still ongoing. IEG sees two of the recommendations as having been reasonably addressed through routine, systematic approaches. IFC’s AIMM system appears to ensure a more systematic approach to ex ante screening of IFC projects. IFC has also done surveys and evaluations. These actions can motivate retiring recommendation 4 for IFC. The World Bank’s gender screening process is likely to ensure a more systematic approach to addressing gender issues, and this motivates retiring recommendation 6 on gender. For the remaining recommendations, IEG acknowledges management’s various actions, including on diagnostics and knowledge sharing, but does not view those actions as constituting a complete response to the evaluation’s recommendations. In IEG’s view, management’s update is lacking in specifics on how actions address the recommendations’ intended outcome. For example, it is unclear that the various surveys, diagnostic studies, and knowledge sharing activities achieve the recommendations’ intended outcome of greater poverty focus through a more informed, coordinated, and tailored approach to rural nonfarm issues. IEG proposes that the World Bank continue to report on progress under a consolidated recommendation which could be reworded as “the World Bank should support and track poverty reduction through rural nonfarm economy activity by (i) identifying context-specific constraints to the rural poor gaining from nonfarm employment, income and productivity growth; (ii) aligning the design of project 14 Chapter 3 The Transition to a New Management Action Record System interventions to target those constraints and (iii) strengthening M&E to determine and report on what works, where and for whom.” Lessons on Process Box 3.3 sums up some lessons on the process for agreeing and reporting on recommendations. 15 Chapter 3 The Transition to a New Management Action Record System Box 3.3. Lessons on Process • The Independent Evaluation Group (IEG) should continue dialogue with management about draft evaluation findings and recommendations. Such dialogue has been useful, resulting in greater stakeholder buy-in. • The Management Action Record (MAR) process is more productive when done in a spirit of constructive dialogue on shared objectives. The previous MAR process was heavy on process and written technical documentation. The process did not generate good information to the Board on progress toward recommendations’ intended outcomes and did not fulfill accountability purposes. • Candor helps make the MAR system more effective. The previous MAR process sometimes masked that management did not agree much with the recommendations. • Implementation updates that were more outcome oriented and the summary self- assessment proved useful. There is room to expand this. Future IEG evaluations could include a brief summary statement on outcomes sought to promote outcome-oriented dialogue and reporting. • IEG and management could consider defining criteria or milestones to indicate when recommendations for the World Bank Group to do more or do better (ongoing nature recommendations) are fully or largely implemented. In the past, managem ent’s action plans defined targets. With the discontinuation of action plans, clear progress criteria are lacking. In the future, recommendation completion criteria or milestones could come from a combination of evaluations defining the outcome sought, man agement’s responses and self-assessments, and MAR discussions between IEG and management. Ultimately, however, judgment will be needed. • Direct engagement between IEG and the technical units in charge of implementing the recommendations is useful. Central units in the World Bank, International Finance Corporation, and Multilateral Investment Guarantee Agency act as intermediaries and focal points for evaluations. These central units perform needed roles to prioritize and provide coherence across many issues and agendas and serve as permanent institutional points of contact for IEG. However, for some implementation updates, IEG felt there was too little direct engagement with the relevant technical units, which led to insufficient up-to-date information on implementation progress, among other things. One contrast to this general pattern was the capital market evaluation (World Bank 2016f); technical teams and central units from IEG, the International Finance Corporation, and the World Bank discussed follow-up to this evaluation directly. Conclusions from the discussion informed decisions about how to treat that evaluation’s recommendations and are also reflected in box 3.2. 16 4. Enhancing the Value Added of IEG Recommendations The focus on fewer and more strategic recommendations and more meaningful postevaluation follow-up is an opportunity for IEG to enhance the value added of its recommendations. The MAR reform process resulted in insights and lessons on how to do this. These insights and lessons are based on the MAR reform discussions and management’s self-assessment, not on a comprehensive assessment of evaluations’ quality or overall usefulness. The use value of IEG evaluations goes well beyond actions to implement their recommendations. Insights on Evaluation Topics Overall, the perspective that emerges from the MAR reform is that IEG evaluations add value when they facilitate change, challenge perceived views, take strategic perspectives, and review whether the Bank Group is doing the right things and where it is at risk of missing the big picture. IEG appears more impactful when it evaluates topics of growing importance and topics in which the Board and management have a learning interest. Evaluations of mature portfolios can also be useful, particularly when they detect blind spots. Timely choice of evaluation topics is strongly correlated with evaluation impact. Many of IEG's evaluations of early implementation progress were impactful because they helped management articulate the Bank Group's role in a new area or took the early pulse on a new instrument or approach. Examples of evaluations that successfully covered new areas or areas of growing importance include those on industry competitiveness and jobs (World Bank 2016c) and fragility, conflict, and violence (World Bank 2016e; box 4.1). Examples of evaluations that informed new instruments or approaches include the Systematic Country Diagnostics and Country Partnership Frameworks (World Bank 2017g), Program-for-Results (PforR; World Bank 2016d), the citizen engagement evaluation (World Bank 2018b), and the knowledge flow and collaboration evaluation (World Bank 2019c). An evaluation with unfortunate timing was carbon finance (World Bank 2018a). The collapse of international carbon credit trading markets undermined the context for some of the evaluation’s findings and recommendations. IFC ceased its carbon market business line, which rendered the evaluation’s IFC recommendation moot. 17 Chapter 4 Enhancing the Value Added of IEG recommendations Box 4.1. Engagement in Tough Places Management follow-up to the Independent Evaluation Group’s evaluation of World Bank Group engagement in situations of fragility, conflict, and violence (FCV) has been very strong. Progress has been made on recommendations to increase staff presence in FCV areas, financing mechanisms for FCV in middle-income countries, and a broader strategic approach to FCV issues, among others (World Bank 2016e). The evaluation informed the FCV strategy and the International Development Association Replenishment discussions. Appendix 1 of the FCV strategy puts forth a multidimensional public system for measuring and monitoring fragility, as a complement to the traditional unidimensional FCV list, in line with recommendation 2 of the evaluation (World Bank n.d.). No clear picture emerges on the relative traction of different evaluation foci among sectoral, thematic, and corporate topics. Of the 28 evaluations covered by the MAR reform, 14 covered a single sector or subject. Of the remainder, seven covered themes such as jobs; fragility, conflict, and violence; regional integration; and creating markets or instruments such as PforR. The final seven evaluations covered corporate effectiveness (the twin goals, the operating model, and IFC’s strategic client engagement) and M&E and data. Insights on Recommendation Types Although it is difficult to generalize, some big-picture recommendations appear more likely to have strong implementation. IEG’s deep dive of 50 environmental sustainability–focused recommendations made over a 10-year period found that action plans for 47 percent of these evaluations’ recommendations had been implemented to a high or complete degree. Recommendations on policy and strategic directions and how to implement projects and programs had the best traction, with action plans for 56 and 58 percent, respectively, of these implemented completely or to a high degree. Corporate and institutional recommendations had average implementation; recommendations on knowledge work, measurement systems, and assessment had weak traction, with only 31 percent of related action plans implemented completely or to a high degree (World Bank 2018e). According to management’s self-assessment, “during the period under review, the most impactful IEG recommendations were aligned with World Bank Group priorities, contributed to an ongoing change process, informed a decision point, argued in favor of a new direction, and recommended specific actions within management’s control.” Industry competitiveness and jobs, fragile situations, forced displacement, knowledge flow and collaboration, PforR, Systematic Country Diagnostics and Country Partnership 18 Chapter 4 Enhancing the Value Added of IEG recommendations Frameworks, and health services are examples of impactful evaluations that met those criteria (World Bank 2016c, 2016d, 2016e, 2019c, 2018f, and 2019f). Additionally, some IEG findings and recommendations usefully detected blind spots in operations’ coverage. For example, electricity access and urban transport made impactful recommendations to increase the focus on countries with very low electrification rates and on disadvantaged groups’ access to affordable mobility, again according to management’s self-assessment (World Bank 2015c, 2017d). IEG has made many recommendations on M&E, diagnostics, data, knowledge, and learning. Recommendations can be grouped in many ways. One grouping shows that 22 percent of IEG recommendations over FY15–19 focused on M&E, 18 percent on diagnostics, 16 percent on staff's knowledge and incentives, 11 percent on country programs, 8 percent on coordination inside the Bank Group, and 7 percent on strategic approaches (figure 4.1). Figure 4.1. Focus of Independent Evaluation Group Recommendations M&E, results frameworks 22% Diagnostics 18% Recommendation category Knowledge, learning, incentives, other internal 16% Other 15% Country program focus 11% Collaboration inside the WBG 8% Strategy or more coherent approach 7% Partnerships outside WBG 3% Percentage of recommendations Source: Management Action Record reform team. Some recommendations were included in more than one category. Note: M&E = monitoring and evaluation; WBG = World Bank Group. IEG is less positive than management about progress on implementing recommendations on M&E, results frameworks, and learning. Approximately 22 percent of all IEG recommendations covered M&E, results frameworks, and learning from M&E fully or partially. Most of these recommendations approached the topic from the perspective of sectoral investment projects, sometimes motivated by evaluation findings of uneven quality of results frameworks. These recommendations often called for better 19 Chapter 4 Enhancing the Value Added of IEG recommendations results frameworks, explicit theories of change, linking projects to long-term outcomes, and disaggregation by gender, socioeconomic group, or geographic area. Some also linked M&E to operational learning and performance management. Management reports good progress in this area, including better understanding of the importance of M&E among staff, higher IEG ratings for quality of project M&E, and IFC’s AIMM system. IEG's retrospective of progress on M&E and results was less positive. The retrospective noted progress on many tools and templates, staff training, the Bank Group evaluation framework, AIMM, and monitoring of gender gaps. However, it also noted long-standing gaps related to incentives, organizational culture, and staffing capacity on M&E and results. In IEG’s view, M&E quality and use, and the Bank Group’s broader outcome orientation, need continued attention. Recent and ongoing efforts in the World Bank and IFC to renew outcome orientation and improve staff M&E tools and knowledge are useful but could be complemented with broader changes in capacity, approaches, and incentives. Progress on implementing recommendations on diagnostic and analytical tools is hard to assess. Approximately 18 percent of all IEG recommendations covered diagnostic and analytical tools. Examples include recommendations to intensify analytical work related to conflict-induced forced displacement to inform dialogue, programming, and operational design; better identification and monitoring of public policy objectives of trade regulations; and better diagnostics of disparities in access to water and sanitation. Recommendations on diagnostics were sometimes motivated by evaluation findings that the Bank Group had strong analytical tools but did not apply those tools as often as desirable or applied them but did not act on the findings. Management responses to recommendations on diagnostic tools sometimes noted resource and client demand constraints to undertaking more diagnostics or translating their findings into action. Additionally, IEG recognizes that it is difficult for management to report on recommendations on diagnostics because routine internal monitoring systems typically do not cover the production, use, and follow-up to diagnostic tools. Therefore, reporting tends to be based on examples. Several other types of recommendations tend to be challenging for management to act and report on. IEG should not refrain from making challenging or controversial recommendations, but it needs to have realistic expectations of what management is able or willing to do. Paraphrasing management’s self-assessment, some of the predictable recommendation implementation challenges include the following: • Recommendations to collaborate and adopt more harmonized approaches across an area are often challenging for management to implement. This is because they often belong to many departments and have no clear locus for implementation. 20 Chapter 4 Enhancing the Value Added of IEG recommendations The rural nonfarm economy evaluation is an example. Several practices cover rural nonfarm issues and the World Bank has no unit or mechanism with a coordination mandate on rural nonfarm issues, making recommendation implementation challenging. • Recommendations to produce a strategic document to achieve more harmonized or coherent approaches are less likely to be impactful than recommendations to use existing mechanisms; for example, roles or units already charged with coordination. • Recommendations on culture, incentives, and leadership have also proven difficult to act on, again in part because there is no clear locus for implementation and because changing organizational culture is genuinely difficult. • Many recommendations have called on management to do more within the evaluation’s topic area. But, in a flat-budget environment, it can be hard to step up. • Some recommendations in the Bank Group’s client-driven model to do more depend on client demand that is not fully under the Bank Group’s control. IEG can consider making recommendations to use analytical and convening work to stimulate client demand. IEG plans to update and enhance its internal guidance and training to evaluators on recommendation writing. Box 4.2 sums up lessons for IEG on formulating recommendations. Recommendation quality is subjective. Evaluators will need to rely on sound judgment of what changes could improve the Bank Group’s effectiveness and how those changes could occur. 21 Chapter 4 Enhancing the Value Added of IEG recommendations Box 4.2. Lessons for IEG on Recommendations • When deciding what to evaluate, continue to give weight to evaluations’ potential to spark change. • Evaluate early experiences. It could be argued that there should be advanced progress with solid evidence of effectiveness and outcomes before IEG evaluation. However, evaluations of early implementation progress can be valuable by providing insights to help the World Bank Group shape an agenda that is growing in importance. • Recommend major changes. Few IEG recommendations call for the Bank Group to stop doing something or to change an approach that was not working. IEG may want to recommend more often that something be stopped or changed if findings warrant it. • Write fewer recommendations. Evaluations can sum up lessons and make suggestions but provide fewer recommendations that focus on the major changes that are likely to be impactful. • Continue to make recommendations that are directional yet implementable. This means avoiding recommendations that management may perceive as too prescriptive or too specific on how to achieve the desired outcomes while maintaining a realistic sense of what steps management could plausibly take. For example, recommendations to coordinate or harmonize approaches across departments can be reframed in relation to scaling up a promising existing coordination mechanism. Also, consider ways to rethink or reframe recommendations to produce new strategy documents and change culture, and rethink recommendations for which successful implementation depends on factors outside management's control, such as client demand, resources, and partners’ actions. • Continue to pay close attention to drafting. Some recommendations could have been drafted more clearly, more concisely, or in a more constructive tone. The recommendations and their motivating text in the Management Action Record tables need to be easily understood by readers unfamiliar with the evaluation’s context years after the evaluation is completed. • Avoid compound recommendations. Some past IEG recommendations have up to four different elements and, in hindsight, could have been more clearly focused. • Recommendations to deepen the evidence base can read as trivial, since the evidence base is almost never perfect. It might be better to point to specific unresolved operational questions. 22 5. Conclusions This document concludes the MAR reform process. The purpose of the MAR reform is to enhance the use value of IEG recommendations to increase IEG's value added for the Bank Group’s development effectiveness. The process was characterized by more strategic engagement by IEG and management and resulted in agreement between IEG and management on the nature of future IEG recommendations, how to use the MAR process to report on management’s progress toward recommendations’ intended outcomes, and how to transition most pre-FY20 evaluations into the new MAR. The process also resulted in some insights and lessons for future recommendations to ensure greater impact. IEG and management disagreed in two instances. For the pollution and rural nonfarm economy evaluations (World Bank 2017a, 2017f), management proposed to stop reporting on recommendation implementation, although IEG sees value in continued reporting in the MAR on some of those evaluations’ recommendations. IEG does not concur with World Bank management that implementation is largely complete. IEG proposes to CODE that it (i) endorse the MAR reform as described here, and (ii) advise on the areas of disagreements, namely whether to fully retire the pollution and rural nonfarm economy evaluations from the future MAR system. IEG’s position is that management should continue reporting on these, though considering consolidating the recommendations as proposed in section 3, until it can produce adequate evidence of recommendation implementation. There are several risks to the MAR reform’s success that need to be managed in the years ahead. It may be difficult to maintain senior management’s attention on postevaluation follow-up. Deepening the focus on how recommendation implementation affects the Bank Group’s development effectiveness might help sustain senior management’s attention. Better evidence and milestones to assess progress toward recommendations’ intended outcomes could facilitate outcome-oriented reporting and dialogue. Candid self-assessments could prove hard to sustain. And there is a risk that management, IEG, and CODE may have differences in interpretation of the MAR reform’s aims. It will be important to watch for these risks, adjust the MAR system accordingly, and continue to foster Bank Group senior management’s appreciation of IEG evaluations’ role in helping to enhance the Bank Group’s development effectiveness. 23 Notes 1Making evaluation recommendations is one of the avenues the Independent Evaluation Group uses to influence the World Bank Group’s development effectiveness. Other complimentary avenues include IEG's microproducts and its many efforts to promote learning from evaluations. 2 http://ieg.worldbank.org/about-us. 24 References World Bank. 2014. Learning and Results in World Bank Operations: How the Bank Learns. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2015a. Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms and Business Regulations. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2015b. The Poverty Focus of Country Programs: Lessons from World Bank Experience. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2015c. World Bank Group Support to Electricity Access, FY2000 –2014. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2015d. World Bank Support to Early Childhood Development. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2016a. Behind the Mirror: A Report on the Self-Evaluation Systems of the World Bank Group. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2016b. Financial Inclusion: A Foothold on the Ladder toward Prosperity? An Evaluation of World Bank Group Support for Financial Inclusion for Low-Income Households and Microenterprises. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2016c. Industry Competitiveness and Jobs: An Evaluation of World Bank Group Industry-Specific Support to Promote Industry Competitiveness and Its Implications for Jobs. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2016d. Program-for-Results: An Early-Stage Assessment of the Process and Effects of a New Lending Instrument. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2016e. World Bank Group Engagement in Situations of Fragility, Conflict, and Violence. Washington, DC: World Bank. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2016f. The World Bank Group’s Support to Capital Market Development. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2017a. Growing the Rural Nonfarm Economy to Alleviate Poverty: An Evaluation of the Contribution of the World Bank Group. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2017b. Higher Education for Development: An Evaluation of the World Bank Group’s Support . Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2017c. The International Finance Corporation’s Approach to Engaging Clients for Increased Development Impact. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2017d. Mobile Metropolises: Urban Transport Matters: An IEG Evaluation of the World Bank Group’s Support for Urban Transport. Independent Evaluation Group. Washington, DC: World Bank. 25 References World Bank. 2017e. A Thirst for Change: The World Bank Group’s Support for Water Supply and Sanitation, with Focus on the Poor. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2017f. Toward a Clean World for All: An IEG Evaluation of the World Bank Group’s Support to Pollution Management. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2017g. World Bank Group Country Engagement: An Early-Stage Assessment of the Systematic Country Diagnostic and Country Partnership Framework Process and Implementation. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2018a. Carbon Markets for Greenhouse Gas Emission Reduction in a Warming World. Independent Evaluation Group. Washington DC: World Bank. World Bank. 2018b. Data for Development: An Evaluation of World Bank Support for Data and Statistical Capacity. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2018c. Engaging Citizens for Better Development. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2018d. Growth for the Bottom 40 Percent: The World Bank Group’s Support for Shared Prosperity. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2018e. Independent Evaluation Group’s Work on Environmental Sustainability: A Decade of Recommendations. Washington, DC: World Bank. World Bank. 2018f. World Bank Group Support to Health Services: Achievements and Challenges. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019a. ‘Creating Markets’ to Leverage the Private Sector for Sustainable Develo pment and Growth: An Evaluation of the World Bank Group’s Experience through 16 Case Studies. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019b. Grow with the Flow: An Evaluation of World Bank Group Support to Facilitating Trade 2006–17. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019c. Knowledge Flow and Collaboration under the World Bank’s New Operating Model. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019d. Results and Performance of the World Bank Group 2018. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2019e. Two to Tango: An Evaluation of World Bank Group Support to Fostering Regional Integration. Independent Evaluation Group. Washington, DC: World Bank World Bank. 2019f. World Bank Group Support in Situations Involving Conflict-Induced Displacement. Independent Evaluation Group. Washington, DC: World Bank. World Bank. 2020. Implementing IEG Recommendations over FY15–19: A World Bank Group Management Report. Washington, DC: World Bank. 26 References World Bank. n.d. World Bank Group Strategy for Fragility, Conflict, and Violence 2020–2025. Washington, DC: World Bank. 27 The World Bank 1818 H Street NW Washington, DC 20433