98806 June 30, 2015 Concept Note IEG Results and Performance of the World Bank Group 2015 Background 1. Results and Performance of the World Bank Group (RAP) is the annual account of what recent IEG evaluations reveal about the effectiveness of the World Bank Group (WBG) which includes IBRD/IDA, IFC, and MIGA, in addressing current and emerging development challenges. The report synthesizes evidence from recent IEG evaluations complemented by relevant information from other sources. RAP 2015 will be the sixth in a series that began in 2010 with the consolidation of separate annual reports that IEG prepared for the World Bank (IBRD/IDA), IFC, and MIGA. As the methodology for the reporting on results and performance is established, in lieu of an Approach Paper, this Concept Note outlines the context, rationale, and scope of the proposed report. 2. Following the RAP 2014, which adopted a significantly more focused and streamlined approach in assessing WBG performance compared to the previous years’ reports, RAP 2015 will start with this year’s special theme, “mainstreaming gender – progress made in WBG operations”, followed by the reporting on the results and performance of WBG’s activities during recent fiscal years. Context: A Changing WBG 3. The recent WBG organizational changes have resulted in a new structure, with global practices and clusters of global practices. RAP 2015 will group findings using the new WBG structure but it is important to note that RAP 2015 will not evaluate the WBG’s change management process or the new WBG strategy, for several reasons: first, the changes are so recent that it’s unlikely that the effects on project performance could be observed yet; second, RAP is intended to synthesize existing IEG evaluations rather than add to existing IEG evaluations. RAP therefore uses existing data, and new analysis is confined to the thematic chapter on gender and to exploration of performance trends and their drivers. As in previous years, the focus of RAP 2015 will be to assess WBG operations by reviewing operationally mature projects for IFC and MIGA and recently closed operations for the World Bank. For IFC, attempts will be made to present an analysis to predict development 1 outcomes of more recent projects as IEG has an established methodology and access to relevant portfolio. Context: Integration of Gender into Work of the WBG 4. The year 2015 will end with a new WBG gender strategy,1 14 years after the inception of the 2001 Gender Strategy and OP/BP 4.20. For the first time it will be a joint Bank-IFC strategy. The new gender strategy will complete a process that has progressively sharpened the focus on gender at the corporate level. The “Gender Action Plan” (GAP, 2007-2011) had as a main goal to improve women’s economic opportunities and provided incentives to invest in improving women’s access to jobs, land rights, financial services, agricultural inputs and infrastructure. The GAP led to the definition of corporate commitments on gender – gender equality was selected as a “Special Theme” of the IDA16 and IDA17 replenishment. Those commitments have been reflected in IDA indicators; and the WB Corporate Scorecards framework. The World Development Report 2012 focused on gender equality and development. While no specific actions for IFC were included in the 2001 gender strategy, over the past 10-15 years IFC has promoted several initiatives to integrate gender in its portfolio — such as the Gender Entrepreneurship Markets program, the Global Banking Alliance, the Women in Business Program, WINVest, and SheWorks. Moreover, gender indicators (such as female employment) had been included in IFC’s development outcome tracking system (DOTs)2. In 2013, the Gender Cross-Cutting Solution Area was created bringing Bank and IFC under the same umbrella and confirming the prominence of gender for the institution. 5. While there has been progress in integrating gender in the WBG operational, analytical, and policy work, issues remain. In 2010 IEG undertook the evaluation of the effectiveness of the Bank’s gender policy (IEG, 2010) which found that the progress made by WB in gender integration from 2002 to 2008 weakened in the latter half of the review period. Based on its assessment and on the 2001 Gender Strategy approach (embedded in the OP/BP 4.20), IEG recommended WB to institutionalize the management accountability framework, develop a monitoring system to assess how well the Bank’s work program adequately addresses gender-related concerns, establish a results framework to facilitate adoption of an outcome 1 The concept note of the forthcoming gender strategy was discussed at CODE on April 8th, 2015. 2 DOTS data will be used to help IEG identify recently approved projects with a gender focus. DOTS is a monitoring system developed by management and not a self-evaluation system validated by IEG. IEG has reported on DOTS ratings in its Biennial Report on Operations Evaluation (BROE) and will not assess or present DOTS data in the RAP. 2 approach to gender integration, and restore a broader requirement for gender integration at the project level. 6. Since the time of the 2010 evaluation, some of the gaps identified by IEG have been addressed: reporting mechanisms have been institutionalized; efforts to produce gender- disaggregated data and impact evaluation evidence have intensified; and progress has been made in strengthening the results and accountability mechanisms. As a result, the integration of gender in both country strategies and the Bank’s portfolio has improved. 7. Despite clear progress, the meaningful integration of gender at the country, sector and/or project level and, in particular, the measurement and reporting of gender results remain a challenge largely because there are no agreed upon metrics to track and rate results, despite the existence of corporate commitments. For example, a recent IEG learning report on Social Safety Nets and Gender (IEG, 2014 [2]) found that WB Social Safety net projects would benefit from addressing gender differences in their design, but they rarely do. Most project documents included limited discussion of intra-household dynamics and the gender-relevant context of the supported intervention. Women were generally targeted as a vulnerable group or in an instrumental way, without discussing the costs that the intervention may have imposed on them. Gender was often missing from monitoring and evaluation (M&E) frameworks, except for tracking female beneficiaries. Findings of this report identified opportunities to strengthen the integration of gender into projects. 8. Bringing a stronger focus on results –both at the project and country strategy level — is a major goal of the new WBG Gender Strategy, currently under preparation. Its Concept Note “Promoting Gender Equality to Reduce Poverty and Boost Shared Prosperity” highlights a country-based approach, and a strong link between the gender equality agenda and the twin goals of the institution, which makes achieving gender equality a key component of eradicating poverty and achieving shared prosperity and inclusion. The expected launch of the new Gender Strategy at the end of 2015, around the same time of the adoption of the new Sustainable Development Goals (where gender is expected to feature prominently, with one stand-alone goal and a number of gender indicators included in other goals) makes this RAP particularly timely, and the RAP 2015 team will strive to feed early findings into the ongoing gender strategy development. Rationale, Objectives, and Audience 9. Independent evaluations can provide insights into what has worked and how existing approaches can be modified to meet new contingencies. RAP 2015 will draw from and build 3 on IEG’s project, country-level, corporate, and thematic work and on other relevant research in order to distill lessons and guidance that will help the WBG more effectively address these challenges. 10. The main objectives of RAP 2015 are to: (i) review the approach adopted by the WBG to integrate gender in the Bank’s operations and country strategies and measure, assess, and report gender results;3 (ii) report on the results and performance of the WBG’s activities as captured in recent validations, evaluations, and analysis; (iii) distill lessons from evaluations to support efforts to enhance the WBG’s effectiveness; and (iv) report on the level of management actions in response to recommendations of previous IEG evaluations. 11. Since IEG reports directly to the World Bank Group’s Board of Directors and the Board’s Committee on Development Effectiveness (CODE), these are considered the main audience for RAP 2015. Other important stakeholders include WBG’s senior management; the Gender Strategy team; management of the Global Practices and Regions; IFC and MIGA Directors and Senior Managers; Country Directors and Representatives; OPCS; Task Teams of operational projects. RAP 2015 becomes a public document upon completion, and therefore external partners of various kinds are also important stakeholders. A preliminary stakeholder analysis, based on document review and some discussions with management, is attached as Appendix E. Scope and Key Questions 12. RAP 2015 will have three distinct but interrelated parts. Chapter 1 of the RAP will focus on this year’s special theme, “mainstreaming gender – progress made in WBG operations” and will draw lessons for the future. As the system currently in place does not ensure that gender results are routinely measured, tracked, and reported on, the chapter will focus on the approach adopted to address gender issues in projects and country strategies and on the features of M&E frameworks to report on gender results (rather than the results themselves). Specifically, the chapter will examine the following key questions: 3 The Gender Chapter in this year’s RAP represents an opportunity to analyze in depth some of the challenges in defining, measuring, tracking, and reporting results in operations and country strategies. At the same time, IEG is committed to more systematically assess gender results in its many evaluation products. The work undertaken for the RAP, as well as other currently ongoing efforts, will likely generate an approach that IEG will adopt to report with regularity on results regarding gender integration in WBG lending. 4  What is the approach/framework adopted by the WBG to integrate gender in operations and country strategies?  To what extent can M&E frameworks of operations and country strategies measure and report on gender results? 13. It is important to emphasize that RAP 2015 will not analyze the effectiveness of the gender strategy in mainstreaming gender in the WBG’s portfolio, nor will it analyze the effectiveness of WBG’s support in achieving greater gender equality. RAP 2015 will instead focus on gender indicators and document whether they are coherent with the overall results frameworks and relevant to the project’s objectives. It will also assess the extent to which the Bank’s operations have actually collected gender-disaggregated and gender-relevant baseline and follow-up data on outcomes and under which circumstances this has been more successful. 14. Chapter 2 of RAP 2015 will focus on recent results and performance of the WBG and will focus on the following key questions:  How effective have the WBG’s interventions been in delivering development results?  What are the key factors associated with performance4 of WBG projects?  What lessons can be drawn for design and implementation of the WBG’s work in the future? 15. RAP 2015 will synthesize performance ratings from IEG reviews of the WBG’s self- assessments of its country strategies and financing interventions (including IFC advisory services and MIGA guarantees) to gauge project and program performance. The analysis will be organized in line with the WBG’s new structure, providing information specific to Global Practice groups, regions, and IFC’s new categorization for industries. Taking into account the Board discussions surrounding the findings of RAP 2014, this year’s RAP will include a deeper analysis on the possible differences in performance depending on the commitment size of projects. 16. One of the salient findings of the RAPs from the past few years has been a decline in performance of World Bank and IFC operations. Several internal and external evaluations (for example, Geli, Kray, and Nobakht 2014; IEG 2014; Kilby 2000 and 2014) have analyzed key factors affecting the outcomes of WBG interventions, and the Board expressed interest in a more timely feedback from IEG on the extent to which these key factors associated with better performance are present in the ongoing project portfolio. For the World Bank portfolio, IEG has started discussions with OPCS about ways of looking at the portfolio of 4Given the main theme of this year’s RAP, where possible and relevant, gender specific results will also be assessed. 5 active projects to provide early feedback in the RAP in line with the Board request. With the understanding that scrutinizing ongoing projects is properly the role of World Bank management, IEG’s view is that this assessment should be commissioned by OPCS. IEG’s role would then be to help OPCS develop a common shared methodology and to validate the work independently once completed. In discussion, OPCS indicated that they are planning to carry out a review of quality at entry during FY 16. Such an assessment of the extent to which ongoing projects incorporate elements associated with better project outcomes, would shed light on recent developments and provide the early feedback which the Board seeks. This OPCS review is likely to focus on key elements of quality at entry, and an important set of criteria are reflected in the recent TTL Quality Support Checklist (developed jointly between OPCS and IEG). OPCS has indicated that IEG will be kept informed as their work progresses. As of the date of this Concept Note, the detailed timetable of this review is still to be confirmed. IEG will explore further with OPCS when and how IEG could provide inputs to the scope and methodology of this review by OPCS and assess the possibility of completing a validation exercise to be included in this year’s RAP. 17. For IFC, an analysis will be presented on how well the key factors that have been found to influence development outcomes are reflected in ongoing investment projects. 18. Chapter 3 of RAP 2015 will report on the Management Action Record (MAR). MAR serves as an accountability and learning tool that informs the WBG Board of Directors and the general public about the progress of Management’s implementation of their action plans prepared in response to the recommendations from IEG’s evaluations. MAR also serves as an important learning tool highlighting what has and has not worked in Bank Group interventions. 19. The chapter will outline the implementation status of Management’s action plans in FY15 and provide analysis of historical trends of both the implementation ratings and Management’s actions in selected fields. Specifically, this section will examine the following key questions:  What progress has been made by WBG to implement the agreed Management Action Plans and how have they translated into the adoption of IEG recommendations?  What lessons can be learned from the analysis of the MAR process?  What has been the trend in the past adoption ratings and Management’s response to IEG’s recommendations in selected fields? Approach and Methodology 6 20. RAP 2015 will follow the approach used in previous RAPs and analyze data from a variety of sources, synthesize learning from recent evaluations, and seek inputs from WBG management and staff, especially on gender related issues. Given limited time and resources, RAP 2015 will not attempt to collect new data or new evaluative evidence. Where knowledge gaps exist, RAP 2015 will candidly note deficiencies, and suggest issues for future study. 21. Chapter 1 of RAP 2015 will build on two different exercises, leveraging on IEG available sources and resources: (i) an assessment of gender results in WBG operations; and (ii) an assessment of gender integration in country strategies. Details of this methodology on gender related analysis is attached as Appendix C. 22. Chapter 2 of RAP 2015 will build on the analysis of previous RAPs and analyze WBG portfolio performance focusing especially on FY2009-FY20145, looking at trends over time and differences across relevant categories or groupings, including project size. Categories or groupings that may be explored to understand significant factors include:  Country income, IDA / IBRD / Blend  Geographical regions and other country characteristics such as Fragile and Conflict Affected States  Global practices and clusters of Global Practices, including sectors and themes  Lending instruments  Sizes (commitment and actual) of projects, if possible controlling for other factors such the above  Lending volume of projects as well as number of projects in portfolio  Flagged as incorporating gender or not  ISR ratings and flagged as problem project or not  Timing and scope of restructuring 23. To provide sector-specific or theme-specific insight, recently completed IEG evaluations will be used to identify findings applicable to the current WBG portfolio. The list of recently completed evaluations are attached as Appendix B. 24. Using the list of factors that have been found to influence development outcomes for IFC investment projects, IEG will assess to what extent these features are reflected in IFC’s ongoing portfolio. The IEG model that is based on; (i) external factors that define the credit risk rating system (i.e. management quality, corporate governance, project margin, and country conditions) and (ii) internal factors that define the work quality (i.e. screening, appraisal, and structuring, supervision and administration, and role and contribution) will be used to assess the performance of ongoing operations. 5 Analysis of more recent project performance may also be included, using FY2015 performance information if available. 7 25. Chapter 3 of RAP 2015 will provide an update on IEG Recommendations to WBG Management and follow-up on those recommendations. WBG Management and IEG have recently reformed the Management Action Record (MAR) system which is used to track agreed evaluation recommendations and their corresponding Management responses, action plans, and follow up. In prior years, the MAR update has been presented in a short section. This year, more analysis on both the progress that has been made on reforming the system and on implementation of recommendations will be presented. 26. Following the practice in previous RAPs, the regional updates will be presented in an online Appendix to the RAP 2015. In light of the recent reorganization that the WBG has gone through, GP and sector specific updates may be provided as an Appendix as well. Communication and Dissemination 27. The preliminary stakeholder analysis, attached as Appendix E will be updated as and when new information from the stakeholders is collected. This analysis will provide the foundation for the development of a communication strategy for the RAP. 28. As a complement to the narrative RAP report, an interactive dashboard will be created to enable users to view results and trends and do their own analysis. Variables and information related to the analysis for this RAP will be included in the dashboard as appropriate. Work Plan and Timeline 29. Under the direction and guidance of Nick York (Director, IEGCC) and Marie Gaarder (Manager, IEGPS), RAP 2015 will be prepared by a team including; Midori Makino (Task Team Leader), Sidney Edelmann (coordinator for IFC and MIGA), Elena Bardasi (coordinator for the gender chapter), Kavita Mathur, Eduardo Maldonado, Joy Behrens, Gisela M. Garcia, Thao Thi Phuong Nguyen, Lourdes Pagaran, Claude M.L. Leroy-Themeze, Hiroyuki Hatashima, Bahar Salimova, Yoshine Uchimira, Unurjargal Demberel, Jacqueline Andrieu, Jimena Luna, Zukhra Boyette and Nathyeli Acuna. Romayne Pereira will provide administrative support. The team will be supported by short-term consultants and by IEG staff responsible for maintaining the Management Action Record. The team will seek inputs from other IEG staff and consultants across the VPU on specific aspects of the report preparation and dissemination. 30. The peer reviewers for the report are Aart Kraay (Senior Advisor, DECMG), Marco Segone (Director, Evaluation Office, United Nations Entity for Gender Equality and the Empowerment of Women), and Rakesh Nangia (Director of the Operations Evaluation Department of the African Development Bank). Consultations with relevant Bank Group 8 management and staff will be conducted following the usual practice of IEG’s large-scale evaluations. 31. The RAP 2015 report will be prepared with an estimated budget of $650,000 (TBC) of which 15 percent will be spent in FY15. An additional $75,000 (TBC) will be budgeted for the regional updates, Summary Sector Highlights and Top Lessons for IFC and World Bank. A separate budget will be requested for wider outreach and dissemination activities upon completion of the report. The report will be finalized and presented to the Board in the third quarter of FY16. 9 Appendix A: Principal Sources of Data to be used for RAP 2015 Project evaluations or validations such as Project Performance Assessment Reports (PPARs) for both World Bank-financed and IFC projects; Implementation Completion Report Reviews (ICR-Rs) for World Bank-financed projects; Expanded Project Supervision Reports (XPSRs), IEG sector highlights, and IEG cluster notes for mature IFC investments; Project Completion Reports (PCRs) for IFC Advisory Services; and MIGA Project Evaluation Reports (PERs). Specifically, the portfolio review analyses for the World Bank within RAP 2015 will be based primarily on ICRRs for projects exiting the portfolio in FY11-15 and PPARs prepared in FY11- 15; XPSRs for IFC investments reaching early operational maturity in FY2009-14; PCRs for IFC advisory services operations closed in FY2009-14; and MIGA PERs for guarantees reaching early maturity by FY2009-14. Country evaluations, including both CPS Completion Report Reviews (prepared for 87 countries during FY11-14) and more detailed Country Program Evaluations (14 prepared during FY10- 15).   Sector, Global Practice, and thematic studies produced by IEG since FY14, including some reports that are expected to be completed by fall 2015 (see Appendix B). Learning products produced by IEG since FY14, including some studies that are expected to be completed by fall 2015 (see Appendix B). Management Action Records to capture the progress and gaps identified in degree of adoption of past IEG recommendations and to assess ongoing efforts. Databases that include IEG’s ICR-R, XPSR-EvNote, PCR-EvNote, PER-EvNote, and CASCR-R databases, as well as the World Bank’s Business I. Documents and records on relevant corporate strategies, initiatives, programs, and evaluations, including the World Bank’s Scorecard, Implementation Status Results reports of World Bank financed operations, the Country Opinion Surveys, IFC’s Development Goals and institutional reforms, and MIGA’s self-evaluation and monitoring initiatives. Other, external research will be drawn on as relevant / appropriate. 10 Appendix B: Major IEG Evaluations and Learning Products Recently Completed or Expected to Be Completed in FY14-FY16 Main IEG Evaluations Recently Completed or Expected to Be Completed in FY14-FY16 Fiscal Year Evaluation FY 14 1. RAP 2013 - Risk & Results (WB, IFC, MIGA) 2. The World Bank Group and Public Procurement 3. Learning and Results in WB Operations - Phase I 4. Investment Climate Reforms: An Independent Evaluation of World Bank Group Support to Reforms of Business Regulations 5. World Bank Group Support to Health Financing 6. World Bank Group Support to Public-Private Partnerships 7. The Big Business of Small Enterprises: Evaluation of the World Bank Group Experience with Targeted Support to Small and Medium-Size Enterprise 8. World Bank Group Assistance to Low-Income Fragile and Conflict-Affected States 9. Tunisia Country Program Evaluation (CPE) 10. Brazil Country Program Evaluation (CPE) FY 15 1. RAP 2014: Achieving MDGs (WB, IFC, MIGA) 2. The Poverty Focus of Country Programs: Lessons from World Bank Experience 3. Impacts of Interventions during Early Childhood on Later Outcomes: A Systematic Review 4. World Bank Lending for Financial Inclusion 5. Evaluation of the WBG's Support for Electricity Access (IFC, MIGA, WB) 6. Learning and Results in WB Operations - Phase II 7. Cluster CPE: Resource Rich Countries FY 16 1. RAP 2015 - Gender Equality (WB, IFC, MIGA) 2. Tertiary Education 3. Capital Market Development (IFC, MIGA, WB) 4. Job and Competitiveness 5. Process Evaluation of SCD and CPF 6. Review of Instruments: PforR 7. Assessment of WBG's Self-Evaluation Systems (ROSES) 8. WB Support to Fragile Situations in Non-FCS Countries 9. Clustered CPE on Small States 11 Main IEG Learning Products Recently Completed or Expected to Be Completed in FY14- FY16 Small Large Complex FY14  Making Roads Safer: Learning from the  Maternal Health World Bank's Experience systematic review  Avian Flu  Social Safety Nets  Adapting to Climate Change: Assessing and Gender World Bank Group Experience  PPPs in Africa (Cluster Review)  Advisory Services (Cluster Review)  Micro Finance in Africa  Lines of Credit in SME Financing FY15  Good Practice Notes for  DPOs: Public Expenditure  DPOs: Sectoral Results Framework  DPOs: Results Framework and Environment and - Rural Roads Conditionality Social Risks in  DPOs: Macro Fiscal  Additional Financing Instrument DPOs Framework in DPLs  Selectivity in Country Strategies: The  WBG  Outcome Evidence in ICRs Evidence Transformational  Lessons on Results  Housing Finance Engagements Framework - Sustainable  Opportunities and Challenges from Land Management Working in Partnerships  Good Practice Notes on SCD  Results Frameworks in Country  OECD/DAC Evaluation Strategies Network on Evaluating  Past and Future--Bank-IFC Cooperation Partnership Programs at the Country Strategy Level  Synthesis of Country Level  Microfinance in Africa Findings: Albania & Costa  IFC Additionality Rica  Manufacturing  PPP Country Case Studies  Renewable Energy  Guidance Note on Country Case Studies  Annual Sector Highlights  Review of IFC Direct Equity Investment  PSD Synthesis  Bond Market FY16  DEPO’s Financial Sector  Land Administration  Maximizing  DPO’s Environmental Sector  Migration Development  Deferred Drawdown Ops  DPO’s Political Economy Analysis Impact IDA (DDO)  Financial Management Information  Sustainable Access to Water Systems Supply and Sanitation  Review of Instruments: Direct Equity  Policy Based Guarantee  Review of Investment Funds (Instrument)  Gender in CDD  Sustainable Fisheries 12 Appendix C: Methodology for Part I of the RAP i. Assessing results of WBG operations This exercise will build on the database prepared for the 2010 IEG Gender Evaluation. The database includes 890 investment loans in 93 countries6 approved between FY02-FY08, identified as “relevant” for gender integration.7 For each project variables capturing the degree of gender integration at design stage were generated by the gender evaluation team. IEG assessed the extent of gender analysis in program documents, the extent of gender-related measures or actions, and the extent to which gender-related results were targeted and monitored. The exercise proposed for RAP 2015 consists of (i) selecting all projects (both “relevant” and “not relevant” for gender integration) included in the IEG Gender Evaluation database that closed in FY12 and have a complete ICRR. Note that the portfolio review conducted by the IEG Gender Evaluation team included data extracted from PADs and an analysis of the level of gender integration in design. Additional data on these projects from QAG will be derived if available; (ii) deriving additional information about these project (from mid-term reviews, ISRs, ICR, and ICRRs) on the indicators that were actually collected to measure gender results; and (iii) analyzing the gender-relevant results documented by operations, with specific focus on the definition of beneficiaries, gender-relevant outcome and output indicators, and other relevant gender features. Box 1 - Integration of gender in investment projects and DPOs based on the IEG Gender evaluation The 2010 IEG Gender evaluation found that 56 percent of the 890 relevant projects integrated gender into project design. Higher levels of gender integration were observed in regions with higher levels of gender inequality and/or with more binding gender-related constraints to poverty reduction (gender integration was higher in countries with a lower CPIA 7 rating). The evaluation found that there was a significant decline in gender integration in projects in 2008 (down to 46 percent). Gender integration was also assessed for 307 DPOs approved in the same period. The evaluation found that gender integration in DPOs was substantially lower than for investment loans. 6 To be included in the IEG evaluation, countries need to have a population of over 1 million people, have more than two investment projects and a prepared CAS or equivalent during the evaluation period, and have a GDI and HDI index. 7 According to the Gender Evaluation, gender was considered to be of “low” relevance if a project was likely to benefit both men and women (based on a review of its objectives, components, and social analysis) and did not include involuntary resettlement or affect indigenous peoples (IEG Gender Evaluation page 24). 13 Box 2 - Gender relevant results in ICR and ICRRs At the time of the IEG Gender evaluation, ICRs were silent on gender in about one-fourth (22) of closed projects that integrated or discussed gender issues at appraisal and had an IEG ICR review available. In 41 percent (9) of these cases, IEG ICR Reviews addressed gender issues; in the other 59 percent (13), IEG did not note the ICR’s failure to discuss gender issues or consider this in rating outcome or ICR quality. Feasibility: Merging the FY12 ICRR dataset with the IEG Gender Evaluation database, and focusing only on projects considered ‘relevant’ for gender based on the definition adopted in the IEG evaluation, 106 projects were found to respond to the given criteria (closed in FY12, with an ICRR, and gender relevant). These projects are overall well distributed across regions (see table below).8 This sample, while not random, can be considered orthogonal to the outcome of interest. The final sample will include an additional 27 projects that were not considered ‘gender- relevant’ by the 2010 Gender Evaluation (and are expected to include very little or no information on gender). Region Frequency % Sub-Saharan Africa 39 36.8 East Asia and Pacific 19 17.9 East and Central Europe 20 18.9 Latin America and the Caribbean 9 8.5 Middle East and North Africa 5 4.7 South Asia 14 13.2 Total 106 100.0 These projects will be complemented by IFC projects which were evaluated in FY12-FY14. ii. Assessment of gender integration in country strategies and country paper evaluations This exercise consists of reviewing methods and approaches adopted by recently completed country-focused evaluations with the intent to identify successful and (if possible) efficient approaches to assess gender outcomes. Sampling strategy: select country strategy completion reports reviewed by IEG in FY12 to FY14. Analysis: The following information will be collected, based on CPS and CPSCR, CPSCR Reviews, CPE and diagnostic/ESW referred work: (i) gender-relevant analysis included in ESW and other diagnostic work (including Gender Assessments); (ii) Gender-relevant information in 8 Only exception seems to be LCR with only 9 projects. 14 country strategies pillar/projects; and (iii) indicators included in M&E framework to track results/social audits/feedback mechanisms. 15 Appendix D: References APQC. 2012. “Using Metrics that Drive Bottom-Line Value: An APQC Best Practices Study.” Denizer, Cevdet, Daniel Kaufmann, Aart Kraay. World Bank DEC Working Paper. 2011. “Good Countries or Good Projects? Macro and Micro Correlates of World Bank Project Performance.” Ika, Lavagnon, Amadou Diallo, and Denis Thuillier. 2012. “Critical success factors for World Bank projects: an empirical investigation.” International Journal of Project Management 30(2012): 105-116. Independent Evaluation Group (IEG), World Bank. 2010. Gender and Development: An Evaluation of World Bank Support, 2002-08. Washington, DC: World Bank. Independent Evaluation Group (IEG), World Bank. 2013. Results and Performance of the World Bank Group 2012. Independent Evaluation Group (IEG), World Bank. 2014. Results and Performance of the World Bank Group 2013. Independent Evaluation Group (IEG), World Bank. 2014. Social Safety Nets and Gender: Learning from Impact Evaluations and World Bank Projects. Washington, DC: World Bank. Independent Evaluation Group (IEG), World Bank. 2015. Results and Performance of the World Bank Group 2014. Kilby, Christopher. Center for Economic Research. 1995. “Supervision and performance: The case of World Bank projects.” Kilby, Christopher. March 1995; Revised September 2000. “World Bank borrower relations and project supervision.” Kilby, Christopher. 2014. “Assessing the impact of World Bank projects on project outcomes.” (Department of Economics, Villanova University, USA, 2014) Geli, Patricia, Aart Kraay, and Hoveida Nobakht. World Bank DEC/OPCS Working Paper. 2014. “Predicting World Bank project outcome ratings.” Operations Evaluation Department, World Bank. 1994. Annual Review of Evaluation Results 1994. Chapter 2. P.26. World Bank Intranet. Foundations of Bank Operations (FBO) Reference Guide. http://intresources.worldbank.org/INTOPCS/Resources/380831-1294415423637/FBO_RG-November2013-midsize.pdf Accessed 5/1/2015. World Bank Corporate Scorecard. October 2014. Accessed 5/1/2015. http://www- wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2014/10/09/000456286_20141009104938/Rendere d/PDF/913110WP0World00Box385295B00PUBLIC0.pdf Accessed 5/1/2015. Tier III. World Bank Corporate Scorecard. April 2015. Tier III. World Bank/McKinsey. 2012. Gallery Walk presentation on organizational assessment. World Bank OPCS Working Paper. 2012. “Delivering results by enhancing our focus on quality.” World Bank. 2012. World Development Report 2012: Gender Equality and Development. World Bank and Oxford University Press. 2001. “Engendering Development Through Gender Equality in Rights, Resources, and Voice.” World Bank policy research report. 16 Appendix E: Preliminary Stakeholder Analysis Analysis and grouping of stakeholders comprising audiences for this report, based on experience in previous IEG evaluations and discussion with WBG management and advisors to Executive Directors (EDs). Stakeholder Their interests Types of decisions this stakeholder makes Stakeholder Group (or stakeholder Group) (what they want from RAP2015) (that RAP2015 could feed) 1 WBG Board (as a whole)  Information on how the WBG is  Project approval performing – Timely feedback  Institutional priorities  Recent vs. older Board  Large vs. small projects  Meaningful integration of gender considerations in design and implementation 2 WBG Board, CODE  What works (what achieves  Advice to Board on criteria for project development results, both in approval? general and gender-related?)  Topics for future IEG evaluations  What doesn’t work  Priorities for OPCS?  Characteristics of successful vs unsuccessful projects 3 ED Advisors  View of WBG operations as a whole  Recommendations to EDs and gender-specific  Comparative advantage of WBG 4 WBG Senior Management  Twin goals: Eliminating poverty,  WBG strategy, priorities Boosting shared prosperity  WBG budget  Prosperity shared across genders WBG Senior  A successful change process Management  Meaningful integration of gender considerations into project design and implementation 5 GP Senior Management  How this GP is doing vs other GPs  GP strategy, priorities  GP budget 17 Stakeholder Their interests Types of decisions this stakeholder makes Stakeholder Group (or stakeholder Group) (what they want from RAP2015) (that RAP2015 could feed)  What kinds of work to prioritize to achieve better results in the practice area 6 Region Senior Management  What results are being achieved in  Region priorities this Region vs others, overall and  Relationships with country clients gender-related?  What results are being achieved in particular countries, overall and gender-related? 7 Country Directors  What results are being achieved in  Content of country portfolio the country, overall and gender- related?  Productive relationships with country counterparts  Attention to priorities of the country 8 Gender Cross Cutting Solution  How does integration of gender  How best to integrate gender meaningfully Area and Gender Strategy Team relate to project outcomes? into CPFs  Lessons for new gender strategy?  How best to integrate gender meaningfully into projects 9 IFC Senior Management  Financial performance – overall and  Funding priorities gender-specific  Development outcome predictability  Development impacts for  Advisory priorities investment and advisory projects –  Measures to improve work quality overall, gender-related, gender- specific  Work quality  Additionality MIGA Senior Management  Development outcomes for  Prioritization of new projects by sector guarantee projects 18 Stakeholder Their interests Types of decisions this stakeholder makes Stakeholder Group (or stakeholder Group) (what they want from RAP2015) (that RAP2015 could feed) 10 OPCS  Reporting on portfolio (e.g. at  How to analyze operational portfolio ABCDQ meetings)  Content of operational policies and guidance OPCS and other  Reporting on corporate  Improvements to systems, tools; simplification WBG performance (Corporate Scorecard), stakeholders including for indicators related to gender integration  Management of safeguard policies 11 IEG Senior Management  High quality, credible work  Focus of future IEG evaluations  Understandable to stakeholders  Prioritization around communication of  Ability to provide a timely message evaluation results 12 Development effectiveness  How to promote higher quality  Advice on results frameworks, M&E in functions within the GPs, project design, results frameworks, projects Regions M&E  How to integrate gender meaningfully in project design, results frameworks, and M&E 13 TTLs and IOs of Operational  Preparing the next project  Where to spend their time Projects  Supervision of ongoing projects  How to structure future projects  How to integrate gender  What to focus on during supervision meaningfully into project design and implementation – including results frameworks and M&E 14 DEC  Understand trends in WBG  Priorities for future research Others performance, overall and gender-  Opinion on IEG’s work related 15 External stakeholders – NGOs,  Understand trends in WBG  Where to focus advocacy around poverty CSOs performance, overall and gender- reduction related  Where to focus advocacy around gender 16 Others? 19