evaluations that matter 20 16 a n n u a l r e p o r t About IEG The Independent Evaluation Group (IEG) is an independent The goal of IEG’s evaluation work is to influence the World Bank unit within the World Bank Group. It reports directly to the Group’s ability to achieve development outcomes globally and World Bank Group’s Boards of Executive Directors, which with its member countries by providing impartial, evidence- oversee IEG’s work through the Committee on Development based assessments and lessons on drivers of success and Effectiveness. IEG is charged with evaluating the activities of failure. IEG shares these insights through a variety of channels the World Bank (the International Bank for Reconstruction and and engages stakeholders to ensure that lessons are learned Development and the International Development Association), and necessary improvements are made. the International Finance Corporation (IFC), and the Multilateral All evaluations discussed in this report are available on IEG’s Investment Guarantee Agency (MIGA). website: http://ieg.worldbank.org How IEG contributes to the work of the World Bank Group contents IEG seeks to accelerate World Bank Group’s development Message from the Director-General 2 effectiveness through independent evaluation. 1 Evaluations that Matter: An Overview of IEG’s We aim to deliver: Evaluation Activities During the Year 4 Independent, credible, and useful evaluations 2  Influence and Knowledge that Matter 20 Impartiality  through the employment of rigorous methods that promote triangulation of findings from 3R  enewing and Strengthening Our Organization 26 a cross-section of sources 4L  ooking Forward 30 New  insights through the choice of what we evaluate; how we formulate evaluation questions; and which Appendices 36 methodologies we use, including the use of the latest technology Learning  through outreach activities that target our key stakeholders, including the Boards of Executive Directors, World Bank Group management and staff, and the development and evaluation communities Developing  evaluation capacity for the World Bank Group staff, clients, and partners. message from the director general This has been an important year for In order to achieve the twin goals in a sustainable manner, the World the development community and the Bank Group has to be able to learn from its successes and failures, to World Bank Group. better understand what works and the reasons why, and to use these insights to inform decision making and future action. In September 2015, world leaders came together to adopt the new Sustainable As the 2015 External Review of IEG noted, “The success of the Bank Development Goals (SDGs), which will [Group’s] strategy depends heavily on creating a results-oriented underpin a coordinated global effort to end extreme poverty, fight culture of learning within the [World Bank Group], and on the steady inequality and injustice, and fix climate change. In December, flow and use of high-quality, timely, and context-sensitive self and 175 countries and the European Union made commitments to reduce independent evaluation evidence about what does and does not global carbon emissions and limit global warming to 2ºC (3.6ºF). work for cycles of course corrections, continuous improvement and accountability across the larger Bank system….” Along with renewed optimism, the year brought many new challenges —including a slowing global economy, political and economic volatility, For IEG, this means we must be able to deliver evaluation insights new pandemics, and a deepening refugee crisis, among others. when they matter to inform stakeholder decisions in real-time. Amidst all these changes, the World Bank Group remains an To this end, IEG has realigned its work to better support the important actor on the global stage. Demand for World Bank Group implementation of the World Bank Group’s strategic new directions support continues to grow. Total commitments rose to more than and to deliver timely, relevant evaluation evidence to strengthen the $61 billion for the fiscal year, ending June 30, 2016. Internally, the World Bank Group’s development effectiveness. World Bank Group is evolving its strategy in a changing world, building on the twin goals to eliminate extreme poverty by 2030 and boost shared prosperity, measured as the income of the bottom 40 percent in any given country. 2 INDEPENDENT EVALUATION GROUP Evaluations that matter… to the World Bank Group’s operational priorities and maximize IEG’s Our theme for this year’s annual report, “Evaluations that Matter,” contribution. These are inclusive growth, sustained delivery for the reflects IEG’s commitment to influencing the World Bank Group’s poor, and environmental sustainability. agenda through insights that are timely, useful and targeted at the We continued to refocus our work around two strategic objectives: most critical development challenges. deepening evidence on the results of World Bank Group activities (what During 2016, we took several important steps to position IEG’s work works) and generating evidence on the early implementation experience for greater influence and impact within the World Bank Group and the of the World Bank Group’s new strategy (real-time learning). broader development community. IEG continues to engage constructively with the World Bank Group’s Our major evaluation products—at the sector, thematic, country, Board and management. One of the key areas of our engagement and project level—focused on some of the most critical challenges during the last year addressed the recommendations of the 2015 faced by the World Bank Group and its member countries. Key External Review of IEG. In this report, we highlight some of the evaluations during the year looked at the World Bank Group’s role actions that we are implementing, in response to the review. in supporting capital markets development; small states; fragility, I am excited at what we were able to accomplish this year. With the conflict and violence prevention; electricity access; jobs and industrial launch of our new strategic engagement areas, IEG has a unique competitiveness; and a number of other important areas. At the opportunity to contribute more effectively to the World Bank Group’s corporate level, we assessed the Bank Group’s self-evaluation priorities in the coming years. I look forward to our continued systems. To ensure that our evaluations indeed mattered, we also engagement. expanded our outreach efforts to various key stakeholder groups within and outside the World Bank Group. Internally, we reorganized our structure and introduced three Caroline Heider over-arching Strategic Engagement Areas to better link our work Director General, Evaluation 2016 ANNUAL REPORT 3 1 generating insights improving outcomes 4 INDEPENDENT EVALUATION GROUP evaluations that matter An Overview of IEG’s Evaluation Activities during the Year In keeping with our goal of informing the World Bank Group’s activities and operations with timely, relevant evaluations, IEG refocused its work during the year. We introduced three strategic engagement areas (SEAs)—Inclusive Growth, Sustained Service Delivery for the Poor, and Environmental Sustainability. The strategic engagement areas aim to drive a more selective, coordinated, and focused approach within the organization and to align IEG’s evaluation work more closely with the World Bank Group’s priorities. The SEAs are also a platform for more effective stakeholder engagement —ensuring that IEG and Bank Group management engagement, for example, is focused on the most critical development challenges. IEG delivered a number of influential products during the year under each of the strategic engagement areas. 2016 ANNUAL REPORT 5 Inclusive Growth Generating growth that creates jobs, and is inclusive, The SEAs are also a platform for more “ resilient and sustainable is a major development challenge. Over the year, IEG completed three evaluations to assess effective stakeholder engagement— the World Bank Group’s role in this important area. ensuring that IEG and Bank Group management engagement, for example, Jobs and Competitiveness Productivity, competitiveness, and more and better jobs are is focused on the most critical key to economic development and are at the top of government development challenges.” development agendas. Supporting industry competitiveness has been an important part of World Bank Group activities. IEG evaluated the World Bank Group’s support towards promoting industry productivity and competitiveness, and examined the implications of improved industry competitiveness for employment. The evaluation highlighted the problem of deindustrialization in developing countries, particularly in Africa and the World Bank Group’s role in providing support for manufacturing and industry development. IEG found that the World Bank Group’s support was strongest in the areas of supporting country priorities and country-level competitiveness, but less so in terms of promoting competitiveness around management skills and manufacturing. Overall, the WBG has not had a distinct, overarching approach to supporting industry competitiveness in the last decade. IEG’s report recommends that the WBG clarifies its approach to industry level support for competitiveness, while paying closer attention to its job effects. 6 INDEPENDENT EVALUATION GROUP Upcoming evaluations in FY17 in the area of inclusive growth will look at the World Bank Group’s activities in the areas of tertiary education, FINANCING FOR DEVELOPMENT and rural non-farm employment. In July 2015, Finance Ministers Financial Markets and development partners convened Mobilizing financing for development is a critical element in the drive in Addis Ababa, Ethiopia, to discuss towards eliminating extreme poverty, boosting shared prosperity and achieving the new sustainable development goals. Recognizing the strategies for financing the global constraints on public finance and overseas development assistance, development agenda. many countries are increasingly turning towards market-based solutions. IEG participated in the conference and contributed a IEG’s evaluation of the World Bank Group’s contribution to capital synthesis paper to inform the discussions. Drawing relevant markets development sought to assess whether the World Bank insights from evaluating World Bank Group projects, IEG Group has been relevant, effective and efficient in supporting the outlined several examples and experiences, highlighting development of domestic capital markets in client countries. what has worked in raising development finance, and A related learning product on the World Bank Group’s Support to how to increase the value for money through improving Housing Finance drew lessons from past experience to explore how effectiveness, efficiency and outcomes. best to meet the Sustainable Development Goals’ target of ensuring access for all to adequate, safe, and affordable housing by 2030. The The paper is available at: http://ieg.worldbank.org/f4d report also underscored the importance of supporting functioning capital markets, as a basis for mobilizing sustainable housing finance. 2016 ANNUAL REPORT 7 8 INDEPENDENT EVALUATION GROUP Sustained Service Delivery for the Poor A follow-up assessment in 2016 looked at the financial sustainability Access to social services are essential for poverty of the electricity sector in developing countries—an issue of eradication. Although many countries devote increasing paramount importance, particularly in Sub-Saharan Africa. In proportions of their spending to essential public services addition, work is underway to assess the potential for “off-grid such as health and education, it is unclear how much of electrification”—to be finalized the coming fiscal year. this actually reaches poor people. IEG’s work in this area Upcoming evaluations in FY17 will look at four areas—urban focuses on identifying what works and what doesn’t in transport, water supply and sanitation, nutrition, and gender impacts service delivery within an environment that underscores in community driven development. the need for client responsiveness, sustainability, voice for the poor, and overall accountability and good governance. Environmental Sustainability To this end, IEG is refining its methodological frameworks to better Promoting sustainable patterns of consumption and assess the service delivery models and behavior change interventions production and protecting and managing the natural that underpin the World Bank Group’s focus on sustained service resource base are at the core of the World Bank Group’s delivery to the poor. twin goals of eradicating extreme poverty and boosting shared prosperity. IEG’s work in this area aims to galvanize Providing Sustainable Access to Electricity attention and focus on environmental sustainability across In 2015, IEG completed a major evaluation, The World Bank Group all Bank Group projects. Support to Electricity Access, looking back over a 15-year period. The evaluation pointed out that low access countries (most of them Policy Lending for Environment in Sub-Saharan Africa) would collectively need a 10-fold increase in IEG has been working on various aspects of development policy the pace of new connections if the global goal of sustainable energy finance and operations to derive lessons from the World Bank’s for all by 2030 was to be realized. experience with this important instrument. Since the early 1980s, the World Bank has provided over $300 billion in budget support to governments around the world. Over the last decade, budget support averaged about a quarter of total World Bank financing. 2016 ANNUAL REPORT 9 In FY16, IEG completed two reviews on development policy financing. Over the last 12 years, FY2004–15, the World Bank Group approved One looked at the Use of Political Economy Analysis in the a total of 3,870 projects, accounting for approximately US$297 billion Design and Implementation of Development Policy Financing, in commitments that are pollution-relevant. To date, the effectiveness while the second looked at the specific lessons to be drawn from the of these interventions has not yet been evaluated. World Bank’s environmental policy lending. The two products To address this gap, IEG this year launched an evaluation of the followed an earlier review in FY15 that looked at the quality of results World Bank Group’s support for pollution management, as part of frameworks in development policy operations. The products were our ongoing emphasis on deepening the Bank’s knowledge about widely disseminated and well received, in part, because they were its effectiveness in the area of environmental sustainability. IEG’s timed to coincide with and inform the World Bank’s own three year evaluation will be the first stock-taking exercise focusing on those review of its development policy operations. pollution phenomena that affect poor countries the most, that is, IEG’s assessment of the World Bank’s use of development policy air and water pollution, and waste. The evaluation will be a timely financing for projects in the environmental sector aimed at contribution to the global efforts to fight climate change. drawing insights on how to design and implement this instrument, outlining some of the tensions and tradeoffs. Land Administration Secure land rights underpin many development objectives across The World Bank’s environmental policy lending has grown rapidly multiple sectors. In agriculture, housing, transport, energy and since 2005, with operations supporting actions across a broad range extractives, land serves as a critical means of production. Clear of subsectors, including climate change mitigation, climate change and secure land rights can also play a role in reducing conflict and adaptation and disaster risk management, environmental protection, environmental degradation. And for private sector development, pollution management, institutional strengthening, and specific secure and transferable land rights are critical elements of the sectoral reforms in energy, transport, water, agriculture, forestry, enabling environment. and other sectors. The World Bank Group has a long history of developing and The 2012 World Bank Group environmental strategy, Toward a Green, implementing projects aimed at securing land rights. As a global Clean, and Resilient World for All, explicitly defines a “clean world” as development institution, the World Bank Group is often a central a strategic objective—a world with “low pollution and low emission.” player at the intersection of development and land administration or land rights. 10 INDEPENDENT EVALUATION GROUP 2016 ANNUAL REPORT 11 In FY16, IEG completed a review of the World Bank Group’s issues and generate broader development outcomes to which experience with land administration, as part of a renewed effort to land issues are a critical constraint. generate learning insights for practitioners in the land and related Upcoming evaluations under the environmental sustainability sectors. IEG looked at 14 World Bank Group land administration engagement area will cover other key areas, including carbon projects implemented between 1997 and 2011. IEG’s report, Lessons finance, renewable energy, environmental sustainability in from Land Administration Projects: A Review of Project Performance World Bank Group operations, environmental degradation, Assessments, highlights institutional tensions that the World Bank and sustainable cities. Group must navigate to more effectively address land administration TRANSFORMING OUR WORLD—Aiming for Sustainable Development In September 2015, the new Sustainable Development Goals (SDGs) were adopted by the UN General Transforming Our World – Assembly. Over 190 countries signed up to the goals, which set out a path to end extreme poverty, Aiming for Sustainable Development fight inequality and injustice, and fix climate change. Using Independent Evaluation to Transform Aspirations to Achievements IEG contributed a synthesis paper on the experience and results of the Millennium Development Goals in order to draw relevant insights for the new SDGs. The paper highlighted the challenge of translating global goals into sustainable national programs, and pointed to the importance of ensuring country ownership and promoting knowledge flows between countries and with development institutions. IEG’s evaluations also point to the need for the World Bank Group to have a clearly articulated role, approach and expected contribution to the SDGS, both externally for enhancing partnerships and internally to facilitate synergies and priorities. 12 INDEPENDENT EVALUATION GROUP Client Country-Focused Evaluations These include exploring opportunities beyond supporting livelihoods in conflict- With our country-focused evaluations, we draw attention affected communities, such as support for private sector development, using to the shared challenges that the World Bank Group’s the World Bank Group’s expertise in public financial management more client countries face. These evaluations stimulate effectively, and improving its monitoring and evaluation frameworks. discussions to find common solutions across the globe, including through south-to-south knowledge sharing. Small States More than a quarter of the World Bank Group’s members are countries with World Bank Group Engagement in Situations of Fragility, populations under 1.5 million. Although these countries vary greatly in their Conflict, and Violence level of development and the size of their economies, many of them share Following an earlier evaluation into the World Bank Group’s activities a number of intrinsic characteristics and development challenges. In many in low income fragile and conflict-affected states, IEG completed of these countries, the World Bank Group is the biggest development partner. a second evaluation during the year looking at the impact of conflict IEG’s evaluation of the World Bank Group’s engagement in small states and violence in non-fragile situations, focusing mainly on middle- looked at how effectively the World Bank Group supports smaller countries, income countries. given the unique challenges they face, including limited institutional capacity, IEG’s evaluation, World Bank Group Engagement in Situations of acute vulnerability to economic and natural shocks, and an inability to exploit Fragility, Conflict, and Violence, noted that middle-income countries economies of scale. The report covered a ten year period and drew on pose distinct challenges from their low-income counterparts. IEG separate regional evaluations of World Bank Group supported programs in the found that while the Bank Group has been adept at responding and at Organization of Eastern Caribbean States (OECS), the Pacific Island Countries adjusting its strategies and analytical support to situations of violence (PICs), and in Cabo Verde, Djibouti, Mauritius, and the Seychelles. and conflict, its operational response has been constrained by its The report shares key findings that can inform future Bank Group limited menu of instrument choices. Moreover, institutional and staff activities and help to further the effectiveness of Bank Group engagement incentives seem to be lagging behind the spirit of its strategic approach, including issues such as regional approaches, selectivity of engagement, as expressed in various Bank documents, including the 2011 World responsiveness to shocks, client institutional capacity constraints, Development Report on conflict, security, and development. and partnerships. IEG’s evaluation highlights several findings and IEG’s evaluation outlines a number of recommendations for how the recommendations, particularly with respect to strengthening resilience World Bank Group can further enhance its impact in middle-income and improving competitiveness of small states. countries facing fragility, conflict and violence. 2016 ANNUAL REPORT 13 EVALUATION INSIGHTS THAT MATTER—Sharing Experiences Across Resource-rich Developing Countries With a population of 15 million, Zambia has a nagging and stubbornly persistent average poverty rate of over 60 percent as of 2015, rising to nearly 80 percent in rural areas. The country is highly dependent on the copper industry that itself is ailing under dramatic commodity price volatility. The industry is both a challenge and an opportunity for the country, as it is for many similar ones. At the invitation of the government of Zambia and the World Bank Zambia country office, IEG shared the findings of its report, World Bank Group Engagement in Resource-Rich Developing Countries: The Cases of the Plurinational States of Bolivia, Kazakhstan, Mongolia, and Zambia. Over 50 senior government officials attended a workshop to discuss how Zambia could learn from IEG’s presentation. This was followed by another high-level presentation, hosted by hosted by Zambia’s Cabinet Secretary, Dr. Roland Msiska and attended by permanent secretaries and heads from Zambia’s key ministries. 14 INDEPENDENT EVALUATION GROUP Corporate Evaluations IFC’s advisory services projects continued to perform well. IFC projects in IDA and blend countries also showed stable results, Results and Performance of the World Bank Group despite the perceived higher risk associated with investing in these IEG’s annual flagship report, the 2015 Annual Results and markets. However, the rating for investment projects declined, Performance of the World Bank Group report, looks at the continuing the downturn trend first reported in 2013. IFC investment performance of projects completed during FY12–14 and provides a project development outcomes were rated at 58 percent for 2008–14, detailed assessment of the factors driving project outcomes across far short of the 65 percent World Bank Group corporate scorecard regions, business units, and sectors. target for FY17. At institutional level, IEG found that project performance at the The report also featured a thematic chapter on the World Bank International Bank for Reconstruction and Development (IBRD) and Group’s experience in mainstreaming gender throughout its International Development Association (IDA) remained stable over the operations. IEG’s analysis suggests that while the World Bank Group review period, in line with previous years. The overall performance has made significant progress over the last fifteen years—including of World Bank (IBRD and IDA) projects with outcomes rated as the recent launch of a new gender strategy—a number of key moderately satisfactory and above stabilized at 70 percent. This is challenges remain. still short of the corporate target of 75 percent by FY17. Investment project financing performance—the largest lending instrument type in Overall, IEG found that while the number of Bank operations and number and commitment—mirrored overall World Bank performance country strategies that address gender is up significantly, the quality during FY12–14 at 69 percent. About 78 percent of development and depth of gender integration remains uneven. To address the policy financing projects were rated moderately satisfactory and quality gaps, IEG calls on the World Bank to move decisively towards above. Weighted by net commitment, the performance of both documenting on results rather than intentions. Gender must be an investment and development policy financing exceeded the FY17 integral part of World Bank Group country programs. corporate target of 80 percent. Over the year, IEG engaged the World Bank’s management to At the Multilateral Investment Guarantee Agency (MIGA), 63 percent of explore how to address these issues. The new gender strategy, the 56 guarantee projects evaluated by IEG in the FY09–14 period were which was launched in December 2016, reflects many of IEG’s rated as having achieved satisfactory or above in terms of development recommendations. outcomes, which represents a stable trend over the review period. 2016 ANNUAL REPORT 15 Reviewing the World Bank Group’s Self-Evaluation Systems Robust results frameworks and monitoring and evaluation systems EVALUATION INSIGHTS THAT MATTER— are key determinants of project success. When they function well, Maximizing Development Impact in IDA self-evaluation systems can generate critical insights to improve performance and promote knowledge sharing within operations. IEG’s Every three years, donors come together to review assessment of the World Bank Group’s self-evaluation systems the work of the International Development Association (ROSES), completed during the year, looked at how effective current self-evaluation systems are in supporting performance management, and discuss its funding requirements. This year marks reporting project outcomes, and generating relevant insights for the next cycle in the replenishment of IDA. The last knowledge sharing and enhanced operational quality. replenishment (IDA 17), which was finalized in 2013, IEG found that while compliance levels are mostly strong across raised $52.1 billion to finance projects over a three-year the World Bank Group, they focus too much on reporting and period ending June 30, 2017. accountability, and do not provide the information necessary to help the Bank Group learn and enhance performance. Information In order to support the IDA18 replenishment generated through the systems is not regularly mined for knowledge discussions, IEG prepared a synthesis report to share and learning, and its use for project and portfolio performance management can be improved. findings from recent evaluations, offering evaluative evidence on what has worked and has not worked Review of Program for Results in IDA priority areas. The report was shared with At the request of the World Bank Group’s Board, IEG undertook an IDA Deputies, as an input into the first of four formal evaluation of the Program-for-Results (PforR) lending instrument meetings taking place in 2016. in order to assess its design and implementation performance, and identify potential opportunities to further strengthen the instrument. Although a relatively new product, the PforR instrument has built up a significant portfolio in the three years since it was launched 16 INDEPENDENT EVALUATION GROUP 2016 ANNUAL REPORT 17 WORLD BANK GROUP TRANSFORMATIONAL ENGAGEMENTS Successful transformational engagements change the fortunes of countries, which is why they are a critical component of the World Bank Group’s strategy to end extreme poverty and promote shared prosperity in sustainable ways. Using evaluative evidence from 2000 to 2014 and data from 1990 to 2013, an IEG report, Supporting Transformational Change for Poverty Reduction and Shared Prosperity, looked at what differentiates transformational projects from other types of engagement. IEG’s report extracts lessons from the Bank Group’s experience to inform the design and implementation of interventions with the potential for fundamentally improving the lives of poor and disadvantaged people. The report identified several factors that contributed to successful transformational change, including: a supportive enabling environment such as the full alignment of the interest of the client with the transformational objectives; a rigorous diagnosis and clear understanding of the most binding constraints; adapting interventions to local context, capabilities, and social norms; early and broad engagement with stakeholders; building and strengthening local institutions; employing program design that combines a strategic plan to address the main constraints through focused interventions; providing continuing, sustained support to clients; the quality and technical and entrepreneurial skills of Bank Group staff; and robust monitoring and reporting systems to target interventions and adjust programs in implementation. 18 INDEPENDENT EVALUATION GROUP in 2012. As of As of March 31, 2016, the Board had approved Validation of Self-Evaluations 39 operations, providing US$9.4 billion of financing to support The World Bank Group conducts self-evaluations for most US$49.9 billion in government programs, with a pipeline of of its projects in order to assess performance, results 21 operations under preparation, totaling US$5.4 billion in and impact. A key part of IEG’s work program is the expected Bank financing. validation of project self-evaluations completed by World Bank Group staff. In FY16, IEG validated 266 World Bank IEG’s evaluation found that overall, the PforR instrument has proven implementation completion reports, 22 country assistance to be appropriate, well-structured and relatively effective in driving strategy completion reports (CLRs), 47 project completion a stronger results focus than other Bank lending instruments. IEG reports for IFC advisory services, 76 expanded project recommendations mostly touched on the need to strengthen program supervision reports for IFC investments, and 17 MIGA design, monitoring and results reporting. project evaluation reports. Project-Level Evaluations IEG project-level evaluations mainly consist of project performance assessment reports and implementation completion report reviews. Both are carried out after World Bank funds have been fully disbursed to a project. In FY16, IEG realigned its selection criteria for project performance assessments in order prioritize projects that relate to future evaluation work, high-risk projects, innovative projects, and projects where self- evaluation ratings significantly differ from IEG ratings. IEG completed 49 project performance assessments in FY16. These included two assessments of private sector projects—part of a new pilot initiative to introduce project-level assessments for IFC projects. 2016 ANNUAL REPORT 19 2 knowledge shared lessons applied 20 INDEPENDENT EVALUATION GROUP influence and knowledge that matter IEG continues to be rated highly by its stakeholders for the relevance, quality, and impact of its work. In the 2016 client survey, the World Bank Group board, World Bank Group staff and management, and external stakeholders rated IEG positively on most indicators. 85 percent of respondents rated IEG’s independence as very high. Between 85 and 95 percent of respondents across all three audiences perceived IEG’s work as very relevant to the World Bank Group’s mission. On quality, overall satisfaction with IEG products was rated 74 percent. Over the last year, IEG has taken steps to enhance knowledge, learning and communications, as part of a concerted effort to increase the influence and uptake of its evaluation insights. 2016 ANNUAL REPORT 21 Sharing IEG knowledge with the broader development community is also a priority for IEG. And to this end, IEG cohosted a number “ IEG’s relationship with the World of external outreach events. These included a seminar on financial inclusion cohosted with the Central Bank of Tanzania, a workshop Bank Group management has been on electricity access cosponsored with the African Development positive and open, creating Bank, and a high-level workshop for government leaders in Zambia cosponsored with the World Bank country office, which shared the a more conducive environment findings of IEG’s report on resource-rich countries. for greater ownership of IEG continues to invest in growing its online presence through its recommendations and absorption website, social media and other online distribution partnerships. In FY16, the IEG website registered 689,509 page views and 129,851 of lessons.” unique visitors. Total report downloads for the year reached 32,021, a 57 percent increase from the year before. Our social media channels on LinkedIn, Facebook, Twitter, and YouTube continue to be important vehicles for engaging with IEG’s stakeholders and all performed well in terms of subscriber and usage engagement. The IEG email newsletter grew its subscriber base in FY16 to over 15,000, In FY16, IEG hosted a number of internal dissemination events and making it one our most important external communications vehicles. three external events in support of key evaluations. IEG also delivered several learning-focused engagements over the year, reaching IEG continued to test innovative formats and templates to make hundreds of participants across the World Bank Group. Highlights IEG reports more readable and accessible. For example, a new included the IEG-hosted workshops for staff working on electricity interactive version of the FY15 Social Safety Nets and Gender report access, land administration, housing finance, financial inclusion. was piloted, which enables end-users to rapidly search for and IEG was also part of the annual Results and Measurement (RMES), locate specific content on the most common topics of interest where we hosted 18 out of the 33 sessions, reaching almost 400 (see: http://ieg.worldbankgroup.org/ssn/). participants. 22 INDEPENDENT EVALUATION GROUP IEG ONLINE DISSEMINATION AND OUTREACH FY16 Full Year Highlights Most Read Blogs 689,509 website views 12,559 followers Fin4Dev: Learning from the Past for (down 21%) (up 10%) a New Future (7,618 views) 151,558 blog views 77,374 followers Influencing Change through Evaluation: (up 46%) (up 12%) What is the Theory of Change? (7,424 views) Average time on site 2,454 connections Institutionalizing Evaluation: What is (up 100%) 740 company page followers the Theory of Change? (3,504 views) (up 51%) Pages per session Transformational development projects— (up 69%) 268 subscribers what makes them different? (3,448 views) (up 36%) Evaluation Beyond 2015: Implications of the SDGs (3,421 views) Report Downloads  32,031 (up 57%) Most Downloaded Reports Guidelines IEG Work for Reviewing Program and ICRRs— Budget A Manual CONFE R E NCE E DITION OV E RV IE W OV E RV IE W CONFE R E NCE E DITION OV E RV IE W OV E RV IE W OV E RV IE W World Bank Group Engagement in Resource-Rich Results and Performance of the Learning and Results in Financial Inclusion: A Foothold on Developing Countries: The Cases of the Plurinational A N N UA L R E P O R T 20 15 World Bank Operations: Toward Results and Performance of the World World Bank Group Support to World Bank Support State of Bolivia, Kazakhstan, Mongolia, and Zambia World Bank Group 2015 the Ladder toward Prosperity? Clustered Country Program Evaluation Synthesis Report A N I N D E PE N D E N T E VA LUATI O N deepening impact a New Learning Strategy Bank Group 2014 Electricity Access, FY2000-2014 to Early Childhood Development An Evaluation of World Bank Group Support for Financial Inclusion A N I N D E PE N D E N T E VA LUAT I O N A N I N D E PE N D E N T E VA LUAT I O N A N I N D E PE N D E N T E VA LUAT I O N for Low-Income Households and Microenterprises A N I N D E PE N D E N T E VA LUATI O N E VA LUAT I O N 2 Resource- 2015 RAP 2015 Annual Learning and 2014 RAP Electricity Guidelines Early Financial IEG work Rich Cluster (964) Report (928) Results 2 (613) Access for reviewing Childhood Inclusion program and Report (714) (560) ICRRs—a Development (446) budget (443) (1,211) manual (553) (488) 2016 ANNUAL REPORT 23 A key part of IEG’s outreach strategy is build external partnerships. Management Action Record In FY16, IEG continued to explore content distribution partnerships IEG systematically and publicly tracks the progress made by World inside and outside the World Bank Group. For the first time ever, IEG Bank Group management in implementing actions the organization materials are now being distributed through the World Bank Group’s committed to in response to IEG evaluations’ recommendations. Open Learning Campus portal, bring IEG’s evaluation knowledge The Management Action Record (MAR) offers insights into the accessible to the millions of people who visit the World Bank Group’s nature of individual recommendations and the status of action external website. IEG reports are also accessible through the World implementation—and progress here is separately assessed both by Bank Group’s Open Knowledge Repository and publications unit. Bank Group management and by IEG for four years subsequent to an evaluation’s disclosure. Every year, the MAR goes through an update cycle to track progress made by World Bank Group in implementing agreed upon actions to IEG evaluations. This information is shared publicly through the IEG website. In FY16, IEG and World Bank Group management completed a round “IEG’s work continues to be of ratings and feedback on many of the recommendations from IEG perceived as very relevant to the evaluations. There are currently 152 recommendations in the MAR, dating back to 2012, the majority of which show substantial adoption World Bank Group’s mission, with and progress by World Bank Group Management. Overall, IEG’s positive ratings [from the annual relationship with the Management of the World Bank Group has been positive and open, creating a more conducive environment for greater client survey] ranging between ownership of recommendations and absorption of lessons generated 85 and 95 percent across all three by IEG. respondent audiences.” At the same time, we have been able to “agree to disagree” when our positions substantively differ. In our role as independent evaluators, this is an important tension that we must hold and have done so well in the many evaluations we have done this year and in the past. 24 INDEPENDENT EVALUATION GROUP Championing Evaluation Capacity Development International Program for Development Training (IPDET) IEG supports evaluation capacity development for World Bank Group Founded in 2001 as a joint initiative of IEG and Carleton University, client countries through a number of initiatives, primarily through IPDET is an executive training program that aims to provide managers the Centers for Learning on Evaluation and Results (CLEAR) global and practitioners with the tools required to evaluate development partnership and the International Program for Development Training policies, programs, and projects. Over 3,500 participants from 125 (IPDET). countries have attended the program since its inception. Following a strategic review, concluded in December 2015, IEG Centers for Learning on Evaluation and Results (CLEAR) together with Carleton University launched a new initiative to reshape IEG hosts and provides leadership to the global partnership program IPDET, update the curriculum, and make the necessary content and CLEAR (Centers for Learning on Evaluation and Results) that is pedagogical improvements to ensure IPDET remains cutting edge supported by 10 donors, including, among others, The Rockefeller and a global public good. This work is progressing and is scheduled Foundation, the Inter-American Development Bank (IDB), and the to be completed in the next year. Swedish International Development Cooperation Agency. In FY16, CLEAR issued a new grant for the South Asia Center, based in India. Two grants were concluded, and new grants are under preparation, respectively with for the Francophone CLEAR Center in Senegal and the Spanish-speaking Latin America CLEAR Center in Mexico. During the year, CLEAR launched a new website (www.theclearinitiative.org) to improve user experience and strengthen global branding. A new external collaboration platform was introduced, allowing the CLEAR global network to connect, exchange and learn online in a secure environment. 2016 ANNUAL REPORT 25 3 aligning the business building out 26 INDEPENDENT EVALUATION GROUP renewing and strengthening our organization Internally, 2016 was also a year of organizational renewal. IEG implemented an organizational change process to deepen internal collaboration and synergy, and ensure that IEG’s products, processes, and structure are “fit for purpose.” In January 2016, we completed the transition to a new organizational structure, designed to reduce overhead, better align with World Bank Group norms for managerial span of control, ensure clearer interface and engagement with the new World Bank Group units, and facilitate increased staff collaboration and mobility in larger labor pools. 2016 ANNUAL REPORT 27 IEG ORGANIZATIONAL STRUCTURE Office of Director General, Evaluation Human Development & Economic Management Financial, Private Sector & Sustainable Development Communication, Knowledge, and Learning Services Country Programs Corporate Financial and Sustainable and Economic and Human Private Sector Development Management Unit Development Unit Development Unit Unit Progress was made in a number other important areas. We fully The External Review operationalized the new Strategic Engagement Areas in our work The Committee on Development Effectiveness (CODE), which program planning. We deepened our focus on methods with the oversees IEG’s work, commissioned an independent panel in 2014 appointment of the new Methods Advisor. We continued to implement to conduct an external review of IEG. The panel was asked to identify the recommendations of the Task Force on Collaboration and IEG’s main strengths and any areas for improvement to enhance Synergy, particularly with the creation of the Thematic Coordinators IEG’s impact. The final report was released in June 2015. function to create cross-cutting communities of practice within IEG, The External Review report reaffirmed that “IEG is an essential and to establish a stronger liaison role between IEG operational corporate asset and safeguard, and a vitally important global teams and their World Bank Group counterparts for more effective public good.” knowledge sharing. 28 INDEPENDENT EVALUATION GROUP The report identified a set of eight recommendations to enhance 88 percent said they were satisfied with either their mentor IEG’s influence and role as an independent evaluator. To enhance or mentee pair. IEG’s impact, the External Review recommended: increased Other initiatives to support staff development include weekly emphasis on learning in IEG’s work; improvements in the quality, learning series for staff facilitated by internal and external experts, timing, volume, length, and utility of IEG evaluations; deeper a coaching program where more experienced evaluators and engagement with external experts and stakeholders; more and better subject matter experts run master classes on topics identified as strategic engagement with Management, particularly during the areas of development. In FY16, master class topics included impact process of developing recommendations; more attention to capacity evaluation, evaluation of knowledge products, and sovereign risk development; the need for regular independent reviews of IEG; clarity analysis. on the Director General’s term; and more CODE oversight of IEG. For the next year, IEG will re-launch a face-to-face onboarding course Many of the recommendations give further impetus to ongoing for new staff. initiatives and open the dialogue to push some of them further. IEG welcomed the comprehensive and insightful report and, over the year, we engaged with the Board and the World Bank Group management Resource Management on specific next steps. A road map and action place has been agreed Actual total expenditures at end of the year amounted to $34.06 with the Board and Management. million or 99.4% against budget resources of $34.3 million, compared with $33.35 or 97.7% against budget resources of $34.1 million in A link to the report and IEG’s response are available on IEG’s website. the previous year. Actual bank executed trust funds as of end June amounted to $1.27 million, compared to $0.97 million in the previous Career Development and the IEG Mentoring Program fiscal year. Total staff on board was 107 as of end June 2016. In June this year, IEG won an award for having the best mentoring During the fiscal year, the Board commissioned an external review of program in the World Bank Group. This achievement demonstrates IEG’s FY17 work program and budget. Overall the review rated IEG’s IEG’s continued commitment to career development. Since the budget formulation and management as broadly satisfactory. program launched in 2014, over 70 staff have participated as mentors and mentees. In a recent survey, 96 percent of our participants said that they achieved their expected objectives in the program, while 2016 ANNUAL REPORT 29 4 enhancing impact IEG strategic directions 30 INDEPENDENT EVALUATION GROUP looking forward Following the adoption of recommendations proposed by last year’s external review, IEG is looking forward to implementing several new initiatives in the coming months. Our work program for the coming year will focus on a number of important areas. 2016 ANNUAL REPORT 31 Deepening the Influence of IEG’s Work IEG has taken further steps to align its major evaluations more “ IEG is introducing a new Rapid Results closely with World Bank Group strategic priorities. In line with IEG’s results framework, upcoming evaluations will seek to more effectively Learning Fund of $750,000 under the generate relevant and timely evaluation evidence to accelerate World Director-General, which will support Bank Group development effectiveness. larger IEG learning engagements on Improving the Process of Engaging with World Bank a rolling basis, based on clear eligibility Group Management IEG will look to work with World Bank Group management on criteria.” improving the engagement process on evaluation recommendations and findings. A planned stock-taking of the Management Action Record (MAR) process will help identify opportunities to further enhance impact and effectiveness. Enhancing the Quality and Credibility of IEG Work A new Methods Adviser position was created within IEG during FY16. Over the coming year, the Methods Adviser will lead efforts to deepen IEG’s quality assurance framework; update our standards and tools for data, including more use of beneficiary feedback mechanisms provide strategic; and provide technical leadership to enhance our methods and data work, including strengthening methodologies related to the assessment of inclusive growth, gender equality, and value for money; and continued. 32 INDEPENDENT EVALUATION GROUP Improving Learning and Knowledge Sharing The External Review urged IEG to pay greater attention to utilization of its work, and the value of fewer, shorter, and sharper products that build in double-loop learning. To this end, IEG has significantly redesigned the approach to learning and knowledge sharing. Instead of focusing on the production of lengthy reports, we will shift to short, timely, and focused products in more user-friendly formats that respond to high priority, real time learning requests from operational staff. IEG is also introducing a new Rapid Results Learning Fund of $750,000 under the Director General, which will support larger IEG learning engagements on a rolling basis, based on clear eligibility criteria. All proposals for funding will require upfront “joint ownership” with a relevant Bank Group operational unit, and spell out the purpose, audience, scope, engagement plan, and intended use of the work. In the coming year, IEG will also work on developing stronger knowledge partnerships with the Results Measurement and Evidence Stream, Global Practices, and other technical operational units; upgrading our knowledge management tools and systems; and introducing more innovative formats and templates to make IEG reports more readable and accessible 2016 ANNUAL REPORT 33 FY17 Work Program Major Evaluations Other Evaluation Products Thematic and Sector Evaluations PPARs • Higher Education • World Bank: 55 • Shared Prosperity in Country Programs • IFC/MIGA: 8 • Rural Non-Farm Economy Learning Engagements • Water Supply and Sanitation •  BD, based on client demand under T • Urban Transport Services new IEG approach to learning Corporate and Process Evaluations Validation Products • Process Evaluation of SCD and CPF Country Strategy Completion Report • Data for Development Validations • IFC Client Engagement Model • 22 expected Country-Focused Evaluations Project Completion Report Validations • n/a • XPSR: 40% coverage (96 projects expected) • PER: (20 expected) Systematic Reviews and Impact Evaluations • TBD (2) • ICRR: 100% coverage (250 projects expected) Results and Performance Report AAA/AS Completion Report Validations • RAP 2016: Managing for Results • PCR: 51% coverage (87 projects expected) 34 INDEPENDENT EVALUATION GROUP FY18 Work Program FY19 Work Program Major Evaluations Only (Indicative) Major Evaluations Only (Indicative) Thematic and Sector Evaluations Thematic and Sector Evaluations • Clean World for All • Sustainable Cities • Public Finance for Investment and Jobs • Facilitating Trade • Mobilization of Private Capital • Sustainable and Inclusive Business • Essential Health Care Services • Environmental Degradation • Carbon Finance • Decentralization • Renewable Energy Corporate and Process Evaluations Corporate and Process Evaluations • Process Evaluation of GP/CCSAs • Engaging Citizens Country-Focused Evaluations Country-Focused Evaluations • C  lustered Country Program Evaluation • Clustered Country Program Evaluation  on Middle Income Countries on Middle Income Countries Systematic Reviews and Impact Evaluations Systematic Reviews and Impact Evaluations • n/a • TBD (2) Results and Performance Report Results and Performance Report • RAP 2018: Working in Partnership • RAP 2017: Sustainability in Operations 2016 ANNUAL REPORT 35 appendices IEG Deliverables and Spending Deliverables (Number) FY12 FY13 FY14 FY15 FY16 Planned Actual Percent Total Major Evaluations 8 10 10 7 9 7 78 Corporate Evaluations 2 3 4 2 4 3 75 Sector and Thematic Evaluations 4 5 4 4 3 2 67 Country Focused Program Evaluations 1 1 2 1 2 2 100 Impact Evaluations 1 1 0 0 0 2 n/a Learning Products 0 0 0 0 12 6 50 Project Evaluations and Global Program Reviews Project Performance Assessment Reports 40 31 47 39 58 51 88 Global Program Reviews 1 2 2 0 0 0 n/a Total Validations 414 478 573 571 602 485 Implementation Completion Reports 178 295 361 392 350 267 Country Assistance Strategy Completion Reports 20 19 22 13 22 22 Project Completion Reports 130 76 87 72 102 47 Expanded Project Supervision Reports 75 80 88 78 103 76 MIGA Validations 11 8 15 16 25 17 Total Spending ($ millions) 32.5 33.8 34.3 33.4 43.3 34.1 99.4 36 INDEPENDENT EVALUATION GROUP IEG Staffing Staffing (Number) FY12 FY13 FY14 FY15 FY16 Total Staff 98 107 109 105 109 By Grade GA-GD 22 22 22 23 23 GE 9 7 6 7 8 GF 20 22 25 21 27 GG 20 26 30 30 30 GH 20 22 20 19 17 GI and above 7 7 6 3 4 Photography Caroline Heider, page 2, Rhoda Baer Zambia, page 11, John Rae Zambia, page 33, John Rae Honduras, page 4, John Rae Coal, page 14, Shutterstock South Africa, page 33, John Rae Uzbekistan, page 6, John Rae Nepal, page 17, John Rae Lake Titicaca, page 33, John Rae India, page 6, John Rae Mozambique, page 20, John Rae Myanmar, page 6, John Rae Peru, page 26, John Rae Design Mozambique, page 8, John Rae India, page 30, John Rae Crabtree + Company 2016 ANNUAL REPORT 37 1818 H Street NW, Washington, DC 20433 http://ieg.worldbank.org ieg@worldbank.org