Approach Paper World Bank Group’s Support for Crisis Preparedness: Addressing Fiscal and Financial Sector Vulnerabilities 6/21/2019 1. Context and motivation Crises hurt. The global financial crisis alone reduced growth trajectories of many countries by as much as between 3.4 and 8.0 percent (World Bank 2010b) over the long run. Lower government revenues, public investment, and higher borrowing costs weakened governments’ ability to finance productive expenditures and pro-poor programs. Poverty reduction was slowed and, in some cases, even reversed. Through its impact on unemployment, household incomes, remittances as well as access to and the cost of credit, it is estimated that the global financial crisis resulted in the increase in the number of the poor by 64 million (Chen and Ravallion 2009, World Bank 2010b). Banking crises have been found to have deep and lasting impacts, resulting in an average increase in unemployment of 7 percentage points and an average drop in output of over 9 percent (Reinhart and Rogoff 2009a). Banking crises are also associated with a drop in the incomes of the poor of more than 10 percent, while currency crises have more detrimental impact of about 15 percent of the income of the poor (Rewilak 2018). Macroeconomic vulnerabilities that are frequently associated with crises include those originating on the fiscal side and in the financial sector; they can result in crises with significant social impact. These vulnerabilities can be especially pronounced in an environment of limited institutional capacity (e.g., for small economies), or for economies heavily dependent on a narrow commodity base. As economies are increasingly integrated into the global trade and capital markets and supply chains, this integration itself can become a source of vulnerability when global conditions take a turn for the worse. Of course, crises can and have been self-inflicted through bad policies which increase vulnerabilities. The focus of this evaluation is on crisis preparedness and not crisis response (See Box 1.1 which provides definitions of key concepts used in this evaluation.). The experience of the last 25 years has demonstrated that even stable or fast- growing economies can fall victim to large and destabilizing shocks, often exacerbated by their own vulnerabilities and policy shortcomings, and that unanticipated transmission of shocks is inevitable in an open and interconnected world (IMF 2011, Claessens et al, 2010). This has led to growing awareness among policy makers of the need to shift from reactive, ad hoc responses to shocks to an approach characterized by 1 proactive and integrated risk reduction (World Bank 2013). 1 A country’s ability to withstand shocks, minimize their impact, and recover is often a function of conditions and policies that ex ante reduce the vulnerabilities that can turn an exogenous shock into a full-blown crisis. These factors include the adequacy of macroeconomic buffers, the strength of financial sector balance sheets, the ability to implement timely countercyclical policy, the establishment of frameworks and regulations to facilitate crisis management, and flexible and adequate social protection systems, all of which contribute to a country’s preparedness for crises. Box 1.1. Definitions For this evaluation, the focus is on whether the Bank Group has contributed to preparedness by identifying and responding to fiscal and financial sector vulnerabilities which, if unaddressed, can lead to crises with severe social impacts. For this evaluation, shocks can come from any source, but they will be relevant only to the extent that they affect fiscal and financial vulnerabilities. Finally, “preparedness” can be reflected in terms of institutions, policies, regulations and balance sheets that help (i) identify and /or manage fiscal and financial sector vulnerabilities and (ii) efforts to reduce key vulnerabilities to minimize their economic impact, improve adaptability of social protection systems to chocks, and expedite recovery. The focus is on risk identification and ex ante preparedness to establish mechanisms and frameworks for crisis management, not on ex post crisis response. A clear lesson of the 2008 global financial crisis and other macroeconomic and financial sector crises was the importance of identifying and addressing country-specific vulnerabilities to ensure strong initial conditions and resilience if/when a shock occurs (World Bank 2008, 2009, 2010, 2011, 2012a). Ten years after the onset of the global financial crisis, questions remain about whether developing countries have internalized the importance of crisis preparedness to withstand not only crises of a global nature but also shocks specific to their economies, including to commodity prices, or capital, 1 The World Development Report on Risk (World Bank 2013) emphasizes the importance of preparedness, which includes intelligence, protection, and insurance. These elements of preparedness are consistent with the key building blocks of this evaluation’s framework (see figure 5.1). Furthermore, an important channel of crisis impact is through jobs. Two recent World Development reports have emphasized the importance of jobs as drivers of development, not just as labor demand that reacts to changes in economic conditions, and recognized risks and opportunities in the changing nature of work in response to the unprecedented pace of technological innovation (World Bank 2012b, 2019b). 2 financial and trade flows. Related to this are questions concerning the extent to which the Bank Group gives sufficient attention in operations, technical assistance, analytical work, investments and policy advice to help clients identify vulnerabilities to particular types of exogenous shocks and address key vulnerabilities to bolster crisis preparedness. 2. Relevance and timeliness This evaluation is relevant and timely for several reasons. In the “Forward Look” (World Bank and the IMF, 2016), the Bank Group emphasizes the need for “building flexibility and buffers to respond to changes in the global context and strategic priorities, as for example during the financial crisis” as part of the global public good agenda of strengthening capacity for crisis response. Also, the WBG capital increase (World Bank and the IMF 2018) recognizes the need for Bank Group leadership on global issues, including crisis preparedness and management, as well as the need to serve all clients, including IDA and MICs, small states, as well as FCV countries, maximizing finance for development and improving effectiveness of its business model. The evaluation’s timeliness reflects the following elements of the global economic and development context. • First, global economic expansion is slowing with many risks on the horizon including from rising global interest rates and reversal of capital flows, increasingly unpredictable national policies, volatility in environmental conditions in the face of accelerating climate change, and trade tensions between major economies as well as spillovers from global and regional shocks. These developments have historically been associated with macroeconomic and financial crises (Minsky 1989, Kalemli, Reinhart and Rogoff 2016, IMF 2018f, World Bank 2016b, 2018a, 2019a, Zoellick 2019). • Second, many developing countries, for a combination of reasons, have allowed macroeconomic buffers to erode, making their economies more vulnerable to exogenous shocks (Huidron, Kose and Ohnsorge 2016). Public and private debts have risen substantially as have contingent liabilities, including from public- private partnerships, with hidden or unpredictable fiscal costs. Related assessment has been voiced recently by the IMF which concluded that “rising vulnerabilities are evident in multiple indicators, which show increased financial leverage, a deterioration in underwriting standards, and increased reaching-for- yield behavior by investors in corporate and sovereign debt markets around the world.” (IMF 2018f). • Third, since 2013, median government debt in low-income countries has risen by 20 percent of GDP, increasingly from non-concessional and private sources. And 3 contingent liabilities, including those associated with PPPs, as well as hidden debt suggest that the real debt burden may be even higher (Devarajan 2018, Roubini and Rosa 2018). As a result, former heavily indebted poor countries (HIPCs) are increasingly experiencing debt distress including as the result of a return to commercial and short-term borrowing, often in foreign currency. Eleven LICs are currently in debt distress or at a high risk of debt distress, compared to only six in 2015 (World Bank 2019, Gill 2019). Moreover, shortcomings in the transparency of sovereign bilateral borrowing suggest that the debt problem may be bigger still. (Reinhart and Rogoff 2009b, 2010, 2011, Reinhart and Trebesch 2016, Gill and Karakullah 2018). • Fourth, a recent evaluation of international financial surveillance (FSAP) concludes that, despite improvements in the quality of the diagnostic over time, the reach of financial sector surveillance has fallen short of client expectations, with many countries having had at most one FSAP since 2010 (IMF 2018g). • Finally, the rapid pace of technological and financial innovation continues to influence global financial markets in new and unprecedented ways (e.g., cryptocurrencies, blockchain) which may have implications for systemic or country-specific vulnerabilities. 3. Past evaluations Several IEG evaluations have analyzed the Bank Group’s support to countries impacted by crises 2. This evaluation will take stock of the lessons and recommendations from these evaluations and assess the extent to which the Bank Group has learned from past crises and has adapted its approach to supporting its clients by giving greater 2The IEG has evaluated support for financial sector reforms, including in crisis situations (World Bank 2006) and for middle-income countries (World Bank 2007), which had evaluated the Bank’s experiences with earlier financial crises during the Asian Crisis, Russia, Argentina, and Turkey. The IMF’s Independent Evaluation Office (IEO) has also undertaken crisis-related evaluations, several of which contain lessons of relevance to the World Bank (International Monetary Fund 2011, 2014, 2018g). Other recent IMF analyses of global financial conditions also have important insights and implications for the evaluation (IMF 2018a, 2018b, 2018e, 2018f). IEG will draw on this work as part of its stock taking of lessons learned. 4 attention to ex ante risk and vulnerability identification complemented by support to help reduce vulnerability to shocks. IEG evaluations of the Bank Group’s response to the global financial crisis (World Bank 2008, 2010a, 2012a) identified several lessons and recommendations for the Bank group, including the need to develop early warning mechanisms and strengthen the Bank’s financial and operational readiness (including skills mix) and blending country-level responses within a global strategy to apply scarce resources where they are most effective. And it emphasized the need to strengthen synergies across the Bank, IFC, and MIGA and better leverage external partnerships. IEG’s Social safety net (SSN) evaluation (World Bank 2011) reviewed World Bank support for SSNs from 2000 to 2010. It found that the Bank’s approach had evolved from a project-centered approach toward support for systems and institutions to deal with vulnerabilities. But it also concluded that the 2008-09 crisis had revealed major weaknesses in countries’ SSNs: many middle-income countries found that their poverty- targeted SSNs were not sufficiently flexible to increase coverage or benefits as needed, while low-income countries lacked poverty data and systems to target and deliver benefits. It recommended that the Bank engage more systematically with its client countries, particularly with low-income countries, so that they could more adequately protect against future shocks by putting in place SSNs and strengthen institutional capacities, that would more effectively protect the poorest and most vulnerable. 3 Finally, there was also a broad synthesis of recent IEG evaluations focused on crisis response and resilience (2017b). IEG has also assessed the Bank Group’s support to PPPs (World Bank 2015a), financial inclusion (2015b), and prepared a clustered country program evaluation of small states (2016a) and a synthesis of its evaluations of Bank support to upper middle- income countries (2017a). While not focused on crises preparedness per se, these 3Partly in response to crises with severe social impacts, new thinking about social safety nets emphasizes adaptive social protection systems (ASP) which go beyond traditional systems by helping ensure that those vulnerable to the crises (not just those who are already poor) are identified and that critical investments in human capital are not undermined by a crisis (Rutkowski 2018). With more and better data increasingly available, and with strengthening of institutions, social safety net systems have the potential to become more flexible and better able to respond to crises than in the past (World Bank 2018). 5 evaluations generated evaluative insights relevant to crisis preparedness that the present evaluation will consider. 4. Purpose, objectives, and audience The purpose of this evaluation is to assess the Bank Group’s support to client countries to enhance their preparedness for exogenous shocks through more systematic ex ante identification of vulnerabilities complemented by support to address these vulnerabilities. The evaluation is focused on the period between FY2010 and FY2018, after the global recession, to evaluate whether the Bank Group had integrated lessons from the global crisis. This evaluation aims to inform the design and use of WBG strategies, operations, diagnostics and knowledge products that support crisis preparedness in both low-income and middle-income countries. It will also assess the extent to which the Bank Group is itself equipped to act on earlier lessons from IEG’s crisis-focused evaluations (Crisis Response and Food Crisis evaluations, IEG 2008, 2009, 2010, 2011, 2012a) to improve the relevance and effectiveness of diagnostic and analytical work in support of better crisis preparedness. The evaluation’s scope will include IBRD, IDA, IFC and MIGA. Thematically, the evaluation will focus on macroeconomic and financial sector vulnerabilities, including those originating in the private sector, with potential social impact. IFC and MIGA operations will be evaluated, particularly in the areas of financial sector, as will aspects of PPPs and financial inclusion to the extent that they impact vulnerabilities and affect fiscal, financial, and social safety net preparedness. 4 Primary focus is on crisis preparedness through the prism of fiscal and financial-sector vulnerabilities while the focus on social protection will be limited to adaptability and flexibility of social safety nets to respond to the impact of crises on households. 5 Synergies between the agencies 4The evaluation will look, within the framework of case studies, at the IFC portfolio of financial sector, SMEs and housing investments and activities, for example, to assess the extent to which these activities and interventions may impact countries’ macro-financial and social safety net vulnerabilities and through which channels. 5Social safety net (SSN) systems could include a variety of SSN programs that can be used in crisis response, such as cash (conditional and unconditional) and in kind-transfers, vouchers, food stamps, and public but the specific set of programs will be country specific and will be 6 in helping countries identify and address vulnerabilities will be explored as will any activities or interventions that may themselves have enhanced vulnerabilities. The internal audience and primary intended users consist of the Board, relevant Global Practices (especially Equitable Growth Finance and Institutions (EFI)--both MTI and FCI), the Social Protection and Jobs (SPL) global practices), the Global Crisis Risk Platform (GCRP), as well as the Bank Group country teams working on SCDs and other diagnostics, country strategies, and interventions that contribute to crisis preparedness, IFC and MIGA, and country teams (including staff working directly on macro-fiscal, financial sector and social safety net issues). External groups that may find the evaluation relevant include client governments, other international organizations with mandates to support countries in strengthening crisis preparedness (IMF, BIS, the Financial Stability Board), and other institutions with macroeconomic, financial and social safety net mandates, as well as evaluators, academics and think tanks and NGO communities with an interest in these issues. 5. Evaluation scope and questions Because of the heterogeneity of shocks and country experiences, the Bank Group does not have a single, overarching approach to ex ante crisis preparedness, nor a single institutional locus to improve crisis preparedness through operations and knowledge products. Instead, support for crisis preparedness is embedded in instruments such as Country Partnership Frameworks. 6 Nevertheless, it may be possible to ascertain from a variety of documents key elements of the approach that underpins the Bank Group interventions to strengthen crisis preparedness. (Figure 5.1 and 5.2). evaluated based on the country case studies. In this context, it may include unemployment insurance, which can be a key automatic stabilizer and shock absorber. Overall, and to build resilience at the household level, these programs seek to promote links to human capital, skills, and labor market insertion. 6 Recently, efforts in the direction of greater and more systematic, Bank-wide efforts at crisis preparedness are being made with the Global Crisis Risk Platform (GCRP), which includes a broader range of crises and vulnerabilities that go beyond the fiscal, financial and social, which are the focus of this evaluation (World Bank 2018b). 7 Figure 5.1. Crisis Preparedness: A Conceptual Framework 8 Figure 5.2. Preliminary Theory of Change for Reducing Key Vulnerabilities in Client Countries Outputs Intermediate Outcomes Medium Term Bank Group inputs/ Outcomes activities to identify over 2-3 years and address fiscal and I. Identifying vulnerabilities I. Reducing vulnerabilities financial sector • Systematic monitoring of country conditions and • Bank Group partnership strategy under • Greater macroeconomic vulnerabilities challenges to anticipate risks and identify implementation reflects importance of and financial sector vulnerabilities, drawing on inputs from other addressing fiscal and financial sector stability; improved development partners, particularly the IMF vulnerabilities identified. resilience to • Assessment of coverage, availability, and quality of macroeconomic shocks • Capacity building and TA underway, in Strategies, macroeconomic and fiscal data, including private coordination with key development • Enhanced fiscal space projects, policy and public debt statistics partners (particularly IMF) to improve • Improved debt advice, • Regular debt sustainability analysis client’s ability to identify and monitor management capacity investments • Application of relevant fiscal, financial sector, and fiscal and financial sector risks and practices in client social safety diagnostics identifying vulnerabilities • Improved quality and timeliness of fiscal countries • Identification of any shortcomings in the design or and financial sector data relevant for • Reduced risk of financial operation of social safety nets from the stand point decision making sector crises of flexibility and adaptability • Adoption of policies and frameworks that • Greater fiscal and Country encourage financial stability and financial sector knowledge (data, II. Responding to vulnerabilities soundness and strengthen supervisory resilience in face of monitoring, • Bank policy advice, country dialogue and economic capacity over financial sector exogenous shocks analysis) monitoring clearly identifies fiscal and financial sector vulnerabilities and reflects the relative • Greater diversification of sources of • Progress on poverty importance of addressing them finance for governments and private reduction and shared sector prosperity • Training, technical assistance and capacity building needs identified to strength fiscal and financial • Stronger and more adaptive social • Reduced risks of Partnerships and sector oversight and management protection systems capacity to absorb escalating poverty and Collaboration shocks affecting the poor and vulnerable exclusion during crises • Strategies formulated to build flexibility and implementation capacity for social safety nets • Stronger institutional capacity for social protection to respond to crises Impacts: Progress Towards the Twin Goals and SDGs: Sustained and Inclusive Growth and Poverty Reduction Source: IEG staff, based on the World Bank strategy (2013), Forward Look, Financial Sector Strategy, and Social Protection Strategy. 9 The preliminary theory of change posits that Bank Group’s country strategies, lending/project and knowledge services, and convening power are key elements of support to client countries (both public and private sectors) to help reduce vulnerabilities to shocks (Figure 5.2). 7 This contributes to outputs and intermediate outcomes that improve institutional capacity and policy readiness to reduce macro- financial and financial sector vulnerabilities and those related to social safety nets. The evaluation asks five related questions, with additional sub-questions detailed below. The evaluation design matrix shows how different methods apply to different questions and different levels of analysis: country, thematic, and partnerships/synergies (Appendix A). The evaluation covers the following World Bank Group Agencies: IDA, IBRD, IFC and MIGA. Given the criticality of country context for macro-financial country preparedness, the first four questions will be answered based on the carefully selected case studies. The fifth question on corporate readiness will be answered drawing on the findings of case studies and Bank Group-wide information and assessment of certain aspects of readiness (see below). 1. Did the WBG do the right analysis and diagnostic work at the right time for its client countries? How relevant and timely were Bank Group diagnostics in identifying major fiscal and financial vulnerabilities at the country level? 2. Did the WBG identify key fiscal and financial sector vulnerabilities? Did the WBG help countries identify fiscal and financial sector vulnerabilities and any 7 Importantly, the evaluation will look at interconnectedness of fiscal in and financial sector vulnerabilities as well as with other risks that may affect fiscal and financial sector vulnerabilities. For example, a bankruptcy of a major private bank could not only make the financial system as a whole vulnerable to systemic and confidence risks, but a potential bail out could lead to a major increase in public spending and public debt if it is necessary to stabilize the banking system. Private debt will be subject of analysis to the extent it creates fiscal or financial sector vulnerabilities at the country level. For example, excessive private sector indebtedness can pose a risk to the health of the financial sector. The evaluation will also document and analyze the use of Bank Group hedging and insurance instruments as they are used at the country level. This may include contingent instruments such as DPFs-policy-based guarantees, deferred drawdown options—and other instruments and advice from the Bank and IFC, as well as, where appropriate, support to the development of insurance sectors. 10 needed adjustments in social safety nets to strengthen crisis preparedness? Did the Bank Group clearly and candidly identify key vulnerabilities? 3. Did WBG follow up on diagnostics and analysis in a timely manner to address the identified key fiscal and financial sector vulnerabilities including by helping to develop and put in place mechanisms and frameworks for crisis management? Did the WBG adapt its policy advice and formulate strategies and interventions at the country level to address identified vulnerabilities? 4. Were WBG efforts to help clients reduce key fiscal and financial sector vulnerabilities and develop mechanisms and frameworks for crisis management and social protection effective? In supporting its clients to address identified vulnerabilities, did the Bank Group coordinate with key development partners (particularly the IMF) with relevant expertise? 5. How well prepared is the WBG to identify and respond to client vulnerabilities in a timely manner? Does the Bank Group have staffing, resources and incentives for effective engagement in addressing fiscal and financial sector vulnerabilities as part of its support for crisis preparedness? Answers to these questions will lead to potentially valuable lessons and recommendations on how the Bank Group can improve the impact of its support for crisis preparedness in client countries. Given the Bank’s comparative advantage, previous evaluation work on resilience, and the importance of focused and operationally relevant lessons and recommendations, this evaluation will focus on preparedness related to fiscal and financial sector vulnerabilities. Given the importance of the private sector both as a source of fiscal and financial sector vulnerabilities and as a channel for building resilience to such vulnerabilities, the evaluation will include assessments of IFC and MIGA activities in country case studies and cross-cutting analysis. 6. Evaluation design and evaluability assessment The evaluation will employ a mixed methods approach. Data collection will essentially cover two levels of analysis, the corporate level and the level of selected countries (with a focus on selected aspects: fiscal, financial and social safety net dimensions of crisis preparedness). The main component of the evaluation approach is case study analysis, including cross-cutting syntheses of thematic/sector issues and findings arising from individual case studies. The case is defined as the WBG’s engagement at country level on selected issues relating to crisis preparedness. Case studies will be selected from among WBG client countries to reflect diverse country circumstances, 11 country vulnerabilities and their experience with exogenous shocks as well as regional diversity and income levels. A purposive sample of seven field-based case studies will be selected to assess how the Bank Group supported preparedness in these countries, based on its ability to identify and respond to key fiscal, financial sector vulnerabilities and identified limitations of social safety nets including though key diagnostic instruments (e.g., SCDs, Country Private-Sector Diagnostics (CPSDs), DeMPAs, PEFAs, FSAPs). These studies will also benefit from the existing project-level evaluative evidence (e.g., ICRRs, PPARs, CLRRs) for the respective countries. A workshop was held with the IEG methods team to discuss methods in more depth and provide guidance to the team on the design and protocols of case studies. The approach to country selection is as follows. A reference group of countries was identified that have a minimum portfolio of Bank Group strategic country engagement including knowledge and lending interventions, diagnostics and related evaluative evidence (SCD/CPF, CLRR/CASCR). Next, countries were selected that also show evidence of direct Bank engagement in fiscal and financial sector areas (e.g., FSAP, CEMs, PEFA, PERs, PFRs), including those that are specific to IDA countries (e.g., Support provided through the Debt Management Facility (DMF)). Specific case study countries will be selected from these groups of IDA and IBRD countries to ensure a range of fiscal and financial vulnerabilities. The range of vulnerabilities is measured using a set of indicators of fiscal health (e.g., debt-to-GDP ratios, primary and overall fiscal balances, DeMPA and PEFA scores) and indicators of financial sector health (e.g., financial sector balance sheet, credit, and liquidity indicators), supplemented by the qualitative judgement about vulnerabilities based on the review of the Bank’s strategic, fiscal and financial diagnostics (SCD, CEM, FSAP) as well as IMF assessments (e.g., Article IV consultation reports). Based on the agreed conceptual framework (Figure 5.1 and Figure 5.2 above), detailed theory of change for fiscal and financial areas is being developed, which will inform the template for extracting cross-case study information, and protocols for individual case studies. The evaluation will feature a consistent overarching evaluative framework combining the conceptual framework with main evaluative questions, theory of change, as well as the template and protocols for the preparation of case studies. Two pilot case studies will be conducted early in the evaluation to test the approach, set the standard for the remaining studies, and facilitate cross country comparison. The main methods employed in the framework of the case studies are semi- structured interviews and desk reviews. 12 Preliminary review of the IEG, World Bank, IMF and academic literature related to crisis preparedness has been carried out and will be further developed as the evaluation is conducted. Selection and design of case studies will enhance the potential to derive cross-cutting lessons from individual studies. In addition, the team will review Bank Group strategies and diagnostics and country strategies, through a combination of country visits and desk reviews. Cross cutting analysis of fiscal and financial sector findings will be undertaken drawing on case study countries to extract comparative information, insights patterns, and lessons. External validity will derive from objective selection criteria, consistent implementation of the evaluation framework, and the degree to which findings and patterns converge across case studies. Issues of Bank readiness to support crisis preparedness will rely on findings from case studies but will be augmented with more broad-based assessments of skills, institutional incentives, diagnostic tools, and early warning systems. The above conceptual framework and theory of change drive the evaluation design. This case study-driven evaluation will use a mixed methods approach relying on appropriate quantitative as well as qualitative analyses. The appendix A details the mapping of the methods to specific questions and different levels of analysis. The evaluation will use three main sources of data and information. First, it will use Bank Group data, documents and information in country strategy related documents (e.g., CPFs, SCDs, CLRs, CPEs, CPSDs), projects (ICRR, PPARs, XSPRs), and knowledge products (e.g., GEP, GMR, FSAPs, MFR, DSAs, DeMPAs, PEFAs) 8 that are relevant to crisis preparedness. Second, it will use data and information collected in internal and external stakeholder interviews, including during field visits. And third, it will use macro-fiscal and financial sector data and on data on social safety nets for comparison and benchmarking purposes in case studies. Mixed methods will be used to triangulate findings and extract comparative information and insights to improve the robustness of the analysis. 8For example, we will review Global Economic Prospects reports for the identification of emerging global or regional risks that could affect national fiscal and financial sector vulnerabilities. 13 In addition to client governments and the Bank Group, there are other institutions involved in crisis preparedness (e.g., IMF and FSB). The evaluation, while focused on the performance of the Bank Group, will pay particular attention to collaboration between the Bank and the IMF in areas of overlapping mandate. Collaboration with other development partners will be addressed where relevant to individual country case studies. 7. Limitations Limitations of the methodology are several. A limitation of the case study approach is that the sample may be biased and underlying information difficult to evaluate in that it may exclude country programs that may have rising fiscal and financial vulnerabilities and in which the Bank Group had attempted engagement but for various reasons was not able to develop it into a tangible program. Also, poor quality or availability of data in some countries may limit the granularity and precision of the analysis; but it might also point out to vulnerabilities that need to be addressed in the future. In addition, when assessing effectiveness of World Bank Group interventions related to social safety nets, the evaluation might not find unambiguous benchmarks and it may be difficult to separate the assessment of the quality of social safety net overall and its flexibility to absorb major shocks. 9 Finally, there will be challenges in ensuring comparative insights and lessons with sufficient external validity, which could be mitigated through careful selection of case studies, consistent implementation of evaluation strategy, and convergence of findings and patterns. 8. Quality assurance process The quality control will be ensured through IEG’s review process, including inputs from external peer reviewers and the internal quality assurance mechanism. Zia Qureshi (Brookings, former Director, DEC), Ajay Chhibber (former WB Country Director, Director at IEG, UN Assistant Secretary General), and Mary Goodman (Assistant Director, Strategy, Policy and Review Department; IMF) have agreed to serve as external peer reviewers. Together, the peer reviewers offer expertise in a range of 9Other limitations include, for example, difficulties in evaluating Bank contributions at the country level when there is a gap between what the Bank advises and what the client is willing to do towards crisis preparedness. Also, there is an inherent difficulty in evaluating success with crisis preparedness in cases when crises do not happen. 14 thematic and operational matters pertinent to the evaluation. The evaluation will be prepared under the supervision and guidance of the IEGEC manager, Jeff Chelsky, and the IEGHE director. The approach paper has benefitted considerably from the guidance and advice of Auguste Kouame. The internal review process will engage all IEG units. 9. Expected outputs, outreach, and tracking The main output of this evaluation will be a formal evaluation report that presents relevant findings, lessons, and recommendations to the Bank Group and to external audiences. In addition to the final report, the evaluation will produce seven field-based country case studies and cross cutting analyses on fiscal and financial sector findings drawn from case studies, as well as a focused note on adaptability of social safety nets in those countries. These outputs will inform the final report. They will contribute to the overall evaluation findings and the IEG internal body of knowledge on the topic. Enhanced engagement with the Bank Group throughout the evaluation is envisaged with the view to enhancing learning feedback loops. A dissemination and outreach plan are being developed to reach both internal and external audiences through a combination of events (e.g., launch, regional or GP centered events), IEG online, and social media activities. Suggestions for in-country events to engage stakeholders will be canvassed in consultation with the WBG country teams, government counterparts and think tanks as part of a dissemination and outreach strategy prior to “One Stop” review of the final evaluation report. The strategy will be refined in response to feedback received. 10. Resources Evaluation team. The evaluation team consists of Željko Bogetić (TTL, macroeconomic policy), Felix Oppong (debt and LIC issues, case studies), Stefan Apfalter (financial sector, IFC and MIGA), Shahrokh Fardoust (senior consultant, macroeconomic policy, debt); Augusto de la Torre (senior consultant, macro-prudential, financial sector); Amit Banerjee (senior consultant, IFC/MIGA operations); Ana-Maria Arriagada (senior consultant, social safety nets); Amshika Amar (portfolio analysis and research analyst); Dung Thi Kim Chu (team assistance). The team has benefitted from upstream discussions and advice on methodology and portfolio review approach from the methods advisory team, Jos Vaessen and Marie-Noelle Lantin Roquiz. Budget. Total, two-year budget (FY19-20) for the evaluation is $950,000, including $550,000 for fixed costs, $350,000 for variable costs as well as $50,000 for outreach and dissemination. 15 Consultations. Preliminary consultations and interviews were held with GPs and a number of Bank and the IMF staff in the preparation of this Approach Paper. The evaluation team has also benefitted from discussion with and feedback from the IEG Methods workshop, the Bank Group’s Chief Economists Council as well as the economics group of the 1818 Society on an early concept of the evaluation. The team intends to consult the two groups on emerging findings. The Approach Paper benefitted from the PROACT workshop, which the IEG team held on February 28, 2019, with a select group of experts from across the Bank. The team may also organize an informal workshop involving senior experts to discuss and obtain feedback on early findings of the evaluation. 16 References Chen, Shaohua and Martin Ravallion. 2009. The Impact of the Global Financial Crisis on the World’s Poorest. https://voxeu.org/article/impact-global-financial-crisis-world-s-poorest Claessens, Stijin, Giovanni Dell’Ariccia, Deniz Igan, and Luc Laeven. 2010. “Lessons and Policy Implications from the Global Financial Crisis.” IMF Working Paper 10/44, International Monetary Fund, Washington, DC. Devarajan, Shanta. 2018. Low-Income Country debt: Déjà vu all over again, Brookings Blog, https://www.brookings.edu/blog/future-development/2018/04/25/low-income-country- debt-deja-vu-all-over-again/ Huidron, Raju, Ayhan M. Kose, and Franziska Ohnsorge. 2016. “Challenges of Fiscal Policy in Emerging and Developing Economies.” World Bank Policy Research Working Paper 7725, World Bank, Washington, DC. International Monetary Fund (IMF). 2011. IMF Performance in the Run-up to the Financial and Economic Crisis: IMF Surveillance in 2004- 07. Independent Evaluation Office. Washington DC: IMF. ———. 2014. IMF Response to Financial and Economic Crisis. Independent Evaluation Office. Washington DC: IMF. ———. 2018a. Fiscal Monitor April 2018: Capitalizing on Good Times. Washington DC: IMF. ———. 2018b. World Economic Outlook: Challenges to Steady Growth. Washington DC: IMF. ———. 2018e. New Opportunities and Future Global Challenges. IMF Research Perspectives. Volume 19, Number 2. Washington DC: IMF. ———. 2018f. Global Financial Stability Report: A Bumpy Road Ahead. Washington DC: IMF. ———. 2018g. IMF Financial Surveillance. Independent Evaluation Office. Washington DC: IMF. Kalemli-Ozcan, Sebnem, Carmen Reinhart, and Kenneth Rogoff. 2016. “Sovereign Debt and Financial Crises: Theory and Historical Evidence”. Journal of European Economic Association, vol. 14 (1): 1-6. Minsky, Hyman P. 1989. "The Financial Instability Hypothesis: A Clarification." Hyman P. Minsky Archive Paper 145. Bard College Bard Digital Commons. http://digitalcommons.bard.edu/hm_archive/145 Reinhart, Carmen and Kenneth Rogoff. 2009a. The Aftermath of Financial Crises. NBER Working Paper No. 14656 (January). Reinhart, Carmen and Kenneth Rogoff. 2009b. This Time is Different: Eight Centuries of Financial Folly. Princeton: Princeton University Press. 17 Reinhart, Carmen and Kenneth Rogoff. 2010. “Growth in Times of Debt.” American Economic Review, Papers and Proceeding 100, 573–578. Reinhart, Carmen and Kenneth Rogoff. 2011. “The Forgotten History of Domestic Debt.” Economic Journal, vol. 121, 319–350. Reinhart, Carmen, and Christoph Trebesch. 2016. “Sovereign Debt Relief and its Aftermath.” Journal of the European Economic Association, vol. 14 (1), 215–251. Rewilak, John. 2018. “The Impact of Financial Crises on the Poor.” Journal of International development. 30, 3-19. Rutkowski, Jan. 2018. Using Adaptive Social Safety Nets to Respond to Cope with Crises and Build Resilience. Voices: Perspectives on development blog. (February 18) http://blogs.worldbank.org/voices/using-adaptive-social-protection-cope-crisis-and- build-resilience Roubini, Nouriel, and Brunello Rosa. 2018. “Is the next Financial Crisis Already Brewing.” Financial Times (September 11). Schmukler, Sergio, Didier, T, C Hevia. 2011. “How Resilient Were Emerging Economies to the Global Economic Crisis?”, World Bank Policy Research Working Paper 5637. Stiglitz, Joseph. 2018. “Lessons from the Financial Crisis and Their Implications for Global Economic Policy.” Columbia Business School paper, Columbia Business School, New World Bank. 2006. IEG Review of World Bank Support for Financial Sector Reforms. Independent Evaluation Group. Washington, D.C.: World Bank. ———. 2007. Development Results in Middle-Income Countries. An Evaluation of the World Bank’s Support. Independent Evaluation Group. Washington, D.C.: World Bank. ———. 2008. Lessons from World Bank Group Responses to Past Financial Crises. Evaluation Brief 6. Independent Evaluation Group. Washington, DC: World Bank. ———. 2009. The World Bank Group’s Response to the Global Crisis: Update on an Ongoing IEG Evaluation. Evaluation Brief 8. Independent Evaluation Group. Washington, DC: World Bank. ———. 2010a. The World Bank Group’s Response to the Global Economic Crisis. Phase 1. Independent Evaluation Group. Washington, DC: World Bank. ———. 2010b. Global Economic Prospects: Crisis, Finance and Growth. Washington, DC: World Bank. ———. 2011. “Social Safety Nets - An Evaluation of World Bank Support, 2000-2010.” Independent Evaluation Group, World Bank, 2011. 18 ———. 2012a. The World Bank Group’s Response to the Global Economic Crisis. Phase 2. Independent Evaluation Group. Washington, DC: World Bank. ———2012b. World Development Report 2013: Jobs. World Development Report. Washington, DC. World Bank. ——— 2013. World Development Report 2014: Risk and Opportunity—Managing Risk for Development. Washington, DC. World Bank. ———. 2015a. World Bank Group Support for Public-Private Partnership: Lessons from Experience in Client Countries, FY2002-12. Independent Evaluation Group. Washington, DC: World Bank. ———. 2015b. Financial Inclusion: A Foothold on the Ladder to Prosperity: An Evaluation of World Bank Group’s Support for Financial Inclusion of Low-Income Households and Microenterprises. Independent Evaluation Group. Washington, DC: World Bank. ———. 2016a. World Bank Group Engagement in Small States: The Cases of OECS, PICs, Mauritius, Djibuti, Cabo Verde and Seychelles—Clustered Country Program Evaluaton. Independent Evaluation Group. Washington, DC: World Bank. ———. 2016b. Global Economic Prospects: Spillovers Amid Weak Growth. Washington, DC: World Bank. (January). ———. 2017. World Bank Group Engagement in Upper Middle-Income Countries. Independent Evaluation Group. Washington, DC: World Bank. ———. 2017b. Crisis Response and Resilience to Systemic Shocks: Lessons from IEG evaluations. Independent Evaluation Group. Washington, DC: World Bank. ———. 2018a. The State of Social Safety Nets 2018. Washington, DC: World Bank. ———. 2018b. Global Crisis Risk Platform. (June 16). Washington D.C.: World Bank. ———. 2019a. Global Economic Prospects. (January). Washington, DC: World Bank. ———. 2019b. World Development Report: The Changing Nature of Work. Washington D.C.: World Bank. World Bank and the IMF. 2016. “Forward Look: A Vision of the World Bank Group in 2030,” Development Committee. (September 20). Washington, DC: World Bank and the IMF. World Bank and the IMF. 2018. “World Bank/IMF Spring Meetings Communiqué. (April 21). Washington, DC: World Bank and the IMF. World Economic Forum. 2018. Global Risks Report 2018, 13th edition. Geneva: World Economic Forum. 19 Zoellick, Robert. 2019. “Whoever Runs the World Bank Needs a Plan for Emerging Markets”. Financial Times. February 6, 2019. 20 Appendix A. Evaluation Design Matrix Table A.1. Evaluation Design Matrix: Levels of evaluation, questions, methods Level of analysis Questions 1 & 2 Question 3 Question 4 Question 5 Diagnostic work Operational Effectiveness of Corporate response operational readiness response literature review Cross-cutting portfolio review and analysis (PRA) for identification and case selection Theory of Change development review of relevant data on crisis preparedness indicators semi-structured Corporate interviews review of relevant documentation and corporate (e.g. HR) data review of key review of country PRA (e.g. ICRRs) semi-structured Selected diagnostic strategies semi-structured interviews countries instruments and PRA (e.g. PADs) interviews knowledge semi-structured products interviews semi-structured interviews pattern analysis Cross-case narrative synthesis across countries on fiscal, financial and social safety net perspective dimensions of crisis preparedness 21