Operational Guidance for Monitoring and Evaluation (M&E) in Climate and Disaster Resilience-Building Operations Table of Contents Acknowledgements....................................................................................................................................... 3 Introduction .................................................................................................................................................. 4 Section 1. Operationalizing Resilience Definitions and Concepts................................................................. 7 Recommendations for Task Team Guidance ............................................................................................ 9 Section 2. Addressing Unique Challenges of Resilience M&E and Guiding Principles................................ 12 Recommendations for Task Team Guidance .......................................................................................... 14 Section 3. Developing Resilience Results Frameworks ............................................................................... 17 Recommendations for Task Team Guidance .......................................................................................... 23 Section 4. Selecting and Defining Resilience-Related Results Indicators ................................................... 26 Recommendations for Task Team Guidance .......................................................................................... 33 Section 5. Monitoring and Reporting Resilience-Related Results .............................................................. 36 Recommendations for Task Team Guidance .......................................................................................... 40 Section 6. Evaluating Resilience-Related Results........................................................................................ 42 Recommendations for Task Team Guidance .......................................................................................... 47 Section 7. Moving forward.......................................................................................................................... 50 References .................................................................................................................................................. 51 ReM&E December 2017 Appendix 1: An Illustrative List of Climate and Disaster Resilience-related Results Indicators from World Bank Investment Operations ...................................................................................................................... 54 Summary ................................................................................................................................................. 54 How to Use the Indicator List.................................................................................................................. 54 Resilience- Relevant Indicators Lists ....................................................................................................... 55 Description of Methodology ................................................................................................................... 70 Appendix 2: Some Considerations for Climate & Disaster Resilience-Building DPF Operations ................ 73 Background ............................................................................................................................................. 73 Defining Policy Focus and Scope ............................................................................................................. 73 Selecting Policy Actions........................................................................................................................... 79 Measuring Policy Results ........................................................................................................................ 82 Description of Methodology ................................................................................................................... 88 2 ReM&E December 2017 Acknowledgements This publication is a product of the World Bank ‘Results Monitoring and Evaluation for Resilience Building Operations’ (ReM&E) project, which is part of a broader Programmatic Approach on ‘Enhancing Climate and Disaster Resilience of World Bank Sustainable Development Operations.’ This project is managed by the Climate Change Strategy and Operations Team and the Sustainable Development Chief Economist Office, under the Sustainable Development Vice Presidency and the Global Themes Vice Presidency. The ReM&E team is led by Nathan Engle and Ulf Narloch. Core team members include Sundus Siddiqi and Karima Ben Bih. Additional support to the team has been provided by Anna Williams, Calli VanderWilde, Christina Irene, Saher Tanvir, and Silvia Marquina-Leon. The World Bank’s Independent Evaluation Group has played a key role in facilitating the learning components of this project. Management oversight is provided by Marianne Fay, Stephen Hammer, Neeraj Prasad, and Genevieve Connors. The team also expresses gratitude to the following individuals for their support in serving on the project’s Advisory Group: Rosina Bierbaum, Nancy MacPherson, Heather McGray, Christine Roehrer, Christopher Nelson, Niels Holm-Nielsen, Holger Kray, and Luis Andres. This guidance note was written by Ulf Narloch, Nathan Engle, and Sundus Siddiqi with key inputs from Karima Ben Bih. Useful comments and edits were received from Anna Williams, Saher Tanvir, and Calli VanderWilde. This publication greatly benefited from the feedback provided by World Bank reviewers Hoveida Nobakht, Niels Holm-Nielsen and Greg Browder, and external reviewer Dorcas Robinson (Co- director, Resilience Measurement, Evidence, & Learning Community of Practice). Sundus Siddiqi prepared the review of resilience-relevant indicators in World Bank operations, presented in Appendices 1 and 2. Niels Holm Nielsen provided oversight for the development policy evaluations referenced in Appendix 2. The team is grateful to the focal points and technical experts within the six Sustainable Development Global Practices (Agriculture, Energy, Environment, Social and Urban Resilience, Transport, and Water) of the World Bank for their valuable consultations and feedback throughout the duration of the project. Finally, the team is deeply appreciative of funding from the Global Facility for Disaster Reduction and Recovery (GFDRR), which has supported the ReM&E project and the drafting of this product. 3 ReM&E December 2017 Introduction This note offers practical guidance for monitoring and evaluation (M&E) of World Bank operations that aim to increase resilience to climate-related natural disasters and long-term climatic changes (henceforth “resilience M&E”), through activities that include climate change adaptation and disaster risk management (henceforth “resilience-building”) components. Operations that are themselves vulnerable to climate and disaster risks, and are consequently screened and made more “climate-resilient,” do not fall within this scope when they do not have resilience-building as an objective or targeted component. Such projects follow more of a “safeguards type” approach, whereby climate and disaster risk screening1 is performed, and then mitigation measures are integrated into the project design to reduce any risks to the success of the project. This note proposes to enhance the understanding of design options available for M&E systems in the context of resilience-building operations. While many of these design options are Global Practice (GP) dependent, which has been reflected in the work undertaken to develop sector-specific approaches to resilience M&E under the ‘Results Monitoring and Evaluation for Resilience Building Operations’ (“ReM&E”) project, this note primarily serves as a synthesis of generally relevant, cross-cutting recommendations and applications for all sectors. This guidance note aims to bring about an improvement to M&E and enable teams to design evidence-based resilience-building projects. The application of improved M&E may provide a platform of evidence that can guide implementation and recommend corrective measures. This note presents findings gleaned from an extensive review of resilience-building World Bank operations, that are complemented by international resilience-related initiatives such as: the Global Facility for Disaster Risk Reduction and recovery (GFDRR), the Climate Investment Fund’s (CIF) Pilot Program for Climate Resilience (PPCR), the UK’s International Climate Fund (ICF), and the Adaptation Fund (AF). The primary audience is Task Team Leaders (TTLs) and operational staff of the World Bank, particularly within the Sustainable Development Vice Presidency (SD VP), who design and implement M&E systems for resilience-building operations. Recommendations may also be applicable to resilience M&E for other Vice Presidencies of the World Bank, as well as for projects funded and supported by other development partners including governments, multilateral development banks, UN-agencies, bilateral development agencies, non-governmental organizations, and nationally-funded projects. Government and corporate entities implementing or coordinating resilience-building projects may also find this guidance note helpful. The note is structured around the main elements for designing a resilience M&E system. After first providing key definitions and concepts of resilience-building (Section 1), it highlights general challenges and offers guiding principles for resilience M&E (Section 2). This note then discusses resilience-related design considerations for respective components of a typical M&E system (Figure 1); results frameworks (Section 3) where an illustrative overarching results framework for the SD VP at the World Bank is presented (Figure 2); indicators (Section 4); monitoring and reporting (Section 5); and evaluation (Section 6). 1 More details on World Bank risk screening tools are presented on Page 13. 4 ReM&E December 2017 Appendix 1 presents an illustrative list (Table 4) of 276 climate and disaster resilience-related results indicators, organized by sectors, gleaned from 70 resilience-relevant World Bank investment operations (IPFs). Appendix 2 provides a similar illustrative list (Table 11) of 96 resilience-relevant indicators for Development Policy Financing operations (DPFs/DPOs), organized by policy pillars, gathered from an analysis of 26 resilience-building World Bank DPFs. Appendix 2 further outlines some considerations for the design of key elements of a Resilience DPF: (1) defining the policy focus and scope including potential policy pillars and illustrative policy areas, (2) selecting policy actions that can be fit and sequenced within a broader results chain (Figure 5), and (3) measuring policy results through resilience-relevant indicators. How to Read This Note Each section starts with a “Main Messages” box that describes the role of main M&E elements in general before discussing relevant resilience-related considerations. “Recommendations for Task Team Guidance” and “Good Practices” from case studies are listed at the end of each section, along with “Further Reading” resources. The examples presented throughout the note are for illustrative purposes only; they should not be viewed as exclusive of other potential applications, as prescriptive of how to frame or measure resilience objectives, or as comprehensively representative of the complexity of resilience-building. Figure 1. Primary Components of an M&E System Results Framework Indicators Monitoring & Serves as Serves as Reporting basis for basis for What are we trying to How do we track and How are we doing? achieve and how are we measure progress toward going to achieve it? results? Informs Evaluation What results—outcomes and impact—have been achieved? What worked well and did not work well, and why? What lessons can be identified from implementation? How can the project design be improved? Adapted from Williams (2016) 5 ReM&E December 2017 Summary of ReM&E Engagement Other deliverables produced under the ReM&E initiative include: • Evaluation Guidance Note2 for resilience-building operations • Good-practice case studies3of three World Bank resilience-building projects • Scoping study4 to help identify emerging lessons and to define the key steps to develop a M&E system for climate and disaster resilience-building operations • Toolkit to address TTLs’ most frequently asked questions on ReM&E This initiative also delivered working lunches with World Bank task teams to enhance learning, and two external workshops (2015 and 2017) to bring together experts in the field and exchange ideas. The analyses presented, written materials produced, and the overarching results framework5 have been well-received. For instance, projects that are part of the World Bank’s Africa Climate Business Plan have been analyzed using the ReM&E framework and presented in its 2017 Progress Report to illustrate how these projects are building resilience.6 The indicator analysis described in the Appendices of this note have also been instrumental to enhance TTLs’ approaches to resilience M&E. 2 See: http://documents.worldbank.org/curated/en/669941506093754016/Evaluation-of-resilience-building- operations-operational-guidance-paper-for-project-task-teams. 3 See: http://documents.worldbank.org/curated/en/400851506100481060/World-Bank-resilience-M-E-ReM-E- good-practice-case-studies. 4 See: http://documents.worldbank.org/curated/en/577241468184764561/Options-for-results-monitoring-and- evaluation-for-resilience-building-operations. 5 See Section 3: Developing Resilience Results Frameworks. 6 See: http://documents.worldbank.org/curated/en/247501510166915125/Accelerating-climate-resilient-and-low- carbon-development-second-progress-report-on-the-implementation-of-the-Africa-climate-business-plan- overview. 6 ReM&E December 2017 Section 1. Operationalizing Resilience Definitions and Concepts Main Messages Resilience definitions and concepts underpin resilience M&E. They determine the broader objectives supported by resilience-building operations and hence, the higher-level results to be monitored and evaluated. They build the common ground upon which to develop resilience- relevant results frameworks, to select indicators, and to monitor, report, and evaluate results. A common understanding of “resilience-building” is a necessary (but in-and-of-itself insufficient) condition for a harmonized resilience M&E approach that includes results chains, indicators, monitoring systems, and evaluation techniques. Each operation should clearly define and conceptualize what is meant by resilience-building and how resilience-building is operationalized on the ground through project activities. There is no standard resilience definition, but existing definitions have some common elements. The term “resilience,” even in the context of climate and disaster risks, has no standard definition. Accordingly, no clear boundaries define the types of activities considered to be resilience-building. Part of the challenge arises from the context-specific nature of resilience. For example, large-scale irrigation projects could be categorized as resilience-building in environments with adequate water resources, but would increase vulnerability and threaten development in water scarce environments. Examples of commonly used resilience definitions include: • Inter-governmental Panel on Climate Change (IPCC), 5th Assessment Report: “The capacity of a social, economic, and environmental systems to cope with a hazardous event or disturbance, responding or reorganizing in ways that maintain its essential function, identity, and structure, while also maintaining the capacity for adaptation, learning, and transformation” (IPCC, 2014). • Sendai Framework for Disaster Risk Reduction 2015-2030: “The ability of a system, community or society exposed to hazards to resist, absorb, accommodate to and recover from the effects of a hazard in a timely and efficient manner, including through the preservation and restoration of its essential basic structures and functions” (UNISDR, 2009). • United Kingdom’s Department for International Development (DFID): “The ability of countries, communities, and households to manage change, by maintaining or transforming living standards in the face of shocks or stresses without compromising their long-term prospects” (UK DFID, 2011). • United States Agency for International Development (USAID): “The ability of people, households, communities, countries, and systems to mitigate, adapt to, and recover from shocks and stresses in a manner that reduces chronic vulnerability and facilitates inclusive growth” (USAID, 2012). • Rockefeller Foundation: “The capacity of individuals, communities, and systems to survive, adapt, and grow in the face of stress and shocks, and even transform when conditions require it. Building resilience is about making people, communities and systems better prepared to withstand catastrophic events—both natural and manmade—and able to bounce back more quickly and emerge stronger from these shocks and stresses” (Rockefeller Foundation, 2015). 7 ReM&E December 2017 • Resilience Alliance: “Resilience deals with the tension between persistence and change (i.e., on the one hand understanding and managing the capacity to absorb shocks and maintain function, but on the other hand also to maintain the capacity for renewal, reorganization, and development at a variety of scales” (Biggs et al., 2015; Folke, 2006). Although these definitions vary in theme and foci, they are concurrently united by a few common elements: • Hazards through shocks and stresses, which in the context of climate change and natural disasters can be caused by acute, high-impact extreme events and slow-onset, long-term climatic changes. • People, economic assets, and/or bio-physical systems (e.g., ecosystems and physical infrastructure) at different-levels (farm/household, community, cities, provinces, state, country) exposed to hazards. • Ability and/or capacity to anticipate, respond, and recover from hazards in a way that maintains, if not improves, welfare, assets, structures, and functions. In general, “resilience” and “resilience-building” refer to heightened system capacity to anticipate, respond to, and recover from hazards. Finer details—who or what is exposed to what kind of hazards— alter the scale, nature, and timeframe of resilience-related interventions (e.g., protecting coastal cities from infrequent, but large impact events, such as typhoons vs. supporting subsistence farmers during repeated, low-impact events, such as precipitation shortfalls). A variety of resilience concepts exist and their roles in resilience-building operations often vary. Based on the above definitions, resilience-building involves strengthening three specific capacities (OECD, 2014a): • Absorptive capacity: The ability of people, assets, and systems to prepare for, mitigate, or prevent negative impacts of hazards so as to preserve and restore essential basic structures and functions, for example through protection, robustness, preparedness, and/or recovery. • Adaptive capacity: The ability of people, assets, and systems to adjust, modify or change characteristics and actions to moderate potential future impacts from hazards so as to continue to function without major qualitative changes, for example through diversity, redundancy, integration, connectedness, and/or flexibility. • Transformative capacity: The ability to create a fundamentally new system so as to avoid negative impacts from hazards. These capacities are associated with a number of resilience-building concepts that can be realized through a variety of activities. These concepts are illustrated in Table 1, and their potential linkages with building absorptive, adaptive, and transformative capacity are highlighted in Section 3. 8 ReM&E December 2017 Table 1: Activity examples for resilience-building concepts that support the building of absorptive, adaptive, and/or transformative capacity Concept Activity Example Building/ Reinforcing protective infrastructure (e.g., seawalls) Protection Improving tree canopy cover for environmental protection Resettling communities away from risk-prone areas Robustness Maintaining and upgrading roads and other critical infrastructure Developing contingency plans/ funds Preparedness Developing early warning systems, emergency shelters and evacuation routes Providing disaster risk insurance for vulnerable populations Reconstructing damaged housing and infrastructure Recovery Rehabilitating damaged ecosystems (e.g., forest restoration) Food transfers to disaster affected households Providing less weather-sensitive income generating activities, (e.g., jobs in the Diversity tourism industry) Establishing multiple power-generation back-up systems at different physical Redundancy locations Integration/ Managing/governing water resources across jurisdictions (e.g., with states and connectedness local governments at the basin scale) and sources (e.g., surface and ground water) Building an irrigation system for farmers previously dependent on invariable rainfall to water their crops Flexibility Constructing BRT public transport to provide residents with access to the city center within 60 minutes Budgeting and planning for contingencies to allow for swift action in the face/wake of a disaster It would be impossible for operations to address all these capacities and concepts. Instead, the geographic, socio-economic, and sector-context should inform how the operation seeks to build resilience and, consequently, which concepts, and thus capacities, are prioritized. Generally, concepts related to “adaptive capacity” and “transformative capacity” are much harder to define and operationalize than concepts related to “absorptive capacity,” but all are important to resilience- building in a dynamically evolving world. Recommendations for Task Team Guidance ➢ Operationalize resilience by translating resilience definitions and concepts to the specific context of an operation. This includes the geographic, socio-economic, and sectoral conditions related to the operation. ➢ Adopt an operation-specific definition of resilience. The chosen definition should characterize: (i) hazard type(s) in terms of shocks and stresses; (ii) the people, economic assets, and bio-physical systems vulnerable to the hazards; and (iii) the ability of such entities to anticipate, respond to, and recover from the shocks and stresses. The definitions, formalized by task teams, should clearly define whom or what should be made resilient through which channels against what kind of 9 ReM&E December 2017 stresses and shocks, all while considering geographic, socio-economic, and sectoral particularities of the operation. ➢ Consider resilience concepts related to absorptive, adaptive and transformative capacities. Prioritize those that are most relevant for resilience-building in the context of the operation. Task teams should identify how the operation addresses resilience concepts given specific geographic, socio-economic, and sectoral contexts. Good Practices As part of the ReM&E initiative, three case studies have been developed to provide practical examples of World Bank projects that reflect the outlined principles and highlight various good practices. The case studies are based on the following projects: 1. Kenya Climate-Smart Agriculture Project; 2. Mozambique PROIRRI —Sustainable Irrigation Development Project; and 3. Mekong Delta Integrated Climate Resilience and Sustainable Livelihoods Project. The following good practices arise from one or more case studies: ✓ Strengthening project design with the help of resilience M&E experts ✓ Engaging relevant stakeholders in the project’s M&E design ✓ Embedding strong resilience framing in project design ✓ Building multiple M&E approaches into project design ✓ Clearly defining resilience-relevant indicators and providing guidance on measurement approaches ✓ Balancing indicator ambition with practicality ✓ Securing resources needed for robust M&E ✓ Making a clear case and choosing clear objectives for impact evaluation ✓ Undertaking evidence-based learning throughout the course of the project to improve implementation and enhance results, in addition to accountability These good practices are briefly expanded upon in highlighted boxes throughout this note. The case studies have also been published in their entirety by the World Bank. 10 ReM&E December 2017 Further Reading ✓ Biggs, R., M. Schlüter, and M. Schoon (eds). 2015. Principles for building resilience: Sustaining ecosystem services in social-ecological systems. Cambridge University Press, United Kingdom. ✓ Brooks, N., E. Aure, and M. Whiteside. 2014. Assessing the impact of ICF programmes on household and community resilience to climate variability and climate change. Final report, UK Department for International Development (DFID), London, UK: Section 2. ✓ Folke, C. 2006. Resilience: The emergence of a perspective for social-ecological systems analyses. Global Environmental Change 16: 253-267. ✓ OECD (Organization for Economic Co-operation and Development). 2014a. Guidelines for Resilience Systems Analysis: How to analyse risk and build a roadmap to resilience. Paris: OECD: Section “Conceptual Framework.” ✓ World Bank. 2017. Resilience M&E Good Practice Case Studies. World Bank, Washington, DC, USA. 11 ReM&E December 2017 Section 2. Addressing Unique Challenges of Resilience M&E and Guiding Principles Main Messages The set of principles provided in this note should be used to establish the “cultural contract” within an institution of how to approach and operationalize resilience M&E (Williams, 2016). They should be incorporated into M&E design in a manner that reflects resilience specific characteristics and challenges across different M&E system components (results frameworks, indicators, monitoring and reporting, and evaluation). Lessons from experience bring important insights to the growing field of resilience M&E (Williams, 2016). Recent developments illustrate the unique challenges facing resilience M&E design and re-emphasize broad challenges encountered in designing any good M&E system. Resilience is often addressing ‘wicked problems’ that require creative and adaptive solutions. Building resilience to climate change and disaster risks is often entangled with “wicked problems”, or those issues for which stakeholders differ on defining the problem itself, let alone the solution(s) to the problem. Due to incomplete, contradictory, and changing operation requirements, relevant pathways to resilience to climate and disaster risk are often difficult to recognize or agree upon. Thus, best practices for resilience-building are neither straightforward nor easily agreed upon. The complexities inherent to managing climate and disaster risk require innovative, cross sectoral, and context-specific solutions that can adapt to continuously evolving conditions. The implications for climate and disaster resilience M&E include the need to recognize that: • The transferability, replicability, and scalability of any given resilience-building solution—not only across operations but sometimes even across different components of the same operation—may be limited; • Linear cause-and-effect relationships may not exist, making it difficult to identify clear links between interventions and resilience outcomes/impacts; • Resilience-building is a long-term endeavor that requires addressing a comprehensive set of interacting conditions—as such, changes in the degree of resilience often occur outside of the control and lifetime of given operation; • Due to its complexity and relative nascence, resilience-building should be approached as an evolving area—myriad solutions are continuously being piloted and made available to task teams. Resilience M&E is a quickly evolving field. Resilience M&E has been rapidly evolving over the past few years. The field is relatively young and learning quickly from experience, in part through trial and error. For example, early applications have often been overly ambitious. The original systems (e.g., those of the Global Environment Facility (GEF) and PPCR) have 12 ReM&E December 2017 subsequently been revised and simplified. The general direction has been to move away from complicated systems that are overly burdensome and impractical. While replication is difficult, it is key to learning and resilience M&E will be a “learning by doing” field for some time. It will take accumulated experience— trial and error, shared challenges, and innovations—to amass a body of evidence that will improve resilience M&E strategies, implementation, and ultimately its impact in the years ahead. Risk characterization and understanding vulnerabilities are key to operationalizing resilience. Considering climate change and disasters in today’s plans and projects can increase the long-term success of development efforts. Thus, the World Bank has various tools and methodologies in place to screen for risks. Climate and disaster risk screening represents a proactive approach to considering short- and long- term climate and disaster risks in project and national/sector planning processes at the early concept stage. Screening is an initial, but essential, step to ensure these risks are assessed and managed to support mainstreaming of climate and disaster resilience into key development policies, programs, and projects. Two examples of tools used by the World Bank are given below: • The Climate and Disaster Risk Screening tools7 provides links to climate projections, country adaptation profiles, and disaster risk data sources from the World Bank’s Climate Change Knowledge Portal.8 The data, combined with the user's understanding of the subject matter and country context, generates a characterization of risks to help inform dialogue, consultation, and planning processes at the project and program level. Screening tools exist on a General level, National/Policy level, for Coastal Flood Protection, and for Roads Sectors. Enhanced screening tools have been developed for the following four sectors: Water, Agriculture, Health and Energy, while more are being developed. • GFDRR’s ThinkHazard! Tool9 provides a general view of the hazards, for a given location, that should be considered in project design and implementation to promote disaster and climate resilience. The tool highlights the likelihood of different natural hazards affecting project areas, provides guidance on how to reduce the impact of these hazards, and where to find more information. Resilience M&E poses many methodological challenges. Resilience-building occurs in the context of uncertain climate futures, with inevitable, but often unpredictable, disasters. These conditions pose numerous methodological challenges for resilience M&E, including: • Directly measuring the degree of resilience of people, assets, or systems. The task is challenging, if not impossible, due to the often-unobservable nature of resilience—for an intervention that addresses an infrequent extreme event, resilience results can be evaluated only if the event occurs. In the absence of shock or stress, it may be difficult to determine the degree to which the intervention has delivered on objectives. 7 See: https://climatescreeningtools.worldbank.org/. 8 See: http://sdwebx.worldbank.org/climateportal/. 9 See: http://thinkhazard.org/en/. 13 ReM&E December 2017 • Establishing surrogate or proxy measures. While alternative indicators can often be used to reveal performance trends, proxies for resilience are difficult to establish due to the combined effect of multiple stressors, long time-frames for building and realizing resilience, and cross-scalar (household, community, provincial, region, watershed, national, etc.) intervention design. These complexities lead to shifting baselines that impede the creation of common indicators and targets that can be compared across space and time. • Evaluating and attributing intervention results. Tracking resilience requires flexible systems that recognize change over long time horizons. This challenges the traditional design of M&E systems, particularly with regards to evaluating and attributing the results of interventions. For interventions addressing long-term risks from climate change or extreme events, results attribution can be especially difficult—long-term climatic changes may not yet be evident at the time of project evaluation. Resilience M&E requires sufficient time for outcome-level results to mature and be measured against a set of context and intervention specific process/outcome indicators. • Declaring success. “Successfully” building resilience at one place, at one point in time, through a given intervention, does not invariably entail broader, durable, systems-level resilience. These unique challenges translate into four Guiding Principles (listed under “Recommendations”) for designing a resilience M&E system. Recommendations for Task Team Guidance ➢ Principle 1: Build innovative and flexible M&E systems that can be improved over time, and expand M&E to not only focus on accountability and building transparency, but also learning. The components of M&E systems designed for accountability (holding projects responsible for the delivery of a set of key results) often contrast—but need not to—with those of M&E systems that also integrate learning objectives. Both accountability and learning are important for all operations—not only those in the resilience field. Having robust M&E systems also leads to improved transparency, and thus improved decision-making, justifying their use. However, due to the aforementioned challenges associated with resilience M&E and the continuous opportunities for improvement, learning should be centrally incorporated into every resilience M&E system. Resilience M&E should intentionally focus on innovation, creativity, and experimentation. It should go beyond traditional good practice methods by recognizing the experimental, learning-by-doing nature of complex interventions and by adopting flexible results frameworks and indicators (e.g., moving away from fixed targets and defining evolving targets and regular course-corrections). Furthermore, including a professional M&E expert to help design the system is pivotal for improving performance. Specifically, resilience M&E specialists have in-depth knowledge from both a climate resilience and an M&E perspective, making them indispensable. They bring real- world examples and evidence-based practices that can significantly improve design and implementation of resilience M&E concepts and principles. ➢ Principle 2: Emphasize local-contexts and a beneficiary focus by building on participatory approaches. Given that resilience depends on context, it is essential that resilience-building operations and their M&E systems are not only specifically designed for but also with the 14 ReM&E December 2017 program’s intended beneficiaries. Design considerations for resilience M&E systems may require expert judgements and/or quantitative data that precisely identifies and measures subtle but critically important local features. A participatory approach that draws beneficiaries into the M&E process provides a means to not only overcome such constraints, but also the opportunity to strengthen task team understanding and interpretation of interacting factors and changing conditions with local knowledge. ➢ Principle 3: Build from existing reporting frameworks, systems, and requirements to keep data and capacity needs manageable. As far as these are relevant and can properly capture resilience results, resilience M&E systems should look to align with existing M&E frameworks including: indicator systems of relevant international agreements (e.g., the Sustainable Development Goals or the Sendai Framework of Disaster Risk Reduction); resilience related funds and initiatives (e.g., PPCR, GFDRR, and the Green Climate Fund (GCF)); and corporate results frameworks (e.g., World Bank Corporate Scorecard, Corporate Results Indicators, and other results indicators from sector strategies and/or action plans). ➢ Principle 4: Integrate multi-dimensionality, interactions between sectors and actors, and feedback-loops: Resilience M&E should consider the complexity and numerous dimensions of resilience through multiple climate and disaster hazards and their relationship with other stressors. Vertical interaction of different scales of decision making and project implementation (local, regional, national, etc.), as well as horizontal interactions (different stakeholders and sectors), different timescales (short, medium, and long term), and a variety of uncertain factors and drivers should also be considered. Multiplier, spill-over, and demonstration effects may be difficult to identify and characterize ex ante (e.g., maladaptation)—these and other impacts that go beyond the intervention’s direct scope should still be reflected in the resilience M&E system. Good Practices ✓ Bringing in resilience M&E experts to work with the design team; ✓ Engaging stakeholders in designing the project components, theory of change (results framework), and indicators Excerpt from the Case Study on the Kenya Climate-Smart Agriculture (CSA) project Approved in 2017, the Kenya CSA project aims to increase agricultural productivity and build resilience to climate shocks in arid and semi-arid regions. The project is scheduled to be implemented between 2017 and 2022. To understand the current ReM&E landscape and good practices, the project design team worked with a consulting firm, UNIQUE forestry and land use GmbH (UNIQUE), to help think through the design of the Kenya CSA project. UNIQUE worked with stakeholders to develop a capacity “lens” to resilience building that they then applied, along with applicable resilience definitions and concepts, to the project’s Project Development Objective (PDO) and theory of change. The UNIQUE consultants helped the project design team refine how project activities would tie to a resilience-focused theory of change; to consider what progress tracking activities might be missing (e.g., feedback loops between farmers and developers of the 15 ReM&E December 2017 ICT10 system); and to identify appropriate SMART11 indicators that consider aspects such as aggregation and relevance at different scales. Further Reading ✓ Bours D., C. McGinn, and P. Pringle. 2014. Guidance Note 1: Twelve Reasons Why Climate Change Adaptation M&E is Challenging. Phnom Penh and Oxford: SEA Change CoP & UKCIP. ✓ Dinshaw, A., S. Fisher, H. McGray, N. Rai, and J. Schaar. 2014. Monitoring and Evaluation of Climate Change Adaptation: Methodological Approaches. OECD Environment Working Papers, No. 74, OECD Publishing. ✓ GIZ (Gesellschaft fur Technische Zusammenarbeit). 2013. Adaptation Made to Measure: A Guidebook to the Design and Results-Based Monitoring of Climate Change Adaptation Projects. GIZ, Bonn, Germany. ✓ Leiter, T. 2015. Linking Monitoring and Evaluation of Adaptation to Climate Change across Scales: Avenues and Practical Approaches. In D. Bours, P. Pringle & C. McGinn (Eds.), Monitoring and Evaluation of Climate Change Adaptation and Resilience Interventions. New Directions for Evaluation, 147. ✓ OECD (Organization for Economic Co-operation and Development). 2014a. Guidelines for Resilience Systems Analysis: How to analyse risk and build a roadmap to resilience. Paris: OECD. ✓ OECD (Organization for Economic Co-operation and Development). 2014b. Monitoring and Evaluation of Climate Change Adaptation. Paris: OECD. ✓ Spearman, M. and H. McGray. 2011. Making Adaptation Count: Concepts and Options for Monitoring and Evaluation of Climate Change Adaptation. Eschborn, Germany: GIZ. ✓ Williams, A., 2016. Options for resilience Results Monitoring and Evaluation for Resilience- Building Operations. World Bank Group & GFDRR Scoping Paper. 10 Information and Communication Technology. 11 Specific, Measurable, Attributable, Realistic, and Time-bound. 16 ReM&E December 2017 Section 3. Developing Resilience Results Frameworks Main Messages Resilience results frameworks provide overarching conceptual guideposts for resilience M&E by defining anticipated results and their connections to project objectives (Williams, 2016). Results frameworks tend to utilize visual logic tools—including results chains, theories of change, log frames, logic models, and outcome mapping—that clearly illustrate the causal mechanisms connecting inputs and activities to anticipated outcomes. Results frameworks serve as the basis for indicator selection, monitoring and reporting, and (oftentimes) evaluation. All individual operations should have clearly articulated results frameworks that represent the logic underlying how given intervention activities contribute to the Project Development Objective (PDO) and broader development goals.12 This includes identifying the areas along the results chain where operation activities contribute to climate and disaster resilience. These individual operations can be supported by the development of broader, sector-relevant theories/results frameworks that help to frame the “resilience story” for a given sector. The sectoral focus can guide the projects toward potential pathways of resilience facilitated by these operations, and as these resilience-building project accumulate, they can help consolidate a more solid resilience narrative for the sector. Finally, an overarching M&E theory of change for resilience-building operations in the SD VP can help to frame the broader narrative even further by supporting alignment across GPs’ resilience M&E frameworks. It should include cross-sectoral components that are applicable universally in resilience-building designs and offer a broad overview of general goals associated with resilience-building operations in the context of sector-specific modus operandi. This section begins by introducing the overarching results framework for the SD VP that was developed by the ReM&E project and validated through numerous internal and external consultations. It then briefly describes the progress made to-date on working with the GPs to develop sector-specific theories of change/results frameworks, that build into indicator menus and inform other elements of the sectors’ resilience M&E systems. The section ends with guidance to task teams working to develop resilience results frameworks for their individual projects. This section aligns with the basic principles of World Bank’s Operations Policy and Country Services (OPCS) guidance on results frameworks.13 An overarching results framework helps to synthesize the resilience story across the SD VP The overarching results framework developed through the ReM&E project (see Figure 2) can help 12 IEG defines results frameworks as “explicit articulation (graphic display, matrix, or summary) of the different levels, or chains, of results expected from a particular intervention - project, program, or development strategy” (IEG, 2012). World Bank results frameworks include three main elements: (a) a statement of PDO; (b) indicators to track progress toward achieving outcomes and to ultimately measure PDO linked outcomes; and (c) clear unit specifications for each indicator and baseline, annual and final targets for each indicator, and defined roles and responsibilities for collecting, reporting, and analyzing indicator data (World Bank, 2013). 13 For more information and additional guidance, See: http://siteresources.worldbank.org/PROJECTS/Resources/40940- 1365611011935/Guidance_Note_Results_and_M&E.pdf. 17 ReM&E December 2017 articulate the linkages between relevant project areas and resilience-related capacities, outcomes, themes, and practices across the various categories of SD GPs’ operations (agriculture, water, transportation, etc.).14 Figure 2: Illustrative World Bank ReM&E Overarching Results Framework Resilience lies at the core of the overarching results framework. Absorptive, adaptive, and transformative capacities, along with eight resilience concepts (longer-term outcomes), concentrically frame the core, and they are enclosed by a broader set of resilience themes and clusters of resilience practices. The 14 The ReM&E project team conducted a series of consultations with resilience M&E and technical focal points in the SD GPs to make progress on sector-specific resilience theories of change/results frameworks. These spanned roughly across a year, and included one-on-one meetings (including multiple meetings with each GP), as well as broader group consultations across the SD VP. During these meetings, the team introduced and received feedback on the overarching results framework for the SD VP. The meetings were followed by an internal working lunch amongst the collection of GPs, as well as a workshop with the focal points and external resilience M&E experts, to discuss experiences, resilience M&E design and implementation challenges, examples of good practices to be shared across GPs and external partners, and feedback on the progress and outputs from the ReM&E project, including the overarching results framework. The team received validation on the overarching results framework through these various meetings and events. 18 ReM&E December 2017 framework, as a whole, bridges resilience capacities and concepts with various levels of potential outcomes of World Bank projects. In blending classic elements of M&E with the complex context of resilience, the overarching results framework shifts away from the traditionally linear format of M&E logic frames. Each concentric circle in the results framework can be (conceptually) rotated clockwise/counterclockwise to differentially align with the other circles and wedges. This allows the framework to accommodate multiple pathways towards resilience and thus, more appropriately account for constantly changing and unpredictable conditions that surround issues such as climate change and disaster risk, as well as the myriad ways of building resilience within and across sectors. This framework should be used to create scenarios or templates that articulate sector-specific theories of change, and task teams can ultimately plug-in specific activities, outputs, and intermediate outcomes that can align with this overarching results framework. Figure 3 demonstrates how a specific project might align activities, outputs, and outcomes with the overarching results framework. Figure 3: Pathways Toward Resilience Using the ReM&E Results Framework Based on a Resilient Cities Project Example Note that the resilience themes and clusters on the overarching results framework are there only to guide task teams and GPs/sectors as they develop their respective theories of change. Teams might choose to remove these aspects of the framework and rather link their projects directly to the concepts and capacities, as illustrated in Figure 4. 19 ReM&E December 2017 Figure 4. Example of a Project Linking to the ReM&E Results Framework without the Resilience Themes and Clusters Sector-specific theories of change help to further organize resilience results narratives Sectors are best positioned to develop the relevant themes and clusters of resilience that pertain to the numerous ways in which resilience is built through that sector’s various activities and projects. Developing and refining these within each sector will help to improve the narrative and storyline for how the sector is contributing to building resilience to climate and disaster risks, and help to improve other aspects of the M&E system, such as through the development of indicator menus, etc. As noted earlier, the ReM&E project has supported World Bank GPs in the development of such sector-specific approaches, but this note does note delve into further detail on these sector specific theories of change/results frameworks. 20 ReM&E December 2017 For individual projects, the PDO determines to what degree the results framework captures resilience- building The PDO stipulates what benefits an operational task team will be held accountable for delivering to its target group given project scope, resources, and timeframe. Formalizing a PDO requires a comprehensive assessment of the risks facing relevant people, assets, and systems, as well as the practical design of effective (and efficient) actions to increasing resilience to those risks. A PDO is outcome-focused, avoids mixing different results chain levels (e.g., project outputs and higher- level or longer-term outcomes), and only expresses results that are: (i) within the control of the operation; and (ii) achievable during its lifetime. Resilience-building is a long-term endeavor that requires several (often systems level) interventions—this not only makes increased resilience hard to measure, but also generally results in resilience outcomes occurring beyond the control and lifetime of an operation. Thus, again, the theory of change/results framework is critical for describing where on the pathway toward resilience the project is able to tangibly deliver results. The centrality of resilience-building to the PDO varies across resilience-building operations. The PDO and results framework should be articulated to appropriately reflect resilience as a primary objective, partial objective, or unintended objective. • Primary objective: Resilience-building is a central objective around which all other operation components and activities are formulated and structured (e.g., strengthened institutional framework for climate resilience and improved adaptive capacity of vulnerable communities. • Partial objective: Resilience-building is a sub-set of a larger objective that addresses a broader set of issues. Only some of the operation components and activities are formulated and structured around resilience-building (e.g., increased adoption of improved and climate-smart agricultural practices. • Unintended objective:15 Resilience-building is not part of a direct operation objective but strengthened resilience occurs as a positive externality (e.g., improved sustainable natural resources management and promoted livelihoods diversification in the selected oases. Intervention logic should be formulated around the different results levels of resilience-building A 2013 study by the World Bank’s Independent Evaluation Group (IEG) concluded that current World Bank results frameworks on climate change adaptation and resilience are not outcome-oriented, and as such, risk emphasizing spending over achieving results. Even when operations do define more outcome- oriented results, they have often struggled to measure them. Measurement challenges can be attributed in part to the debate surrounding the distinction between outputs, outcomes, and impact—there is a fine line between these results levels, and how these are defined also often depends on nuances embedded in the PDO. For example, establishing a climate 15 The recognition of “unintended objectives” as a category is subject to debate given that any operation that increases the well-being of vulnerable people (e.g. through improved incomes, health, and/or access to basic services) can be viewed as resilience-building (Hallegatte et al., 2016). 21 ReM&E December 2017 information system can be considered as an outcome if the PDO is to increase the availability of climate information or an output if the objective is to enable climate-informed decision-making. Resilience results frameworks and their underlying intervention logic should reflect resilience-building at different results levels: • Activities: Resilience-building actions and interventions facilitated and financed by the operation. Activities will vary between sectors/GPs, and can be categorized in different ways. Examples include: o Sharing Central Highland Hydrometeorology forecasting information; o Building, upgrading, or recovering physical infrastructure (such as roads, housing, public buildings); o Preparing climate-resilient road design guidelines; o Rehabilitating flood protection infrastructure; o Developing national resilient recovery institutional plans and programs; o Designing climate- and disaster-focused financial instruments and social protection programs. • Outputs: Resilience-building policies, services, and products generated by the activities. Examples include: o Climate change adaptation (CCA) and disaster risk management (DRM) focused institutional reforms, policies, plans, data and information services are developed; o Trainings or awareness raising campaigns to increase understanding of CCA, DRM and/or resilience best practices given; o Access to climate-adapted agricultural practices/ technologies; o Critical ecosystems replanted, protected, or placed under improved management; o Physical infrastructure built, upgraded, or recovered; o Financial instruments/ social protection programs deployed. • Intermediate Outcomes: Resilience-related short/medium-term benefits (i.e., changes) generated for the target group by the policies, products, and/or services financed by the operation. Examples include: o Improved regulation, incentives, and information for climate and disaster-informed decision-making; o Increased adoption of risk-reducing practices and behaviors; o Improved stability and functionality of ecosystems and improved availability of natural resources; o Increased protection and resistance to damage of physical infrastructure and human settlements; o Increased access to and coverage from financial insurance and social protection programs. • Final outcomes/impacts: Durable/long-term contribution to increased resilience. These results are often outside of the operation’s control and will only become evident in the years after a World Bank operation’s completion. This level encompasses the resilience-building of the operation (e.g., redundancy, integration, robustness, etc.) that directly contribute to the ability of people, assets, and systems to resist, absorb, accommodate, and recover in a timely and efficient 22 ReM&E December 2017 manner from a hazard without undermining its welfare, value, structure, and functioning in the long term. • Transformational goals: Higher-order development goals the operation contributes to— generally envisioned as poverty eradication and shared prosperity in a sustainable manner through climate and disaster resilient development. These are nearly universally outside of an individual operation’s control. The intervention logic for resilience-building can be illustrated in a variety of ways—from simple, linear results chains, to complex models that allow for feedback-loops and non-hierarchical relationships—and they can be presented as a graphic illustration, matrix, or text description. There is no standardized intervention logic that can represent the full complexity, multi-dimensionality, and situational dependency of resilience-building for every single project context. However, task teams are advised to start with the overarching results framework and/or the sector-specific results framework within their respective GP that are discussed earlier in this section (e.g., see Figures 2, 3, and 4). Recommendations for Task Team Guidance ➢ Spend sufficient time defining the intervention logic through a sound theory of change for the operation to capture the complexities associated with resilience-building. Carefully consider where along the results chain the operation intends to increase resilience. The intervention logic requires a sound problem statement that describes existing vulnerabilities to and impacts of hazards, and how these can be reduced (i.e., a development hypothesis). ➢ Strike the right balance between avoiding objectives that are too low (i.e., output focused) and too high (i.e., not directly in the control of the operation) on the results chain when formulating the PDO. Refrain from PDO formulations that cross-cut multiple levels of the results chain with broad statements about increasing resilience (i.e., try to avoid blanket and non-specific statements about increased resilience in the PDO). It is generally good practice to clearly define what specific aspect of resilience the project is targeting, thus making its measurement possible. Resilience should be woven into tangible and easily evaluated outcomes that can be viewed as proxy measures of resilience-building. Examples include: o Enable the adoption and implementation of land use management plans that integrate climate and disaster risk considerations; o Improve the use of weather- and/or hydro-meteorological information that enables climate-informed decision making; o Increase the adoption rate of climate-adapted agricultural practices by vulnerable farmers; o Improve the restoration/management of ecosystems that serve as natural buffer zones or stabilize critical areas (e.g., steep slopes); o Upgrade hazard-exposed infrastructure to bring it in line with climate-informed design standards (e.g., roads, housing, public buildings, etc.); o Expand the coverage and disaster-responsiveness of social safety nets; o Increase the capacity of disaster-prone state(s) to respond to and recover from disaster- induced emergencies. 23 ReM&E December 2017 ➢ Integrate resilience into the results framework based on centrality to the PDO: o Primary objective: If resilience is the focus of the operation, the results framework should be structured around resilience-building. The resilience-building contributions made by all operation components, activities, and results should be clearly articulated. o Partial objective: If resilience is one of many operation goals the results framework should reflect resilience-building within a broader set of results. Links to resilience- concepts should be clearly established for all relevant resilience-building components and activities. o Unintended objective: If a PDO is not specific to climate and disaster hazards (e.g., increase ability of household to manage price volatility, or improve management of water resources) but seeks to address some of the underlying causes of vulnerability, resilience concepts could still be reflected in the results framework as robustness, diversity, flexibility etc., as long as achieving these is realistically in the purview of the project’s scope and timeframe. While operation components that increase the well-being of at risk groups without directly addressing causes of vulnerability may be able to capture links to climate and disaster resilience in the project storyline, they may not be able to enter the results framework. Task teams should strive to link the theory of change, activities, outputs and outcomes to the capacities (absorptive, adaptive, and transformative), and the PDO level indicators ought to be indicative of this. ➢ Carefully define measurable and meaningful outcomes to be achieved by the operation, as well as the outputs enabling those outcomes. A well-articulated and balanced results framework captures both the outputs and outcomes of an operation within its broader intervention logic. Outcomes are generally more challenging than outputs to measure—especially with regards to resilience-building—making it crucial to formulate outcomes so that their status can be tracked against suitable indicators (see Section 4). 24 ReM&E December 2017 Good Practices ✓ Strong resilience framing embedded in project design Excerpt from the Case Study on the Mekong Delta Integrated Climate Resilience and Sustainable Livelihoods Project The World Bank has carried out several projects in the Mekong Delta area, each based in a specific sector. The Mekong Delta project is groundbreaking as it develops a systems approach to build resilience, rather than operating in “silos.” It focuses on a complex hydro-ecological system –with upstream and downstream linkages- that requires integrated solutions to build resilience to climate change and development impacts. Resilience is not a secondary consideration or co-benefit; it is the core objective approached by the Bank’s design team through a systems lens. This has been achieved through careful planning and analysis. Unlike other relatively simplistic results frameworks that rely on common, often generic, sector- based measures (e.g., km of dykes built), the multi-sectoral design team developed the project’s results framework to reflect the spatial context of the planned sub-projects. These sub-projects comprised of integrated “packages” of water- and agriculture-related infrastructure, as well as climate-resilient livelihood practices, articulated within a resilience framework. Component 1 of the project focuses on enhancing water, salinity monitoring systems, providing decision-support, and developing an Integrated Master Plan for the Delta. Components 2-4, are spatially cast to reflect the resilience framing versus more traditional sector-based components. Further Reading ✓ Bours D., C. McGinn, and P. Pringle. 2014. Theory of Change for Climate Change Adaptation Programming. Guidance for M&E of Climate Change Interventions Series. Guidance Note 3. SEA Change, UK CIP. ✓ GFDRR (Global Facility for Disaster Reduction and Recovery). 2016. GFDRR Program Logic. GFDRR, Washington, DC, USA. ✓ GIZ (Gesellschaft fur Technische Zusammenarbeit). 2013. Adaptation Made to Measure: A guidebook to the Design and Results-based Monitoring of Climate Change Adaptation Projects. GIZ, Bonn, Germany. ✓ IEG (Independent Evaluation Group). 2012. Designing a Results Framework for Achieving Results A How-to Guide. IEG, World Bank Group, Washington, DC, USA. ✓ IIED (International Institute for Environment and Development). 2014. The Tracking Adaptation and Measuring Development (TAMD) framework. IIED. London, UK. ✓ World Bank. 2013. Results Frameworks and M&E: A Guidance Note. Prepared by OPCS, World Bank, Washington, DC, USA. 25 ReM&E December 2017 Section 4. Selecting and Defining Resilience-Related Results Indicators Main Messages Resilience-related results indicators should be selected to monitor the status of resilience-building operations at different levels of the results chain. Indicators should be activity-specific and do not need to be framed around the term “resilience.” Due to the complexity of resilience-building, indicator development requires careful consideration. Comprehensive indicators will capture changes to the degree of resilience experienced by people, assets, and/or systems vis-à-vis the intervention logic/results framework. A variety of options for measuring relevant resilience-building/resilience-related results exist. Different indicator types can be used to measure resilience-related results. Any resilience-building operation must identify resilience-related indicators to track progress towards the PDO. Typical World Bank results frameworks include both intermediate (i.e., output-focused) and high- level (PDO) results indicators (i.e., outcome-focused). Resilience-related results indicators conform with these standards to allow for measurement at different levels of the results chain (see specific examples in Table 2): • Output indicators measure policy changes, products, and services delivered by resilience-building activities. They provide useful data on progress towards achieving results along the results chain. Resilience-related examples include: o Number of people trained in emergency management and response; o Number of households provided with improved (i.e., more resilient) technology (e.g., production inputs in agriculture); o Length of flood protection infrastructure (e.g., river embankments) constructed or rehabilitated. • Outcome indicators measure the short- and medium-term resilience-related benefits generated for the target group. They indicate if the PDO is being met as well as the degree to which the PDO contributes to resilience-building. Resilience-related examples include: o (Decrease in) Average number of months of household food insecurity; o (Increase in) Number of households with uninterrupted (water, sanitation, and/or energy) service during a hazard event; o (Decrease in) Number of people at risk of travel or transport disruptions. • Proxy indicators signal the ability or capacity of people, assets, and systems to be more resilient without directly measuring outcomes. Proxy indicators serve as acceptable substitutes especially when it is too difficult or costly to measure the outcome itself (World Bank, 2013). Predictive proxy indicators (PPIs) are a proxy indicator sub-type estimate of future outcomes; outcomes that go beyond the lifetime of the operation. PPIs have been utilized for forestry projects that face long-term 26 ReM&E December 2017 timeframes akin to those of resilience-building operations (PROFOR, 2014). Sometimes PPIs inform task teams of crucial progress along a multi-step processes by indicating milestone achievements towards a longer-term outcome (Bours et al., 2014). Resilience-related examples include: o (Increase in) Number of households/or communities using improved tools, information, and instruments to respond to climate change; o (Increase in) Number of people/households with access to more resilient-building services (e.g., early warning systems or safety nets); o (Increase in) Agricultural/forest area under improved (i.e., more resilient) management. Measurement units of resilience-related indicators can be quantitative or qualitative. While M&E tends to favor quantitative indicators for ease of aggregation and comparison, it is also common practice to include qualitative indicators. Qualitative information (e.g., content of training component) may be added to a quantitative indicator (e.g., number of trainings) to provide greater detail. Both indicator types offer advantages and disadvantages in resilience-building contexts: • Quantitative indicators are measured in numerical units (e.g., number of people, land area, kilometers of roads, etc.). They are often based on hard data such as physical or economic measurements, making them easier to verify, aggregate, and compare across (sub)projects. While individual indicators are typically designed to capture a singular aspect (of resilience), they may be combined into indices that express a wider range of factors. However, it is difficult and often subjective to determine the weighting and aggregation methods for such indices, particularly in data-scarce environments. Quantitative indicators are distinct for the degree of observability of change they offer. Quantitative indicators can be: o Observable indicators: Conventional M&E systems only include indicators for results that can be observed and monitored on the ground. While new technologies can reduce costs, these indicators are traditionally expensive due to data collection requirements across varying project implementation locations. To obtain measurable data for assessing actual hazard management capacities and abilities, these indicators require the realization of events. Yet, even when this is the case, observable indicators often lack established baselines for comparison. o Model-based indicators: Indicators that predict results based on hazard and risk modelling have more recently come under consideration. Model-based indicators allow teams to conduct estimates of hazard management capacities and abilities without observing actual changes. While model calibration has intensive data requirements, outcome prediction—even for multiple actions across multiple scenarios—has relatively low data requirements. • Qualitative indicators: By engaging project beneficiaries and stakeholders in participatory approaches (including surveys, expert interviews, focus group discussion, social media, etc.), qualitative indicators can collect more nuanced data, but it is much more difficult to track and compare indicators across space and time. One example where qualitative indicators are in essence transformed into 27 ReM&E December 2017 quantitative indicators is the scorecard method, which measures a variety of (resilience) dimensions against qualitative indicators.16 Task teams can choose amongst a variety of indicators for measuring resilience-related results. Due to the multi-dimensionality of resilience-building (and the hierarchies embedded in capturing results), there is no one-size-fits-all resilience indicator or standardized set of resilience indicators. Indicators vary between operations. They should be determined by operation-specific resilience concepts which in turn depend on geographic, socio-economic, and sector-contexts. As mentioned earlier, sector-specific indicator menus could be one result of the ongoing work with the GPs on developing sector-relevant theories of change/results frameworks, from which task teams can pick and adapt to be most relevant in the specific project context. For most operations, a multi-dimensional framework with a set of indicators is necessary to track progress towards resilience-building. However, resilience-related indicators will not necessarily differ from traditional indicators. In resilience M&E it is not the indicators themselves that are unique, but rather the manner in which they are combined into a suite that holistically reflects and assesses the status of resilience-building over time (Bours et al., 2014). As relevant for all indicators, resilience-related indicators should meet the SMART criteria:17 • Specific: Indicators are precisely formulated to measure only the intended changes in specific resilience-related outputs and outcomes—and not any other changes. For example, if the goal is to make agricultural practices more drought resilient, indicators should measure the adoption of improved irrigation practices, instead of the adoption of improved agricultural practices in general that are not necessarily relevant to drought resilience. • Measurable: Indicators are based on practical ways to measure changes in resilience-related outputs and outcomes in quantitative and/or qualitative units. Measurement requires the clear definition of all indicator elements (including what is meant by terms like “improved” or “resistant”) so that there is no ambiguity. Measurement also requires clearly defined data collection and analysis methodologies to ensure that the data are kept congruent, consistent, and comparable between different researchers working at different times. • Attributable: Indicators only measure elements within operation’s control and can therefore be attributed to project financed components and activities. For example, if a project promotes the adoption of more resilient seeds, an appropriate indicator could measure reductions in harvest 16 The PPCR Toolkit includes multiple scorecard examples, one of which is the PPCR core indicator, “Quality and extent to which climate responsive instruments/ investment models are developed and tested. ” See: https://www.climateinvestmentfunds.org/sites/default/files/knowledge- documents/ppcr_monitoring_and_reporting_toolkit_march_2016_revised.pdf. 17 There is, however, increasing recognition that the SMART criteria do not necessarily capture shifts in underlying drivers of vulnerability and risk, or the nuances and complexities inherent to human and socioeconomic dimensions of resilience. Capturing processes of change and reflecting the wider context within which change takes place requires additional criteria (Villanueva, 2011). 28 ReM&E December 2017 losses—but not reductions in loss of marketed output (which depends on other factors such as storage and transportation). • Realistic: Indicators are realistically measurable given the skills and resources available for data collection and analysis. It is futile to identify sophisticated indicators that measure avoided impacts from disasters through modelling and other advanced analytical approaches if the capacity required to undertake the analytical work is not available. • Time-bound: Indicators measure a change in resilience-related outputs and outcomes that can realistically be achieved within the timeframe of the operation, and the time needed for data collection and analysis corresponds to the timeframe allotted for such work. Depending on the sector-focus of the resilience-building operation, task teams should consider applying sector-specific indicators. Although many resilience-building operations include a cross-sectoral component (e.g., improve availability of climate information), they often concentrate on applications in one or a few sectors. Each sector houses several resilience-related indicators that differ greatly in terms of focus and results-levels (Table 2).18 Table 2: Some Examples of Sector-Specific Indicators from Existing World Bank Operations Sector Indicator Reduction in average length of household food insecurity faced by farm households Reduction in annual crop losses due to weather-related events Number of households that acquire farm inputs and assets to recover from losses Provision of increased grain storage capacity in public and household facilities Agriculture & Agricultural area under improved (resilient) management practices Rural Agricultural area provided with irrigation and drainage services Development Number of farmers adopting improved (resilient) production practices Number of farmers that have received food vouchers Amount of improved rice seeds distributed Increase in number of households whose grain needs can be met Institutional capacity for disaster risk management enhanced Reduction in time to commit funds from the contingency emergency response Increase in satisfaction of users of risk data and information system/services Increase in population covered by early warning systems Capacity of emergency management response personnel enhanced People with access to emergency shelter Disaster Risk Emergency shelters constructed/strengthened Management Length of upgraded embankment Number of early warning systems installed Number of weather and hydromet monitoring and forecasting stations installed Number of emergency centers equipped with search and rescue equipment Number of multi-agency exercise drills completed Number of contingency funding mechanism established Number of households receiving improved climate/disaster flood protection 18 A review of 96 resilience-building operations (70 IPFs and 26 DPOs) at the WB produced about 400 different resilience-relevant indicators. See Appendix 1 and 2 for additional detail. 29 ReM&E December 2017 Area of land under sustainable land and water management practices Area protected against recurrent flooding through drainage works Improved drainage conditions of roads Households’ strengthened awareness and ownership of adaptation and climate change risk reduction processes/measures Increased coverage of hazard forecast and warning messages to population at risk Number of cyclone affected population benefitted due to rehabilitation and construction of protective infrastructure (e.g. embankments, disaster shelters) Number of electric consumer accounts covered by underground cable network Area with electrical cabling transferred underground Energy Power supply restored to pre-flood levels through additional domestic generation/imports People with increased (non-)/monetary benefits from forests Tree canopy in areas under improved protection and management Environment & People with access to restored and improved beach front Natural Resources Forest area under improved protection and management Area restored or re/afforested Number of health centers with reduced vulnerability to hazards Health Number of health facilities constructed, renovated, and/or equipped Number of vulnerable schools protected from hazards Households with restored/reconstructed hazard resilient housing Housing & People served by restored and improved public buildings Buildings Number of hazard resilient houses restored/constructed Revised (resilient) building code applied Capacity to implement community-based DRM plans Time lapse between drought warning and identifying people in need Social Protection Households benefitting from Safety Nets programs and Development Community-based disaster risk management plans developed Number of people protected by enhanced priority infrastructure built by the project Decrease in people at risk of transport interruptions Decrease in days of interrupted traffic due to hazards Households with access to restored and/or upgraded roads (or bridges) Transport & Roads Length of roads constructed or upgraded to resiliency standards Length of road with improved slope protection Climate resilient road design guidelines prepared Amount of rural population with access to an all-weather road Households with uninterrupted water service during hazard events Potential retention of peak flow runoff from upstream tributaries People benefiting from improved flood management Water Resource People benefiting from restored flood protection infrastructure Management Area of watershed protected Improved community water points constructed or rehabilitated Amount of water storage capacity enhanced due to infrastructure investments 30 ReM&E December 2017 Length of improved flood drainage canals Communities with watershed development plans River morphology study completed Climate change adaptation mainstreamed in rural water supply sector’s development frameworks, policies and investment tools at the national level Number of frameworks, policies and investment tools with CCA/DRM developed (in the rural water supply sector) at the national level Number of reforms in climate/disaster resilient forest policy, legislation or other regulations supported Evidence of strengthened government capacity and coordination mechanism to Governance mainstream climate resilience Improved capacity and preparedness for disaster risk management in the government and communities Improved institutional capacity for flood risk management Number of institutions (national/sub-national) with functioning DRM arrangements Number of ministries with annual work plans and budget that integrate CC DRM measures Cross-cutting indicators can be relevant for a variety of projects if the measurement methodologies are flexible enough to be tailored to different project contexts and sectors. In general, cross-cutting indicators best capture higher-level outcomes that exist across sectors (e.g., increased coping capacity or reduced damages), while sector-specific indicators best capture outputs which are often very sector-specific, as shown in Table 2, above. Cross-cutting indicators are theoretically advantageous for their ability to be compared and aggregated across operations (provided they are calculated using a standard methodology). Their potential universality makes them of great interest for corporate reporting. However, the cost to produce such indicators has proven expensive, time consuming, and often not well-reflective of the actual situation on the ground. Thus, their universality diminishes their ultimate value because in most cases the specific project context cannot be credibly captured. Nevertheless, examples of potential (quantitative) cross-cutting indicator concepts for resilience-building include: • Number of people/households benefitting from resilience-building operations: This indicator is output-oriented—it counts the number of beneficiaries (the people or groups who directly derive benefits) from the resilience-building project activities without assessing the degree to which resilience was built. All World Bank projects are encouraged to measure the number of project beneficiaries. This indicator can also be used to monitor beneficiary sub-groups by separately accounting for those benefitting from a particular (resilience-building) activity (e.g., restoring housing). Although straightforward conceptually, counting direct project beneficiaries is often difficult—especially for infrastructure or ecosystem-based interventions (e.g., households benefitting from an upgraded road or improved watershed management). Furthermore, varying degrees of benefits between project components or across different operations make it difficult to compare and aggregate numbers of beneficiaries. For example, it is hard to compare and aggregate the number of beneficiaries of a new climate information system with those receiving restored and upgraded housing. Detailed measurement methodologies for this indicator concept have been developed and tested by the PPCR, DFID’s International Climate Fund (ICF) and the Adaptation Fund (CIF, 2016; DFID, 2016a). Examples of current applications of this indicator concept include: 31 ReM&E December 2017 o Number of direct project beneficiaries from projects with resilience-building objectives; o Number of people who received support for restoration of livelihoods; o Number of households supported to adapt to the effects of climate change; o Number of people supported to cope with the effects of climate change.19 • People/households with increased resilience/protection: This indicator is outcome-oriented, and goes beyond counting beneficiaries. It assesses who (likely) has experienced increased resilience as a result of the project. As actual changes in resilience are difficult to observe, existing measures related to this indicator tend to be based proxy measures of resilience-related attributes. Proxy data demonstrating personal/household improvements in access to services, social safety nets, income, assets, livelihood diversification, etc. over time can be interpreted as evidence for increased resilience. “Increased protection levels from climate and disaster risks” (e.g., through the provision of protective infrastructure) is a specific proxy example. This indicator concept of increased resilience/protection can be difficult to implement however—it is often challenging to clearly define whether improvement in a particular attribute does effectively increase resilience. Further challenges arise from the inability to differentiate between small changes for single attributes and bigger changes for a collection of attributes. Detailed measurement methodologies for this indicator concept have been developed and tested by the ICF.20 Examples of current applications of this indicator concept include: o Number of households who adopt less weather-sensitive livelihoods; o Number of people with increased flood protection; o Number of people whose resilience has been improved.21 • Area/people/households protected from hazards: This indicator is outcome-oriented as well. It measures the area, people, or households, protected from hazards through different risk mitigation measures. A variety of risk mitigation metrics can be considered including: hazard monitoring installation (e.g., early warning system), infrastructure improvement and upgrading (e.g., seawalls), and ecosystem restoration and protection (e.g., mangroves). Defining the level of risk mitigation needed to sufficiently provide protection (e.g., increasing dyke height by 10cm or 1m), is difficult—especially in the context of uncertain and changing climate hazards. Examples of current applications of this indicator concept include: o Number of people protected by enhanced priority infrastructure; o Area protected from coastal flooding; o Area protected by improved river flood protection measures. • Avoided impacts: This indicator measures the final outcomes of an operation in terms of impacts averted by resilience-building activities. Varying types of avoided impacts may be counted, and also aggregated, if expressed in common metrics. Impact types may include asset and infrastructure damages, income and consumption losses, and disability adjusted life years (DALYs). Though other impacts—such as loss of ecosystem services—may also be relevant to the operation, 19 For additional guidance refer to CIF (2016) and DFID (2016a) in the Section 4 “Further Reading” box. 20 For additional guidance refer to DFID (2016b) in the Section 4 “Further Reading” box. 21 For additional guidance refer to DFID (2016b) in the Section 4 “Further Reading” box. 32 ReM&E December 2017 they are often even more difficult to quantify in common metrics. The “avoided impacts” indicator requires not only detailed household socio-economic data from the project catchment area, but a defined counterfactual or business-as-usual scenario (i.e., what would have happened in the absence of the operation) which can be challenging. Detailed measurement methodologies for this indicator concept have been developed by GIZ.22 • Reduced welfare impacts of asset losses: This indicator is related to “avoided impacts,” but it specifically measures a system’s capacity to minimize the impact of asset losses on welfare by differentiating income and consumption losses of different groups (e.g., differentiating between poor and non-poor individuals). It assumes that the asset to welfare losses ratio represents the degree to which a system can absorb, accommodate, and recover from asset losses, and that this in turn indicates socio-economic resilience (Hallegatte et al., 2016). This model-based indicator estimates how different categories of interventions may reduce the ratio between expected asset and welfare losses. The World Bank currently applies a detailed measurement methodology developed for flood risks, but it does so at the national-level (not the project/operation level).23 Depending on data availability, the methodology may be extended to other hazard contexts and applied at other geographic scales (e.g., province-level). Recommendations for Task Team Guidance ➢ Select indicators that capture resilience concepts relevant to the operation and the results framework. There is no one-size-fits all or standardized set of indicators. Task teams should select indicators that appropriately reflect the operation’s geographic, socioeconomic and sectoral context. In some instances, relevant indicators may resemble traditional World Bank metrics, while others may adopt more innovative approaches. ➢ Minimize the number of indicators to the greatest extent possible while keeping measurements comprehensive enough to capture different dimensions and results-levels. Measuring and tracking too many indicators detracts from the strength of the most important aspects, and can overwhelm client capacity. Yet, the number and mix of indicators must sufficiently represent the operation’s range of resilience-building activities and (anticipated) results. ➢ Apply sector-specific indicators and adopt cross-cutting indicators where measurable, meaningful, and practicable. Sector-specific indicators should be used by operations to properly measure the results of sector-specific activities. Cross-cutting indicators might also be adopted if data, technical, and financial capacities allow, and if the indicators provide meaningful information on resilience-building (e.g., while counting the population in a state served by a climate information center is not likely to be meaningful, counting the number people living in upgraded houses would be). ➢ Select technical indicators that match measurement needs and capacities. Output indicators are important for tracking the progress of resilience-building activities. Outcome indicators are critical for understanding the pathway toward resilience-building impacts. When outcomes 22 For additional guidance refer to GIZ (2013) in the Section 4 “Further Reading” box. 23 For additional guidance refer to Hallegatte et al. (2016) in the Section 4 “Further Reading” box. 33 ReM&E December 2017 cannot be directly measured, they can be substituted with meaningful resilience-related proxy indicators. Quantitative indicators are generally easier to interpret than qualitative indicators, yet the latter can contribute to a fuller picture of and capture more accurate progress towards resilience-building. While many indicators are based on observed data, others can be constructed from hazard- and risk-modelling. ➢ Develop and/or adopt clear definitions and measurement methodologies. Having a clear definition of the specific terms, all potential components, and how data is to be collected, analyzed, and interpreted is needed for each indicator and should be indicated clearly in the results framework. Although this recommendation is relevant for any indicator, it is particularly important with respect to resilience-related indicators that are often new to task teams and project implementation units. Good Practices ✓ Clearly defining indicators and providing guidance on measurement approach(es) ✓ Balancing indicator ambition with practicality Excerpt from the Case Study on the Mekong Delta Integrated Climate Resilience and Sustainable Livelihoods Project The Mekong Delta project design team worked on being ambitious, yet realistic, to avoid overpromising on resilience-building objectives and resilience-relevant indicators, recognizing that this is an easy mistake to make during the project planning stage. The team considered what the project could practically achieve in its six-year period, bearing in mind resource constraints such as in-country capacity. The team struck a balance between keeping the project objectives simple and achievable, while still clearly aiming for transformational turning points to build resilience. Additionally, the project design team took the time to carefully document each indicator’s definition and measurement approach, thus creating more transparent and robust indicators. This will also help to instill institutional memory, which will likely be useful if/when project staff turnover during this six-year project, and if/when replicating and adapting the use of similar resilience- building indicators for other projects. 34 ReM&E December 2017 Further Reading ✓ Adaptation Fund. 2014. Methodologies for Reporting Adaptation Fund Core Impact Indicators. Adaptation Fund, Washington, DC, USA. ✓ Bours, D, C. McGinn, and P. Pringle. 2014. Selecting Indicators for Climate Change Adaptation Programming. Guidance for M&E of Climate Change Interventions Series. Guidance Note 2. SEA Change, UK CIP. ✓ CIF (Climate Investment Fund). 2016. PPCR Monitoring and Reporting Toolkit. Revised Version. CIF, Washington, DC, USA. ✓ DFID (Department of International Development). 2016a. ICF KPI 1: Number of people supported by DFID Programmes to Cope with the Effects of Climate Change. BRACED Methodology, DFID, London, UK. ✓ DFID (Department of International Development). 2016b. ICF KPI 4: Number of People whose Resilience has been Improved as a Result of Project Support. BRACED Methodology, DFID, London, UK. ✓ GIZ (Deutsche Gesellschaft für Internationale Zusammenarbeit). 2013. Adaptation Made to Measure: A Guidebook to the Design and Results-Based Monitoring of Climate Change Adaptation Projects. GIZ, Bonn, Germany. ✓ GIZ (Deutsche Gesellschaft für Internationale Zusammenarbeit). 2013. Saved Health, Saved Wealth: An Approach to Quantifying the Benefits of Climate Change Adaptation, Practical Application in Coastal Protection Projects in Vietnam. GIZ, Bonn, Germany. ✓ GIZ (Deutsche Gesellschaft für Internationale Zusammenarbeit). 2014. Repository of Adaptation Indicators, Real case examples from national Monitoring and Evaluation Systems. GIZ, Bonn, Germany. ✓ Hallegatte, S., Bangalore, M., Vogt-Schilb, A. 2016. Assessing Socioeconomic Resilience to Floods in 90 Countries. Policy Research working Paper No 7663. World Bank, Washington, D.C., USA. ✓ PROFOR (The Program on Forests). 2014. Understanding Long-Term Impacts in the Forest Sector: Predictive Proxy Indicators. PROFOR Working Paper. World Bank, Washington, DC, USA. ✓ World Bank. 2013. Results Frameworks and M&E: A Guidance Note. Prepared by OPCS, World Bank, Washington, DC, USA. 35 ReM&E December 2017 Section 5. Monitoring and Reporting Resilience-Related Results Main Messages Systematic tracking of resilience-related results indicators (as well as other pertinent contextual, management and/or fiduciary data) will occur at different intervals depending on an operation’s context and application. Monitoring commonly includes daily, weekly, monthly, or continuous intervals as well as longer-term monitoring commitments (i.e., every two to five years) to observe measures that are slow to change. The frequency for reporting on monitored data/indicators also includes a range of intervals; however, reporting for most World Bank projects occurs on a semi- annual basis. This section examines the role existing methodologies and approaches play in monitoring and reporting in resilience-building operations. Clearly defining the relationship between monitoring and reporting elements and resilience can determine an operation’s success in strengthening the human, asset, and/or system resilience. Developing a monitoring and reporting plan ahead of the implementation phase (i.e., during the concept or appraisal stages) not only helps the task team set specific, targeted monitoring goals, but also helps the task team determine what aspects of resilience-building to focus on in the project. Monitoring Baselines Monitoring starts well before the implementation an operation with the establishment of a robust baseline that describes the situation prior to a development intervention to provide a standard against which progress can be assessed/comparisons made. Setting clear baselines orients task teams towards setting realistic targets. An operation’s results framework, indicators, and risk/vulnerability context determine the most relevant suite of baseline variables. While monitoring of resilience-building results presents a set of challenges—such as data availability, changing baseline conditions, and overall complex contexts—one can adopt practical strategies to establish baselines: • Reconstructing baselines: retroactively creating conditions unaccounted for in ex ante data or existing documentation is key. The process might include mapping techniques, such as Geographic Information Systems (GIS) that draw on historical data, as well as recall techniques that entail surveying participants about their previous socio-economic situation and access to services. • Estimating or projecting baselines: climate projections can inform baselines for resilience- building interventions. Depending on project conditions, relevant projection models may be either stochastic (randomized; parameter values and initial conditions lead to a suite of different outputs) or deterministic (the relationship between variable properties and different states observed are well-known; parameter values and initial conditions lead to a determined output). 36 ReM&E December 2017 • Normalization: when using quantitative cross-cutting indicators that require a counterfactual (as described in Section 4), the contribution of the intervention on project outcomes must be standardized against average or usual trends to make data measured on different scales more comparable. To standardize findings, outcomes are commonly assessed as deviations from impacts expected under “usual” conditions (defined by data such as market demand, shocks, or infrastructure use, and other sector related aspects). • Contextualized baselines: changing contextual factors can undermine an operation’s ability to achieve its objectives—resilience-building projects often take place in evolving, complex environments. To avoid difficulty, it is important that task teams monitor contextual indicators and adjust operation implementation accordingly. “Contextualizing” an operation in this manner requires adjusting the evaluation logic and the baseline for outcome monitoring and assessment. One approach to such “contextualization” is to identify a baseline for the context as well as for the intervention, and then to monitor both contextual and project indicators. For example, measuring relevant environmental or physical variables associated with the project’s location might not be included in the results framework, but these data could prove useful in the evaluation phase to understand the contribution of the project to the resilience building. Related to contextualized baselines, common M&E practice is to assume the base time-period as when an intervention is introduced; however, this is not always an adequate reflection of the realities of climate change. Task teams might consider addressing these challenges in defining “baseline conditions” in a changing climate (particularly with respect to physical and environmental data) by either deriving the baseline from a specific model or ensemble estimate across different models for the relevant time period of the investment, or by defining an anticipated operating range with historical data (such as seasonal rainfall, water-table levels) against which progress can be measured. Setting Targets When setting targets for resilience, teams should consider that an operation’s ultimate outcomes may be complex and achievable only in the long-term. This again relates to the importance of having a well- defined and realistic logic model/results framework. Sequential/iterative targeting is a particularly relevant strategy for managing the complexity of resilience M&E Establishing targets as short-term objectives for achieving an outcome. TTLs should set interim targets or milestones that relate to expected performance over short intervals. This is particularly relevant to resilience, as it involves regular appraisal of goals in a changing context. Focusing on the process and intermediate targets helps identify unanticipated problems, thus allowing for corrective action is possible while the project is ongoing. Following this logic, it is critical for teams implementing resilience-building operations to consider Mid-term Reviews (MTRs) and subsequent project restructurings as necessary and planned elements of the project to be expected, rather than avoided by the task team (i.e., treating mid- course project adjustments as “opt-outs” rather than “opt ins” in cases of resilience-building operations). Methods of Data Collection and Analysis Providing evidence of the actual contributions of projects to resilience-building objectives is of central concern to resilience M&E. Information management for resilience monitoring and reporting therefore 37 ReM&E December 2017 entails the collection, analysis, and dissemination of information to support this. Good information management—across collection, analysis, and dissemination stages—ensures that data is useful and accessible to stakeholders, and may involve awareness-raising or capacity-building among stakeholders to facilitate their use of the information for capturing characterizing results. Task teams can collect the relevant data via traditional means such as local authority statistics, data polls from international organizations, field surveys, and primary and/or secondary studies of project documents. Participatory methods, such as interviews or focus groups, provide information on the experience of the local population, and are an important consideration for resilience-building M&E. The collected data can serve several purposes; applied to inform baselines, revisited for periodic reporting (as will be discussed in the following subsection), and utilized to reconstruct historical conditions that have not been collected in the relevant region in the past and/or are unavailable. Data can be mined from technological sources, including early warning systems, Information and Communication Technology (ICT), big data pools, and satellite imagery, amongst other innovative data collection approaches. These innovative methods create value as they can provide high-frequency data that better depict social, behavioral, natural, and systems trends (e.g., usage of technology per geographic area or land use change over time) rather than one-off data collected at periodic points in time. To deal with specific challenges of monitoring resilience-building in operations, task teams might consider using a mix of monitoring-related techniques/approaches/methodologies to properly collect and analyze data in the field. A sample of these include:24 • Counterfactual analysis: Comparison of project results with corresponding development aspects in a control region or group where no equivalent resilience-building measures were conducted. • Dynamic baseline: Adjustment of baseline data collected at the start of the project to account for substantial changes in conditions (e.g., climate factors and their consequences) to maintain benchmark relevance. • Opportunistic results measurement: Comparison of the impact of extreme events occurring during the project term with the impact of similar events at the start of the project, or simultaneously comparing the impact of extreme events in the project region and other similar regions lacking the resilience-building measures. • Universal/cross-cutting metrics: Quantification of the results of resilience-building projects in a cross-sectoral and cross-project indicator such as the number of saved lives or the value of protected assets. • Triangulation: Concurrent generation of information about the same topic from several information sources and different methods. • Social sciences methods: Utilization of surveys, interviews (structured, semi-structured), and focus groups. 24 The approaches listed do not necessarily pertain solely to “monitoring” aspects of M&E, as they often relate to “indicator” and “evaluation” aspects of M&E. Thus, the reader will notice some overlaps between this list and similar concepts that appear in Sections 4 and 6, respectively. 38 ReM&E December 2017 • Econometrics and statistics: Utilization of tools ranging from modeling, statistical analyses, stochastic baselines, deterministic baselines, and normalization. Predictive modeling, or use of a model to best predict the extent to which an outcome will occur under a given project, is an innovative approach that can benefit resilience M&E. • Participatory monitoring: Includes Most Significant Change (MSC) analysis, beneficiary monitoring, limiting factor analysis, outcome mapping, and recall techniques. • Experiment-related methods: Utilization of case studies, experimental design, quasi- experimental design, propensity score matching, phased pipeline, purposeful sampling, and regression analysis. • Big data and ICT for resilience M&E: Extraction of information through novel approaches, including data mining (i.e., the process of discovering and extracting new patterns in large data sets). Data analytics has developed new approaches to monitoring and reporting using predictive modeling that employs different approaches from the experimental (randomized control trial) methodologies generally used by development evaluators. Though they have potential, these approaches have not yet been widely applied in resilience M&E. The provided approaches are complementary—using them in conjunction with one another can improve the clarity validity of the results by piecing together an approach that combines the strengths of each, depending on the data collection and monitoring challenges a given project faces. When applying different methods, however, careful assessment and transparency regarding the underlying assumptions is required to ensure compatibility between such different sources of evidence. Note that some methods may require extra capacity to generate and interpret (e.g., complex computational modeling data). Successful monitoring depends on the selected sampling strategy, data collection methodology, and team capacity for data processing. Indicators should be tweaked to reflect the realities of the resilience-building context and the constraints of the project (e.g., if data are highly complex or the indicator the data support requires a large sample, efforts should be made to simplify the approach and/or reduce the sample size to make it more realistically monitorable). Reporting The main goal of reporting is to provide and publish comprehensive and regular information on project implementation progress. For resilience-building operations especially, it is important that monitoring systems have strong embedded reporting mechanisms, as reporting is key to promoting resilience M&E elements of continuous learning and stakeholder engagement. To establish accountability and transparency, and to appropriately make progress towards targets, reporting should be iteratively conducted over a project’s lifetime. Setting realistic reporting strategies (intervals, mechanisms, and dissemination) helps ensure the success of operation monitoring. To determine the frequency and intervals of observation needed, consider an operation’s: • Steering/revision needs: each project has specific information needs for project revisions and accountability check-points that occur on a monthly, quarterly, or annual basis. The frequency may change depending on the baseline data and targeting strategies chosen in the design phase. 39 ReM&E December 2017 • Velocity of change: the rate at which change and transformation occurs will vary between projects. For example, while rainfall or temperature need to be monitored daily, progress made towards increasing the diversity of water source supply can be monitored over a longer interval. • Resource availability: includes material, financial, and human considerations. For most World Bank projects, observation intervals of six months to a year sufficiently capture changes in output indicators. Recording the progress made towards outcome indicators may be done simultaneously, but it is highly likely that outcome information may only be available towards the end of, or after, the project cycle. Nevertheless, teams should aim to report more frequently than the end of the project for resilience-related outcomes, especially if the resources required are not too burdensome to do so, as it increases the likelihood of learning and capturing important (unexpected) results. Recommendations for Task Team Guidance ➢ Monitoring and reporting methods must be open, flexible, and adapted to changing contexts. During the concept and appraisal phase, task teams should build robust monitoring and reporting into the project structure and financing, including appropriate baseline and target strategies, data collection methodologies, and analysis approaches. ➢ Resilience-building operations necessitates an “opt-out” approach to MTRs and subsequent project restructurings Course corrections should be expected and embraced as a necessary element of project design, which will help to accommodate the complexity of resilience and the dynamism of the concepts/results that resilience-building projects are intending to demonstrate. For example, teams should build an MTR (with the anticipation of a restructuring to ensue) into the concept note of resilience-building projects, and then discuss with clients during the remainder of the project preparation process so that expectations are established early around the need for the project implementation to be as flexible as possible. ➢ Take the various data collection methods and sources into consideration when designing M&E approaches. Many resilience-building projects seek to support complex processes of behavioral change, which often require innovative, mixed–method approaches for capturing and analyzing new forms of data. ➢ Develop consistent reporting tools. This includes establishing information sharing processes and feedback mechanisms. 40 ReM&E December 2017 Further Reading ✓ Bours, D., C. McGinn, and P. Pringle. 2013. Monitoring and evaluation for climate change adaptation: A synthesis of tools, frameworks and approaches. ✓ CIF (Climate Investment Fund). 2016. PPCR Monitoring and Reporting Toolkit. Climate Investment Funds, Washington, DC, USA. ✓ Dinshaw, A., S. Fisher, H. McGray, N. Rai, and J. Schaar. 2014. Monitoring and Evaluation of Climate Change Adaptation: Methodological Approaches. OECD Environment Working Papers, No. 74, OECD Publishing. ✓ GIZ (Deutsche Gesellschaft für Internationale Zusammenarbeit). 2013. Adaptation made to measure: A guidebook to the design and results-based monitoring of climate change adaptation projects. GIZ. ✓ Naswa P., C. Traerup, C. Bouroncle, C. Medellín, P, Imbach, B. Louman, and J. Spensley. 2015: Good Practice in Designing and Implementing National Monitoring Systems for Adaptation to Climate Change. Climate Technology Centre & Network, Denmark. ✓ OECD (Organization for Economic Co-operation and Development). 2014a. Guidelines for Resilience Systems Analysis: How to analyse risk and build a roadmap to resilience. Paris: OECD. 41 ReM&E December 2017 Section 6. Evaluating Resilience-Related Results Main Messages25 Systematic and impartial project assessment occurs at discrete points in time to robustly identify links between the project intervention and resilience-related results. Evaluations often focus on processes and higher-level outcomes and impacts—results not necessarily directly identifiable just through project monitoring. However, evaluation inputs may include monitoring data alongside other information. Evaluations can be done at strategic points in time during project implementation, as well as after project completion, and can support accountability and/or learning objectives. For resilience-building operations, learning-focused evaluations are particularly important. There is no standardized approach to evaluation. Rather, each evaluation must be tailored to meet the needs of an operation’s stakeholders as well as specific contexts (geographic, socio-economic, institutional, sectoral, etc.). A handful of general considerations influence the success of any evaluation; resilience-building operations are also subject to several additional recommendations. The evaluation of resilience-building operations is especially important for building a body of knowledge on what does (not) work; the accumulated knowledge will in turn feed into the design and implementation of future resilience-building projects. When to do an evaluation Not all projects justify a very thorough evaluation. Therefore, when designing a project, a task teams should consider whether the project is: • Innovative and/or untested: evaluation is important for testing innovative/new approaches or interventions for which there is limited robust evidence of results and impact in the given context; • Strategic: evaluation is important for flagship initiatives, of which results can be used to inform key policy decisions; and, • Replicable: evaluation is important for pilot programs which are due to be scaled up or replicated in a different context. There are five steps to designing and implementing a resilience related evaluation, demarcated in each step below by core/general evaluation considerations and resilience-specific considerations, with the emphasis here in this note on the latter. 25 This section “Evaluating Resilience Related Results” represents a synthesis of the report; “Evaluation of Resilience-Building Operations: Operational Guidance Paper for Project Task Teams”, which has also been produced by the ReM&E project. Task teams are suggested to review this complete guidance paper for more detailed guidance on resilience-building evaluations. See: http://documents.worldbank.org/curated/en/669941506093754016/Evaluation-of-resilience-building-operations- operational-guidance-paper-for-project-task-teams. 42 ReM&E December 2017 Step 1. Establishing purpose, scope, feasibility, and audience • Core evaluation considerations: An evaluation addresses purpose, audience, scope, and feasibility by answering questions such as the following: o What is the primary purpose of the evaluation? o What are the key evaluation questions? o Who is the primary audience for the evaluation? o Is the evaluation feasible in theory and practice? • Resilience-specific considerations: The main criteria that define a sound resilience-building operation’s evaluation include should be based on the Organization for Economic Co-operation and Development (OECD) Development Assistance Committee’s (OECD-DAC) evaluation criteria;26 relevance, effectiveness, efficiency, impact, and sustainability, but must be tailored to the nature and context of resilience-building operations. While task teams should ideally aim to address all five criteria, evaluations meant to address a specific learning purpose or mandate may instead focus in-depth on, and develop specific evaluation questions around, one or two criteria. o Relevance: Extent to which the activity is suited to the priorities and policies of the target group, recipient, and donor. The relevance of the resilience-building project/program intervention might shift over time with changing climatic context and circumstances. Assessing risk, capacities, and vulnerabilities is key to evaluating relevance. If a risk and vulnerability analysis has already been carried out, its accuracy and use should be assessed. o Effectiveness: Extent to which an activity attains its objectives. Evaluation against this criterion should therefore be relatively straightforward, provided that measurable objectives and indicators have been stated and clearly defined at the outset. While effectiveness depends on resilience outcomes, it also depends on the resilience process, including capacity building, information exchange, policy formulation, and learning. o Efficiency: Ratio of outputs (qualitative and quantitative) to inputs. This generally requires comparing alternative approaches to achieving the same outputs, to assess the relative efficiency of the adopted process. Resilience-building projects/programs involve deciding on acceptable levels of risk and vulnerability (defined to some extent by communities, policy-makers, and funders in a collaborative way) as potential trade-offs with the resource investments needed to reduce these risk risks (and thus increase resilience). o Impact: Positive and negative changes—either direct or indirect, intended or unintended— produced by an operation. In the context of resilience-building projects, the criterion should be used to identify and evaluate the overall development impact as articulated in the theory of change/results framework. o Sustainability: Degree to which the benefits of an activity are likely to continue after donor funding has been withdrawn. Projects need to be environmentally, socially, and financially sustainable. Sustainable resilience-building operations are likely to include strong elements of partnership-building and stakeholder engagement. They are likely to focus on interventions that integrate among existing development processes and mechanisms, and that cut across key sectors (such as water management, agriculture, health, and education). 26 See OECD-DAC (1991). 43 ReM&E December 2017 In the context of resilience-building projects, four additional criteria should be considered alongside those emphasized by OECD-DAC: o Connectedness: Degree to which activities are carried out in a context that takes longer- term and interconnected problems into account. Projects should ensure that partners and sectors work together and adopt complementary, synergistic strategies to promoting resilience. Connectedness thus questions the extent to which different types of activities have been integrated and sequenced to address multidimensional challenges, and the extent to which the project worked with key actors in the context within which it operates, so as to maximize long-term impact and transformative change o Flexibility: Extent to which a project is adaptable enough to address/respond to the uncertain conditions of recurrent crises and unexpected events. o Legitimacy: Degree to which project beneficiaries are “active” agents. While active participation of project beneficiaries is not unique to resilience interventions, the actors within a system need to have sufficient capacity and resources at their disposal to achieve resilience in the face of a shock or stress. o Equity: Distribution of project benefits, costs, and risks amongst the population. Resilience-building projects aim to reduce vulnerability in the context of climate change shocks and stresses, specifically. Vulnerability heavily depends on socioeconomic factors, however. This not only implies that different groups may have to be uniquely targeted, but also that the results/impacts of any given project may vary across groups. Step 2. Setting up Evaluation decision-making, quality standards, and communication requirements Formal evaluation plans include clear guidelines for decision-making, ethics, and communication from the operation’s outset. Resilience-building is often assessed over a span of several years, making multiple interactions with beneficiaries and stakeholders a must for evaluation design. • Core evaluation considerations: Core evaluation considerations entail answering questions related to decision-making, management, quality standards, and communications: o Who will make decisions and ensure the evaluation is effectively managed and quality assured? o How to define and ensure the quality standards of the evaluation? o How to define and address potential ethical concerns? o How to communicate the findings of the evaluation? • Resilience-specific considerations: Learning is an important aspect of the evaluation of resilience- building projects. Learning can be facilitated by: o Involving key stakeholders in the evaluation process and sharing learning from the beginning: In practice, this means ensuring that key stakeholders who contribute data or evidence to the evaluation are involved in shaping the findings, conclusions, and recommendations. It also means that, wherever possible, the evidence generated through the evaluation is made directly relevant to contributors. o Establishing internal learning mechanisms: Planning for internal knowledge management and sharing learning processes as a key activity and evaluation outcome means 44 ReM&E December 2017 considering the evaluation related knowledge demands on GPs from the outset. Where possible, evaluations should produce final knowledge products that are respectively tailored to the respective GP as well as other relevant GPs and World Bank units. Step 3. Selecting the appropriate evaluation approach and design Evaluation designs should be selected to meet the core attributes and main objectives of the resilience- building project. Because resilience-building is a complex process with fundamental uncertainties about the causal relationships in the change process, attribution is unlikely to be clear, simple, or linear— evaluation designs should therefore take a “complex systems” perspective that situates the project in question within its wider change context. • Core evaluation considerations: Table 3 offers an overview of the range of appropriate evaluation designs and a headline analysis of their perceived advantages and disadvantages. • Resilience-specific considerations: The complexity of resilience-building is most clear when projects take an integrated approach to resilience-building by delivering different “packages” of interventions to various beneficiary or stakeholder groups, and/or when projects aim to improve policies, planning, and decision-making processes. In these cases, results typically depend on complex interacting factors that are often outside of the scope and control of the project. Resilience-building projects are unlikely to generate robust evidence at outcome and impact levels if they are evaluated too soon after the implementation phase ends. Teams are often challenged by the mismatch between the relatively short timescale associated with resilience- building projects and the lengthy timescales over which evidence of improvements in resilience at the longer-term outcome and impact levels can be witnessed. Because of the “latent” nature of resilience, it is only possible to conclusively assert a project’s contribution to resilience if beneficiaries are observed to have strengthened resilience in the occurrence of a climate or disaster event. This is an unreliable method to evaluate the degree to which the project has met its expected resilience-strengthening outcomes and impact. Again, as noted in Section 3, a well-articulated theory of change and associated results framework can help task teams devise innovative assessment approaches that do not necessarily depend on the occurrence of a climate or disaster event. 45 ReM&E December 2017 Table 3: Overview of Evaluation Designs Design Description Basis for causal inference Advantages Disadvantages Experimental Designs Compares Counterfactuals; the co-presence of Causal relationships between Limited external validity intervention with cause and effects variables can be established (generalizability) due to non-intervention High internal validity the controlled area experimental environment Less suitable for more complex, Uses controls that long-term interventions, where Ethical concerns are randomly many factors seek to produce Resource intensive assigned change Quasi- experimental Compares Counterfactuals; the co-presence of Enables experimentation when Does not control for Designs intervention with cause and effects random assignment is not possible extraneous variables that non-intervention Avoids ethical issues caused by may influence findings area random assignment Uses controls or comparison groups that are not randomly assigned Qualitative designs Includes theory- Identification/confirmation of causal Can often be undertaken in Does not provide a based, processes or “chains” circumstances where other quantitative measure of participatory and Supporting factors and mechanisms at approaches (e.g., experimental the size of attribution case study-based work in context designs) cannot be used approaches. Allows conclusions to be drawn on Validation by participants that their actions and experienced effects are the relative cause-effect elements ‘caused’ by program of an intervention. Comparison across and within cases of Enables evaluators to arrive at combinations of causal factors findings on why interventions are working or are not working in Analytic generalization particular contexts based on theory 46 ReM&E December 2017 Step 4. Identifying data collection tools and methods: Sound evaluation design requires task teams address three questions: • What set of quantitative tools and methods are most appropriate and feasible? • What set of qualitative tools and methods are most appropriate and feasible? • What sampling strategies are most appropriate and feasible? Note that Sections 4 and 5 on “indicators” and “monitoring and reporting”, respectively, provide further details on data collection, particularly as it relates to resilience-building. Step 5. Managing the implementation of the evaluation • Core evaluation considerations: Tasks under this step include: o Task 1: Commission the evaluation implementation–this entails the preparation of a Terms of Reference (ToR) for external evaluators, emphasizing critical skills of an effective evaluator (ability to listen, negotiate, analyze situations, and assist in developing a design with the task team). o Task 2: Manage the implementation of the evaluation–this means developing a clear work plan with realistic timeframes and clear deliverables and milestones. o Task 3: Present, share, and communicate the findings–the learning component of resilience-building benefits from a robust sharing of findings and clear communication of both successes and shortcomings of projects. • Resilience-specific considerations: Due to the inevitable but unpredictable nature of climate change and disaster events, resilience-related recommendations made today may prove to be off the mark later on. Task teams must tie recommendations to an operation’s intentions and theory of change. The team should also include project stakeholders while drafting evaluation recommendations to ensure engagement and create continuity in resilience-building practice and experience. It may also be necessary for teams to clarify the spatial and temporal validity of recommendations as well as any assumptions made during framework development. Recommendations for Task Team Guidance ➢ Decide whether it is sound to embed evaluation in your project, preferably in the concept phase. Evaluations are most robust and useful when they are built into the project from the outset, and when they can build on and utilize the project’s/country’s wider M&E efforts. ➢ Consider multiple evaluation approaches/mixed methods. Different design options are not mutually exclusive. In fact, recently there has been increased interest in evaluation designs and approaches that combine quantitative and qualitative methods to support both robust causal inference and explanation, and are likely better positioned when used in conjunction with one another to capture the complexities of resilience-building results. 47 ReM&E December 2017 ➢ Include learning and explanatory questions about how the project worked in your evaluation. The learning component is a pillar in resilience-building M&E, as it establishes a possibility to link with future projects, build on success, and learn from failures. ➢ Be realistic and pragmatic about results. Evidence of resilience strengthening results can take time. Good Practices ✓ Use of impact evaluation designs to better understand the mechanisms that drive resilience (impact) ✓ Choosing clear objectives for the impact evaluation, including establishing an evidence base where evidence is lacking ✓ Utilizing the impact evaluation designs to inform and help adapt ongoing implementation of the program Excerpt from the Case Study on the Mozambique PROIRRI – Sustainable Irrigation Development Project The World Bank PROIRRI Sustainable Irrigation Development Project, approved in 2011, seeks to increase agricultural production and raise farm productivity in new/ improved irrigation schemes in select provinces in Mozambique. Based on extensive consultations with M&E experts during the project design phase, a rigorous impact evaluation was emphasized for this project. This project was well-suited for this kind of impact evaluation because there was a lack of existing evidence on small-scale irrigation development. The impact evaluation, with a budget of approximately US$1.35 million, is designed to answer central operational questions, such as when to scale-up extension service delivery mechanisms, how to measure farmers’ knowledge and adoption of improved technology, and how to assess the relative impacts of simultaneously occurring interventions. PROIRRI’s impact evaluation uses randomized control trials to establish carefully identified control and treatment groups to generate statistically rigorous information on the impact of the program. A pilot stage of the impact evaluation has been completed, and based on preliminary findings, a second piloting stage began in November 2016. Having been informed by the impact evaluation, the PROIRRI program is planning to scale-up its measurement to additional schemes, and to build further capacity in country, including offering financial literacy and matching grants. 48 ReM&E December 2017 Further Reading ✓ Frankenberger, T., J. Kurtz, and B. Sagara. 2015. Mercy Corps’ Approach to Measuring Resilience. Resilience Discussion Paper No.2. Mercy Corps. ✓ GIZ (Gesellschaft fur Technische Zusammenarbeit). 2016. Impact Evaluation Guidebook for Climate Change Adaptation Projects. GIZ. ✓ Organization for Economic Co-operation and Development - Development Assistance Committee’s (OECD-DAC). 1991. Principles for The Evaluation of Development Assistance. ✓ World Bank. 2017. Evaluation of Resilience-Building Operations: Operational Guidance Paper for Project Task Teams. World Bank, Washington, DC, USA. ✓ World Bank 2006. Impact Evaluation and the Project Cycle. Doing Impact Evaluation Series, No 1. World Bank, Washington, DC, USA. 49 ReM&E December 2017 Section 7. Moving forward Resilience M&E is a relatively new focus area for M&E, and as such this paper consolidates a set of emerging good practices. The ReM&E team considers this note to be the first step in advancing guidance on resilience M&E. Several resilience-focused projects, particularly those that have embedded strong M&E design, are in the early stages of planning or implementation. Their evolution will offer additional lessons over the next several years. It is important to update this note on a regular basis, so the institution can continue learning by doing, and iteratively improve upon the collective understanding of resilience M&E. The guidance note should be viewed in this light—as good emerging practices to continue to observe and learn from—just as understanding of resilience will continue to evolve over time. 50 ReM&E December 2017 References Adaptation Fund. 2014. Methodologies for Reporting Adaptation Fund Core Impact Indicators. Adaptation Fund, Washington, DC, USA. Retrieved from: https://www.adaptation-fund.org/wp- content/uploads/2016/04/AF-Core-Indicator-Methodologies.pdf. Bours D., C. McGinn, and P. Pringle. 2014. Selecting Indicators for Climate Change Adaptation Programming. Guidance for M&E of Climate Change Interventions Series. Guidance Note 2. SEA Change, UKCIP. Retrieved from: http://www.seachangecop.org/sites/default/files/documents/2014%2001%20SEA%20Change%20UKCIP %20GN2%20Selecting%20indicators%20for%20CCA_0.pdf. Biggs, R., M. Schlüter, and M. Schoon (eds). 2015. Principles for building resilience: Sustaining ecosystem services in social-ecological systems. Cambridge University Press, United Kingdom. Retrieved from: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/186874/defining- disaster-resilience-approach-paper.pdf. Cervigni, Raffaello; Wang, Tao. 2017. Accelerating climate-resilient and low-carbon development: second progress report on the implementation of the Africa climate business plan. World Bank, Washington, DC, USA. Retrieved from: http://documents.worldbank.org/curated/en/247501510166915125/Accelerating- climate-resilient-and-low-carbon-development-second-progress-report-on-the-implementation-of-the- Africa-climate-business-plan-overview. CIF (Climate Investment Fund). 2016. PPCR Monitoring and Reporting Toolkit. Revised Version. CIF, Washington, DC, USA. Retrieved from: https://www.climateinvestmentfunds.org/sites/default/files/knowledge- documents/ppcr_monitoring_and_reporting_toolkit_march_2016_revised.pdf. DFID (Department for International Development). 2011. Defining Disaster Resilience: A DFID Approach Paper. Department for International Development, London, UK. Retrieved from: https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/186874/defining- disaster-resilience-approach-paper.pdf. Folke, C. 2006. Resilience: The emergence of a perspective for social-ecological systems analyses. Global Environmental Change 16: 253-267. Retrieved from: http://www.sciencedirect.com/science/article/pii/S0959378006000379?via%3Dihub. GIZ (Deutsche Gesellschaft für Internationale Zusammenarbeit). 2013. Saved Health, Saved Wealth: An Approach to Quantifying the Benefits of Climate Change Adaptation, Practical Application in Coastal Protection Projects in Vietnam. GIZ, Bonn, Germany. Retrieved from: https://gc21.giz.de/ibt/var/app/wp342deP/1443/wp-content/uploads/filebase/me/project-level- me/giz_2013_Saved_health_saved_wealth_- _an_approach_to_quantifying_the_benefits_of_climate_change_adaptation.pdf. Hallegatte, S., M. Bangalore, and A. Vogt-Schilb. 2016. Assessing Socioeconomic Resilience to Floods in 90 Countries. Policy Research Working Paper No 7663. World Bank, Washington, D.C., USA. Retrieved from: http://documents.worldbank.org/curated/en/387821467309551281/Assessing-socioeconomic- resilience-to-floods-in-90-countries. 51 ReM&E December 2017 IEG (Independent Evaluation Group). 2012. Designing a Results Framework for Achieving Results: A How- to Guide. Independent Evaluation Group, World Bank Group, Washington, DC, USA. Retrieved from: http://siteresources.worldbank.org/EXTEVACAPDEV/Resources/designing_results_framework.pdf. IEG (Independent Evaluation Group). 2013. Adapting to Climate Change: Assessing World Bank Group Experience. World Bank, Washington, DC, USA. Retrieved from: http://ieg.worldbankgroup.org/Data/reports/cc3_full_eval_0.pdf. IPCC (International Panel on Climate Change). 2014. Climate Change 2014: Synthesis Report. Contribution of Working Groups I, II and III to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change. [Core Writing Team, R.K. Pachauri and L.A. Meyer (eds.)]. IPCC, Geneva, Switzerland, 151 pp. Iskandarani, Maria and Wilkes, Andreas. 2016. UNIQUE forestry and land use GmbH. Working Paper: A Conceptual Approach For Monitoring And Measuring Triple Wins In Climate-Smart Agriculture Projects— The Example Of Kenya. Prepared for World Bank Group. Washington, DC, USA. Organization for Economic Co-operation and Development - Development Assistance Committee’s (OECD-DAC). 1991. Principles for The Evaluation of Development Assistance. Retrieved from: http://www.oecd.org/dac/evaluation/50584880.pdf. OECD (Organization for Economic Co-operation and Development). 2014a. Guidelines for Resilience Systems Analysis: How to Analyze Risk and Build a Roadmap to Resilience. OECD, Paris, France. Retrieved from: http://www.oecd.org/dac/Resilience%20Systems%20Analysis%20FINAL.pdf. OECD (Organization for Economic Co-operation and Development). 2014b. Monitoring and Evaluation of Climate Change Adaptation. OECD, Paris, France. Retrieved from: http://www.oecd- ilibrary.org/docserver/download/5jxrclr0ntjd.pdf?expires=1474493657&id=id&accname=guest&checksu m=5B5FA66FCA76AAD5722DC15F59AED0F8. PROFOR (The Program on Forests). 2014. Understanding Long-Term Impacts in the Forest Sector: Predictive Proxy Indicators. PROFOR Working Paper. World Bank, Washington, DC, USA, Retrieved from: http://documents.worldbank.org/curated/en/335601468197091736/Understanding-long-term-impacts- in-the-forest-sector-predictive-proxy-indicators. Rockefeller Foundation. 2015. Resilience webpage. New York: Rockefeller Foundation. Retrieved from: https://www.rockefellerfoundation.org/our-work/topics/resilience/. UNISDR (United Nations Office for Disaster Risk Reduction). 2009. UNISDR Terminology on Disaster Risk Reduction. United Nations Office for Disaster Risk Reduction, Geneva, Switzerland. Retrieved from: http://www.unisdr.org/we/inform/terminology. USAID (United States Agency for International Development). 2012. Building Resilience to Recurrent Crisis. USAID Policy and Program Guidance. United States Agency for International Development, Washington, DC, USA. Retrieved from: https://www.usaid.gov/sites/default/files/documents/1870/USAIDResiliencePolicyGuidanceDocument.p df. 52 ReM&E December 2017 Villanueva, P.S. 2011. Learning to ADAPT: Monitoring and Evaluation Approaches in Climate Change Adaptation and Disaster Risk Reduction—Challenges, Gaps and Ways Forward. SCR Discussion Paper 9. Brighton, UK: Institute of Development Studies. Retrieved from: http://community.eldis.org/.5a093c0d. Williams, A. 2016. Options for Resilience Results Monitoring and Evaluation for Resilience-Building Operations. World Bank Group & GFDRR Scoping Paper. Retrieved from: http://documents.worldbank.org/curated/en/577241468184764561/pdf/105159-WP-P155632-PUBLIC- Resilience-ME-Scoping-Paper-5April2016-FOR-PUBLICATION.pdf. World Bank. 2017. Evaluation of Resilience-Building Operations: Operational Guidance Paper for Project Task Teams. World Bank, Washington, DC, USA. Retrieved from: http://documents.worldbank.org/curated/en/669941506093754016/Evaluation-of-resilience-building- operations-operational-guidance-paper-for-project-task-teams. World Bank. 2014. Core Sector Indicators and Definitions. Updated December 2014. World Bank, Washington, DC. USA. Retrieved from: http://siteresources.worldbank.org/PROJECTS/Resources/40940- 1367867968385/CoreSectorIndicatorsList.pdf. World Bank. 2011. PROIRRI - Sustainable Irrigation Development Project. World Bank Group, Washington, DC, USA. World Bank. 2017. Kenya Climate-Smart Agriculture Project. World Bank Group, Washington, DC, USA. World Bank. 2016. Proposed Project Restructuring of the Mozambique Sustainable Irrigation Development Project. Report No: RES21715. World Bank Group, Washington, DC, USA. World Bank. 2017. Resilience M&E Good Practice Case Studies. World Bank, Washington, DC, USA. Retrieved from: http://documents.worldbank.org/curated/en/400851506100481060/World-Bank- resilience-M-E-ReM-E-good-practice-case-studies. World Bank. 2013. Results Frameworks and M&E: A Guidance Note. Prepared by OPCS, World Bank, Washington, DC, USA. Retrieved from: http://siteresources.worldbank.org/PROJECTS/Resources/40940- 1365611011935/Guidance_Note_Results_and_M&E.pdf. World Bank. 2016. Vietnam - Mekong Delta Integrated Climate Resilience and Sustainable Livelihoods Project. World Bank Group, Washington, DC, USA. 53 ReM&E December 2017 Appendix 1: An Illustrative List of Climate and Disaster Resilience-related Results Indicators from World Bank Investment Operations27 Summary This appendix presents climate and disaster resilience-related results indicators currently used in World Bank operations. Based on a review of results frameworks from 70 investment projects in FY13-15 with climate and disaster resilience building objectives, an illustrative list of about 323 relevant indicators is organized by different resilience-building sectors/themes. The detailed methodology is presented toward the end of this appendix. How to Use the Indicator List The list illustrates the breadth and variety of climate and disaster resilience-related indicator concepts currently used in World Bank operations and gives examples of how outputs and outcomes from resilience-building activities could be measured. The list can fulfill the following functions: 1. Provide a starting point for project teams to define resilience-related indicators for results frameworks and tailor them to specific project contexts; 2. Develop more standardized indicator menus for different sectors and types of resilience-building activities; 3. Analyze how results of resilience related World Bank operations are currently measured and identify gaps. The list does not necessarily include the exact indicator names used in the projects, as these are often framed in a very concrete way that reflects the specific context of the project. Also, indicators that are applied in a resilience context, but have not been framed accordingly, are revised so as to clarify the resilience connection. To create a manageable list, indicators are grouped, generalized and simplified so as to represent an indicator concept that is applicable in a variety of contexts: • Indicators generally refer to disaster or climate resilience, or climate change adaptation (CCA) and disaster risk management (DRM), but can be modified and tailored to specific disaster or climate hazard contexts (e.g., cyclones, floods, droughts, rainfall variability, temperature extremes, etc.). • Indicators combine a number of measurement units (e.g., people/ households/communities or states/municipalities/districts), which can be further specified depending on the target group. • Indicators combine different activities and actions that can be further defined depending on the intervention (e.g., infrastructure constructed/improved/rehabilitated or tools developed/tested/applied). • Indicators are not further disaggregated by gender or rural/urban classifications. In principle, all people/household indicators can be gender disaggregated and most indicators can be disaggregated by rural and urban areas. 27 This review of investment operations was prepared by Sundus Siddiqi, under the supervision of Ulf Narloch and Nathan Engle for the ReM&E project. 54 ReM&E December 2017 • Most indicators are expressed in absolute terms (e.g., kilometers of roads retrofitted). They can also be modified to measure the percentage of a target value (e.g., percentage of destroyed houses reconstructed). In some instances, indicator units are not clearly defined (e.g., institutional capacity increased) in the Project Appraisal Document, or they are measured by a discrete value (e.g., study completed or not, institution established or not). Resilience- Relevant Indicators Lists Table 4 provides an overview of results indicators related to climate and disaster resilience, differentiating between indicators with a clear link to resilience-building and those where it is context dependent and the link to resilience needs to be established.28 The list is organized by sector: 1. Agriculture & Rural Livelihoods (35) 2. Energy & Mining (3) 3. Environment & Natural Resources (33) 4. Social Development (5) 5. Transport (30) 6. Urban Development & Cities (6) 7. Water, Flood & Drought Management (37) 8. Water & Sanitation (7) 9. Cross-Cutting (120) Table 4. Resilience-Relevant Results Indicators for Investment Projects by Sector/Theme 1 Agriculture & Rural Livelihoods Clear link Food security and farm income 1 Number of people/households whose grain needs can be met 2 Number of people/households with adequate meals all year 3 Number of months of people/households who would have been food insecure w/o program 4 Number of livestock deaths reported during a climate/disaster event 5 Number of farmers/pastoralists/agricultural households that have received government support (e.g. food vouchers, farm assets or inputs) to deal with disasters/climate shocks 6 Income of farm households recovered to pre-flood levels 7 Number of food grain silos constructed/ operational 28 For example, increased access to irrigation is often considered as a resilience-building activity. However, in water-scarce environments this could actually be a form of maladaptation. 55 ReM&E December 2017 8 Number of climate/disaster recovery plans of eligible commercial farmers developed/approved 9 Monitoring, governance and management of food stocks improved Agricultural production and technologies 10 Area of agricultural land receiving improved flood protection 11 Average yield from farms applying climate/disaster production practices/ technologies 12 Agricultural area under climate/disaster resilient production practices/ technologies 13 Number of farmers/pastoralists/agricultural households engaged in climate/disaster resilient production practices/technologies 14 Number of farmers/pastoralists/agricultural households provided with improved access to water 15 Number of farmers/pastoralists/agricultural households trained in climate/disaster resilient production practices 16 Number of training days provided to farmers on climate/disaster resilient production practices/ technologies 17 Number of farmers’ field schools with CCA/DRM teaching practices formed/operated 18 Amount of climate/disaster resilient crop seeds/varieties distributed 19 Number of climate/disaster resilient pasture management plans implemented 20 Value of investments in climate/disaster resilient agricultural production and land resource management Agricultural information and monitoring and forecasting tools 21 Number of farmers/pastoralists/agricultural households receiving daily weather forecast 22 Number of farmers/pastoralists/agricultural households adopting agricultural monitoring and information tools 23 Number of farmers/pastoralists/agricultural households with access to agricultural monitoring and information data 24 Methodology developed to assess agricultural damages and loss from climate/disaster shocks 25 Staff trained to assess agricultural damages and loss from climate/disaster shocks 26 Number of available modern devices for measurement of soil moisture and heat regime of agricultural lands Context dependent 27 Area with irrigation and drainage services provided/rehabilitated 28 Number of people provided with new/ improved irrigation and drainage services 29 Area with improved irrigation water use efficiency 56 ReM&E December 2017 30 Average crop production income 31 Average crop yield 32 Value/number of agricultural products produced/traded 33 Value/number of livestock traded 34 Number of households satisfied with livestock health services 35 Amount of decreased grain storage cost 2 Energy & Mining Clear link Supply of and access to energy services 1 Number of climate/disaster resilient distribution lines constructed/rehabilitated 2 Number of power supplies restored to pre-climate/disaster emergency levels 3 Number of electric consumer accounts covered by underground cable network 3 Environment & Natural Resources Clear link Forestry and fishing incomes 1 Number of people/households with increased monetary or nonmonetary benefits from CCA/DRM-related forests/land/coastal management 2 Number of beneficiaries from CCA/DRM-related forest/land/coastal management 3 Average income for land/forest users/fishermen from alternative (e.g. less climate/disaster sensitive) income generating activities 4 Number of alternative (e.g. less climate/disaster sensitive) income generating activities financed by the project Forest, land, coastal management 5 Number of tree species in replanted/rehabilitated forests to improve forest resilience 6 Area with increased tree/vegetation cover (i.e. re/afforested/restored, etc.) 7 Area of slope stabilization through agroforestry/re/afforestation 8 Area of marine environments under improved forms of management related to CCA/DRM 9 Area of beach front restored/improved/protected 57 ReM&E December 2017 10 Area benefitting from CCA/DRM-related forest/land/coastal management practices/plans (i.e. protection/enhancement/restoration of biodiversity, tree cover, natural vegetation, etc.) 11 Number of jetties/wharves requiring high priority attention due to climate/disaster risks 12 Number of people/households with access to beach front restored/improved/protected 13 Number of land users that adopted climate/disaster resilient forest/land/coastal management practices (i.e. protection/enhancement/restoration of biodiversity, tree cover, natural vegetation, etc.) 14 Number of communities/districts/provinces with development plans that integrate climate/disaster risks considerations in forest/land/coastal management 15 Number of CCA/DRM-related forest/land/coastal use master plans developed/revised/adopted 16 Number of households/communities/municipalities with approved CCA/DRM-related environmental/drainage plans 17 Number of people/households with access to restored/improved environmental services/ facilities to increase climate/disaster resilience 18 Number of environmental services/facilities restored/improved to increase climate/disaster resilience 19 Number of forest cooperatives established/strengthened with regard to CCA/DRM 20 Number of forest/land users/fishermen trained in CCA/DRM-related forest/land/coastal management 21 Number of sessions organized on climate/disaster resilient forest/land/water management Land use/coastal planning and policies 22 Improved coastal climate/disaster risk monitoring 23 Number of policy instruments utilized for strengthened forest/land/coastal planning and management supporting CCA/DRM 24 Number of reforms in forest/land/ coastal policy, legislation or other regulations supporting CCA/DRM 25 Number of government institutions provided capacity building to improve CCA/DRM- related management of forests/land/coasts 26 Number of knowledge products on forest/land/coastal management with CCA/DRM considerations disseminated 27 Number of sensitization materials on forest/land/water management with CCA/DRM considerations disseminated through media 58 ReM&E December 2017 Context dependent 28 Number of people/households in targeted forest and adjacent communities with increased monetary or non-monetary benefits from forests 29 Change of Normalized Difference Vegetation Index (NDVI) 30 Number of nurseries established/restored to produce high quality seedlings of indigenous species 31 Number of seedlings produced for reforestation and rehabilitation 32 Number of proposals to expand conservation corridors 33 Number of forest field offices rehabilitated/reconstructed 4 Social Development Clear link Social aspects, including community-driven development 1 Community mechanisms established and functioning to respond effectively to climate/disaster risks 2 Percentage of targeted communities which demonstrate capacity to implement community-based CCA/DRM plans 3 Number of community-based CCA/DRM plans/projects developed 4 Degree to which villages have integrated CCA/DRM into community action plans and are implementing appropriate investments 5 Number of toolkits/guidelines prepared for community-based CCA/DRM 5 Transport Clear link Connectivity and travel disruptions 1 Travel time along road sections improved/reconstructed/rehabilitated during/after climate/disaster events 2 Number of days of interrupted traffic due to climate/disaster events 3 Reduced major incidents requiring emergency repairs due to climate/disaster events 4 Number of people at risk to climate/disaster- related transport interruptions 5 Number of roads in good and fair condition due to climate/disaster resilient measures 6 Number of roads classified as vulnerable to climate/disaster risks 7 Number of interventions that enhance all-weather accessibility of roads 59 ReM&E December 2017 Road and transport upgrading and rehabilitation 8 Number of project beneficiaries from transport infrastructure improvement/ reconstruction/ rehabilitation to climate/disaster resilient standards 9 Number of people/households/communities with access to improved/reconstructed/repaired climate/disaster resilient roads 10 Kilometers of roads that are constructed or upgraded in compliance with climate/disaster resilient design standards 11 Kilometers of roads adequately maintained in line with climate/disaster resilience standards 12 Kilometers of road-side slopes stabilized against landslides 13 Kilometers of road with unsealed shoulders 14 Kilometers of evacuation/rescue roads established/rehabilitated 15 Number of locations with improved slope protection 16 Number of detailed road-side slope stabilization completed 17 Number of climate/disaster resilient road design guidelines prepared 18 Number of people/households/communities with access to improved/reconstructed/repaired climate/disaster resilient bridges 19 Number of crossing structures/culverts/small bridges improved/reconstructed/ rehabilitated to climate/disaster resilient standards 20 Number of bridges improved/reconstructed/rehabilitated to climate/disaster resilient standards 21 Number of climate/disaster resilient designs and studies completed for road and bridge improvement/ reconstruction/rehabilitation 22 Number of people/engineers/officials trained on climate/disaster resilient road construction/maintenance 23 Number of improvements launched to strengthen road resilience with climate/disaster resilience standards 24 Amount of budget disbursed for (climate/disaster resilient) annual road maintenance Transport planning and policies 25 Number of climate/disaster resilient transport strategies and road investment plans completed 26 Number of transport management agencies with climate/disaster contingency plans Road safety 27 Amount of reduced fatalities due to road safety measures 28 Number of road projects that conduct/incorporate road safety audit recommendations 60 ReM&E December 2017 29 Kilometers of road network surveyed for road safety 30 Number of road safety design audit guidelines prepared (with climate/disaster resilience considerations) 6 Urban development & Cities Clear link Urban planning and cities 1 Number of urban climate/disaster resilient plans developed 2 Number of urban climate/disaster risk vulnerability analyses completed 3 Improved knowledge sharing between communities/municipalities on climate/disaster resilient city practices 4 Number of knowledge sharing activities on investments that improve climate/disaster resilience in urban areas 5 Number of local officials/stakeholders/technical experts trained in the use of climate/disaster resilient urban and territorial planning tools 6 Number of knowledge products and tools related to urban climate/disaster resilience developed 7 Water, Flood & Drought Management Clear link Flood risks and water scarcity 1 Expected annual economic damages due to flood risks 2 Number of people/households/communities benefitting from improved flood protection 3 Area protected by improved flood risk mitigation measures River basin, watershed and water resource management 4 Number of beneficiaries from climate/disaster resilient river basin/watershed management 5 Number of communities which adopted climate/disaster resilient river basin/watershed management plans 6 Number of plots managed under climate/disaster resilient practices in the target watersheds 7 Area of protected/restored wetlands in climate/disaster risk areas 8 Number of river basin/watershed management plans integrating climate/disaster risks 9 Number of river basin/watershed/water resource management organizations established/operating related to CCA/DRM 61 ReM&E December 2017 10 Number of local committees/organizations/communities engaged in storm water management activities 11 Number of flood protection or river basin/watershed management interventions contributing to increased resilience Flood protection and water storage infrastructure 12 Change in water storage capacity 13 Average flow velocity of floodplain canals 14 Number of floodplain canals operating near optimal flow velocity 15 Number/length/height of dam strengthened/rehabilitated 16 Length of dyke/river embankments/flood bunds constructed/improved/rehabilitated 17 Length of constructed/improved/rehabilitated distribution canals/flood drainage canals/spill tail canals 18 Number of distribution canals/flood drainage canals/spill tail canals properly maintained 19 Number of gabion baskets used in construction of flood mitigation works 20 Number of storm drains constructed 21 Number of structures/designs (e.g., flushing inlets, river culvert, river gauge stations, pumping stations, etc.) upgraded/rehabilitated/operated to climate/disaster resilience standards 22 Number of feasibility studies conducted for new large scale multi-purpose climate/disaster resilient water storage investments 23 Climate/disaster risk resilient drainage and water infrastructure design standards (Y/N) Hydrology information and water resource monitoring and forecasting 24 Time from drought warning to identification and information flowing to affected people 25 Number of new/upgraded water resource monitoring stations (hydrologic, meteorological, and groundwater) 26 Funded operation and maintenance cost of water resources monitoring network 27 Number of people/staff trained in areas related to CCA/DRM-related water management/planning/information systems 28 Number of improved climate/disaster informed tools used in decision making on water allocation 29 Climate/disaster risk mapping and hydrological model developed 30 Number of spatial knowledge portals at river basin/watershed-level 31 Number of river basin/watershed level flood risk models and analyses completed 62 ReM&E December 2017 Context dependent 32 Amount of groundwater extracted 33 Number of water users provided with new/improved irrigation and drainage services 34 Area provided with new/improved irrigation and drainage services 35 Number of canals supplying irrigation areas 36 Number of rain gauges operating in line with standard operating procedures 37 Number of river morphology studies completed 8 Water & Sanitation Clear link Supply of and access to water and sanitation services 1 Number of households with uninterrupted water service in project area during climate/disaster events 2 Number of people/households satisfied with the quality and security of water supply 3 Change in non-revenue water/water lost 4 Improved climate/disaster resilience of water supply and sanitation 5 Number of climate/disaster resilient drinking water sources rehabilitated/developed 6 Adoption of land uses that improve climate/disaster resilience in areas that provide supply water 7 Number of frameworks/policies/investment tools to increase climate/disaster resilience of water supply 9 Cross-Cutting Clear link People, households, communities 1 Number of people/households /communities reporting improvement in well- being/livelihoods/assets due to CCA/DRM project activities 2 Number of people/households/communities benefitting from CCA/DRM project activities 3 Number of beneficiaries satisfied with CCA/DRM project activities 4 Number of people/households/communities considered vulnerable in target communities/municipalities/districts/states 63 ReM&E December 2017 5 Number of people/households/communities supported to manage/cope with climate/disaster risks 6 Number of people enrolled in climate/disaster resilient safety nets programs/other cash transfers programs 7 Number of people/households/communities with increased capacity to respond to disaster events 8 Number of people receiving regular payments within the agreed timeframe after a climate/disaster emergency 9 Number of disaster-affected people/households/communities whose livelihoods have been recovered 10 Number of disaster–affected people/households/communities that benefitted from rehabilitation/recovery and/or reconstruction activities 11 Number of eligible people/households/communities that accessed recovery funding/emergency support/transfers 12 Number of people/households that adopt climate/disaster resilient practices/livelihoods 13 Number of people/households/communities that incorporate climate/disaster information into decision-making 14 Number of people/households that show awareness and ownership of CCA/DRM activities 15 Number of people/households/communities with access to early warning systems (e.g. through mobile short messaging services) 16 Number of people/households/communities with access to improved weather forecasting information 17 Number of people/households/communities with increased awareness of climate/disaster risks 18 Number of people/households/communities that participate in evacuation drills 19 Number of people/households/communities with increased awareness of emergency management 20 Number of people/households/communities trained in CCA/DRM technologies 21 Number of people/households/communities in climate/disaster vulnerable states/municipalities/ districts benefiting from a CCA/DRM action program 22 Number of people trained in climate adaptation and/or disaster risk mitigation 23 Number of people/households benefitting from alternative income generating activities supported/implemented 24 Amount of diversification of sources of income away from climate/disaster-sensitive sectors 64 ReM&E December 2017 25 Number of household livelihoods that have benefitted from public works (gender disaggregated) Physical assets, structures, equipment 26 Number of people/households/communities benefiting from infrastructure investment to reduce human vulnerability to climate/disaster risks 27 Number of people/households protected by climate/disaster resilient infrastructure 28 Number of emergency management warehouses equipped with specialized search and rescue equipment 29 Number of emergency vehicles procured for effective communication and response to disasters 30 Number of communication and search and rescue equipment provided to the state national disaster response 31 Number of people trained in disaster preparedness and emergency management to conduct contingency planning for key hydraulic infrastructure 32 Reduction in vulnerability of physical assets 33 Number of households with access to repaired/re-/constructed emergency shelters (and improved with hazard resilient design standards (and connectivity)) 34 Number of emergency shelters repaired/re-/constructed (and improved with hazard resilient design standards (and connectivity)) 35 Number of people/households/communities with reconstructed/ repaired/ retrofitted/ improved houses with climate/disaster resilience standards 36 Number of houses reconstructed/repaired/retrofitted/improved with climate/disaster resilience standards 37 Number of people/households/communities served by reconstructed/ repaired/ retrofitted/improved public buildings with climate/disaster resilience standards 38 Number of public buildings reconstructed/repaired/retrofitted/improved with climate/disaster resilience standards 39 Number of people/households/communities with awareness of retrofitting/improved construction with climate/disaster resilience standards 40 Number of craftsman/engineers trained to retrofit/improve houses/buildings/roads/bridges with climate/disaster resilience standards 41 Number of climate/disaster resilient building codes strengthened/ developed/ adopted/ enforced 42 Number of public assets with risk information to inform resilient planning 43 Number of vulnerability assessments and design work underway with climate/disaster resilience standards 65 ReM&E December 2017 Budgeting, financial instruments and funds 44 Number of projects implemented/completed under CCA/DRM investments 45 Number of projects/investment developed with climate/disaster resilience considerations 46 Amount of changes in budget allocation at national/sub-national/local/sector level to CCA/DRM activities 47 Number of agencies/states/municipalities/districts that budget required resources needs for CCA/DRM as identified in plans/strategies 48 Amount of national/sub-national/local/sector level budgeting/financing for CCA/DRM improved 49 Number of local-level institutions access to financial support for CCA/DRM through convergence with national-level programs 50 Number of active CCA and DRM loan accounts 51 Volume of active CCA and DRM loan accounts 52 (Number of) climate/disaster emergency insurance funds developed 53 Operating procedures for climate/disaster emergency (insurance) fund established 54 Amount of financial resources for climate/disaster emergency fund generated 55 Amount of time taken to disburse contingency funds in the event of an emergency 56 Amount of contingency funds disbursed (within critical time period after identification of need) Policies, plans and institutions 57 Improved institutional capacity for CCA/DRM at the national/sub- national/local/sector-level 58 Number of agencies/states/districts/municipalities that have developed/adopted climate/disaster risk reduction plan 59 Number of agencies/states/districts/municipalities that have developed/adopted disaster response protocols/emergency/contingency plans 60 Number of agencies/states/municipalities integrating climate/disaster risk considerations and/or resilience standards/guidelines in (updated/revised) development plans/strategies 61 Number of agencies/states/municipalities adopting CCA/DRM actions 62 Strengthened national/sub-national/local-/sector-level capacity to implement CCA/DRM 63 Strengthened national/sub-national/local-/sector-level coordination to implement CCA/DRM 64 Capacity in the Secretariat/PMU to provide effective oversight and implementation guidance for CCA/DRM 66 ReM&E December 2017 65 Number of CCA/DRM polices/plans/investments from participating agencies/states/ municipalities/countries harmonized 66 Number of government officials trained in CCA/DRM 67 Number of national/sub-national/sector CCA/DRM strategies/plans developed/adopted 68 Number of national/sub-national/sector CCA/DRM guidelines/operating procedures developed/tested 69 Number of adopted guidelines defining roles and coordination mechanisms for emergency and disaster response at national/sub-national/local/sector-level 70 Number of inter-agency exercises and drills for disaster response completed 71 Number of programs/plans/strategies integrating climate/disaster risk considerations and/or resilience standards/guidelines developed/adopted 72 Number of sector/state/municipality/district plans/strategies mainstreaming CCA/DRM 73 Number of integrated national policy frameworks that support mainstreaming of climate/disaster resilience across sectors developed/validated/implemented 74 Number of local-level institutions access technical support for CCA/DRM through convergence with national-level programs 75 Number of national/sub-national entities planning/prioritization for CCA/DRM improved 76 Number of government institutions that have prioritized/scaled-up/implemented CCA/DRM actions across selected sectors/states/municipalities/districts 77 Number of ministry agencies regulating climate/disaster resilience to incorporate risk analysis in public investment projects adopted 78 Number of national/provincial/local government officials trained on CCA/DRM 79 Number of functional early warning and response system from local to national level 80 Strengthened national/sub-national/local-/sector-level capacity to implement climate/disaster emergency response 81 National/sub-national/local/sector-level emergency center fully staffed and equipped 82 Improved capacity of emergency response personnel at national/sub- national/local/sector-level to respond to climate/disaster risks 83 Number of post-disaster housing eligibility policies that maximizes a supported resilient recovery approach developed 84 Number of agencies/states/districts/municipalities/sectors that approve guidelines for resilient post-disaster reconstruction process 85 Number of civil society organizations trained for providing CCA/DRM support 67 ReM&E December 2017 Information, weather forecasting and early warning 86 Number of climate/disaster event early warning systems installed/operating 87 Amount of increased coverage of climate/disaster event by early warning system 88 Number of weather/river gauge/hydro-met monitoring/recording/measurement/forecasting centers/stations/equipment 89 Time lapse between weather event information and early warning notation/response 90 Number of provisions of real-time metrological/hydrological data 91 Number of seismic monitoring networks established/operating 92 Number of data communication links established to climate/disaster hazardous sites 93 Number of daily weather forecast provided to agencies/ states/districts/municipalities 94 Number of people/households/communities that use CCA/DRM information and tools 95 Number of agencies/states/municipalities/districts that use improved CCA/DRM information and tools 96 Number of states/municipalities/districts that make effective use of climate/disaster risks/vulnerability information 97 Number of users of climate/disaster risk data and information systems 98 Number of users satisfied with the climate/disaster risk data and information systems 99 Number of agency/state/municipality/district officials trained on using/analyzing climate/disaster risk information/mapping/ modeling 100 Number of education and training programs on CCA/DRM are formulated/implemented 101 Number of trainings on climate/disaster data management/analysis/sharing 102 Number of workshops/events/activities for CCA/DRM knowledge dissemination 103 Number of agencies/states/municipalities/districts reporting on climate/disaster risks/vulnerability information 104 Number of agencies/states/municipalities/districts connected to data sharing platforms 105 Number of inter-agency/ministerial/state/province protocols on climate/disaster data sharing approved and implemented 106 Number of inventory databases of settlements in high risk areas available 107 Number of public buildings geo-referenced in a national exposure database 108 Number of states/municipalities/districts with concluded climate/disaster risk mapping/modelling 109 Number of climate/disaster risk mapping/modelling and other analytical tools produced 68 ReM&E December 2017 110 National/sub-national/local/sector-level climate/disaster risk data dashboard/platform/ information system available 111 CCA/DRM module developed in agency/sector/state/municipal/district information systems 112 CCA/DRM databases of climate/disaster risk information/vulnerability assessments established/maintained (Y/N) 113 Number of CCA/DRM decision-support system developed/launched/publicly available 114 Number of climate/disaster information products/services made available for decision- making 115 Number of technology packages developed/tested/implemented to identify climate/disaster risks 116 Number of climate/disaster risks identification studies completed 117 Number of feasibility studies for CCA/DRM projects completed 118 Number of operational manuals providing procedures/protocols for CCA/DRM at national/sub-national/sector-level developed/updated 119 Number of guidelines/manuals for climate/disaster resilient construction design and materials developed/updated 120 Number of guidelines based on good local and international practices for guiding CCA/DRM actions (including cost-benefit analysis) 69 ReM&E December 2017 Description of Methodology Purpose of Exercise To generate an illustrative list of indicators that could be used for monitoring results from climate and disaster resilience building activities in World Bank operations. Approach Based on a review of World Bank project documents, investment lending operations are identified that have climate and disaster resilience building Project Development Objectives (PDOs). The review includes operations tagged as providing Climate Change Adaptation (CCA) and/or Disaster Risk Management (DRM) co-benefits between FY13 and FY15 to capture an illustrative set of current resilience-building projects being implemented or planned within the institution. The PDO and intermediate results indicators of these projects, as well as other essential project information, has been consolidated in a Master database in order to analyze these indicators and produce sector lists of indicators relevant to resilience building activities. Building the Database A database of indicators from resilience-building operations has been built based on the World Bank Climate Finance database (downloaded on 10/24/15), which contains 1945 World Bank projects between FY11-15. For FY13-15, there are 340 projects with CCA co-benefits. 206 of these (61 percent) also have DRM co-benefits. This list has been cross-checked with the GFDRR database of all World Bank projects with DRM co-benefits for FY 12-15. From these 340 projects, only those are included in the database that have climate and disaster resilience building objectives by identifying operations that have at least one of the following four trigger “words”: “resil”, “clim”, “adapt”, and/or “disaster” in their PDOs. These parameters result in a final database of 80 projects: 70 of these are Investment Project Financing (IPF)/ Specific Investment Loans (SIL) and 10 are Development Policy Loans (DPL).29 This appendix focuses on indicators for IPF projects. The database contains every project development and intermediate results indicator currently being used in these 80 projects, segregated by the PDO/ Component/ Intermediate Result they correspond to. This provides a total of approximately 1200 indicators, including duplicates between several projects and indicators not related to resilience-building activities. The indicators are categorized by resilience-related sector/theme and classified considering the following 10 categories: Table 5. Categorization of Indicators Sector/theme Explanation Agriculture & rural Indicators related to food security and farm income, climate-smart livelihoods agricultural production (cropping and livestock), technologies and agricultural information and tools Energy & mining Indicators related to access to and supply of resilient energy services 29 The project documents of four projects and indicator matrix for one from FY 13-15 could not be found. These projects have thus been removed from the database. 70 ReM&E December 2017 Environment & Indicators related to forestry and fishing incomes, sustainable natural resources land/forest/coastal management (including ecosystem-based adaptation, ecosystem services and biodiversity, and integrated landscape and coastal management), and land use planning and policies Social development Indicators related to social aspects of resilience-building, including community-based approaches to CCA and DRM Transport Indicators related to connectivity and travel disruptions, road and transport link upgrading and rehabilitation, transport planning and policies, and road safety Urban development Indicators related to resilient urban planning and cities & cities Water, flood & Indicators related to flood risks and water scarcity, river basin, drought watershed and water resource management, flood protection and management water storage infrastructure, hydrology information and water resource monitoring and forecasting Water & sanitation Indicators related to access to and supply of resilient water and sanitation services Cross-cutting Indicators of general relevance for CCA and DRM cutting across several sectors/themes including robustness, and protection from change/shocks, preparedness to emergencies, recovery from emergencies, and understanding of risks. Other categories Indicators related to other resilience sectors/themes that are not the focus of the Sustainable Development Practice Group, such as social safety nets Simultaneously, the identified indicators are screened for their relevance for climate and disaster resilience building. Indicators are considered as resilience-relevant if they are related to one or more of the following: - Preparedness to manage and cope with climate change/disasters - Robustness to withstand climate change/disasters - Protection against climate change/disasters - Recovery from climate/disaster emergencies - Diversity of a system to mitigate risks - Redundancy of a system to withstand failure - Integration/connectedness of a system - Flexibility of a system to respond to uncertainty Indicators are initially categorized as: • Y: Yes - the indicator is directly relevant to resilience. These indicators are directly included in the list. • Y*: Yes - the indicator could be relevant to resilience if it measures dimensions specific to climate and disaster risks. These indicators are included in the list, after adding more resilience-specific language to its name. • M: Maybe: Casual link between the indicator and resilience is not clearly established or understood. It could be resilience relevant in some contexts but not in others (and even lead to 71 ReM&E December 2017 mal-adaptation), thus these indicators are included in each sector/them indicator list as a separate category denoted as “context dependent”. • N: Not related to resilience. These indicators are dropped from the analysis. Next, indicators with similar names and metrics or those based on the same idea or concept are merged to simplify the analysis and remove redundancies. Where indicators are framed so as to reflect a very specific project context, they are further generalized and simplified for broader applicability. For example, “Number of farmers adopting climate resilient production practices” represents both of the following indicators: 1. Farmers in areas targeted under the project have adopted climate resilient food crop production practices 2. At least 8000 farmers demonstrate climate resilient agricultural practices Multiple iterations of this process have taken place in order to ensure the production of efficient and holistic groupings of indicators. Attempts have also been made by the team to achieve this without losing indicators related to different types of resilience-building or indicator variations that reflect different levels of resilience-building (e.g., differentiating between and keeping both the Number of farmers adopting climate resilient production practices and agricultural area under climate resilient production practices). Many of these indicator categorizations, generalizations, and simplifications are subjective and are not always reflective of the specific project context in which the indicators are applied. As such, the list should only be used for illustrative and instructive purposes to show the breadth and variety of indicators currently being applied to reflect results related to climate and disaster resilience. 72 ReM&E December 2017 Appendix 2: Some Considerations for Climate & Disaster Resilience- Building DPF Operations30 1) Background This appendix targets World Bank Task Teams preparing climate and disaster resilience Development Policy Financing (DPF) operations (henceforth ‘Resilience DPF’). The focus is on operations that aim to build resilience to (natural) disasters and long-term climatic changes, including through climate change adaptation (CCA) and disaster risk management (DRM)-focused activities (henceforth ‘resilience- building’). While not being the focus of this work, resilience-building can also involve addressing non- climate and disaster-related stressors (e.g. economic instability, conflict, etc.). The objective of this note is to outline some considerations for the design of key elements of a Resilience DPF: (1) defining the policy focus and scope including potential policy pillars and illustrative policy areas, (2) selecting policy actions that can be fit and sequenced within a broader results chain, and (3) measuring policy results through resilience-relevant indicators. These considerations have been formulated based on a review of 26 DPF operations with resilience- building objectives. These do not include all DPFs that could be considered as resilience-building; the detailed methodology for the selection of DPFs can be found at the end of this appendix. The examples and categorizations derived from this review serve illustrative purposes only. In some instances, the specific framing of PDOs, policy pillars, tracks, actions and indicators are generalized or simplified to ease presentation in this note. 2) Defining Policy Focus and Scope When preparing a Resilience DPF, there are several options for defining the focus and scope of the operation in terms of: (1) the centrality of resilience-building to the development objectives, (2) the sector applications of the operation, (3) the policy areas covered by the operation. Decisions on (1), (2) and (3) are interdependent. For example, the selected sector applications could determine specific policy areas to be covered, and vice versa. 2.1 Resilience-Building in the PDO Concerning the role of resilience-building in the program-development objective (PDO) of the DPF, there are two types of Resilience DPFs (see Table 6): (i) DPFs with objectives focused on resilience-building only; (ii) DPFs with broader objectives that go beyond resilience-building. 30 This appendix was written by Ulf Narloch based on data preparation and analysis by Sundus Siddiqi, under supervision of Niels Holm-Nielsen. This work was undertaken under the Safer School Program KSB and the ReM&E project, including financial support from the Global Facility for Disaster reduction and Recovery (GFDRR). 73 ReM&E December 2017 Table 6. Role of Resilience-Building in the DPF Objectives PDO Focus 1. Resilience- Strengthen institutional/policy frameworks/capacities for comprehensive building is management of climate and disaster risks the only Strengthen the government’s program/reform agenda/ capacities to reduce disaster objective risks impacts and efficiently respond to disasters (with CAT DDO) Support recovery from disaster emergency and build resilience against future shocks Support state-/municipal-/community-level policies/actions to strengthen social resilience to climate change 2.Resilience Address causes of state fragility and vulnerability, including those emanating from building is climate and disaster risks part of a Support the national green growth strategy and climate change action, including CCA broader or/and DRM objective Strengthening the institutional framework and monitoring capacity in integrated water resources management and mainstream CCA or/and DRM 2.2 Sector Applications Most Resilience DPFs cover policy actions of relevance for various sectors given the cross-sectoral dimensions of resilience-building. Some of them include resilience-building policy actions relevant to applications specific to one or several sectors. Generally, three different types of Resilience DPF can be found: i. DPF only covering cross-sectoral policy actions; ii. DPFs with cross-sectoral policy actions and sector applications; iii. DPFs with one sector focus, such as agriculture and water. 2.3 Policy Pillars and Areas Resilience DPFs can cover a variety of policy actions within a number of policy pillars and areas. As developed by Hallegatte et al. (2016), risks depend on physical exposure to hazards, asset vulnerability and potential income and consumption losses, which determine the relation between asset losses and welfare losses.31 Overall, Resilience DPFs can include any of the following 6 policy pillars to address these risk dimensions, which could include policy actions relevant for different policy areas in these pillars (Table 7): i. General planning and financing (cross-cutting) ii. Spatial planning and risk prevention (reduce physical exposure) iii. Asset vulnerability reduction (reduce asset vulnerability) iv. Risk monitoring and response (reduce asset vulnerability) 31 Hallegatte, S., Bangalore, M., Vogt-Schilb, A. 2016. Assessing socioeconomic resilience to floods in 90 countries. Policy Research working paper; no. WPS 7663. Washington, D.C. See http://documents.worldbank.org/curated/en/387821467309551281/Assessing-socioeconomic-resilience-to- floods-in-90-countries. 74 ReM&E December 2017 v. Financial resilience (reduce income and consumption losses) vi. Resilient livelihoods (reduce income and consumption losses) 75 ReM&E December 2017 Table 7. Policy Pillars and Illustrative Policy Areas Covered by Resilience DPFs Policy Pillars Illustrative Policy Area Examples from DPFs General planning Legal and institutional - Strengthening institutional capacity to for planning, prioritization and financing of and financing: frameworks and capacities to CCA Improve public formulate, prioritize, finance, - Strengthening the strategic planning and institutional coordination for response to frameworks, implement and monitor DRM & climate change capacities and CCA- actions -Creating mechanisms for inter-agency coordination of DRM processes to manage climate & disaster risks Strengthening the development -Integrating disaster risks into national development plans and management of public planning and public investment investments systems to reduce disaster & - Mainstream climate and disaster risks management into public investment program climate risks processes Developing/updating -Formulation of DRM plans at municipal-level DRM&CCA plans at -Development of national DRM master plan Cross-cutting national/sector/local-level Mobilizing public financing for -Mobilization and budgeting of additional financial resources for CCA specific DRM&CCA actions -Strengthening of financing mechanisms for the implementation of the National DRM plan Spatial planning Identification of high risk zones -Identification and zoning of high-risk areas and risk and exposed assets and people - Development of a methodology for the formulation and consolidation of a national prevention: Policy inventory of settlements in high risk areas Reduce Physical Exposure measures to -Preparation of urban land use plan reduce physical exposure to Supporting protection and risk -Inclusion of risk reduction in territorial development plans climate & disaster mitigation measures -Improving the design of public flood risk prevention/mitigation programs hazards - Establishing program(s) for risk prevention in human settlements and urban zones Improving land use and -Improving inter-sectoral coastal planning ecosystem management and - Integration of risk-mitigation measures in land use and watershed management planning plans 76 ReM&E December 2017 -Establish operating rules for the program on protection and conservation of micro- watersheds Improving water resource - Adoption of programs, law and action plans to improve resilient use of water management resources - Promote recharge of aquifers and conservation of groundwater resources -Establish water protection corridors Asset Developing building codes, -Construction and upgrading of district roads according to revised design standards vulnerability design standard and asset - Adoption of revised road use regulation to limit vehicle overloading and road reduction: Policy management plans for road and damages measures to other transport infrastructure increase the Developing building codes, -Rebuilding houses according to cyclone and flood resiliency standards resistance of design standards, and asset - Increase the housing sector’s capacity to support and facilitate the reduction of physical assets management plans for housing seismic vulnerability in low-income housing and infrastructure to climate & Developing building codes, - Integrating resilience measures into the revisions of the national building code disaster hazards design standards plans and -Increasing the education sector’s capacity to implement disaster risk reduction asset management plans for measures in school infrastructure public buildings and other - Adoption of Safe Hospitals policy infrastructure Risk monitoring Improving generation and use -Improving application of disaster information in policy making through data sharing and response: of disaster and climate risk- platform and disaster risk profiles related information services, -Updating policy to establish fully functional early warning system Reduce Asset Vulnerability Policy measures to minimize hazards including early warning - Establishment of national and regional meteorological centers for weather impacts through forecasting and early warning improved preparedness to Increasing public awareness -Updating policy to carry out information dissemination activities manage climate about climate & disaster risks and disaster Strengthening emergency -Adoption of legal framework for disaster risk preparedness and response protocols hazards preparedness and response and exercises capacity -Adoption of emergency operations procedures integrating the role of firefighters 77 ReM&E December 2017 --Establish coordination system between national and sub-national level for disaster emergency response -Develop sectoral emergency response plans and commissions Financial Increasing financial capacity for -Establishing a national disaster emergency fund resilience: emergency recovery -Building mechanisms for financial protection from disasters Policy measures -Improved institutional mechanism for disaster risk financing and insurance that reduce fiscal -Adopting legal and institutional framework for risk transfer instruments and to secure impact of climate contingent financing and disaster hazards through Supporting contingency -Budgeting funds for post-disaster recovery improved planning and emergency -Developing/adopting sector contingency plans emergency recovery -Strengthening the institutional capacity to effectively plan, finance and implement recovery capacity post-disaster reconstruction Improving coverage and climate -Expansion of coverage of households in vulnerable districts by social protection and disaster responsiveness of program social protection systems - Improving targeting of natural disaster insurance program to low-income farmers - Increasing the capacity of community development and social protection programs Reduce income or Consumption losses to address disaster risks Resilient Increase food price stability and -Improve functioning of Strategic Grain Reserve to reduce price volatility in staple food livelihoods: Policy food security measures to Promoting better adapted -Strengthening budget allocation and policy instruments to scale-up climate-resilient reduce the income portfolios and agriculture for small-holder farmers vulnerability of livelihood practices (e.g. in -Promoting efficient irrigation for priority crops households and agriculture and forestry) communities to -Promoting sustainable forestry at community-level climate and disaster hazards Provision of reliable basic -Adopting measures for sanitation provision in emergency situations through increased services (health, water & - Adopting measures to address health risks from weather-related shocks coping and sanitation, etc.) adaptive capacity 78 ReM&E December 2017 3) Selecting Policy Actions In selecting policy actions for Resilience DPFs, these should, among others, meet the following criteria: (1) fit the results chain through which the desired resilience objectives are to be achieved; (2) be sequenced in a way they can achieve the end-of-program resilience results. 3.1 Formulating a Results Chain Policy actions are to be placed within a results framework for the DPF operation. This results framework should be informed by a broader results chain or technical theory of chain through which the policy actions within the different policy pillars and areas contribute to climate and disaster resilience. There are many different ways to formulate a results chain (from simple linear chains to multi-level frameworks with feedback loops etc.) and there is no standardized approach for Resilience DPFs that can target a number of very different policy areas and actions depending on the risk and country context. A simplified, linear framework can be organized around the following levels (Figure 5 for illustrative purposes): i. Activities: Policy actions supported by the DPF to unlock policy and institutional constraints for resilience-building; ii. Outputs: Direct policy changes delivered by the DPF that provide policy and institutional frameworks, reforms and capacities that support resilience-building; iii. Intermediate outcomes: Changes induced by the DPF policy actions that can be considered as resilience-building; iv. Final outcomes: Higher-level change in physical exposure, asset vulnerability, emergency preparedness, recovery capacity and livelihood vulnerability supported by the DPF but dependent on other factors; v. Impacts: Durable (‘resilient’) long-term condition supported by the DPF by enabling the country to resist, absorb, accommodate and recover in a timely and efficient manner from climate and disaster hazards; vi. Transformation: Societal shift supported by the DPF, which can be envisioned as poverty eradication and shared prosperity in a sustainable manner through increased climate and disaster resilience. 79 ReM&E December 2017 Figure 5. An illustrative Results Chain for Resilience DPFs Activities: Policy actions Outputs Intermediate Final outcome: Impact: Transformation: supported by DPF Direct policy Outcome: Higher level Durable condition Long-term societal shift changes Change induced by change supported supported by DPF supported by DPF delivered by DPF DPF policy actions by DPF Risk identification Zoning Reduced physical Spatial planning and measures Risk prevention exposure: Better protected human Country is able to risk prevention Spatial/land use plans measures implemented settlements, assets, resist, absorb, Ecosystem mgmt plans Improved management accommodate and Country is able infrastructure and of ecosystems recover in a timely to eradicate General ecosystems planning and efficient manner poverty and Building codes Upgraded/retroffited without undermining and Reduced asset promote shared Asset vulnerability roads, housing and the welfare of people financing: reduction Design standards public buildings and vulnerability : prosperity in a Actions that Increased resistance of and the functions and Asset mgmt plans other public sustainable Improve physical assets and structure of socio- infrastructure economic and bio- manner through public infrastructure increased frameworks, physical systems Awareness campaigns Increased access to resilience capacities Risk monitoring and Information platforms hazard information Improved emergency and response preparedness: Reduced For example: Early warning systems Emergency drills processes to hazard impacts manage Emergency protocols Evacuation/shelter • Reduced asset climate & losses disaster risks Contingency plans Recovery/reconstruction Improved recovery • Reduced Safety net instruments Increased coverage of capacity: Reduced fiscal consumption/ Financial resilience Risk financing & safety net & insurance impacts income losses insurance strategy Emergency fund Reduced livelihood Grain reserves Food price stability vulnerability : Agric adaptation plan Resilient agric practices Increased coping and Livelihood resilience Hazard (exogenous) shock or Sector dvlpmt strategies Income diversification adaptive capacity of households and stress caused by climate Measures for service Reliable/responsive communities change or natural disaster provision basic services DPF/program level: under the direct influence of the DPF policy actions National/system: dependent on other factors 80 ReM&E December 2017 3.2 Sequencing Policy Actions To facilitate a country’s reform agenda and overcome the main constraints in one policy area, sometimes multiple policy actions are required that can be addressed within the same operation or be tackled over several consecutive operations. DPFs can be organized as: i. Standalone operation with a set of policy actions to be met at the same time; ii. Programmatic series of 2-3 operations with prior actions and policy triggers that can be sequenced over a multi-year period. Programmatic series offer the advantage of supporting a longer-term engagement and multiple actions that can build on each other if the spacing between operations matches the time needed to complete the policy actions.32 Standalone DPF provide some flexibility on when and how the government’s reform agenda will be further supported. In programmatic series (but also standalone operations with 2nd operation) policy actions can be sequenced in a way they build a results chain or theory of change to achieve the end-of-program result. The prior actions in the first operation facilitate the policy triggers in the second operation, which then facilitate those triggers in the third operation. All operations together enable the end-of-program result. Some existing programmatic series represent a results chain through the sequencing of their policy actions (Table 8). Table 8. Examples of Sequenced Policy Actions from Programmatic Resilience DPFs Trigger Trigger End of Prior Action DPF1 Program DPF2 DPF3 Result National strategy or Policy framework or Operational DRM/CCA Fund development plan with action plan for DRM/CCA framework for accounts DRM/CCA DRM/CCA Fund operationalized Update provincial level Develop National CCA Adopt CCA Improved planning, climate change scenarios Strategy Prioritization prioritization and Framework financing for CCA Approval of strategic plan Establishment of regional A protocol for the Reduced lead time for National Institute of meteorological and new management and between early Meteorology 2013-2016 organizational statute to exchange of data is warning issued and deliver forecasts and approved through weather monitoring early warnings at regional a joint ministerial (in 2 river basins) level more efficiently diploma of relevant line Ministries 32 See also IEG, 2016. Lessons from Environmental Policy Lending. Independent Evaluation Group Learning Product. World Bank Group, Washington, DC. 81 ReM&E December 2017 None National Health Strategy Protocols and Increase of high-risk adopting measures to guidelines for districts/municipaliti address severe and disaster es that have longer-term climate risks preparedness and introduced disaster response to disease preparedness and pandemics following response protocols weather-related for health service shocks delivery Adopt national action plan Adopt guidance for Adopt the coastal Increase of for the integrated development of functional zoning provinces that have management of coastal integrated coastal zone plan to guide adopted and started zones management programs at integrated coastal implementation of provincial level zone management integrated coastal programs at zone management provincial level programs Establish technical and Adopt operational None Reduced intra- financial guidelines for market seasonal price reporting and interventions from volatility accountability for maize Strategic Grain Reserve drawn from Strategic Grain Reserve 4) Measuring Policy Results Identifying appropriate indicators to measure whether the expected end-of-program results have been achieved (i.e. indicator that are measurable and indicate a meaningful change in policies) is critical. The indicator choice depends on several factors (including government capacity and available data). A key criterion is the timeframe within which end-of-program results are expected to be realized: i. Short-term outcomes (ca. 2-3 years): those that can directly be achieved through a policy action (e.g. guidelines and policies in place). ii. Medium-term outcomes (ca. 3-5 years): those that require a sequence of policy actions or/and complementary conditions to be achieved (e.g. investments mobilized) iii. Long-term outcomes (ca. >5 years): those that need a longer time to unfold, or for which change can only be observed over longer time horizons (e.g. actual reduction in exposure). In current DPFs there is a variety of indicators used to measure the end-of-program results from resilience- building policy actions – many of which are rather output focused directly linked to meeting specific policy actions. Such indicators are useful if for standalone operations with a short time-frame and to establish if critical policy changes are achieved.33 For programmatic series and operations with a medium or long- 33 See also IEG, 2016. Lessons from Environmental Policy Lending. Independent Evaluation Group Learning Product. World Bank Group, Washington, DC. 82 ReM&E December 2017 term timeframe, more outcome-focused indicators are preferable so as to measure if the policy actions actually triggered the desired changes. Most (all) indicators currently used are proxies for resilience, which can signal if the ability or capacity of a country to manage a climate/disaster event is increased, but cannot not measure the actual change in this ability or capacity. There are some indicator options that could monitor the country’s ability or capacity to manage a climate or disaster event based on observed data (O). The difficulty of such indicators is that it requires the realization of such an event to produce measurable data and often lacks a baseline to be compared with. Alternatively, new approaches can be based economic and hazard modelling to predict the potential impacts of an event (P). Estimates can be produced for different scenarios and policy actions and then be expressed in terms of ‘saved losses’ or ‘effectiveness of selected policy measures.’ Depending on the policy focus and time-frame task teams can choose from a broad range of results indicators (see Table 9). Table 9. Resilience-Relevant Results Indicators for Development Policy operations by Policy Pillar Indicator Metric Timeframe Cross-cutting planning and financing Guidelines adopted for creation/sharing/use of climate and disaster 1 Yes/No Short relevant data and information services 2 Guidelines for prioritization of CCA/DRM investments is available Yes/No Short Guidelines for integration of climate and disaster risks into investment 3 Yes/No Short planning is available Mechanism (e.g. data sharing platform) established to coordinate CCA Yes/No 4 Short and DRM actions 5 Legal instrument in place for CCA and DRM coordination and planning Yes/No Short Short- 6 National monitoring system in place to track CCA and DRM financing Yes/No Medium National monitoring system in place to report results of CCA or DRM Yes/No 7 Medium investments Ministries/sectors or states/provinces/districts/municipalities reporting # 8 Medium on CCA and DRM budgets Share of line ministry annual programs that include risk risks analysis and 9 % Medium mitigation in key annual programs Share of (public) investment projects/programs that integrate risks 10 % Medium assessments and mitigation into planning process States/provinces/districts /municipalities that have mainstreamed 11 # Medium CCA/DRM into local development plans Ministries/sectors or states/provinces/districts/municipalities with 12 # Medium adopted CCA/DRM action plans Ministries/sectors or states/provinces/districts/municipalities that Medium- 13 # budget CCA/DRM action according to plans Long 83 ReM&E December 2017 Welfare risks at national-level from selected hazards or ability to reduce 14 % Long the ratio of welfare losses to asset losses (P) Spatial planning and risk prevention 1 Guidelines adopted for hazard mapping to identify high-risk areas Yes/No Short 2 Guidelines adopted for inventory of settlements in high risk areas Yes/No Short Guidelines adopted for integrated spatial planning for risk prevention at 3 Yes/No Short national/sub-national-level Feasibilities studies completed for protective infrastructure projects (e.g. Short- 4 # dyke/river embankments/flood bunds, etc.) Medium States/provinces/districts/municipalities that have completed hazard 5 # Medium mapping and identified high-risk areas Districts/municipalities with completed inventory of settlements in high 6 # Medium risk areas States/provinces/districts/municipalities that have issued integrated # 7 Medium spatial plan for risk prevention States/provinces/districts/municipalities that have action plans for 8 # Medium inclusion of risk reduction in territorial development plans Coastal provinces/districts that have adopted their integrated coastal 9 # Medium zone management programs Length of large-scale protective infrastructure constructed and upgraded 10 Km Medium (e.g. dyke/river embankments/flood bunds, etc.) Area of critical restored/protected ecosystems (mangroves, steep Hectar 11 Medium slopes, wetlands, etc.) e Area benefitting from protective infrastructure investments (e.g. Hectar 12 Medium dyke/river embankments/flood bunds, etc.) e Hectar 13 Area exposed to high risks (P) Long e People living in high risk areas if possible disaggregated by age, gender, 14 # Long poverty and other vulnerability proxies (P) 15 Value of physical assets located in high risk areas (P) $ Long Asset vulnerability reduction Adoption of building guidelines that meet (multi-hazard) resiliency 1 design standards by relevant line ministries (housing, public buildings, Yes/No Short roads) States/provinces/districts/municipalities with climate and disaster Short- 2 # resilient asset management plan Medium States/provinces/districts/municipalities prioritizing asset investments 3 # Medium according to vulnerability assessments and resiliency requirements Roads constructed/retrofitted/upgraded based on vulnerability 4 Km Medium assessments and resiliency design standards Bridges constructed/retrofitted/upgraded based on vulnerability # 5 Medium assessments and resiliency design standards Houses constructed/retrofitted that meet multi-hazard resiliency 6 # Medium standards 84 ReM&E December 2017 Public buildings (hospitals, schools, etc.) 7 constructed/retrofitted/upgraded based on vulnerability assessments # Medium and resiliency design standards Medium- 8 Total value of assets constructed according to resiliency design standards # Long Vulnerability of assets to selected hazards (no standardized methodology 9 % Long available) 10 Total asset losses during climate/disaster event (O) $ Long 11 Saved asset losses due to reduced asset vulnerability (P) $ Long Risk monitoring and response 1 Annual climate and disaster risks awareness campaign is conducted Yes/No Short Adoption of emergency response protocols for critical agencies (e.g. Yes/No 2 Short firefighters) 3 Local-level emergency response focal point appointed Yes/No Short 4 Emergency Operation Committees established and operating Yes/No Short National center for weather forecasting and early warning established Short- 5 Yes/No and operational Medium Short- 6 National/sub-national emergency center fully staffed and equipped Yes/No Medium Agencies or districts/municipalities that have undertaken emergency Short- 7 # response drills and exercises medium Guidelines adopted for creation and use of climate and disaster relevant Short- 8 Yes/No data and information services medium Stations (seismic and/or hydrometereological) operated for hazard Short- 9 # monitoring within national/sub-national network Medium Ministries/sectors or states/provinces/districts/municipalities with plans Short- 10 # and protocols for disaster preparedness and response Medium National data platform for climate and disaster relevant data and 11 Yes/No Medium information services established and operational Users of national data platform for climate and disaster relevant data and 12 # Medium information Emergency management warehouses across the country quipped with 13 specialized search and rescue equipment and communication # Medium technologies 14 Evacuation routes constructed from high-risk areas Km Medium 15 Emergency shelters constructed # Medium District/municipalities with weather forecasting and/or early warning 16 # Medium systems established and operating Medium- 17 Households with access to evacuation routes and/or emergency shelters # Long Households with access to weather forecasting information and/or early Medium- 18 # warning system Long 85 ReM&E December 2017 Lead time between natural hazard information and issuing emergency 19 Time Long warning $ 20 Emergency personnel mobilized during climate/disaster event (O) Long $ 21 People evacuated during climate/disaster event (O) Long $ 22 Assets protected during climate/disaster event (O) Long Saved asset losses due do emergency preparedness and/or early warning 22 $ Long (P) Fiscal resilience Guidelines adopted for post-disaster needs assessments. and Short- 1 Yes/No reconstructions process and financing Medium Short- 2 National catastrophe risk financing strategy and insurance developed Yes/No Medium Guidelines adopted to design and implement risk transfer instruments at Short- 3 Yes/No the national-/subnational- level. Medium National budget includes a specific budget line for emergency response Short- 4 Yes/No and/or reconstruction Medium Number of ministries/sectors or states/provinces/districts with Short- 5 # emergency contingency plans Medium Catastrophe risk insurance schemes (e.g. for housing) designed and 6 Yes/No Medium operational 7 Emergency response fund created and operating procedures defined Yes/No Medium Households covered under climate/disaster-risk focused safety net 8 # Medium program 9 Roads reconstructed (based on resiliency design standards) Km Medium 10 Bridges reconstructed (based on resiliency design standards) # Medium 11 Houses reconstructed (based on resiliency design standards) # Medium 12 Public buildings reconstructed (based on resiliency design standards) # Medium Households subscribing catastrophe risk insurance scheme (e.g. for 13 # (%) Long property) Affected households receiving payments under safety net program # (%) 14 Long during climate/disaster event (O) Emergency financing mobilized within XX months/years after 15 $ Long climate/disaster event (O) Contingency funds disbursed after climate/disaster event (within critical 16 $ Long time period after identification of need) (O) 17 Duration of reconstruction process after climate/disaster event (O) Years Long 18 Reduced reconstruction time (P) Years Long Long 19 Saved income losses due to fiscal resilience measures (P) Years 86 ReM&E December 2017 Livelihood resilience National/local strategy/plan for development of climate and disaster 1 Yes/No Short resilient income activities National/local stagey/plan for climate and disaster resilient provision of 2 Yes/No Short water & sanitation services Yes/No National/local strategy/plan for development of climate-resilient 3 Short agricultural practices adopted Number of states/provinces/districts/municipalities with climate Short- 4 Yes/No resilient agriculture development plan Medium Water & sanitation service providers (e.g. companies) that have 5 # Medium incorporated climate and disaster risks into their management plans Government support for development of less climate-sensitive sectors 6 # Medium and income activities # or Farms utilizing more advanced and efficient irrigation practices for 7 hectare Medium selected crops s Area planted according to conservation (or other climate resilient) Hectar 8 Medium agricultural practices e Area with access to more advanced and efficient irrigation practices for Hectar 9 Medium selected crops e Average yields for selected crops on farms engaged in conservation (or Tons/ 10 Long other climate resilient) agricultural practices hectare Households engaged in employed in less sensitive climate-sectors and Medium- 11 # income activities Long Share of household incomes from less climate sensitive income sources 12 % Long (e.g. transfers, non-agricultural wages, etc.) Households considered as vulnerable to climate and disaster risks (no 13 % Long standardized methodology available) Household income/consumption losses during climate/disaster event 14 $ Long (O) 15 Saved household consumption due to livelihood measures (P) % Long Notes: N/A indicates that these indicators are not yet applied in World Bank DPFs. These indicators come from a wider review of indicators applied in World Bank operations or/and international best practices. O=Observed impacts based on the ground monitoring, which requires the actual realization of an event. P=Predicted impacts based on economic, risk and hazard modelling, e.g. as in Hallegatte et al., 2016. 87 ReM&E December 2017 5) Description of Methodology 5.1 Approach Resilience-relevant World Bank Development Policy Financing (DPF) Operations are identified that have climate and disaster resilience building objectives, which resulted in an illustrative (not complete) list of 26 resilience-building DPF, of which 7 are the 2nd or 3rd operation in a series or represent a direct follow- up from an earlier DPF. The list includes the following DPFs: 1. All Catastrophe Deferred Drawdown Option (CAT DDOs) DPFs from the pipeline/portfolio between FY09 and FY17 − Source: Operations Portal 2. All DPFs with Climate Change Adaptation (CCA) co-benefits (between FY 09 and FY15) and/or Disaster Risk Management (DRM) co-benefits (between FY13 and FY15) that have resilience- specific Project Development Objectives (PDOs). Resilience-relevant DPLs are identified as those that have one of the following four trigger “words”: “resil”, “clim”, “adapt”, and/or “disaster” in their PDOs. Those projects have also been included where the goal/description of the project captured resilience but the PDO did not. The projects were identified based on: − FY13-15 projects with DRM co-benefits - Source: DRM Co-benefits FY12-15 (detailed)- Oct ’15 update − FY11-15 projects with CCA co-benefits - Source: FY11-15 Climate Finance Portfolio Data- June ’16 update − FY09-10 projects with CCA co-benefits - Source: Coding sheets- March ‘16 update − Pipeline projects with CCA co-benefits- Source: GP Online Climate Data- April update 3. Relevant Pipeline projects that are known by the task team and found to add to the list in terms of representative action. 5.2 Building the Database For the selected 26 projects, the database captures the following information: 1. Basic information about project: FY, Name, Country, Region, TTL, GP, Approval Date, Closing Date, Status, Project Sequence, Sectors and Theme classification, Financing information, DRM and/or CCA co-benefits (if assigned). 2. Information relevant to forming results chains from the Program Documents: a. Resilience-relevant policy pillars and policy tracks. For all projects the resilience-relevant policy pillars and policy tracks are listed. The pillars and tracks are categorized and cross referenced with 7 Resilience DPF Strategic pillars: (i) Financial resilience, (ii) Resilient Livelihoods, (iii) Risk monitoring and emergency response, (iv) Reduction of vulnerability 88 ReM&E December 2017 of physical assets, (v) Cross cutting DRM & CCA planning and financing, (vi) Territorial and spatial planning and risk mitigation, and (vii) General service category. b. Policy actions for each resilience-relevant policy pillar/track, including the prior actions and any policy triggers, where DPF forms a programmatic series. c. End-of program results indicators corresponding to the policy actions are categorized, along with how they are measured, including indicator description, measurement units, baseline and end target, and timeframes, as applicable. 89