91807 Social Protection & Labor Systems Social Protection & Labor Technical Note June 2014 | Number 4 Why Is It Important to Monitor and Evaluate Monitoring and Evaluating Disaster- and Climate Social Protection Programs’ Shock-sensitive Components of Social Efforts to Respond to Natural Protection Programs? Early integration of disaster Disasters and Climate risk management/climate change adaptation criteria into Change-Related Shocks social protection programs’ Rachel Cipryk and Mirey Ovadiya monitoring and evaluation plans, systems, and budgets Key Elements of an M&E System allows for more effective capture of necessary information, Disaster risk management/climate change adaptation activities are expected to contribute to two fundamental objectives of social protection: including proxy indicators to measure the reduction of 1. Before disaster hits, they aim to reduce and mitigate the risks associated with risk exposure. Organizations disaster- and climate-related hazards by reducing poverty, increasing resilience, such as the Active Learning and promoting opportunities and livelihoods both before and after disasters strike. These objectives are part of the prevention and promotion functions of Network for Accountability and social protection. Performance in Humanitarian Action (ALNAP) have 2. After a disaster hits, they aim to protect poor and vulnerable households and documented international help them cope with their impacts through relief activities and recovery and experience in developing reconstruction interventions. monitoring and evaluation The following steps can make social protection M&E more disaster and climate systems that measure disaster sensitive: risk management and climate 1. During program preparation, select performance outcomes, outputs, change adaptation. Monitoring indicators, and realistic interim targets. and evaluation systems have also been used after disasters to 2. During program preparation and early implementation, gather baseline data provide rapid real-time feedback on: disaster- and climate-related risks or post disaster/climate change impacts, vulnerable households, and existing coping/adaptation mechanisms. on the appropriateness and coverage of the response, so that 3. During program preparation, build a monitoring system that includes adjustments can disaster risk management/climate change adaptation that can be adjusted be made. and expanded if disaster strikes: Technical Notes June 2014 | Number 4 ■■ Assess the capacity of the program to achieve program Risk reduction and mitigation (ex ante measures) objectives, including those related to disaster risk management/climate change adaptation (for example, Outcomes, outputs, and indicators should measure the building emergency response capacity) ex ante and ex post. extent to which activities help beneficiaries reduce the risks from disasters and adapt to climate change, increase ■■ Establish partnerships to coordinate data collection household and community resilience, and contribute to and define collective data collection plans for post broader social protection or poverty reduction objectives. disaster contexts. Examples include improving the diversity and sustainability of livelihood assets, improving infrastructure that builds ■■ Identify the methods and tools to use for the various resilience to shocks, increasing and improving institutional components of the M&E plan both before and after a capacities, and ensuring social inclusiveness in the disaster. distribution of benefits. ■■ Design an M&E feedback loop to inform program If direct measurement of change is not possible, proxy implementation. indicators can be used. One type of proxy is the quantity 4. Evaluate all program components for efficiency and and quality of physical mitigation measures constructed, for effectiveness with regards to future programming. example the number of people/hectares of land protected by Consider conducting an impact evaluation. strengthened and improved embankments. Another is changes in awareness, attitudes, skills, and practices for risk reduction Step 1: Select Performance Outcomes, Outputs, and climate adaptation, which may indicate the degree to Indicators, and Interim Targets which a community is prepared to respond to disaster. Social protection programs need to be able to track changes to Risk coping (ex post measures) vulnerability and resilience so that performance measurement systems can be appropriately adapted. It is therefore important In the wake of a disaster, social protection projects are well to build some flexibility into the M&E plan. In addition, all positioned to respond quickly to the needs of their regular M&E plans should be participatory in nature and include both beneficiaries and to take on additional beneficiaries. Outputs, quantitative and qualitative methodologies. outcomes, and indicators for disaster response need to be identified to measure the results. A sample of these measures can be found in Ethiopia’s Products Safety Net Program (Table 1) which addresses chronic food insecurity among a highly climate-vulnerable population. Table 1. Outcomes, Outputs, and Indicators for Risk Financing Mechanism in Ethiopia’s Productive Safety Net Program (PSNP) Measure Objectively verifiable indicator Goal Livelihoods and lives protected from shocks in PSNP districts Outcome (Program’s own impact) Transitory cash and food needs addressed 1. Consumption ensured and assets protected by existing PSNP beneficiaries, effectively in PSNP districts, to the limit of with rapid response team (RRT) risk financing resources 2. Consumption ensured and assets protected by non–PSNP beneficiaries receiving risk financing assistance, with RRT Social Protection & Labor | World Bank Group Outputs 1. Accurate early warning of shocks achieved 1.1 Early warning issued within x weeks of first indication 1.2 Early warning messages balance triangulated data sources and resolve inconsistencies, with RRT 1.3. Early warning message is accepted and agreed by State Minister for Disaster Reduction and Food Security 1.4 Ex post evaluation of early warning shows acceptable accuracy 2. Appropriate contingency plans ready when 2.1. Contingency plans for all PSNP districts submitted to Early Warning needed Response Department in July each year 2.2. Contingency plans updated every 12 months, following feedback from regional and federal level 2.3. Quality review process for contingency plans operates effectively 3. Adequate contingent financing resources 3.1. Pooled fund is at intended level before shock available where and when needed 3.2. Agreements made with key donors on nature, timing, and scale of response to contingent appeal 3.3. Early warning system provides adequate early notice of scale, nature, location, and timing of resources needed 3.4. National committee approves fund release as guided 3.5. Performance standards for funds flows achieved at all levels 3.6. Communication and coordination between key stakeholders on funding need, utilization, and problems is effective 4. Planned systems and processes for risk 4.1. Early warning system functions according to performance standards financing mechanism function effectively 4.2. Contingency plans meet quality standards 4.3. Funding flows function according to performance standards 4.4. Coordination meets performance standards 4.5. Decision-making systems follow technical guidance 4.6. Clear guidance for roles and responsibilities and response to transitory needs in PSNP districts followed by key actors 4.7. Transitions between instruments and actors meet performance standards 4.8. Staff capacity able to scale up as needed and meet performance standards 4.9. Logistical capacity able to scale up as necessary and meet performance standards 5. Effective coordination with other financial 5.1. Shared policy and strategy framework agreed for transitory response in and delivery instruments and actors PSNP districts achieved 5.2. Guidelines for transitory response followed by key actors 5.3. Joint planning for transitory response between actors 5.4. Effective communication of transitory response information between actors 5.5. Transitions between instruments smooth Source: IDL Group, 2009 Technical Notes June 2014 | Number 4 Step 2: Gather Baseline Data and Conduct a Post Step 3: Build the M&E System Disaster Needs Assessments Capacity assessment and preparedness To measure the contribution of an intervention, it is first necessary to obtain a baseline that assesses current and Providing training, using specialist expertise on evaluations, projected risks, frequency of the disaster, the disaster’s impact and forming partnerships with disaster risk management/ on vulnerable groups, and coping mechanisms used mitigate the climate change adaptation organizations will help prepare disaster’s impact. This analysis should be carried out as part of teams to monitor the program and expand monitoring the overall poverty and risk assessment process. It will also act capacity to respond quickly to a disaster (box 2). Necessary as the foundation for designing project objectives, outcomes, skills include the ability to conduct social analysis for post outputs, and performance indicators in the M&E plan. disaster contexts, use of participatory approaches, and an understanding of both the social protection system and the Risk coping disaster and climate risk context. Post Disaster Needs Assessments (PDNAs) are government-led efforts, supported by bilateral and multilateral development partners. They are typically undertaken three to four weeks Box 2. Training Indonesian officials in after a disaster. A PDNA uses two methodologies: 1) the World social research after the tsunami Bank–led Damage and Loss Assessment (DaLA) methodology, To help them assess the ongoing impacts of the 2004 Asian tsunami which examines damage, losses, and sector-specific needs; 2) in Aceh, Indonesia, the Aceh Community Assistance Research the United Nations–led Human Recovery Needs Assessment Project (ACARP) provided its evaluation team with a two-week (HRNA) methodology, which assesses the impact of the training course on social research methodologies. The course disaster on human needs and development achievements. covered basic concepts of quantitative and qualitative research and research techniques, practical skills in interviewing and research A qualitative methodology called social impact analysis (SIA) notation, rapid and participatory methods, and gender balance in research and reporting. Participants learned how to design and has recently been introduced into the PDNA processes to use questionnaires and conduct a range of research engagements, provide insights into the post disaster social consequences including focus group discussions, structured and semistructured that are not easily observable. (Box 1). The SIA assesses areas interviews, village histories, and data analysis. such as community dynamics, social cohesion, demographic marginalization, and household-level challenges to livelihoods restoration The information gathered from a PDNA and SIA It is important to have agreements in place with agencies or form a baseline for identifying key social protection needs and organizations with which the social protection program can monitoring whether and how these needs are met. partner (Box 3). Such partnerships allow a qualified team to be quickly deployed after a disaster. Box 1. Using social impact analysis to assess the effectiveness of social protection in Thailand Box 3. Expanding community outreach in Pakistan after an earthquake During the 2011 floods in Thailand, a social impact analysis team was deployed with the Post Disaster Needs Assessments team. For The Earthquake Relief, Rehabilitation and Reconstruction Program the first time, SIA findings were used to inform recommendations (E3RP) of Pakistan’s Poverty Alleviation Fund’s (PPAF) deployed for post disaster recovery actions by the social protection system. 47 social mobilization teams in Azad Jammu Kashmir and 60 in Integration of the social impact and social protection system North West Frontier Province. The teams played a critical role by analysis provided the basis for recommendations that responded carrying out damage assessments, inspiring social mobilization, to the difficulties many households were facing but which had providing training, and conducting quality control. escaped attention in the Damage and Loss Assessment (DaLA) Each team was supposed to include an engineer and a male and process. Pairing the two sets of recommendations, the team female social organizer and be responsible for 800–1,000 households. identified existing social protection mechanisms that could have In the event, some teams lacked adequate numbers of women, been used to uncover particularly vulnerable households and reducing the capacity of PPAF to work with vulnerable households, channel resources to them. The recommendations suggested that particularly households headed by women. Partner organizations pensions for the elderly and grants for poor households be doubled did not appear to understand gender issues or housing design that for three months in order to provide additional support until the met the needs of people with disabilities. The PPAF concluded that, floods abated; that livelihood interventions target women and not in the future, it would be desirable to train and monitor partner only men (as the first round of assistance had done); and that low- organizations on vulnerability and gender issues. interest rate loans be made available to facilitate the repayment of high-interest, informal loans that had become a necessity for many low-income households. Social Protection & Labor | World Bank Group Partnerships and coordination Step 4: Evaluate Performance M&E may involve a variety of implementing partners including Performance assessment measures efficiency, effectiveness, and different national ministries, local governments, and outside impact. To measure efficiency and effectiveness, it is important agencies. Ensuring that all parties contribute to the ongoing to cover the following dimensions: needs of the M&E framework requires that responsibilities, lines of communication, and coordination be clear and agreed • Communications at multiple levels, including within the to by all parties. implementation team at the ground and higher up, with partnering organizations and with core and disaster- Coordination and information sharing are particularly related beneficiary groups. important after disasters. Establishing shared protocols regarding who is responsible for what information is useful. • Operations related to planned and unplanned responses, Existing data sources, such as predisaster baselines and social including readiness, the availability of resources for swift program monitoring information, should be shared with all response, plans of action, and partnerships to maximize post disaster assessment teams. resources. • Preparedness of the implementation team, program re- Methods and instruments sources, and communities with which the program has been working (if preparedness activities had been taking place). An M&E plan guides ongoing assessment of progress. It should include both qualitative and quantitative methods, • Targeting the most vulnerable households as well as which can be used to manage project risks, oversee quality providing an adequate level of benefits that meet the control, and measure performance. Existing social protection needs and priorities of beneficiaries (such as debt relief in methodologies and tools can be modified to capture disaster addition to the meeting of immediate needs). risk management/climate change adaptation information. Such methodologies include participatory approaches, financial • M&E of the disaster event, including the accuracy of and technical audits, management information systems (MIS), post disaster assessments, to ensure that resources and social audits, and other beneficiary feedback methods, such as assistance are appropriately distributed and to determine grievance and redress systems. whether the M&E system contributed to corrective action. MIS is an important tool for targeting, beneficiary coverage, Evaluating the impact of disaster and climate resilience and fiduciary control. It can also be used to track disaster risk components of social protection programs can be difficult outcomes and overlapping them with data about beneficiary if there are few measurable or observable changes in the coverage or poverty density. environment or a disaster does not occur within the project’s duration. As discussed above, proxy indicators may have to The grievance and redress systems that are in place for normal suffice as a means of evaluating the degree to which a program operations should continue during the post disaster period. has built resilience to a hazard or climate shocks. Generally, However, they may need to be adjusted and streamlined to however, the high cost of impact assessments means that they deal with the temporary surge in caseloads. Advance planning are conducted only when a program demonstrates particularly is needed to ensure that they can be scaled up quickly on innovative or important results. demand. Grievance systems ensure two-way communications with the public and help clarify program eligibility, objectives, In disaster response, it is important to ensure that the impact and benefits. of the mitigation measures be tracked following the event in order to assess the impact. The social impact assessment methodology can be used for medium- to long-term monitoring of the social impacts of the disaster. Comparing these findings against findings for a control group provides an indication of the effects and impacts of mitigation and preparedness activities. Technical Notes June 2014 | Number 4 Tips for Practitioners: Principles to Follow in Conducting M&E The following tips can help practitioners monitor and evaluate 4. Plan ahead for disasters. social protection programs’ efforts to mitigate and respond to disaster and climate risks: 5. Create contingency plans for financing and implementing post disaster M&E. 1. Ensure that the M&E system captures all areas of programming. All elements should be reflected in the 6. Use existing systems, to the extent possible. Post disaster M&E instruments. Ensuring sufficient human and environments are often complex, rushed, and confusing. financial resourcing for activities such as logframes, Avoid complicating programs or adding new activities and M&E plans, operational guides, and needs assessments is formats when these needs can be met through existing essential. mechanisms. 2. Ensure that program staff have the skills to conduct M&E, 7. Ensure that social accountability mechanisms remain in including the ability to adjust programs based on post place during and after disasters. Although post disaster disaster assessment findings chaos can cause social accountability to be missed or deemed too difficult, it is important to ensure that 3. Partner with other agencies where possible, particularly transparency and accountability are maintained. when monitoring slow-onset disasters or responding to disasters. Establish protocols to identify who has what 8. Continue to monitor. Although easy to drop amidst post information and will be responsible sharing it. Identify disaster chaos, M&E provides valuable information. gaps and make plans to fill them. Additional Resources ALNAP.org is a learning network that supports the humanitarian sector in its efforts to improve humanitarian performance through learning, peer-to-peer sharing, and research. Learning to ADAPT: Monitoring and Evaluation Approaches in Climate Change Adaptation and Disaster Risk Reduction: Challenges, Gaps and Ways Forward http://community.eldis.org/.5a093c0d Making Livelihoods and Social Protection Gender Sensitive http://documents.worldbank.org/curated/en/2012/10/16875747/making-livelihoods-social-protection-gender-sensitive Monitoring and Evaluation in Disaster Risk Management http://www-wds.worldbank.org/external/default/WDSContentServer/WDSP/IB/2011/01/05/000333038_20110105000313/ Rendered/PDF/587940BRI0211M10BOX353819B01PUBLIC1.pdf Social Impact Assessment Guidelines: Analyzing the Social Impacts of Disasters. Vol. I: Methodology http://siteresources.worldbank.org/INTEAPREGTOPSOCDEV/Resources/PostDisasterocialAnalysisToolsVolumeI.pdf Social Impact Assessment Guidelines: Analyzing the Social Impacts of Disasters. Vol. II: Tools http://siteresources.worldbank.org/INTEAPREGTOPSOCDEV/Resources/PostDisasterSocialAnalysisToolsVolumeII.pdf The findings, interpretations, and conclusions expressed herein are those of the author(s), and do not necessarily reflect the views of the International Bank for Reconstruction and Development/The World Bank and its affiliated organizations, or those of the Executive Directors of The World Bank or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work.