101749 PROFOR WORKING PAPER Understanding Long-Term Impacts in the Forest Sector: PREDICTIVE PROXY INDICATORS Acknowledgements This report was written by Daniel C. Miller (Task Team Leader and Sr. Forestry Specialist, World Bank) and Catherine Benson Wahlén (Independent Consultant). Helpful inputs were provided by Selene Castillo, Tuukka Castrén, Diji Chandrasekharan, JongA Choi, Neeta Hooda, Anders Jensen, Peter Jipp, Stig Johansson, Julian Lee, Nalin Kishor, and Nga Phuong Nguyen. Exchanges with the Climate Smart Agriculture indicators team led by Ademola Braimoh helped inform this report. Laura Ivers, Veronica Jarrín, and Sujatha Venkat Ganeshan provided essential support to this activity. The team benefited from insightful comments and suggestions from Rocio Sanz Cortes, Jeneen Reyes Garcia, Carole Megevand, and Anthony Waldron as well as from members of the Science for Nature and People (SNAP) working group on evidence-based conservation and the Forests and Livelihoods: Assessment, Research, and Engagement (FLARE) network. The peer reviewers were Julien Colomer (Evaluation and Impact Assessment Officer, IUCN), Gerhard Dieterle (Adviser, GCCPT, World Bank), Anders Jensen (Sr. Monitoring and Evaluation Specialist, GPSOS, World Bank), and Avjeet Singh (Sr. Operations Officer, GWADR, World Bank). The work was done under the overall guidance of Valerie Hickey, Practice Manager, Environment and Natural Resources Global Practice, at the World Bank. Financial support for this activity from the Program on Forests is gratefully acknowledged. Disclaimer All omissions and inaccuracies in this document are the responsibility of the authors. The views expressed do not necessarily represent those of the institutions involved, nor do they necessarily represent official policies of PROFOR or the World Bank. Suggested Citation Miller, Daniel C. and Catherine Benson Wahlén, 2015. Understanding Long-Term Impacts in the Forest Sector: Predictive Proxy Indicators. Washington, DC: Program on Forests (PROFOR), World Bank. For a full list of publications please contact: Program on Forests (PROFOR) 1818 H Street, NW Washington, DC 20433, USA profor@worldbank.org www.profor.info/knowledge PROFOR is a multi-donor partnership supported by the European Commission, Finland, Germany, Italy, Japan, the Netherlands, Switzerland, the United Kingdom and the World Bank Group. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 3 Ta b l e o f C o n t e n t s ACKNOWLEDGEMENTS 2 LIST OF BOXES, FIGURES, AND TABLES 4 ACRONYMS 5 EXECUTIVE SUMMARY 6 1. INTRODUCTION 10 2. USE OF PREDICTIVE PROXY INDICTORS IN FORESTRY AND OTHER SECTORS 13 2.1 DEFINING PROXY INDICATORS 13 2.2 LITERATURE REVIEW 13 2.3 STRENGTHS AND LIMITATIONS OF PROXY INDICATORS 14 3. IDENTIFYING PREDICTIVE PROXIES IN OPERATIONAL INVESTMENTS 16 3.1 OVERALL APPROACH 16 3.2 PROJECT IDENTIFICATION 17 3.3 DATA COLLECTION 18 3.4 CRITERIA FOR EVALUATING QUALITY INDICATORS AND PREDICTIVE PROXY INDICATORS 20 3.5 VALIDATION OF INDICATORS AND DEVELOPMENT OF AN INDICATOR MENU 21 4. RESULTS 22 4.1 OVERALL ANALYSIS OF INDICATORS 22 4.2 IDENTIFICATION OF PREDICTIVE PROXIES FROM THE WORLD BANK FORESTRY PORTFOLIO 24 4.3 POTENTIAL PREDICTIVE PROXY INDICATORS 25 4.4 POTENTIAL PREDICTIVE PROXY INDICATORS WITH BROAD RELEVANCE 38 4.5 THE UTILITY AND IMPORTANCE OF WORLD BANK CORE SECTOR INDICATORS 40 4.6 ASSESSING MONITORING AND EVALUATION IN THE WORLD BANK’S FORESTRY PORTFOLIO 43 5. OPPORTUNITIES FOR AND CONSTRAINTS ON USE OF PROXIES IN WORLD BANK PROJECTS 50 5.1 OPPORTUNITIES 50 5.2 CONSTRAINTS 51 6. GUIDANCE ON USING PROXIES 52 7. CONCLUSION 54 7.1 SUMMARY OF KEY FINDINGS 54 7.2 TAKING THIS WORK FORWARD 55 Annex A. Data and Methods 56 Annex B. Exploring Potential PPIs in Knowledge Programs: The Case of PROFOR 67 Annex C. Menu of Highly Rated Forest Sector Indicators 74 Annex D Glossary 105 References 107 4 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Boxes Box 1: What is a Predictive Proxy Indicator? 11 Box 2: Clarifying Links in the Results Chain 19 Box 3: Testing Predictive Proxy Indicators in the Argentina Forests and Community Project 28 Box 4: Forestry-Relevant Core Sector Indicators in IDA and IBRD Projects 41 Box 5: From Unsatisfactory to Satisfactory: The India Ecodevelopment Project 43 Box 6: Avoiding Overdesign: The Cape Peninsula Biodiversity Conservation Project 45 Box 7: Another Impact on Project Outcomes? TTL Turnover 48 Box A1: Application of SMART Criteria 62 Figures Figure 1: Approach to Identify Predictive Proxy Indicators 16 Figure 2: Regional Representation of World Bank Forestry Projects Reviewed 17 Figure 3: Project Outcome Ratings 18 Figure 4: Distribution of Projects by PDO Type 19 Figure 5: PDO Indicators SMART Scores 23 Figure 6: GEO Indicators SMART Scores 24 Figure 7: Results Chain for Indicator Cluster 1: Sustainable Forest-Related Income 27 Figure 8: Results Chain for Indicator Cluster 2: Afforestation/Reforestation to Support Livelihoods Improvement 31 Figure 9: Results Chain for Indicator Cluster 4: Positive Environmental Impacts (Climate Change–Related) 34 Figure 10: Uptake of Forestry Core Sector Indicators by Indicator 42 Figure A1: Representation of High-Forest Countries 57 Figure A2: Regional Representation of Projects Reviewed 57 Figure A3: Project Lending Type 59 Figure A4: Project Environmental Category 59 Figure A5: GEO Type 60 Figure A6: Project Area Focus 60 Tables Table 1: Examples of Indicators Relevant to Sustainable Forest-related Income Predictive Proxy 29 Table 2: Examples of Indicators Relevant to Afforestation/Reforestation to Support Livelihoods Improvement Predictive Proxy 32 Table 3: Examples of Indicators Relevant to Positive Environmental Impacts (Biodiversity-Related) Predictive Proxy 33 Table 4: Examples of Indicators Relevant to Participatory Project Design and Implementation Predictive Proxy 37 Table 5: Examples of Indicators Relevant to Strengthened Land Tenure Predictive Proxy 39 Table 6: Examples of Indicators Relevant to Predictable, Sustainable Financing Predictive Proxy 40 Table 7: Uptake of Forestry Core Sector Indicators in World Bank Projects 42 Table 8: Common Lessons Learned on Indicators 46 Table B1: Summary of Information on PROFOR Indicators Reviewed 68 Table B2: Top 25 Scoring Indicators in PROFOR Indicators Review 70   UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 5 Acronyms AFR Africa Region CBFM community-based forest management CIFOR Center for International Forestry Research CODE Committee on Development Effectiveness CSI Core Sector Indicator EAP East Asia and the Pacific ECA Europe and Central Asia EOP End of Project ESP Environmental Service Program FAO Food and Agriculture Organization FLARE Forests and Livelihoods: Assessment, Research, and Engagement FOREST focused, outcome-oriented, replicable, evidence-based, short-term, and timeless GEF Global Environment Facility GEO Global Environment Objective ICF International Climate Fund ICR Implementation Completion Report IEG Independent Evaluation Group IO intermediate outcome IUCN International Union for Conservation of Nature JFM joint forest management LAC Latin America and the Caribbean M&E monitoring and evaluation MENA Middle East and North Africa MTR mid-term review NTFPs non-timber forest products PA protected area PCR Project Completion Report PDO Project Development Objective PES payment for environmental services PPI predictive proxy indicator PROFOR Program on Forests RCT randomized control trial REDD+ Reduced Emissions from Deforestation and Degradation SAR South Asia SDR Special Drawing Rate SFM sustainable forest management SMART specific, measurable, attributable, realistic, and time-bound SME small and medium-size enterprise SNAP Science for Nature and People TTL Task Team Leader 6 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS E X E C U T I V E S U M M A RY Background and Study Rationale The international development community is increasingly demanding better evidence on the effectiveness of policies and programs across different sectors. The forest sector is no exception. Governments and donor agencies explicitly seek to link investment to proven impact. Yet the evidence base necessary to inform interventions in the forest sector that can successfully enhance the livelihoods of the forest-dependent poor, foster economic growth, reduce emissions from deforestation and degradation, and conserve forest biodiversity remains weak. There is a particular need to identify robust indicators to track and assess the impacts of forest-related investments. The Independent Evaluation Group (IEG) 2013 review of the Forests Strategy of the World Bank, the largest multilateral funding source in the sector, and the subsequent Committee on Development Effectiveness (CODE) report highlighted this need. The IEG recommended the development of outcome indicators on sustainable forest management to track progress across the World Bank’s Forests Strategy. The 2013 CODE report highlighted the imperative to develop short- term proxy indicators for long-term impacts in the forest sector. To date, however, there is little systematic knowledge on the availability of such predictive proxies in the sector, what form they should take, and the conditions under which they are effective. This PROFOR-financed study responds to this gap in knowledge and to broader demand from donors, government agencies, and implementing organizations to develop robust, yet practical means to better understand the impacts of forest sector investments. It focuses in particular on potential predictive proxies for longer-term term outcomes and suggests that such indicators do in fact exist. The report identifies a set of theory-based predictive proxy indicators (PPIs) relevant to one or more overarching development objectives: poverty reduction and economic growth, biodiversity conservation, climate change mitigation and adaption, and good governance. The results and the approach used here lay the foundation for future analytical work to test and refine PPIs in the forest sector. They also have the potential to inform efforts in other complex development sectors seeking reliable information in the short term on likely longer-term outcomes. This report should be of special interest to World Bank Group Task Team Leaders (TTLs) and monitoring and evaluation (M&E) specialists who are working on operational and analytical investments that have a forestry component. The indicators discussed here can help to inform the design and implementation of such investments so that they are able to have more positive impacts on the World Bank’s key development goals of eliminating extreme poverty and boosting shared prosperity in a sustainable manner. The indicators developed in this report also should also be of interest to other actors involved in forest sector investments, and the approach is relevant to other sectors that may also grapple with long time horizons and significant temporal lags between interventions and impacts. Looking ahead, this report should have particular resonance as the international community looks to adopt an ambitious set of Sustainable Development Goals and related targets and indicators to guide development policy over the next 15 years. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 7 Analytical Approach The results presented in this report draw from several sources of evidence. The primary information source is an analysis of the World Bank’s forest lending portfolio over the past 25 years. This portfolio review focused on 80 projects, including the full sample (n=48) of the Bank’s investment during the study period in the top 11 countries in terms of forested area and/or prominence of forest biodiversity: Brazil, China, Colombia, Costa Rica, the Democratic Republic of the Congo, India, Indonesia, Madagascar, Mexico, Peru, and the Russian Federation. In addition, 32 projects were randomly selected from all the relevant projects in the portfolio. Project selection was weighted by region so that regions with a larger number of projects were proportionately represented within the review. Following the selection of projects and the data collection process, a three-step approach to identifying potential predictive proxies was used: developing criteria for good predictive proxies; assessing possible indicators for their potential as predictive proxies, on a scale from 1 to 5; and validating potential proxies. All project indicators were evaluated according to the SMART (specific, measurable, attributable, realistic, and time-bound) criteria, using a Likert scale from 1 to 5. A PPI should provide a credible indication about some future change or state based on currently available evidence. It is a stand-in for what we would like to measure directly (a future change or state) but cannot (yet) measure. To be judged as having potential as a predictive proxy, the indicator had to imply a plausible theory of change explaining why it was likely to accurately predict a desired future change or state that would result at least in part from a given intervention. We augmented potential PPIs identified through the portfolio review with indicators used by other key actors in the forestry sector and suggested by experts, including World Bank TTLs and research specialists. We then used a theory- based approach to validate the PPIs based on expert views gathered at a workshop of TTLs and other World Bank staff and at meetings at the Oxford Center for Tropical Forests and with members of the Science for People and Nature working group on evidence-based conservation and the Forests and Livelihoods: Assessment, Research, and Engagement network. This report also explored validating PPIs by collecting data on hypothesized proxies at different points in time (that is, during a project, at project end, and post-project) and examining whether the outcomes measured persist over time and across a large number of interventions. While some relevant time series data sets were identified, cost considerations prohibited exploring this validation method more fully during this phase of the work. Findings The main overall finding is that predictive proxy indicators do appear to exist and can be used in practice. Given the complexity and diversity of the forest sector and novelty of the task, this conclusion was far from a certainty when this research began. We identified a range of potential PPIs, several of which have already been used in World Bank forest projects. These PPIs focused primarily at the Project Development Objective (PDO) and Global Environment Objective level, although some intermediate-level indicators were identified. The report argues that robust PPIs can provide an important ex ante evaluation tool that enables practitioners and researchers to predict future outcomes and longer-term impacts if certain assumptions hold. To help inform the design and implementation of forestry operations and knowledge, we developed a list of top ranking indicators based on an assessment of their predictive potential and their SMART score. These indicators are presented in an indicator menu organized by major objective (that is, relating to poverty, biodiversity, climate, or governance) 8 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS and including brief notes on how they might be used. The indicator menu is presented as an annex and represents a key product stemming from this analytical work. This review did not identify any standalone “silver bullet” predictive proxies, but a major contribution is the idea that multiple indicators, considered together, can have strong predictive potential. The report describes a series of seven indicator clusters that form PPIs. These clusters consist of two to four indicators and address the following broad development objectives or themes: • Sustainable forest-related income • Afforestation/reforestation to support livelihoods improvement • Positive environmental impacts (biodiversity-related) • Positive environmental impacts (climate change-related) • Increased carbon stocks • Participatory project design and implementation • Effective project monitoring and evaluation The report highlights the importance of indicators on land tenure and sustainable financing. These two indicators are part of several clusters and are also recommended as key indicators in many cases of the likely persistence of project gains over the longer term. Another major finding from the review is that the World Bank’s Core Sector Indicators (CSI) are generally robust and often have strong potential as PPIs. Five of the seven forestry CSIs feature as elements in the PPI clusters that were developed. A sixth, relating to support for policy and regulatory reforms, is also likely an important constituent in additional PPI clusters. Moreover, several CSIs from outside the forestry sector also constituted key elements in PPI clusters. Thus, the use of CSIs can have the double benefit of helping to capture both end-of-project and longer-term outcomes of forest investments in a consistent way across countries and contexts. Uptake of forestry CSIs has improved over time, and 63 percent of active forestry projects approved since the CSIs were launched in July 2012 now include at least one CSI. This finding suggests that some ongoing projects may already have in place clusters of indicators that may have the capacity to predict longer-term outcomes. The review also found that project M&E within World Bank forest sector investments has improved over time, as indicated by an increased emphasis on M&E and tracking progress on indicators, particularly those relating to outcomes as opposed to outputs, in more recent project Implementation Completion Reports (ICRs) compared with Project Completion Reports or earlier ICRs. This is a positive finding, especially given that the quality of project M&E has been shown to influence project outcome ratings. Regression analysis quantified a strong positive association between the quality of project M&E in the forestry projects reviewed and ultimate project outcome ratings: as project M&E scores increased by one category (from moderately satisfactory to satisfactory, for example), project outcome rating scores increased by one-half a rating category. This result indicates a significant return on investment in M&E for overall project outcomes, one that may portend sustained results over time for projects. An important practical contribution of the report is a distillation of key lessons learned on M&E based on review of the World Bank Forests Portfolio. For example, projects should avoid overdesign. A broad-brush results framework that clearly lays out objectives and key performance indicators but avoids restrictive detail can facilitate adaptive management, innovation, and—ultimately—better results. Projects should also use M&E in a proactive manner to assess progress and address weaknesses early during implementation. A well-functioning M&E system has been found to provide the basis UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 9 for stronger dialogue between the donor and client during project implementation. In addition to these general lessons, the report also includes examples of projects with exemplary M&E frameworks recognized for their careful design, adaptive nature, and focus on impact and quality. The report goes beyond identifying potential PPIs and distilling M&E lessons to also analyze opportunities and constraints to the use of PPIs in World Bank projects, with potential relevance to interventions by other donors and actors in the sector. Opportunities highlighted include the potential for such indicators to facilitate the use of a smaller number of indicators in projects, comparisons across projects, regions, and sectors, and more cost- and time-effective measurement. The use of PPIs may also have significant value as a communications tool showing the impacts of forest-related investments. Constraints identified include project incentive structures that promote indicators that are achievable within project time frames but lack imagination or ambition, disincentives to collect and report on data including due to cost and lack of available technical support, and the need to match indicators with larger client priorities and capacity. Drawing from the portfolio review and discussions with TTLs and other forest project implementers, the report provides guidance on using PPIs. This guidance highlights potential factors to consider when developing proxy indicators, including suggestions on criteria for selecting PPIs and looking beyond World Bank projects for examples of proxy indicators. A key innovation presented is a more robust framework of criteria to identify PPIs: the FORESTS criteria. Akin to the SMART criteria for good indicators generally, the FORESTS criteria suggest that effective predictive proxy indicators should be focused, outcome-oriented, replicable/reliable, evidence-based, short-term, and timeless. The extent to which all these criteria must be fulfilled to create a strong PPI remains to be tested. Paths Forward This report highlights the importance of many CSIs, including several outside the forest sector, as well as secure tenure and sustainable financing mechanisms as measures that can be combined with other indicators as part of a cluster or used as stand-alone indicators. It underscores the importance of identifying additional PPIs, testing and validating them in new operations as well as through retrospective analysis. Future development of PPIs must strive to be cost-effective and be integrated into ongoing World Bank and other development agency strategies for realizing key international development objectives not only in forestry but also in other complex sectors.   10 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 1. Introduction The international development community is increasingly demanding better evidence on the effectiveness of policies and programs across different sectors. The forest sector is no exception. Governments and donor agencies explicitly seek to link investment to proven impact. Yet the evidence base necessary to inform interventions in the forest sector that can successfully enhance the livelihoods of the forest-dependent poor, foster economic growth, reduce emissions from deforestation and degradation, and conserve forest biodiversity remains weak (Miteva, Pattanayak, and Ferraro 2012; Wunder, Angelsen, and Belcher 2014; CIFOR 2015). There is a particular need to identify robust indicators to track and assess the impacts of forest-related investments. The Independent Evaluation Group (IEG) review of the Forests Strategy of the World Bank (IEG 2013), the largest multilateral funding source in the sector, and the subsequent Committee on Development Effectiveness (CODE) report highlighted this need (CODE 2013). The IEG evaluation observed that environmental indicators used in forest projects tended to measure effort or process (such as the number of hectares under management plans or the number of hectares planted) rather than project outcomes. Further, the evaluation found that poverty reduction indicators used in World Bank projects were often inadequate for measuring whether programs and projects reached the poorest and most vulnerable members of the community (IEG 2013: xviii, 102). The IEG therefore recommended that technical guidance on sustainable forest management (SFM) outcome indicators be developed and used to better track progress across the objectives of the World Bank’s Forests Strategy. The CODE report specifically highlighted the imperative to develop short-term proxy indicators for long-term impacts in the forest sector (IEG 2013). To date, however, there is little systematic knowledge on the availability of such predictive proxies (see Box 1) in the sector, what form they should take, and the conditions under which they are effective. A study of World Bank environment lending, for example, found that “the data necessary to examine how specific donor supported institutions fare in the medium to long-term simply do not exist in most cases” (Buch, Buntaine, and Parks 2015: 30). Even a basic understanding of the characteristics that would make for a strong predictive proxy remains lacking. A recent review of World Bank investments in forest governance (Kishor and de Rijk 2014) similarly found that the “E” part of M&E (monitoring and evaluation) is rarely used to look at project impacts. The authors recommended the development and testing of predictive proxy indicators (PPIs) along with greater use of more rigorous impact evaluation methods. This Program on Forests (PROFOR) study responds to this gap in knowledge and to broader demand from donors, government agencies, and implementing organizations to develop robust yet practical means to better understand the outcomes and impacts of forest sector investments. It focuses on identifying predictive proxy indicators for operational investments but also considers the prospect of such proxies for analytical work designed to influence policy and practice. Any effort to identify potential predictive proxies and provide recommendations on their use confronts two particular challenges in the forest sector: interventions usually involve considerable complexity, and they often take a long time to show results. For example, results of investments in thinning, tree stand improvement, or natural regeneration under SFM are unlikely be apparent for 10–30 years. At the same time, forestry projects often include multiple objectives, require the integration of socioeconomic and ecological expertise, and entail processes that unfold over different spatial and temporal scales. These characteristics make attribution of impacts to specific interventions (as opposed to other factors) difficult within the forest sector, particularly within the context of landscape and cross-cultural approaches. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 11 BOX 1: WHAT IS A PREDICTIVE PROXY INDICATOR? A predictive proxy indicator is best understood by defining each element in the term. An indicator is commonly understood as “a parameter or a value from parameters which points to, provides information about, or describes the state of a phenomenon with a significance extending beyond that directly associated with a parameter value” (OECD 1993). Indicators can be qualitative or quantitative. They can also be categorized by type according to their function. One example is a proxy indicator, which can be defined as an indirect measurement of a variable lacking direct information (IEG 2012) and a substitute for an indicator that is hard to measure directly and that may reveal performance trends, potential problems, or areas of success (World Bank 2014a). The variable can be lacking information for several reasons, including that direct measurement is too challenging or costly to obtain in a reasonable time period. Proxy indicators are used when the effect of a particular intervention cannot be assessed using direct information. The proxy indicator replaces the use of such direct data with secondary data that are feasible to collect and provides information on the effect of the intervention using this indirect information. As the term suggests, proxy indicators are therefore almost inevitably approximations. An example of a proxy indicator sometimes used in the forestry sector is the development and implementation of forest management plans, which is seen as a proxy for sustainable forest management outcomes. A predictive proxy indicator, or simply predictive proxy, is a specific type of proxy indicator that seeks to provide information about the future. This study uses the term predictive proxy indicator to refer to a measure taken during implementation of a project, program, or policy that stands in for longer-term impact. A predictive proxy thus provides an indication about some future change or state based on currently available evidence. In short, it is a stand-in for what we would like to measure directly (a future change or state) but cannot (yet) measure. The objective of this study is to identify potential predictive proxy indicators to enable an assessment of outcomes that have long incubation periods (that is, long-term outcomes or impacts) and for which attribution is unclear and to provide guidance on their application. It highlights the value of PPIs as early indicators that a project, program, or policy is likely to achieve desired results, that it may need some adjustments to reach stated objectives, or that achievements are likely to be sustainable after implementation. The approach and findings presented here should have broad relevance to those working in the forest sector and even in other sectors where predictive indicators may be of use. This report should be of interest to project teams and M&E specialists in a variety of organizations ranging from bilaterial and multilateral donors to technical agencies in developing countries to NGOs and research institutions. Given the focus on indicators in World Bank projects, it will be of special interest to World Bank Group Task Team Leaders (TTLs) and M&E specialists who are working on operational and analytical investments that have a forestry component. The indicators discussed here can help to inform the design and implementation of such investments so that they are able to have more positive impacts on the key development goals of eliminating extreme poverty and boosting shared prosperity in a sustainable manner. The future- and results-oriented nature of these indicators also mean that they have particular relevance in the context of REDD+ and the World Bank’s Program-for-Results lending instrument, both of which require credible near-term indication that longer-term changes and results will occur. Ideally, robust predictive proxy indicators would be used during implementation to suggest that a given policy, program, or project is on the right track and is likely to have desired impacts for a significant period of time after the intervention has concluded. This study, based largely on review of indicators used in World Bank forestry projects, suggests that such indicators do in fact exist. We identify a set of theory-based PPIs that contain indicators relevant to one or more overarching development objectives: poverty reduction and economic growth, biodiversity conservation, climate change 12 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS mitigation and adaption, and good governance. The report highlights the particular importance of the World Bank’s Core Sector Indicators (CSI) (World Bank 2014b), which are designed to help track and aggregate project results across the organization. The forestry CSIs are central, but the report finds that CSIs from other sectors such as governance are also key PPIs. These results, described in detail below, provide essential guidance for future analytical work to test and refine potential PPIs in the forest sector. These results and the approach used here also have the potential to inform efforts in other complex development sectors seeking reliable information in the short term on likely longer-term outcomes. This review of the World Bank’s forestry portfolio also finds that project M&E and the use of indicators has improved over time, as indicated by an increased emphasis on M&E and on tracking and evaluating progress on indicators in more recent project Implementation Completion Reports (ICRs) compared with Project Completion Reports (PCRs) or earlier ICRs. This finding suggests a positive trend in attention to indicators across the World Bank’s forest portfolio. In addition, the review showed a positive association between the quality of project M&E and project outcome ratings. As project M&E scores increased by category, project outcome rating scores increased by one-half a rating category. These and other findings are discussed in more detail in the sections below. The next section of this report reviews relevant peer-reviewed and grey literature on PPIs from the forestry and other sectors. It defines predictive proxy indicators and draws lessons from different sectors that may be applied to forestry. Sections three and four form the core of this report. They describe the methods and data sources used to identify predictive proxies and present main findings from the research. In addition to the central focus on evidence from the World Bank’s forest lending portfolio over the past 25 years, this study also draws from indicators used by other key agencies working in the forest sector and gleans insights from experts within the World Bank, other donor and implementing agencies, and the research community. It also includes a brief analysis of the PROFOR portfolio to examine the potential of proxies related to knowledge and analytical work. Sections five and six provide analysis on the opportunities for and constraints on using PPIs and present guidance on how they might be used in practice. The last section summarizes key findings and suggests avenues for taking this work forward. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 13 2. Use of Predictive Proxy Indictors in Forestry and Other Sectors 2.1 Defining Proxy Indicators A proxy is something that can be used to represent another thing. A proxy indicator is thus an indirect measurement of a variable lacking direct information (IEG 2012) or an indicator that substitutes for another indicator that is hard to measure directly (World Bank 2014a). Trust, for instance, is a common proxy indicator of social capital (Morrone, Tontoranelli, and Ranuzzi 2009). In the forest sector, development and implementation of forest management plans have been advanced as a proxy indicator for sustainable forest management outcomes (IEG 2013). Proxy indicators are used when a direct measurement is hampered by complexity, cost, or insufficient length of time for data collection on implementation results. As described earlier, the forest sector is beset by each of these challenges. That forest-related interventions typically do not generate impacts during their lifetimes is a defining feature of work in the sector and a central rationale for this study’s focus on predictive proxy indicators. As described in Box 1, a predictive proxy indicator denotes a measure taken during implementation of a project, program, or policy that stands in for longer-term impact. A predictive proxy thus provides an indication about some future change or state based on currently available evidence. Proxy indicators are used when the effect of a particular intervention cannot be assessed using direct information for reasons that range from limited resources for data collection to a lack of data or timing-related challenges. In such situations, a proxy indicator uses data that can be collected as a substitute for direct data. As the term itself suggests, proxy indicators are therefore almost inevitably approximations. 2.2 Literature Review A review of peer-reviewed and grey literature did not reveal any use of the term predictive proxy indicator in the forest sector. However, PPIs have commonly been used in the fields of education, health care, finance, and management. For example, the level of education is frequently used as a predictor for potential earnings, with primary education showing the highest social profitability in all world regions (Psacharopoulous 1994). In their analysis on the effect of education on earnings, Blundell, Dearden, and Sianesi (2005) found an average return of 27 percent for individuals who completed higher education compared with anything less, although they also caution that the relationship between schooling and earnings is not entirely linear, with varying returns depending on qualification levels. Obesity is another factor that has been shown to affect earnings. For example, Cawley (2004) found that a difference in weight of 64 pounds compared with average weight was associated with an 18 percent difference in wages among white women in the United States, or the equivalent of three years of work experience or one and a half years of education. In the field of management, a body of literature has examined the relationship between firm performance and human resource issues, such as staff turnover. For example, Huselid (1995) highlighted the relationship between firm 14 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS performance and high performance work practices (comprehensive employee recruitment and selection procedures, incentive compensation and performance management systems, employee involvement and training). Using a sample of nearly 1,000 firms in the United States, he illustrated that high performance work practices have an economically and statistically significant impact on employee turnover and productivity and on corporate financial performance. The disparate examples of PPIs just mentioned share in common that they are based on historic correlations from large amounts of data. “Big data” is becoming more prevalent in the field of forestry but it has so far been largely limited to biophysical data; large, comparable, socioeconomic datasets remain few and far between in the forestry sector (FAO 2014), hampering efforts to develop and empirically test predictive proxies. For this reason and others mentioned earlier, predictive proxies have not been developed in the forestry sector—a limitation this report seeks to address. 2.3 Strengths and Limitations of Proxy Indicators Among approaches to shed light on the impacts of complex interventions like those characteristic of the forest sector, predictive proxy indicators have the advantage of being future-oriented and relatively cost-effective. Many other forms of evaluation focus on understanding the impact of past actions and struggle to provide information about likely future trajectories. More rigorous forms of impact evaluation, including the increasingly popular use of ex ante randomized controlled trials (RCTs), can provide useful information to inform implementation but are most often carried out at the end of a given project, program, or policy and provide evidence on outcomes at that point in time. To date, such approaches have focused less on what findings have to say about likely future impacts (Woolcock 2013), which is especially true the longer the period of impact considered. Providing near-term information on longer-term results is a key potential comparative advantage of predictive proxy indicators. Further, in comparison to impact evaluation, which can be quite costly, collection and analysis of data on predictive proxy indicators could be relatively inexpensive. In sum, if backed up by time-tested evidence, predictive proxies could form a powerful tool for understanding longer-term impacts in forestry and other sectors where complexity and long time lags are characteristic. Despite these virtues, PPIs will not be perfect. In addition to the challenge of identifying and testing such indicators, predictive proxies will be based on correlation and cannot fully address the problem of attribution. Correlation—a relationship of concomitant occurrence or co-variation—does suggest that a causal relationship may be present and also can help establish the validity of a given indicator (Scriven 1991), such as a predictive proxy. However, such indicators will be limited in the causal claims they can make, as they do not in and of themselves control for other potentially confounding factors, such as other interventions, economic fluctuation, political change, and climatic and ecosystem variability. A further potential limitation is the need for PPIs to account for the distorting nature of the socio-institutional environment that projects create during implementation but that will be absent when the intervention is completed. As this report argues, however, careful identification and development of potential PPIs can help account for such important contextual factors. The foregoing as well as available literature on indicators related to poverty and environment suggest that identifying a single proxy indicator with robust predictive capacity is highly unlikely in a sector as complex as forestry—let alone in the cross-sectoral work that is increasingly popular in a landscapes framework (Sayer et al. 2013; GEF 2014). For example, Stuart and Collen state “the notion of a single perfect indicator misunderstands what biodiversity is all about. Biodiversity is a complex concept and different indicators reveal different facets that are important…Just as there is a suite of indicators in the economic world (Gross Domestic Product, Dow Jones Index, inflation rate, unemployment rate, etc.), a suite of biodiversity indicators is something that is inevitable from the very nature of biodiversity itself. No one measure…of biodiversity can tell us what is happening to all aspects of what is, by definition, a multifaceted term (Stuart and Collen 2013: 434–35).” The same limitation likely applies to PPIs in the forest sector, biodiversity-related or not. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 15 In the absence of a “silver bullet” predictive indicator, this report proposes that clusters of indicators, when taken together, have greater predictive power than when analyzed in isolation. The “pressure-state-response” framework used to analyze environmental change (OECD 1994) includes a cluster of indicators for each environmental problem that suggest the origin of pressures for each environmental problem, what the state of the environmental attribute or good is, and what the policy or practical response is or could be to reduce pressure. For example, Shyamsundar (2002) used this approach to analyze the relationship between deforestation and poverty, providing a series of indicators, some of which may have predictive potential when considered jointly. In conclusion, predictive proxies are a very promising tool for dealing with the challenge of assessing impact in the forest and other complex development sectors, but they should be seen as a complement to, not a substitute for, other forms of evaluation such as RCTs and rigorous forms of impact evaluation. 16 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 3. Identifying Predictive Proxies in Operational Investments 3.1 Overall Approach This study aimed to conduct an in-depth examination of the World Bank forest portfolio and to examine other relevant data sources to develop guidance on the identification and use of PPIs for project impacts. Figure 1 illustrates the three step approach used to identify predictive proxies: developing criteria for good PPIs, assessing possible indicators for their predictive potential on a scale from 1 to 5, and validating potential PPIs. FIGURE 1: APPROACH TO IDENTIFY PREDICTIVE PROXY INDICATORS 1. Develop criteria for 2. Assess possible 3. Validate good predictive proxies predictive proxies potential proxies Expert Opinion Data on hypothesized proxies at different times (during; end; post-project) Potential PPIs can be validated in at least two ways. The first is based on expert knowledge and theory. This theory-based approach (Weiss 1995, 2000) seeks to identify a plausible logic or theory of change for why a given indicator or set of indicators would have predictive power. Such “applied forward reasoning” (Levin et al. 2012) was developed based on interviews with forestry project managers within the World Bank and other key institutions in the forestry sector and academic experts. Potential predictive proxies were also developed and vetted at a workshop of TTLs and other World Bank staff at the World Bank in January 2015. Discussion at three external meetings—at the Oxford Center for Tropical Forests in June 2014, the Science for Nature and People (SNAP) working group meeting on evidence-based conservation in February 2015, and the Forests and Livelihoods: Assessment, Research, and Engagement network (FLARE) on sustainable forests and livelihoods in March 2015—provided additional insights and validation of predictive proxies. A second way to validate PPIs is to collect data on hypothesized proxies at different points in time (during the project, at the end of project, and post-project) and see if the outcomes measured persist over time and across a large number of interventions. This approach would provide concrete evidence of the association of a given indicator or set of indicators with longer-term outcomes, as has been done in other sectors. However, it is an expensive, time-consuming, and challenging task that would require bringing together information from a variety of different data sources and, in many cases, remeasuring indicators in the post-implementation period. As such, it is beyond the scope of the current review. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 17 3.2 Project Identification This study is rooted in a systematic review of the World Bank’s forest portfolio. Details on the methods used to identify projects included in this review and to collect and analyze data on them are provided in Annex A. The primary evidence base for this study is 80 projects chosen to represent the geographic diversity of World Bank’s forestry portfolio, weighted toward high forest cover countries and/or those of exceptional biodiversity importance (Figure 2). To allow variation in the lag time between project completion and the present, closing dates of selected projects ranged from 1991, the year the Global Environment Facility (GEF) was established, to 2013, the last year for which project evaluation data were available. India (n=16), China (n=10), Mexico (n=6), and Brazil (n=5) had the largest number of projects included in the review, with other countries typically having only one or two projects. FIGURE 2: REGIONAL REPRESENTATION OF WORLD BANK FORESTRY PROJECTS REVIEWED 18 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Projects reviewed account for nearly $6 billion in investments, with project budgets ranging from $0.9 million for the Rural Environment Project (P066199) in Azerbaijan to $1.3 billion for the First Programmatic Development Policy Loan for Sustainable Environmental Management (SEM DPL; P095205) in Brazil. These projects included the range of lending types used by the World Bank, with the majority (76 percent) being Specific Investment Loans (SIL). The majority of projects received a satisfactory rating (see Figure 3). FIGURE 3: PROJECT OUTCOME RATINGS 60 52 50 40 30 20 10 8 8 8 2 0 Highly Satisfactory Satisfactory Moderately Moderately Unsatisfactory Satisfactory Unsatisfactory 3.3 Data Collection Project analysis was structured around a results chain framework linking activities to outputs to outcomes to impacts. (See Box 2.) For each project, data were collected from ICRs1 on the project’s Project Development Objective (PDO) or Global Environment Objective (GEO), project indicators, project design, M&E framework, and performance ratings. Data on indicators include baseline values, original target values, revised target values, actual value achieved, and the timeline for the following types of indicators, where applicable: PDO indicators, GEO indicators, intermediate outcome (IO) indicators, outcome/impact indicators, output indicators, and project or management objectives or activities. Additional project data collected included project approval and closing dates, project financing (including disbursed amounts and co-financing), sector distribution, and environmental category. Information about transition to regular operations or about any succeeding World Bank projects was also noted. 1. ICRs were used for all projects for which they were available. For projects that closed in March 1994 or earlier, PCRs were used. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 19 BOX 2: CLARIFYING LINKS IN THE RESULTS CHAIN The World Bank’s results chain framework is best understood by defining its various components. Results encompass outputs, outcomes, or impacts of a development intervention. A results chain describes how particular inputs will likely lead to intended outcomes. Outputs are defined as products and services provided by a project, such as roads or water connections or revisions to legal frameworks. Outputs may also be described as intermediate results. Outcomes are results that occur after the use of outputs, such as reduced travel times or availability of clean drinking water in a village. Outcomes can also include modified behavior, conditions, or situations as a result of program outputs. Outcomes can be short-term, medium-term, or long-term (5–25 years). An Intermediate Outcome is a result that is proximate to an intended final outcome but often more achievable and measurable during a project’s lifetime than an intended final outcome. Longer-term Outcomes are typically not visible at project closure and may not be apparent until 5–25 years after a project closes. Impacts represent the ultimate result of the outcome, which most often becomes evident several years or more after project completion. Source: World Bank 2014a. To gain an understanding of the main focus of projects included in the review, PDOs were coded as focusing primarily on one of four categories: biodiversity conservation, climate change adaptation and mitigation, good governance, poverty reduction (including the concepts of economic growth and shared prosperity), or equally on biodiversity and poverty (Figure 4). FIGURE 4: DISTRIBUTION OF PROJECTS BY PDO TYPE Biodiversity and Poverty 6% Biodiversity 16% Poverty 13% Intitutions 44% 20 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS This report is also informed by several additional sources of data, including interviews with World Bank project managers, M&E specialists, and other experts within and outside the World Bank. In total, more than 100 people, including about 20 World Bank TTLs, were consulted for information and ideas on PPIs. Indicators used by a range of non-World Bank donors in the forestry sector were also collected and assessed for their proxy potential. Indicators from leading donors in the sector with available results frameworks, such as DFID, KfW, and GIZ, as well as trust funds (for example, FIP and FCPF), and relevant policy organizations (such as FAO, Forest and Farm Facility) were collected from websites and directly from staff. In addition to the core focus on operational investments, this study also included a preliminary analysis of potential predictive proxies related to knowledge work in the forest sector. This analysis was based on a review of PROFOR activities, which collected information on activity indicators and outcomes with the aim of identifying knowledge-related proxies and providing recommendations for advancing broader work on the identification and use of predictive proxies related to forest knowledge. More details on this element of the study, including background, methodology, results, and findings, can be found in Annex B). 3.4 Criteria for Evaluating Quality Indicators and Predictive Proxy Indicators Following the data collection process, all World Bank project indicators were evaluated according to the SMART (specific, measurable, attributable, realistic, and time-bound) criteria, using a Likert scale from 1 to 5. Indicators were also rated for their potential as proxies using the same scale. In the absence of established practice, working criteria for scoring each indicator’s potential as a predictive proxy were developed. To be judged as having potential as a predictive proxy, the indicator had to score highly on the SMART criteria but also meet the following minimum requirement—it implied a plausible theory of change explaining why it is likely to accurately predict a desired future change or state resulting at least in part from a given intervention. A theory of change is a logical description of how a given intervention or change process is expected to lead to longer-term outcomes and impacts. Its distinguishing feature is an explicit articulation of assumptions thought to connect specific steps to achievement of longer-term goals (Schorr and Weiss 1995). For instance, a sustainable financing indicator may be a predictive proxy based on a theory of change that arrangements for funding a given intervention over time imply that the intervention will be implemented even after project closure, with the assumption that it will continue to generate positive impacts. This indicator implies that the necessary funding is secured for a given period of time, that institutional arrangements are in place to allocate the funds, and that the use of the funds is effective. Along with having a plausible theory of change, two additional criteria2 were considered in assessing the PPI potential of each indicator: • Active stakeholder support—The indicator suggests “buy-in” by those whose behavior a given intervention wishes to influence such that desired behavior appears likely to persist after the intervention has finished (for example, incentives exist for a given action or behavior independent of project funding). Such indicators may relate to the strength of forest user groups, social capital or cohesion, and various forms of participation, among others. 2. A more comprehensive set of criteria for identifying PPIs was developed subsequent to initial coding as described here. The FOREST criteria were developed based on this initial effort and discussion of results in various forums. See section 6. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 21 • Change in behavior or capacity—The indicator measures a change in capacity to implement actions related to a given intervention or a change in behavior to support the intervention’s actions. Examples include an indicator measuring an increase in capacity to carry out forest management by a forest department or forest user group or one that measures a reduction of community reliance on resources inside a protected area. The highest possible score for a given indicator was 30 (25 based on the SMART criteria and 5 based on the PPI criteria). All indicators were coded into one of five categories based on their overall long-term outcome: poverty alleviation (including economic growth and shared prosperity), biodiversity conservation, climate change mitigation, governance, and other. Annex A provides further details on the assessment of indicators used in the World Bank’s forestry portfolio. 3.5 Validation of Indicators and Development of an Indicator Menu The primary means of validating potential PPIs was through expert opinion gathered through interviews, presentation of results to external audiences, and a workshop with World Bank TTLs and others. Results are discussed in sections 5 and 6. Based on the portfolio review and expert input, a list of top scoring indicators was created. The purpose of this indicator menu is to provide an easy to use resource that can inform the design and implementation of forestry operations. Indicators are grouped according to their main development objective. The menu includes basic information on indicator type, the context in which it has been used, and notes on how it might be used moving forward. This indicator menu is presented as Annex C and represents a key product stemming from this analytical work. 22 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 4. Results This section provides a general overview of results from the portfolio review and indicator analysis, with a particular focus on how indicators used in World Bank projects were evaluated and how this process contributed to the identification of predictive proxies. The section then proposes potential PPIs, including illustrative results chains for three of the proposed cluster indicators. Finally, it highlights the importance of M&E in World Bank projects and underscores several key lessons learned on M&E from the project review. 4.1 Overall Analysis of Indicators Projects included in the review used a range of indicator types, in line with World Bank trends at the time of project appraisal and implementation. For instance, the most recent projects typically include PDO and/or GEO indicators and IO indicators, while older projects used either impact/outcome indicators, output indicators, or a combination of indicators. The evaluation of indicators in this review primarily focused on PDO and GEO indicators, based on the assumption that analysis and recommendations on these would be most broadly relevant for World Bank TTLs and other project designers. IO indicators were also evaluated based on the SMART criteria to facilitate potential future exploration of their utility as predictive proxies. The analysis underscored the importance of evaluating indicators with targets. When indicators were considered without targets, they tended to score lower on all the SMART criteria. An example from the Madhya Pradesh Forestry Project in India (P010506) illustrates how the inclusion of targets can make an indicator more specific and measurable. The indicator—increased management effectiveness in project Natural Protected Areas—lacks specificity on its own because it simply suggests an increase without providing a unit of measurement and uses the term “effectiveness.” But with the target of two new and four updated Natural Protected Area management plans, five Protected Area Management Committees in operation in Project Natural Protected Areas, and three Natural Protected Areas administered by private nonprofit organizations by December 2004, the indicator becomes more specific and therefore more measurable. Although some indicators addressed too many issues together, others group several issues together in a way that makes each component seem more like steps of a whole. One such example is from the same project—increase forest cover and productivity through the development of participatory processes for management and use of forest resources, taking special account of the interests of tribals and other disadvantaged groups. The indicator is: Joint Forest Management (JFM) approach established; Participatory Rural Appraisal–based microplanning methods established; silvicultural practices adapted to multiple objectives of JFM; restoration techniques for degraded areas based on natural regeneration tested; tribal interests in planning measured; and interests of other disadvantaged groups, scheduled castes and women, and the landless measured. This indicator, while quite lengthy, includes details that show how the project envisions increasing forest cover and productivity. The number of PDO indicators included in projects ranged from 1 to 26, with an average of 6 PDO indicators per project. Eleven projects included a GEO indicator, with a range from 1 to 8 indicators per project. Four projects included 1 GEO indicator. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 23 Figure 5 shows the distribution of PDO indicators according to the SMART and PPI criteria. The majority of the indicators (n=80, or 45 percent) received scores between 21 and 23, with only 8 percent receiving the top range of scores. FIGURE 5: PDO INDICATOR SCORES 14-17-4% 27-30-8% 18-20-19% 24-26-24% 21-23-45% The four highest-scoring PDO indicators each received a score of 29: • Increased capacity to finance SINANPE recurrent costs with local resources, from the Peruvian National Trust Fund for PAs–Programme for a Participatory Management of PAs Project (P068250) in Peru, which was rated satisfactory • Tenure of at least 30 percent of the occupants in national lands in the project area regularized through formal, long- term usufruct agreements or title developed with the assistance of the project, from the Forests and Rural Productivity Project (P064914) in Honduras, which was rated moderately satisfactory • Generation of 13.3 million cubic meters of timber and 2.73 million tons of bamboo by December 31, 2025, with RMB 1.1 billion net income from fruit tree crops by year 2022, from the Sustainable Forestry Development Project (P064729) and Sustainable Forestry Development Project (Natural Forest Protection; P060029) in China, which was rated satisfactory • A total of 60,000 terajoules per year produced by renewable energy sources or saved by energy efficiency projects supported by BNDES, once they are fully operational, from the First Programmatic Development Policy Loan for Sustainable Environmental Management (SEM DPL; P095205) in Brazil, which was rated satisfactory Figure 6 shows the distribution of GEO indicator scores. The majority of the indicators (n=9, or 43 percent) received scores between 24 and 26. 24 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS FIGURE 6: GEO INDICATORS’ SCORES 27-30-14% 18-20-19% 21-23-24% 24-26-43% The two highest-scoring GEO indicators received a score of 30: • At least 80 percent of the conservation activities proposed for each protected area designed with a participatory approach, from the Consolidation of the Protected Areas System Project (SINAP II; P065988) in Mexico, which received a satisfactory rating • Trends in the rate of habitat conversion in protected areas included in the Project, also from the SINAP II project in Mexico The highest scoring PDO and GEO indicators are all from projects that closed in 2010, which lends some anecdotal support to the general finding that M&E, as well as indicators, have improved over time. Annex A highlights additional detail and examples of indicators that scored high on the SMART criteria and illustrates limitations of some indicators. It is important to note that while scoring indicators based on the SMART criteria was a useful way to rank and discuss existing indicators, good M&E practice suggests that all five criteria are minimum criteria. 4.2 Identification of Predictive Proxies from the World Bank Forestry Portfolio There are 176 PDO indicators that were rated for their potential as proxy indicators. Eleven indicators (6 percent) were rated as highly recommended (score=5). Fifty-five indicators (31 percent) were recommended (score=4). Eighty- four (48 percent) were rated as recommended with some caveats (score=3). The remaining 27 indicators scored are not recommended. Top-scoring potential PPI indicators are included in the indicator menu in Annex C. Twenty-one GEO Indicators were rated for their potential as proxy indicators. Out of these, nine (41 percent) were rated as highly recommended and are also included in the indicator menu. Like overall coding of PDOs and GEOs, indicators were also coded in terms of their long-term outcome: poverty alleviation (including economic growth and shared prosperity), biodiversity conservation, climate change adaptation and mitigation, good governance, and others. Ten indicators were classified as other because they did not address any of UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 25 the four main themes, either because they focused on issues such as pollution or water or because the indicator lacked sufficient specificity to be classified. The majority of PDO indicators (n=94, or 53 percent) were classified as addressing governance. Forty-four indicators (26 percent) focused on poverty, while 22, or 12 percent, focused on biodiversity. Only 6 indicators (3 percent) focused on climate change. It is also important to note that several indicators could have been coded under multiple themes. For instance, the following three were coded as governance indicators but also address biodiversity concerns: • Raise public awareness of biodiversity values and increase participation in biodiversity (Central Asia Biodiversity GEF Project; P042573) • Six natural forest management areas under effective management (Sustainable Forestry Development Project (P064729) and Sustainable Forestry Development Project (Natural Forest Protection; P060029) in China) • More effective and extensive support for Conservation and Development (India Ecodevelopment Project; P036062) The majority of GEO indicators focused on biodiversity (n=10; 48 percent). Governance was the second highest area of focus for GEO indicators, with nine indicators, or 43 percent of indicators. One indicator focused on poverty and no indicators focused on climate. In addition, one indicator was classified as other. 4.3 Potential Predictive Proxy Indicators PPIs identified through the World Bank forestry portfolio review, the inventory of indicators used by other key donors in the forestry sector, and expert views are described below. This process did not yield standalone predictive proxies in most cases, but it did result in the identification of clusters of two to four indicators that, taken together, were determined to have a strong predictive potential. The description of each indicator or indicator cluster includes the following: • Anticipated outcome(s) • Description of the theory of change detailing how a given intervention is expected to lead to longer-term outcomes and impacts • Rationale for including each indicator as part of the cluster to capture and track the anticipated outcome • Brief guidance notes on practical use of the indicator cluster Following each proposed indicator cluster, indicators that have been used in previous World Bank or other projects are included for reference. Illustrative results chains showing where different indicators in several PPIs might be placed are included for some of the proposed indicators in this section. The PPIs are demarcated in bold, italic font. 26 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS POVERTY Indicator Cluster 1: Sustainable Forest-Related Income a) people in targeted forest and adjacent communities with increased monetary or nonmonetary benefits from forests + b) people in targeted forest and adjacent communities have secure access and use rights + c) forest activities are aligned with biodiversity-friendly management practices = sustainable forest-related income Theory of Change: This indicator cluster is designed to be adaptable based on the time period of interest. All three indicators are likely needed to ensure that forest-related income not only increases but also that it continues to flow over the longer term based on SFM practices. The first two indicators in the cluster (a and b) may be have some predictive power on their own, but without some indication that benefits are derived in a sustainable manner, benefits will not persist over the longer term. Thus, the inclusion of indicator c in the cluster strengthens inference about whether monetary and nonmonetary benefits will continue to be delivered over a longer time period. The first indicator in the cluster measures the extent to which people in the project area have gained monetary benefits or nonmonetary benefits from forests, such as improved access to fuelwood, income from sale of forest products, or cultural and spiritual services. One implication of this indicator is that local people may be more likely to support SFM if they receive benefits from forests, either through increased income, such as through employment in the forest sector, or through direct benefits, such as harvesting non-timber forest products (NTFPs) for consumption or sale or through payments for environmental services (PES) schemes. Although this benefit stream would likely help shift incentives toward SFM so that benefits can continue to be delivered, this indicator is insufficient on its own to ensure sustainable forest-related incomes. Additional indicators are needed to help track whether harvest and use of forest resources is sustainable over time. Two complementary indicators have been added based on the assumption that forest products (wood and non-wood) are more likely to be harvested at sustainable levels if people know that their access to and benefits from forest resources are formally recognized and secure. The third indicator addresses the importance of ensuring sustainable income flows over a longer period of time, in recognition that deriving benefits and having the right to continue deriving them is not enough to ensure sustainable income flows. An additional indicator is required to provide information on the likely ecological sustainability of efforts to extract forest benefits. This last indicator aims to capture whether or not forest use is sustainable by specifying that activities should follow “biodiversity-friendly” management practices. As described below, this indicator is a World Bank CSI, which while desirable in many cases is not the only possible indicator of sustainable forest use. Other indicators that capture ecological sustainability may be substituted here. The importance of including indicators on access and use rights and aligning forest activities with biodiversity-friendly management practices is elaborated upon in a large literature on common property. This literature supports the assertion that individuals will invest in maintaining forest products, NTFPs, and other resources at sustainable levels if they have secure resource rights, including the right to access the resource and rights to establish rules and norms to control overuse of the resource (e.g., Ostrom and Schlager 1996; Mendelsohn 1994). More recent reviews (e.g., Lawry et al. 2014; Kishor et al. forthcoming) further support the economic theory that long-term investment and increased tenure security form plausible pathways through which recognition of property rights can improve the welfare of those who receive title and such recognition (formal or informal). UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 27 FIGURE 7: RESULTS CHAIN FOR INDICATOR CLUSTER 1: SUSTAINABLE FOREST-RELATED INCOME Intermediate Activities/Outputs Outcomes/Indicators Long-term Outcomes Outcomes/Indicators A. Develop income earning A. Increased producer and A. Increased benefits from activities. investor access to markets, forests. • Persons trained in vocational/ value-adding and economic People in targeted forest and business development skills diversification opportunities adjacent communities with (#) • Targeted population starting a increased monetary or non- • New NTFPs processing micro-enterprise (%) monetary benefits from forests technologies (#) • Survival of micro-enterprise xx • Producer organizations months/years after start (%) established (#) • Employment increased in wood production and processing (number) A. Develop infrastructure to • Timber forest products support forest production and developed (#) marketing. • Marketable non-timber forest • Roads constructed, products (#) rural (km) • Markets for ecological • Markets constructed/ services developed (carbon rehabilitated (#) sequestration; water management; conservation) • Share of total timber harvested processed in-country (%) • Share of total NTFPs harvested processed in country (%) • New income-generating activities based on processing of forest products at local level Increased contribution (number) to economic growth and poverty alleviation B. Enhanced rights to forest B. Enhanced rights to forest and tree resources and forest and tree resources and forest products and services products and services. • Forest area with documented People in targeted forest and and gazette tenure rights (ha) adjacent communities have • Timber species with secure access and use rights documented harvest rights (number, %) • Non-timber forest products with documented harvest rights (type, number, %) • Documented rights to land C. Increased forest user C Forest activities are adoption of sustainable forest sustainable. management practices. Forest activities are aligned with • Producers adopting sustainable biodiversity-friendly management forest management practices practices (#, % of targeted households) • Area with improved forest management techniques (ha) 28 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Guidance Notes: Developing a results chain is one of the first steps in considering how the proposed cluster indicator can be used in projects. Figure 7 presents an example of where the proposed cluster indicators fall in the results chain, along with illustrative examples of activities and outputs. For this cluster indicator on sustainable forest-related income, the proposed PPIs are all outcome indicators and are demarcated in bold, italic font. The first indicator is a World Bank forestry CSI. Further information on the use of this indicator can be found at http:// siteresources.worldbank.org/PROJECTS/Resources/40940-1367867968385/CoreSectorIndicatorsList.pdf. The second indicator addresses security of tenure and rights over time as a result of the project and measures the recognition of use rights through forest agreements, management plans, titles, or other formalized use rights. Use or ownership rights cover the full continuum of land tenure situations, customary or statutory, individual or collective, on private or public lands, and can accommodate any land tenure system, in line with the CSI definition. Further information on the World Bank CSI on land administration and management indicators can be found at http://siteresources.worldbank. org/PROJECTS/Resources/40940-1367867968385/CoreSectorIndicatorsList.pdf . The third indicator aims to ensure that forest benefits do not exceed the sustainable yield. For instance, the indicator should show that benefits from logging operations are carried out based on a sustainable management plan or that collection and harvesting of NTFPs is at sustainable levels. The term biodiversity-friendly comes from the World Bank’s biodiversity CSI, “new areas outside protected areas managed as biodiversity-friendly (ha),” (http://siteresources.worldbank.org/PROJECTS/Resources/40940-1367867968385/ CoreSectorIndicatorsList.pdf) and refers to compliance with social and environmental standards in a way that respects civil and indigenous rights, maintains or enhances social and environmental conservation values, prohibits invasive planting, and ensures that harvesting meets national laws and international treaties on biodiversity signed by the country in which the project is located. As described in Box 3, this PPI cluster is being tested in a recently approved World Bank project, the first such empirical test of the predictive potential of indicators. See Table 1 for specific examples of indicators relevant to this predictive proxy. BOX 3: TESTING PREDICTIVE PROXY INDICATORS IN THE ARGENTINA FORESTS AND COMMUNITY PROJECT The Argentina Forests and Community Project (P132846), approved in April 2015, has provided a timely opportunity to begin testing some of the predictive proxy indicators identified in this report. The project seeks to improve forest management and increase access to markets and basic services by small forest producers (including indigenous people and campesinos) in the comparatively poor yet forest-rich areas in the country’s northern provinces. Two-thirds of Argentina’s remaining natural forests are in the Chaco region, but they are increasingly threatened by high deforestation rates. At the same time, more than 70 percent of the population in this remote region lives below the poverty line, and forest loss threatens to deepen levels of impoverishment. To address the rapid loss of natural forests, the government of Argentina created a Forest Law in 2007, which includes a Forest Fund designed to protect natural forests by supporting conservation, restoration, and sustainable management of natural forests and PES. The Fund has totaled more than $50 million in recent years, but to date less than 4 percent of eligible indigenous or poor criollo communities have accessed it. Financing from the Forest Fund supports implementation of approved SFM plans. This World Bank project will support efforts by the government to increase access to the Forest Fund, including by helping communities develop SFM plans and strengthen their tenure. These activities and the project’s focus relate to all three indicators identified in PPI cluster 1 on sustainable forest-related income. It includes the CSI on people in targeted forest and adjacent communities with increased benefits from forests as well as two tailored indicators relating to sustainable financing and tenure security. The Argentina project team thus saw an opportunity to test these indicators for their predictive potential. At the PDO level, the project results framework includes “Increased share of Forest Fund resources allocated to small forest producers” and at the intermediate outcome level it includes “Forest area brought under strengthened tenure.” The project will collect data on these three indicators during implementation and explore the prospect of continued data collection and analysis in the post-project period to provide the first forward-looking test case of how well the indicators performed as predictive proxies. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 29 TABLE 1: EXAMPLES OF INDICATORS RELEVANT TO SUSTAINABLE FOREST-RELATED INCOME PREDICTIVE PROXY General Indicator from PPI cluster Specific example indicators Source a) Monetary and nonmonetary • People in targeted forest and adjacent communities with increased World Bank CSI benefits from forests monetary or nonmonetary benefits from forests • People employed in production and processing of forest products World Bank CSI • Changes in income in forest communities over time FIP • Number of direct jobs created as a result of International Climate DFID Fund (ICF) support • Number of forest-dependent people with livelihoods benefits DFID protected or improved as a result of ICF support • Level of diversity of income-generation activities FAO-FFF • Permanent jobs created through small and medium-size enterprise FAO-FFF (SME) productive activities • Income generated from forest services for forest-dependent people GEF and communities • Jobs created through the SME productive activities GIZ • Annual incremental revenue to villages P046768 in Senegal • Average forest-based product income (cash and kind at 2009 P073094 in India real prices) realized by Vana Samarakshana Committee members resulting from improved forest productivity • Number of jobs created from project investments P064914 in Honduras • 20 percent increase in net value of forest goods and services P035751 in Mexico produced by assisted communities and ejidos • 30 percent increase in jobs available in assisted communities vs. P035751 in Mexico control, from the Community Forestry II • Decline in seasonal outmigration for employment P073094 in India • Poverty reduction: per capita income of project beneficiaries P046952 in China increased by specified percentage b) Tenure and property rights • Percentage of indigenous peoples and local community members/ FIP forest communities (women and men) with legally recognized tenure rights and secure access to economic benefits and/or the means of maintaining traditional livelihoods • Increase in land and resources under legal control and FIP management of indigenous peoples and local communities, including through traditional forest management systems • Increase in area with clear, recognized tenure of land and resources FIP for indigenous peoples and local communities • Legal rights granted by competent authorities to have access to GIZ forest resources c) Sustainability of forest • Improved natural resource management practices FIP activities • The SME has identified and delimited special areas (water bodies, flooding, soil, endangered species, areas of value) within the forest GIZ management plan • The SME applies CITES or IUCN or national lists of endangered GIZ species to avoid the harvesting or collection of endangered species 30 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Indicator Cluster 2: Afforestation/Reforestation to Support Livelihoods Improvement a) seedling survival rate after three years + b) thinning of seedlings/young trees after three to five years = Income from forest over the longer term Theory of Change: This cluster indicator uses intermediate indicators with strong proxy potential for predicting potential income from forests over the long term. The first indicator measures the survival rate of planted seedlings after three years. This addresses the importance of ensuring that planted seedlings survive beyond the life of the project, with the potential to develop into productive young trees and, eventually, into a productive forest plantation that yields benefits for targeted beneficiaries, such as local populations or governments. Additional indicators, however, are needed to help track whether the seedlings will survive and contribute to income from forests in the future. A complementary indicator has therefore been added based on the assumption that productive forest plantations require management, such as the thinning and maintenance of seedlings and young trees over time, to ensure that the seedlings develop into productive forest plantations. Pre-commercial thinning is a technique used to improve the health and quality of seedlings (Smith et al. 1997). It is expected that the combination of these two indicators will help to promote afforestation or reforestation efforts with the aim to contribute to poverty eradication and boost shared prosperity in project areas over the long term. Although not included in the cluster, this indicator may require additional indicators related to the ecological appropriateness of afforestation and/or reforestation efforts, secure property rights, and sustainable financing. Secure property rights, for instance, could help to motivate individuals to invest in seedling survival over the long term. However, ample seedling survival rate or presence of thinning activities may themselves imply that property rights are secure, thus obviating the need for a separate indicator on property rights. Indeed, it may be that seedling survival rate in a given area is sufficient as a stand-alone PPI given that it may imply secure property rights and active management. This indicator cluster is the only one in this report that uses IO indicators. Empirically testing whether these two indicators predict sustainable forest-based income would be useful to better understand whether these indicators are sufficient on their own or need additional indicators to have sufficient predictive power.   UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 31 FIGURE 8: RESULTS CHAIN FOR INDICATOR CLUSTER 2: AFFORESTATION/REFORESTATION TO SUPPORT LIVELIHOODS IMPROVEMENT Intermediate Activities/Outputs Outcomes/Indicators Long-term Outcomes Outcomes/Indicators A and B. Develop income A. Seedling Survival Rate after A and B. Increased benefits earning activities. 3 years from forests. • Persons trained in forest People in targeted forest and processing technologies (#) adjacent communities with • New timber value chains increased monetary or non- developed (#) monetary benefits from forests • Producer organizations Income from Forests over A and B. Develop B. Thinning of Young Seedlings/ A and B. Strengthen provision the Long-term infrastructure to support Trees after 3-4 years of forest management advisory forest production and services. – marketing. • Advisory personnel training in • Roads constructed, rural (km) [topic] (#) • Markets constructed/ • Community members trained rehabilitated (#) in [topic] (#) • Forest User Groups formed (#) Guidance Notes: Developing a results chain is one of the first steps in considering how a proposed cluster indicator can be used in projects. Figure 8 presents an example of where the proposed cluster indicators fall in the results chain, along with illustrative examples of activities and outputs. For this cluster indicator on afforestation/reforestation to support livelihoods improvement, the proposed PPIs are both IO indicators and are demarcated in bold and italic font. The first indicator measures seedling survival rate after three years, using the total number of surviving seedlings as a unit of measurement. The second indicator measures whether or not thinning of seedlings or young trees occurs after three to five years. The time frame for thinning will depend on the species of trees selected and its geographic location. See Table 2 for specific examples of indicators relevant to this predictive proxy. 32 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS TABLE 2: EXAMPLES OF INDICATORS RELEVANT TO AFFORESTATION/REFORESTATION TO SUPPORT LIVELIHOODS IMPROVEMENT PREDICTIVE PROXY General Indicator from PPI cluster Specific example indicators Source a) Seedling survival rate after • Replanting (ha): Upkeep of about 55,000 ha of existing P001168 in Cote d’Ivoire three years forest plantations and the establishment of 9,000 ha of new plantations to complete the planting program of the 1st and 2nd Forestry Projects • Adapt and improve technologies and provide technical P010506 in India advice, including staff acceptance of improved planting stock and nursery technologies; number of improved seedlings to private farmers • Seedlings produced P003287 in Zimbabwe • Increase in forest area covered by improved forest and P053830 in the Russian Federation pest management b) Thinning of seedlings/young • Adapt and improve technologies and provide technical P010506 in India trees after three to five years advice, including staff acceptance of improved planting stock and nursery technologies; number of improved seedlings to private farmers • Increase in forest area covered by improved forest and P053830 in the Russian Federation pest management BIODIVERSITY Indicator Cluster 3: Positive Environmental Impacts (Biodiversity-Related) a) forest area brought under adaptive, biodiversity-friendly management plans + b) predictable, sustainable financing + c) functioning institutions to enforce rules and resolve conflict = Positive environmental impacts (biodiversity-related) Theory of Change: The first indicator, a CSI on forest area brought under management plans, measures the forestland area that as a result of a Bank project has been brought under a management plan that has been prepared, endorsed, and is in the process of implementation. To ensure that activities under the management plan are biodiversity-friendly (including both socially and ecologically appropriate, see below) and to allow for adaptive management in the event that a particular activity or objective needs to be revised, the indicator aims to measure the forest area under adaptive, biodiversity-friendly management plans; consequently, these two terms have been added to the CSI. To ensure sufficient financial resources to implement and enforce the management plan over time, two cluster indicators have been added that address sustainable financing and the existence of institutions to ensure enforcement of the management plan. This combination of indicators has the potential to yield increased carbon stocks that will be sustained, supported, and enforced over time. The inclusion of an indicator on institutions is supported by a wide literature on the importance of institutions. For example, common property literature also highlights examples where local communities have successfully developed institutional arrangements to manage their natural resources (e.g., Ostrom 1990; Agrawal 2001). UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 33 Guidance Notes: Together, this cluster of three indicators aims to provide a predictive proxy for longer-term environmental outcomes. The first indicator, forest area brought under management plans, includes production and protection forests as well as other forests under sustainable management. See http://siteresources.worldbank.org/PROJECTS/ Resources/40940-1367867968385/CoreSectorIndicatorsList.pdf A “biodiversity-friendly” management plan complies with social and environmental standards, as defined by a biodiversity CSI. The term aims to ensure standards that respect civil and indigenous rights, maintain or enhance social and environmental conservation values, prohibit invasive planting, and ensure that harvesting meets national laws and international treaties on biodiversity signed by the country in which the project is located. The sustainable financing indicator can either be the one proposed below or another financing indicator that is appropriate for the project situation. The third indicator on governance institutions aims to ensure that the management plan is implemented and enforced. For instance, if the management plan stipulates that no logging is allowed in the area, an institution would need to be in place to ensure that this rule was followed and that any violators were prosecuted. See Table 3 for specific examples of indicators relevant to this predictive proxy. TABLE 3: EXAMPLES OF INDICATORS RELEVANT TO POSITIVE ENVIRONMENTAL IMPACTS (BIODIVERSITY- RELATED) PREDICTIVE PROXY General Indicator from PPI cluster Specific example indicators Source a) Forest area brought under • Improved natural resource management practices FIP adaptive, biodiversity-friendly • Land area under effective forest management practices GEF management plans • Number of relevant principles for sustainable forest and farm FAO-FFF management mainstreamed into national policies and planning • Area of forests on Tanzania Mainland managed according to P057234, P058706 in Tanzania approved forest management plans (including CBFM and JFM) • Area under sustainable natural resource management, from P046768 in Senegal the Sustainable and Participatory Energy Management Project • Hectares of forests brought under participatory management P049395 in Ethiopia • Increased forest cover and productivity through development P010506 in India of participatory processes for management and use of forest resources, taking special account of the interests of tribals and other disadvantaged groups b) Predictable, sustainable • See examples under a separate entry below for a financing “predictable, sustainable financing” predictive indicator c) Functioning institutions to • Government institutions provided with capacity building CSI enforce rules and resolve conflict support to improve management of forest resources (number) • Improved access to effective justice/recourse mechanisms FIP • Evidence that infractions in the forest sector are detected, FIP reported, and penalized • Evidence that laws and regulations in project/programs FIP are being implemented, monitored, and enforced and that violations are detected, reported, and prosecuted • Number of networks, alliances, and federations formed and FAO-FFF active (number of female and male members) • Perception of representativeness among members (females, FAO-FFF males, youth, indigenous peoples, marginalized groups) • Number of men and women from producer groups who hold a FAO-FFF decision making position in relevant policy making processes • Capacity for sustainable forest management improved in state P067367 in Romania institutions responsible for forest management and among underserved private forest owners   34 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS CLIMATE CHANGE Indicator Cluster 4: Positive Environmental Impacts (Climate Change–Related) a) forest area brought under adaptive, biodiversity-friendly management plans + b) indicator of sustainable financing to implement plans + c) functioning institutions to enforce rules and resolve conflict = positive environmental impacts (climate change-related) Theory of Change: This indicator is similar to the last one but is oriented toward positive climate change effects, such as carbon storage or other positive mitigation or adaptation impacts as a result of forest interventions. The indicator could be used to measure SFM, increased carbon stocks, increased adaptive management, or biodiversity results, depending on the project and management plan focus. To ensure sufficient financial resources to implement and enforce the management plan over time, two cluster indicators have been added that address sustainable financing and the existence of institutions to ensure enforcement of the management plan. This combination of indicators has the potential to yield increased carbon stocks that will be sustained, supported, and enforced over time. FIGURE 9: RESULTS CHAIN FOR INDICATOR CLUSTER 4: POSITIVE ENVIRONMENTAL IMPACTS (CLIMATE CHANGE–RELATED) Intermediate Activities/Outputs Outcomes/Indicators Long-term Outcomes Outcomes/Indicators A. Forest activities are sustainable. Forest area brought under adaptive, biodiversity-friendly management plan B. Increased financing for SFM. B. Financing is predictable • Amount of payment for and sustainable. ecological services (carbon Predictable, sustainable financing Positive environmental sequestration; watershed exists impacts (climate-change management; conservation related) areas) ($) • area under SFM • Private investment in • trends in carbon stock forest-related activities by • increased adaptive type (tourism, environmental management services, logging) ($) C) Institutional Activities/ C) Institutions enforce rules Inputs and resolves conflicts • Forest User Groups formed Functioning institutions present to (number) enforce rules and resolve conflict • Forest extension service established UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 35 Guidance Notes: Developing a results chain is one of the first steps in considering how the proposed cluster indicator can be used in projects. Figure 9 presents an example of where the proposed cluster indicators fall in the results chain, along with illustrative examples of activities and outputs. For this cluster indicator on positive environmental impacts (climate change–related), the proposed PPIs are all outcome indicators and are demarcated in bold, italic font. Please see Indicator Cluster 3 for guidance on and examples of relevant indicators. Indicator Cluster 5: Increased Carbon Stocks a) forest area brought under adaptive, biodiversity-friendly management plans + b) area restored or reforested (ha) + c) predictable, sustainable financing = Increased carbon stocks Theory of Change: This cluster of indicators aims to measure increased carbon stocks. The first indicator, a CSI on forest area brought under management plans, measures the forest land area that, as a result of a Bank project, has been brought under a management plan that has been prepared, endorsed, and is in the process of implementation. To ensure that activities under the management plan are biodiversity-friendly (including both socially and ecologically appropriate, see below) and to allow for adaptive management in the event that a particular activity or objective needs to be revised, the indicator aims to measure the forest area under adaptive, biodiversity friendly management plans; consequently, these two terms have been added to the CSI. To ensure sufficient financial resources to implement and enforce the management plan and support afforestation/ reforestation efforts over time, an indicator has been added that addresses sustainable financing, with the aim of ensuring that efforts to increase carbon stocks will be sustained, supported, and enforced over time. Guidance Notes: Please see Indicator Cluster 3 for guidance on the individual indicators and examples of relevant indicators. GOVERNANCE Indicator Cluster 6: Participatory Project Design and Implementation a) proportion of citizens who consider that design and/or implementation of project subjected to consultation is responsive to their views (percent) + b) changes to project activities as a result of consultations (yes/no) and/or c) grievances registered related to delivery of project benefits that are actually addressed (percent) = project gains likely to persist 36 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Theory of Change: This cluster of indicators aims to ensure that users are engaged in the project and help contribute to its improved results, in line with the CSIs on participation and civic engagement, with the ultimate aim of ensuring that project gains and results continue and persist over time. The cluster of indicators is based on a theory of change that project beneficiaries or people whose behavior a project seeks to change are more likely to perceive the project intervention as benefiting them or to continue to change their behavior if they have a voice in the project design and implementation through some form of consultation or feedback. It is also important to include target groups in project design and implementation, because they are the ones who will ultimately have responsibility for sustaining project activities or changed behaviors in the future. Guidance Notes: The first and third indicators are CSIs, and all three indicators are included as examples in the World Bank’s Results Framework and M&E Guidance Note (World Bank 2014a) and reflect a two-way interaction between target beneficiaries and project staff or other relevant actors. The first two indicators measure consultation: the first one aims to monitor the degree of involvement citizens have in the design and implementation of projects while the second captures whether there is a tangible response to citizen feedback. For the first indicator, the CSI guidance notes that different projects will use different mechanisms to engage communities in project implementation and that the indicator will adopt the project definition of community consultation activities and provide a simple count of how many men and women have participated in a range of consultation activities over a certain time frame, reporting on the highest number of participants in each community and measuring the period between project effectiveness and project completion mechanisms. For the third indicator, CSI guidance (World Bank 2014b) explains that the indicator measures the transparency and accountability mechanisms established by the project so the target beneficiaries have trust in the process and are willing to participate and feel that their grievances are attended to promptly, although it is understood that grievance or redress mechanisms will not be established in all projects. Importantly, each of the above indicators are an option for meeting the World Bank’s new goal to include beneficiary feedback in all projects where beneficiaries are clearly identified. This goal aims for all projects to measure and report on at least one indicative citizen engagement indicator in the project results framework (see World Bank 2014a). The portfolio review yielded a number of indicators related to participatory project design and implementation, though many of these examples from past forest projects need some revision or modification before they could be recommended for use as indicators in current and future projects. (See Table 4 for specific examples of indicators.) For instance, the following two examples aim to ensure increased participation but represent outcome statements, rather than indicators, even though the projects used them as indicators: • Increased participation of women in planning, managing, monitoring, and evaluating Project Natural Protected Areas, from the Peruvian National Trust Fund for PAs–Programme for a Participatory Management of PAs Project (P068250) in Peru • Increase forest cover and productivity through development of participatory processes for management and use of forest resources, taking special account of the interests of tribals and other disadvantaged groups, from the Madhya Pradesh Forestry Project (P010506) in India UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 37 TABLE 4: EXAMPLES OF INDICATORS RELEVANT TO PARTICIPATORY PROJECT DESIGN AND IMPLEMENTATION PREDICTIVE PROXY General Indicator from PPI cluster Specific example indicators Source a) proportion of citizens who • Number of men and women from producer groups who hold a decision FAO-FFF consider that design and/ making position in relevant policy making processes or implementation of project • Perception of representativeness among members (females, males, FAO-FFF subjected to consultation is youth, indigenous peoples, marginalized groups) responsive to their views (percent) • Percent of Forest Fund projects supporting indigenous and criollo P132846 in Argentina smallholders and forest-dependent communities b) changes to project activities as • No indicators were found through the portfolio review or indicators a result of consultations (yes/no) inventory. c) grievances registered related to • Improved access to effective justice/recourse mechanisms FIP delivery of project benefits that are • Evidence that infractions in the forest sector are detected, reported, FIP actually addressed (percent) and penalized • Evidence that laws and regulations in project/programs are being FIP implemented, monitored, and enforced and that violations are detected, reported, and prosecuted Indicator Cluster 7: Effective Project Monitoring and Evaluation a) government officials and relevant project staff provided with capacity building support to improve management practices to design and implement M&E + b) government officials and project staff regularly conduct high-quality M&E + c) predictable, sustainable financing for M&E activities (including after project completion) = Positive longer-term project ecological and economic outcomes Theory of Change: This indicator is based on the assumption that a project with a strong M&E component is more likely to result in positive project results over time. Our review of the World Bank forestry projects showed a positive association between the quality of project M&E and project outcome ratings. As project M&E scores increased, the project outcome rating score increased by nearly one-half a rating category (n=59; p=0.0). Thus, for example, a project with a highly satisfactory M&E component was more likely to have a highly satisfactory outcome rating than a project with only a satisfactory M&E. This finding further suggests that M&E represents a potentially large return on investment given that M&E elements of projects are typically less than 5 percent of overall project budgets.3 Similarly, the World Bank’s Annual Review of Development Effectiveness found a “positive correlation between the quality of project-level M&E and better project outcomes” (World Bank 2009: 107). This finding on the positive association between quality M&E and improved project outcomes finds additional support in literature on the importance of M&E in shaping intervention outcomes and impacts (e.g., Stem et al. 2005). Our review also underscored the importance of ensuring that relevant client counterparts and/or project staff have the capacity to design and implement M&E systems. Consequently, this cluster indicator combines capacity building support for M&E with an indicator on regularly conducting M&E. Further, because M&E requires financial and human resources, 3. We tested this association using Stata and found that the project outcome rating increased by 0.45 units for each 1 unit increase in M&E. The 2009 Annual Review of Development Effectiveness found a 0.60 correlation between the quality of project-level M&E and better project outcomes (World Bank 2009: 107). 38 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS the cluster indicator includes an indicator on predictable, sustainable financing for M&E. Collectively, these indicators are likely to contribute to effective project M&E and to the persistence of project results over the long term. Guidance Notes: The World Bank’s Results Framework and M&E Guidance Note (2014a) proposes indicative citizen engagement indicators that can be modified to match particular project contexts, which it notes can be used as both intermediate and outcome indicators depending on the project context, scope, PDO, and approaches, among other things. The first indicator measures capacity building support provided to government officials and project staff to improve their ability to design and implement M&E. The second indicator measures whether government officials and/or project staff regularly conduct M&E. The term “high-quality” M&E refers to an M&E system that identifies project indicators and targets, evaluates baselines, and tracks and reports on progress toward project indicators and targets on a regular basis. High-quality M&E may also include an adaptive management approach, such as revising indicators and targets based on changes in the project as necessary. The third indicator on predictable, sustainable financing aims to ensure that there are sufficient financial resources to carry out M&E activities over time. 4.4 Potential Predictive Proxy Indicators with Broad Relevance Four of the seven PPIs just described include an indicator on predictable, sustainable financing and secure property rights figures as a proposed or potential indicator in several clusters as well. This review suggests that these two indicators can enhance many cluster indicators as well as having predictive potential as stand-alone indicators. STRENGTHENED LAND TENURE a) forest area brought under strengthened tenure or use rights = strengthened land tenure Theory of Change: Users are more likely to invest their own time, resources, and efforts in SFM if they know that their access to the area is secure over the long term and that they have the potential to benefit from their investments over time. Therefore, formalization of user rights or tenure has the potential to contribute to SFM over time. Evidence from the review of the World Bank’s forest portfolio highlights this relationship between secure access and use rights and investments in sustainable land use practices, such as SFM. For example, the Forest Protection and Rural Development project in Viet Nam (1997–2006; P004839) issued Land Use Rights Certificates (or “red books”) to households in buffer zones surrounding national parks. The project ICR emphasizes that the provision of secure, long-term land use rights for farmers contributed to increased investment in land by farmers who adopted advanced agroforestry farming models introduced by the project. Guidance Notes: This indicator includes the total forest area with strengthened tenure in hectares. Table 5 presents examples of related indicators, and the range of meaning that “strengthened tenure” may cover has been described above. This indicator is one that could be used on its own or combined with other indicators in a cluster. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 39 TABLE 5: EXAMPLES OF INDICATORS RELEVANT TO STRENGTHENED LAND TENURE PREDICTIVE PROXY General Indicator from PPI cluster Specific example indicators Source a) forest area brought under • Increase in land and resources under legal control and management of FIP strengthened tenure or use rights indigenous peoples and local communities including through traditional forest management systems • Increase in area with clear, recognized tenure of land and resources for FIP indigenous peoples and local communities • Evidence that the legal framework and implementation practices FIP provide for nondiscriminatory land tenure rights and land use systems and protect the rights of indigenous peoples and local communities • Area of forest under clear, nondiscriminatory tenure and territorial FIP rights, including the recognition of traditional rights • Legal rights granted by competent authorities to have access to forest GEF resources • Tenure of at least 30 percent of the occupants in national lands in the P064914 in Honduras project area regularized through formal, long-term usufruct agreements or title developed with the assistance of the project • Forest area brought under strengthened tenure P132846 in Argentina PREDICTABLE, SUSTAINABLE FINANCING a) Development, establishment and implementation of a financial mechanism or trust fund to support activities or efforts identified as critical for the continued achievement of forest investment objectives, including capacity building and training to ensure that local actors can manage and disburse funding = predictable, sustainable financing Theory of Change: Predictable, sustainable financing is necessary to support continued efforts and activities after a project intervention. An indicator that measures the amount or percentage of activities funded through local resources or through a co-funding arrangement suggests funding that is independent of the project, with the potential to continue over the long term. It is also important that local actors have the capacity to manage and allocate financing to ensure financing is directed toward intended activities (not siphoned off for corruption) and used in a sustainable manner. In the absence of these indicators, forest activities could end due to a lack of resources, or resources could disappear or be depleted through corruption or mismanagement. Guidance Notes: Financing is a critical component of many indicators and one that can contribute to the indicator’s potential to improve long-term impacts over time. Consequently, this indicator is one that has the potential to be used both as a stand-alone indicator as well as an indicator used in combination with others as part of a cluster indicator. (See Table 6 for specific examples of indicators.) The key characteristic of this indicator is that it implies financing that is regularly available and sufficient for desired activities. 40 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS TABLE 6: EXAMPLES OF INDICATORS RELEVANT TO PREDICTABLE, SUSTAINABLE FINANCING PREDICTIVE PROXY General Indicator from PPI cluster Specific example indicators Source a) Development, establishment, • Increase capacity to finance SINANPE recurrent costs with P068250 in Peru and implementation of a financial local resources mechanism or trust fund to • Sustainably finance and promote investment in forestry sector DPOs in Ghana support activities or efforts P102971/P113172/P118188 identified as critical for the • Mulanje Mountain Conservation Trust established and P035917 in Malawi continued achievement of forest functioning on income being generated by endowment fund investment objectives, including • At least 20 percent of the funds invested at the PA level by P065988 in Mexico capacity building and training non-environmental agencies are compatible with conservation to ensure that local actors can and/or sustainable use of biodiversity manage and disburse funding • At least 80 percent of the development initiatives financed P065988 in Mexico by non-environmental agencies have no negative impacts on biodiversity or include mitigation measures • Establishment of a financial instrument to support easements P061314/P52009 in Costa Rica targeting biodiversity conservation in Costa Rica • Sustainable funding mechanisms established, from The Uttar P035169 in India Pradesh and Uttaranchal Forestry Project • Area of forest plantations under private management P057234/ P058706 in Tanzania agreement (ha) • [Protected areas agency] annual budgetary support from P131965 in Mozambique central state treasury ($) 4.5 The Utility and Importance of World Bank Core Sector Indicators The World Bank Core Sector Indicators deserve special mention. Five of the seven forestry CSIs feature as elements in the PPI clusters developed in this paper. A sixth, relating to support for policy and regulatory reforms, is also likely an important constituent in additional PPI clusters. Thus using CSIs can bring a double benefit of helping to capture in a consistent way both end-of-project and longer-term outcomes of forest investments. The CSIs for forest investments were launched in July 2012 to help capture some key overarching results the World Bank has sought to help its clients to achieve. They are reported in IDA/IBRD project Implementation Status Reports, and they are being tracked and aggregated over time. Beyond forestry, several other CSIs, such as those relating to land management, biodiversity, civic engagement, and social inclusion, are not listed as forestry CSIs but are very relevant to the sector. (See Box 4.) In several cases, these indicators also have potential as PPIs. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 41 BOX 4. FORESTRY-RELEVANT CORE SECTOR INDICATORS IN IDA AND IBRD PROJECTS The use of Core Sector Indicators in IDA and IBRD projects was introduced in 2012, and experience with their use so far has been instructive. The requirement for a project to report on performance against a CSI depends on whether or not its primary objective has been coded to reflect its respective sectors. All projects are expected to report on the number of project beneficiaries as well as the proportion of beneficiaries who are female. But a forestry project would not necessarily report on biodiversity outcomes unless it had been coded accordingly. The following CSIs are indicative of the types of indicators related to forests and trees in landscapes that might be monitored and reported. FORESTS • Area restored or re/afforested (ha) • Forest area brought under management plans (ha) • People in targeted forest and adjacent communities with increased monetary or nonmonetary benefits from forests (number) • People employed in production and processing of forest products (number) • Forest users trained (number) • Reforms in forest policy, legislation, or other regulations supported (yes/no) • Government institutions provided with capacity building support to improve management of forest resources (number) BIODIVERSITY • Areas brought under enhanced biodiversity protection (ha) • New areas outside protected areas managed as biodiversity-friendly (ha) LAND ADMINISTRATION AND MANAGEMENT • Target population with use or ownership rights recorded as a result of the project (number) • Target land area with use or ownership rights recorded as a result of the project (ha) • Land area where sustainable land management practices have been adopted as a result of the project (ha) • Land users adopting sustainable land management practices as a result of the project (number). • Land area brought under a catchment system as a result of the project (ha) PARTICIPATION AND CIVIC ENGAGEMENT • Participants in consultation activities during project implementation (number) • Subprojects or investments for which arrangements for community engagement in post-project sustainability and/or operations and maintenance are established (percentage) • Beneficiaries that feel project investments reflected their needs (percentage) SOCIAL INCLUSION • Share of vulnerable and marginalized people of the total project beneficiaries (percentage) • Representatives in community-based decision making and management structures that are from the vulnerable or marginalized beneficiary population (percentage) • Vulnerable and marginalized beneficiary population who participate in nonproject consultations and decision making forums (percentage) Source: World Bank 2014b. To get a better sense of the use of relevant CSIs and potential for further analysis of their relationship to PPIs, we analyzed uptake of forestry CSIs since their inception in 2012. We found that their use has improved over time, with 63 percent of active forestry projects approved since July 2012 including at least 1 CSI (see Table 7 and Figure 10). The area restored or re/afforested was the most frequently used forestry CSI. The remaining CSIs were all used relatively evenly (in four to five projects) with the exception of “people employed in production and processing of forest products,” which has not been used in any project to date. These broad findings suggests that some ongoing projects may already have in place clusters of indicators that may have the capacity to predict longer-term outcomes. 42 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS TABLE 7. UPTAKE OF FORESTRY CORE SECTOR INDICATORS IN WORLD BANK PROJECTS AFR EAP ECA LAC MNA SAR Total May 2015 # of projects 9 4 4 6 0 4 27 # projects 6 2 4 3 0 2 17 with forestry CSIs % with 67% 50% 100% 50% 0% 50% 63% forestry CSI June 2014 # of projects 10 6 3 4 1 3 27 # projects 1 4 3 2 0 2 11 with forestry CSIs % with 10% 67% 100% 50% 0% 67% 41% forestry CSI June 2013 # of projects 9 4 2 5 1 1 22 # projects 1 3 2 0 0 0 6 with forestry CSIs % with 11% 75% 100% 0% 0% 0% 27% forestry CSI Note: Analysis was based on all projects that had a sector or theme code of Forestry (AT). The 2015 data reflect active forestry projects approved since the adoption of the core sector indicators by the World Bank in July 2012. Earlier years used all active forest projects. FIGURE 10. UPTAKE OF FORESTRY CORE SECTOR INDICATORS BY INDICATOR Total number of projects with forestry code 27 Projects using a forestry CSI 17 People employed in production and 0 processing of forest products* Reforms in forest policy, legislation or 4 other regulations supported Govt institutions provided with capacity 5 building to improve mgt. of forest resources* Forest area brought under management plans* 5 People in forest&adjacent community with 4 monetary/non-monetary benefit from forest* Forest users trained 5 Area restored or re/afforested* 10 0 5 10 15 20 25 30 *= CSIs that were also identified as potential predictive proxy indicators. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 43 4.6 Assessing Monitoring and Evaluation in the World Bank’s Forestry Portfolio In addition to reviewing the forest portfolio to identify potential predictive proxy indicators, the review also examined M&E more generally in the selected projects with an eye toward distilling lessons to inform future project M&E design and practice and use of PPIs. A major finding is that, even if shortcomings remain, project M&E has improved over time. For instance, the most recent ICRs, usually those published after 2000, contain a project overview that highlights progress on PDO indicators, which were not always present in earlier projects. In earlier ICRs, the type of indicators are not always identified clearly; for instance, they are often listed in an annex at the back, not discussed in the main body of the report, and appear to be output indicators or physical indicators. The earliest PCRs also had a much more limited focus on indicators. The majority of ICRs and PCRs addressed M&E (n=74), although several did so in a cursory manner, such as by including a list of output indicators in the report annex and not providing substantial discussion on M&E and indicators in the report. A smaller set of reports offers lessons learned on M&E and indicators and suggests recommendations for the design and implementation of M&E and indicators. The review also includes a few projects that are examples of “turn around” stories— projects that were failing initially and improved with significant World Bank attention and supervision. (See Box 5.) BOX 5: FROM UNSATISFACTORY TO SATISFACTORY: THE INDIA ECODEVELOPMENT PROJECT The India Ecodevelopment project (P036062), which aimed to conserve biodiversity by implementing the government of India’s ecodevelopment strategy in and around seven PAs, is an example of a “turn around” story, according to its ICR. The ICR explains that the project “had a slow take off and the progress was unsatisfactory in the initial periods.” The project failed to disburse funds during its initial year of implementation, suffered from a lack of continuity of task leadership (with four TTLs in the first three years of implementation), and was rated as unsatisfactory in its initial evaluations, among other challenges. At the MTR, the World Bank decided to restructure the project to focus on implementation of ecodevelopment activities around the seven selected PAs and dropped a component on the preparation of future biodiversity projects. Renewed efforts by the government, PA staff, local communities, and other stakeholders helped to turn the project around. Consultants monitored project progress at all sites from the MTR onwards, providing feedback to the main project office for action. The ICR also highlights the World Bank’s “flexibility in adapting the project design and targets consistent with a process oriented and learning approach” and using aide-memoires as guidance for adaptive management and monitoring. By project closing, the project had achieved its objectives, and several of its activities and approaches are now considered best practice. The ICR notes the following on institutional development impact: “The improvement in relationships between PA staff and local communities at all PAs may be one of the most significant contributions to long-term sustainability of the PAs and biodiversity conservation in India.” Six projects revised their M&E frameworks, indicators, or targets, according to the project ICR or PCR. In addition, some projects made minor revisions, such as updating targets to reflect better-than-expected progress or revising targets downwards to better align them with project realities. The Conservation and Sustainable Use of Biodiversity in the High Andes Region (P063317) in Colombia is an example of a project that substantially revised its M&E framework to include an increased emphasis on outcomes. The project originally included a logical framework that focused on activities and outputs, with 40 product indicators. At the MTR, the World Bank revised the M&E framework to focus more on measuring achievement of objectives and progress toward 44 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS the GEO. These revisions included focusing supervision on 15 IO indicators that were most representative of the GEO aims of increased biodiversity knowledge, conservation, and sustainable use and developing six new outcome indicators to capture aggregated improvements in biodiversity knowledge, conservation, and use practices, using the 40 product indicators as inputs. However, the ICR notes that the project partners did not incorporate the revised indicators into the project’s M&E processes, and the six new indicators were only measured at project closing. The Mulanje Mountain Biodiversity Conservation Project (MMBCP; P035917) in Malawi is an example of a project that revised its indicators to facilitate improved results measurement while maintaining its original focus. According to the ICR, the World Bank, Borrower, and Implementing Agency recognized during implementation that the original project indicators were qualitative in nature and difficult to measure. Consequently, they refined the original set of GEO, IO, outcome, and output indicators into quantitative proxy indicators and prepared a refined results framework at MTR that reflected the new set of quantitative indicators. The ICR emphasizes: “The proxy indicators did not replace nor depart from the initial intent or focus of the original set of key indicators; rather they facilitated accurate measurement of what had previously been highly descriptive key indicators.” Model Projects Several projects included exemplary M&E frameworks that are recognized for their detailed design, adaptive nature, and focus on impact and quality. This section highlights a few of these projects and particular elements of their M&E frameworks, as highlighted by their ICRs. The Karnataka Watershed Development Project (P067216) in India highlights several elements of an effective M&E system, including detailed design of the system before project implementation. The ICR states the M&E system provided a strong tracking and learning mechanism, correcting and realigning the project and pushing for better performance and accountability through on-the-ground results. It describes project management as responsive to issues raised and able to proactively take corrective actions. The ICR also highlights ongoing analysis during the project as critical in facilitating implementation changes that resulted in a sharper poverty focus, opportunities for women and the landless, greater equality among small, medium-size, and large farmers and increased cost efficiency in soil and water conservation. The Water Conservation Project (P056516) in China praises the World Bank’s focus on impact in its supervision rating. It notes: “The Bank maintained a focus on development impact, consistently emphasizing the targeted outcomes and the innovations being tested under the project. Bank missions ensured that monitoring and evaluation began early in the project period and that the baseline survey was promptly conducted. Aide-Memoires systematically recalled the development objectives, and variations from target key indicators were raised as issues.” The ICR for the Eastern Anatolia Watershed Rehabilitation Project (P009023) in Turkey describes the project as “noteworthy for having focused on quality rather than aiming at simple area targets and showing considerable flexibility and resulting in learning by doing.” In general, these projects underscore the importance of designing an M&E system before project implementation begins and maintaining a focus on monitoring and evaluating indicators throughout the project. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 45 Lessons Learned on M&E Indicators should support a results-driven process, focus on outcomes, and show a clear link between project activities and outcomes, according to the most common lesson learned on M&E. Approximately one-third of project ICRs or PCRs include lessons learned on M&E (n=26, or 33 percent). These lessons learned address M&E frameworks, indicators, project duration and time considerations, participation and stakeholder involvement, and resources and sustainability, among other issues. M&E FRAMEWORKS Seven ICRs or PCRs explicitly recommend defining the M&E framework at the start of the project, before implementation begins. Reports suggest focusing on the development of an M&E framework during project preparation and design phases, including the identification of baseline targets and data collection plans. For instance, based on its failure to put in place an M&E system, “despite repeated requests from the Bank to design and implement an M&E system,” including in aide-memoires and the MTR, the ICR from the Transfrontier Conservation Areas Pilot and Institutional Strengthening Project (P001759) in Mozambique stresses the importance of having an M&E system in place at project effectiveness. The ICR recommends “future projects should consider not releasing funds until the baseline data for an M&E system has been established.” Additional recommendations on M&E frameworks include the importance of showing causality, M&E as an adaptive, learning, and innovation tool, and the need to match M&E expectations with technical capacity. The Forest Concession Management and Control Pilot Project (P060003) in Cambodia argues that the project should have used M&E in a proactive manner as a way to track the project’s progress and address its weaknesses early on during implementation. The ICR’s lessons learned section explains that “a properly functioning M&E system might have provided the basis for a stronger dialogue between the Bank and the Borrower and might have been developed more pro-actively as the ‘learning and innovation’ tool,” which could have helped the project to adapt during implementation. The Uttar Pradesh and Uttaranchal Forestry Project (P035169) in India recommends ensuring consistency between M&E, management information systems, and technical capacity, noting that the Forest Department staff had a low level of technical expertise on M&E and that senior decision makers possessed a low awareness on the importance of M&E benefits. The Cape Peninsula Biodiversity Conservation project (P036062) in South Africa, recommends avoiding “overdesign” of indicators to ensure sufficient flexibility and innovation during project implementation. (See Box 6.) BOX 6: AVOIDING OVERDESIGN: THE CAPE PENINSULA BIODIVERSITY CONSERVATION PROJECT The Cape Peninsula Biodiversity Conservation project (P036062) in South Africa, a highly satisfactory project, is an example of a project that aimed to ensure flexibility in indicator selection and use. According to the ICR, the project design was “robust and flexible to address the challenges of being implemented in a rapidly changing legal and institutional environment.” Many of the indicators were “fine-tuned” throughout project implementation, as shown in the ICR’s Annex on the log frame matrix. Among its lessons learned, the ICR recommends avoiding overdesign, explaining how the project “benefitted significantly from a broad- brush Logframe, which clearly set out the objectives and key performance indicators but avoided detail (10 percent design and 90 percent implementation).” The ICR further emphasizes that this approach facilitated flexibility and innovation approaches and solutions during project implementation, saying “a free hand was given to the architects of the project to be innovative and to take risks.” This project suggests that, while defining and selecting robust indicators is critical in contributing to project achievements, it is also important to ensure flexibility and innovation during project implementation, as needed. 46 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS General lessons learned on M&E also highlight the importance of ensuring independent M&E , ensuring sufficient and sustainable financial resources to conduct M&E , involving stakeholders in defining M&E frameworks and assessing progress, and ensuring M&E sustainability . For instance, the ICR from the Karnataka Watershed Development Project (P067216) in India states “an independent and credible M&E institution can complement M&E functions in the implementing agency and provide major contributions to project success,” including through providing complementary services such as spatial information or surveys. INDICATORS Twenty-two projects reflect on the definition and use of indicators in the lessons learned section of their reports. As noted, the most common lesson learned emphasizes the importance of focusing on results and ensuring that project activities will contribute to project outcomes (n=10). (See Table 8.) For instance, the Mexico Environmental Sustainability Development Policy Loan (P095510) describes the importance of measurable and specific indicators, stating that “clearly-defined baseline and target values to monitor progress and evaluate outcomes, as well as a relevant set of measurable and meaningful outcome indicators, are essential for a results driven process.” Earlier PCRs suggest that projects focused too much on output indicators and not enough on outcomes and results-based indicators, with several reports noting that the project was implemented before the World Bank began placing emphasis on logical or results frameworks. This last point suggests that the World Bank has placed an increased focus on results-driven M&E over time. TABLE 8: COMMON LESSONS LEARNED ON INDICATORS Indicator should be: Number of projects Results-driven process and focus on outcomes 10 Measurable, realistic, and specific 4 Achievable during the project’s lifespan 3 Unaffected by exogenous, confounding factors 1 Useful for management 1 Reliable and systematic 1 Project documents also note that exogenous, confounding factors can limit the achievement of outcomes, and they recommend linking project development indicators to project activities. For instance, the Sustainable Forestry Pilot Project (P053830) in the Russian Federation included an indicator on “increase in forestry revenues.” The lessons learned section of the ICR explains the project assumed that regulations drafted during the project would help achieve this indicator. It points out, however, that the project failed to consider the effect of wood product demand or the state of the economy, factors over which the project had no control but that affected achievement of the forest revenue indicator. Some ICRs discuss the challenges of showing project impacts and outcomes during short time frames, noting that some outcomes may not be discernible at project closure even though these outcomes may materialize over time, a common challenge for M&E in the forestry and other sectors, as described in the Introduction.. Consequently, some ICRs recommend selecting indicators that can be achieved during the project’s lifetime. One project suggests that post-project evaluation should be incorporated into the project time frame to account for such results over time. These recommendations fit with the overall thrust of this report: that PPIs can be identified that can provide a credible estimate in the near term (during project implementation) of longer-term results. The Ecomarkets Project (P061314 and P52009) in Costa Rica experienced challenges in demonstrating impacts during the project lifetime. The ICR explains that the project did not monitor the actual impact of project activities on the generation of specific services. For example, it did not assess the relationship between increased forest cover and increases in UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 47 biodiversity or water services. The ICR recognizes, however, that monitoring the impact of forest cover on water services is not a simple task, explaining that the impacts will likely be experienced over time and over dispersed geographic areas. The project is also described as having a weak M&E framework because it did not systematically evaluate the extent to which participation in the program changed behavior. Therefore, the ICR explains, “while it can be observed that PSA participants have substantially higher levels of forest cover, it is difficult to ascertain how much of this difference was due to the project.” Defining how M&E data will be gathered and used is also important from the design stage, according to the ICR from the Indigenous and Community Biodiversity Conservation Project (COINBIO; P06674) in Mexico. This project document explains that the information that needs to be collected for reporting to a global entity such as the GEF is not the same type of data that an ejido requires for decision making. In addition to the characteristics of indicators highlighted in Table 8, a few other lessons learned emerged from project documents. The Third Forestry Development Project (P009582) in Bhutan reflects that the project indicators did not capture or show the good professional progress of Bhutanese project partners that resulted from the project’s training and technical assistance components. It states: “Such achievements are equally important mosaic pieces as e.g. the number of kilometers of forest roads constructed. Would the project be considered as a learning process for initiating gentle and sustainable methods of forest management, the assessment would have been better.” The First Programmatic Development Policy Loan for Sustainable Environmental Management (SEM DPL; P095205) in Brazil recommends that projects monitor existing indicators that are already available. Under the lessons learned section, the ICR notes: “The lessons learned in this operation points to the need to use existing indicators regularly monitored by implementing agencies whenever possible instead of creating program specific ones. This is the case, for instance, of the indicator for annual deforestation rate monitored by INPE [the Brazilian Space Agency].” In contrast to this project, the Mexico Environmental Sustainability Development Policy Loan (P095510) defined a set of outcome indicators for measuring project progress that were also included in Mexico’s 2007–12 National Development Plan, which meant that indicators were relevant for measuring progress on both the project and Mexico’s National Development Plan. Three projects highlight the importance of defining baselines and targets as part of the development of an M&E system. Although this lesson may seem overly simplistic, several projects did not define baselines or failed to adapt project targets during implementation. The EcoMarkets project (P061314 and P52009) in Costa Rica reflects that it is difficult to evaluate the effectiveness of conservation programs “when such programs are not designed to be tested and measured against a clear baseline or ‘control’ case.” The Maloti-Drakensberg Transfrontier Conservation and Development Project (P061314 and P52009) in South Africa states that “a baseline not only helps to measure success, but also disciplines the designers to pay attention to realistic and measurable indicators.” It is also important to note that several other projects did not define or include baseline or target indicators yet did not specifically reflect on this shortcoming in the lessons learned section. Another factor that may affect project outcomes relates to project supervision and management. For instance, many project ICRs highlighted the challenges that frequent changes in TTLs pose for project continuity and achievement of project objectives. (See Box 7.) 48 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS BOX 7: ANOTHER IMPACT ON PROJECT OUTCOMES? TTL TURNOVER The review showed that projects generally have several TTLs, which can negatively affect project continuity, momentum, implementation, and achievement of objectives. For instance, the Energy Access Project (P049395) in Ethiopia changed TTLs five times over the 10-year implementation period. The Rural Environment Project (P066199) in Azerbaijan had three TTLs in four years, which the ICR notes disrupted dialogue between the World Bank and the client and “severely undermined supervision efficiency as each new TTL built a learning curve on the Project, stakeholders, and issues.” The ICR from South Africa’s Maloti-Drakensberg Transfrontier Conservation and Development Project (P052368) stressed the following lesson learned on TTLs: “Frequent changes in task management during project design and implementation can cause serious disruption, when task managers are leaving without finishing important milestone tasks. Examples under this project were the finalization and agreement on the logical framework at the design stage, the completion of the restructuring of the Project at mid-term or finishing the complex and time consuming design and procurement process of the environmental centre at the later stage of the Project.” Similarly, the ICR from the Third Forestry Development Project (P009582) in Bhutan found that “frequent turnover of Bank staff handicapped continuity in supervision, with six TTLs over the life of the project and frequent turnover of other supervision staff.” At the same time, when TTLs carried out their roles effectively, the ICRs, and even the Borrower’s ICRs, note the benefits to the project. For example, some ICRs praised projects for selecting TTLs with experience with similar projects in the region, such as a TTL with experience on trust funds in Latin America and the Caribbean who worked on several related projects or TTLs who lived in the country during at least part of the project implementation. ICRs also commended TTLs who stayed through a critical project period or ensured a smooth transition. The Sustainable Forestry Pilot Project (P053830) in the Russian Federation, for instance, noted that the project had four TTLs in a 10-year period but ensured continuity and sustained institutional memory in the team supervising the project. Similarly, two highly satisfactory projects underscored the importance of continuity in World Bank management. The Second Loess Plateau Watershed Rehabilitation Project (P056216) in China emphasized consistency in task management and task team composition as critical in a strong Bank-Borrower partnership. The Sustainable and Participatory Energy Management (PROGEDE; P046768) project in Senegal states that continuity in Bank team staff was a key feature of PROGEDE. The ICR explains that, from project preparation to project closing (eight years), there were only minor changes to the core Bank team, which resulted in “an uncommon level of knowledge about the sector, the project, the Borrower’s institutions, the actors, the issues, and the opportunities.” The ICR recognizes that it may be difficult to replicate such continuity in many other operations but stresses “the level and quality of outcomes of the project does provide sufficient grounds to suggest that increasing continuity of operational teams could improved the quality and poverty alleviation impact of operations in the Bank.” TTL turnover in projects is likely to continue and can bring benefits. However, these findings underscore the importance of efforts to ensure continuity in projects when changes in TTLs and other key supervision staff occur. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 49 Summary of Key M&E Lessons Learned • Indicators should support a results-driven process, focus on outcomes, and show a clear link between project activities and outcomes. • Projects should use M&E in a proactive manner as a way to track progress and address weaknesses early on during implementation: a well-functioning M&E system can provide the basis for stronger dialogue between the donor and client during project implementation. • Avoid overdesign: projects can benefit significantly from a broad-brush results framework that clearly lays out objectives and key performance indicators but avoids restrictive detail. Such an approach can facilitate adaptive management, innovation, and—ultimately—better results. • Clearly define baselines and targets as part of the development of an M&E system and measure them as soon as possible. • Link project development indicators specifically to project activities in order to help account for exogenous, confounding factors that can limit the achievement of outcomes. • Use existing indicators regularly monitored by implementing agencies where relevant and possible rather than creating program or project specific ones. This helps reduce costs, ensures that indicator data will be collected, and creates synergies with related institutions and interventions. • Ensure sufficient and sustainable financial resources to conduct M&E and involve stakeholders in defining M&E frameworks and assessing progress. • Take steps to build client M&E capacity and financial sustainability of M&E in the post-implementation period. 50 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 5. O pportun iti es f or a nd C onst r a i nt s on Use of Proxies in Wor l d B a nk Pr oj ects 5.1 Opportunities When designing project indicators, World Bank guidance emphasizes “less is better,” recommending to “limit the number of outcome indicators to five or fewer and the overall number of indicators to not more than 15” (World Bank 2014a: 2). In contrast to this advice, the review of the World Bank’s forest portfolio found that the average number of PDO indicators per project was greater than six, suggesting a need to ensure a smaller number of indicators in future World Bank projects. The development and use of predictive proxy indicators offer an opportunity to identify a small set of indicators that can be used to predict longer-term results. Such indicators can also help facilitate comparison across projects, regions, and sectors. Participants at the expert workshop in January generally supported the use of PPIs, highlighting different instances in which proxies would be useful. In particular, they supported PPIs as valuable in terms of predicting project and longer- term outcomes and for easier measurements that would demand less time and resources. Participants cautioned, however, that many of the indicators identified through the review of the World Bank forest portfolio are useful but may be too specific to the particular project in which they are embedded. The biodiversity and the poverty breakout groups expressed support for the CSIs as useful in particular. A breakout group on governance also highlighted the importance of ensuring that proxy indicators can be aligned with the priorities and capacity of the relevant government agency or agencies in the country in which the project is being carried out. They observed that indicators might serve their intended purpose but not be well aligned with a government’s ability, capacity, culture, or intention. Participants also recommended including the importance of ensuring good, strong baselines on which to monitor future progress. Similarly, the portfolio reviewed highlighted the importance of clearly linking goals, indicators, and targets, such as through a matrix that shows the relationship among the different goals, indicators, and project components. Discussions with TTLs highlighted the need for strong enabling conditions and indicators to measure these conditions. Suggestions included large-scale datasets on national governance, market prices, gross domestic product, forestry budgets, and national deforestation rates. Some experts interviewed proposed conceptually separating indicators of context or enabling conditions from those directly related to a project activity or objective, suggesting there may be ways to clarify the different elements in predictive indicator clusters. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 51 5.2 Constraints One key constraint to the development and use of PPIs is the project incentive structure.4 TTLs and other project staff have an incentive to use indicators that are achievable within the project results framework in order to demonstrate progress and outcomes. Pressure to produce results within the project time frame and to achieve a high project rating may bias project staff to select indicators that focus on outputs or easier-to-achieve outcomes as opposed to more ambitious indicators that may be more difficult to achieve during the project period. Some TTLs and project staff may also face disincentives to collect and report on data. Further, more ambitious indicators are more likely to produce results and outcomes after the project is completed; while these longer-term outcomes are desirable, the current system is not necessarily structured to promote these types of indicators. Better understanding of the incentives and disincentives faced by World Bank TTLs and other project staff in selecting and using outcome-focused indicators, including PPIs, could help to operationalize the use of proxy indicators. Discussions with TTLs also underscored the effort and time required to gain client buy-in during project preparation, which can lead to a simplifying of indicators. At the same time, clients may sometimes refuse to accept new indicators if, for example, they deviate from established government indicators. These factors may also hamper the use and development of PPIs in some cases. There are also several methodological constraints in the use of indicators, including challenges related to attribution, confounding factors, and the time taken to show impacts. Indicators related to forests and poverty in particular face challenges related to attribution, with project interventions assuming that supporting SFM would lead to benefits for the poor without clearly articulating a theory of change as to how such interventions would deliver these benefits. Additionally, indicators related to poverty alleviation are often imperfect measures of whether projects reach the poor and most vulnerable (IEG 2013). TTLs also described challenges related to attribution and confounding factors in discussions. PPIs, however, are proposed as a means to at least partially address some of these challenges. One constraint described by several TTLs is the potential to have to restructure projects if the results framework changes, which then presents its own challenges. TTLs also pointed out that good indicators can be accompanied by inappropriate targets. In some cases, the lack of appropriate targets may mean that the project needs to be restructured, which again results in additional effort, resources, and time. This point underscores the importance of ensuring that potential PPIs are matched with clear targets that can be adapted to the project context. Additional constraints highlighted by TTLs include the importance of ensuring that institutions and incentives are in place to continue to support behavioral change after project completion; TTL turnover, which results in limited responsibility of TTLs for the success of their former projects; and the importance of ensuring that indicators can be used or adapted across projects and time periods. TTLs also noted that the limited availability of internal World Bank technical expertise on M&E can be a constraint to using more appropriate or innovative indicators. Broad policy-level engagement beyond the project level was also recommended to facilitate the integration and mainstreaming of data collection, including in national statistics agencies. Finally, the cost and feasibility of collecting data, including baseline data, is an important consideration in the selection and use of predictive proxy indicators. There is a need, for example, to share surveys and other data collection instruments that may have relevance across multiple operations and contain questions that yield data on key indicators, including PPIs. While this review, and future efforts, can help to identify good indicators and potential PPIs, such work must also consider the relative cost of measuring, tracking, and reporting on the indicators over time. 4. It is important to note that this incentive structure is not unique to the World Bank but rather representative of development projects in general and the pressure faced by project staff to produce results during project periods. 52 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 6. Guidance on Using Proxies The results presented in this report suggest that single indicator predictive proxies are likely few and far between. Instead, clusters of indicators taken together are more likely to be effective in providing information about potential future impacts. It is worth noting, however, that some indicators do seem to be more effective on their own or in combination than others, including in many cases the CSIs for forestry and other sectors. Specific guidance on the clusters of potential PPIs identified to date is provided in the preceding sections. Discussions with World Bank M&E and other technical experts further highlighted potential factors to consider when developing proxy indicators, including suggestions on criteria for selecting PPIs and looking beyond World Bank projects for examples of proxy indicators. As noted earlier, this review did not find any well-accepted criteria to guide the selection of proxy indicators. Consequently, PPIs were identified and evaluated using the SMART criteria, theories of change, and expert guidance. The process of developing potential predictive proxies has led to innovation in how such proxies may be identified moving forward. Specifically, we propose a complement to the well-known SMART criteria for use in assessing potential predictive proxy indicators—the FOREST criteria:5 • Focused: the suggested PPI is part of a specific, well-developed theory of change and can be described in a results chain • Outcome-oriented: the suggested PPI seeks to provide information on longer-term outcomes • Replicable/reliable: the suggested PPI is appropriate for use in different locations and time periods • Evidence-based: the suggested PPI is developed based on evidence from research and/or practice (qualitative and/ or quantitative) and can be confirmed in longer-term studies • Short-term: change in the suggested PPI is discernable and measurable in the near term (typically within two to four years) as well as over time while linking to the stated long-term objective • Timeless: the suggested PPI can be achieved at any given point in time and still link to the stated long-term objective The above criteria highlight suggested components that can be used to identify or evaluate potential PPIs. To ensure that a given PPI is as effective as possible, it should meet each of these criteria, similar to guidance on the SMART criteria. A key first step in considering how PPIs can be used in projects is to develop a results chain and a theory of change explaining how activities are expected to lead to outputs, outcomes, and eventual impacts. This report includes three illustrative examples of results chains for cluster indicators on poverty and climate. While two indicators in PPI clusters are IO indicators, the results chains show that the majority of proposed PPIs are outcome indicators and fall on the right hand side of the results chain. It is notable that, depending on the focus of a given intervention, governance indicators may be either IO indicators or outcome indicators, as found more generally in a recent review of World Bank forest governance interventions (Kishor and de Rijk 2014). 5. Special thanks to Anders Jensen for this creative formulation and stimulating discussion on the specific criteria included in it. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 53 Future work on the identification and use of PPIs should investigate further whether most PPIs are in fact outcome indicators or if additional IO PPIs would be beneficial. Regardless of whether PPIs are outcome indicators or intermediate indicators, good M&E practice—including early establishment of baselines and tracking and reporting on indicators—will be needed from the beginning of any project, program, or policy. In addition, potential sources of data for measuring the indicators should be carefully considered in their formulation. Finally, where possible, PPIs should be conceptualized and worded quantitatively with specific targets. Two indicators appear in multiple PPI clusters and are worth highlighting here as indicators that many more forestry projects may wish to consider including in their M&E frameworks: first, secure tenure and property rights and, second, sustainable financing mechanisms to take project activities forward. This review has highlighted the importance and predictive potential of these two indicators. Consequently, they are recommended at minimum for use as stand-alone indicators where appropriate as well as part of a cluster of indicators. Finally, as highlighted throughout the report and elsewhere, it is critical that the development of predictive proxy indicators be cost-effective. The World Bank has already indicated its commitment to the cost-effective development and implementation of such indicators (IEG 2013). The findings on predictive proxy indicators in this report should contribute to the further development of this wider World Bank effort as well as more generally in and beyond the forest sector. 54 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 7. C o n c l u s i o n 7.1 Summary of Key Findings This review aimed to increase understanding on the potential of short-term proxy indicators for longer-term impacts of forest sector investments and how they may be used in practice. The findings suggest the existence of potential proxy indicators within World Bank forests projects as well as other sources. The review also demonstrated an increased interest and focus among World Bank staff in M&E and the development of robust indicators to track and measure project achievements. This interest underscores the critical importance of careful work to identify a set of PPIs that can be used by World Bank staff in future projects. The proposed indicator menu, included as Annex F, is a first step at identifying such indicators, illustrating the conditions under which such indicators have been used, and providing guidance on their use. The proposed PPI clusters represent an additional step in identifying such indicators. As noted in section 6, secure tenure and property rights and sustainable financing emerged as two indicators that are recommended for inclusion as part of a cluster or used as stand-alone indicators. This finding is consistent with the IEG’s evaluation of the World Bank’s forest strategy, which highlights shifts in the ownership and management of forests and underscores the importance of equitable, efficient ownership and management of forests (IEG 2013). As reforms transfer forest ownership and management rights to communities and individuals, indicators related to secure tenure and property rights are likely to emerge as helpful in predicting the long-term outcomes of forest investments. Similarly, as this review highlighted, predictable, sustainable financing is essential to ensure continued efforts after project completion. This report has also demonstrated improvements in project M&E and indicators over time in forest-related investments at the World Bank, based on increased emphasis on M&E and on tracking and evaluating progress on indicators in more recent project ICRs compared with project PCRs or earlier ICRs. The analysis showed a positive association between the quality of project M&E and project outcome ratings. As project M&E scores increased by category, project outcome rating scores increased by one-half a rating category, a finding that underscores the potentially high return of investments in project M&E. Though rooted primarily in the experience of World Bank forestry investments, this work has broader relevance beyond the Bank and beyond the sector. Indicators can play a key role in galvanizing support for focused action on particular topics, as shown by the attention of governments, donor agencies, and others on achieving the Millennium Development Goals and their associated targets and indicators. As the international community turns its attention to the development and implementation of the post-2015 Sustainable Development Goals, targets and indicators are likely to play a key communications role. Predictive proxy indicators in particular could help to make the case for the likely impacts of forestry investments beyond the forestry sector, including on climate mitigation, environmental protection, jobs and income, and other areas. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 55 7.2 Taking This Work Forward There are multiple avenues for effectively taking this work forward. First, there is a need to empirically test the PPIs identified through this review. An important component in identifying robust proxy indicators to track and assess the impacts of forest-related financing is to validate them empirically to assess their predictive capacity. Few institutions collect post-project data in a systematic way, which represents a challenge in evaluating project success and indicators over time. For instance, Buch and colleagues searched for post-project evaluations completed by members of the Development Assistance Committee of the Organisation for Economic Co-operation and Development and found that “only the Japan International Cooperation Agency (JICA) has a program in place to monitor the impacts of its projects following completion” (Buch, Buntaine, and Parks 2015: 29). There are administrative and financial constraints to collecting such data, but it would be useful to explore ways to do so based on existing examples. In the absence of such systematic post-project data tracking, it is possible to collect data on key indicators using historical data in order to conduct retroactive analysis. A challenging, but key next step in work on this topic will be to identify and analyze post-project data to evaluate a select number of projects and indicators over time. This kind of analysis might draw inspiration from studies from other fields that can suggest paths for moving forward on the development and implementation of PPIs. In the field of paleoclimatology, for instance, proxy indicators are used to construct a “window back in time,” to show changes in climate variability over a 400-year period (Mann, Bradley, and Hughes 1998) or to reconstruct the makeup of forests and aquatic environments (Deforce, Storme, and Bastiaens 2014). Out of the 80 projects included in this review, 26 projects specifically mentioned follow-up projects that were expected to continue to work toward the project’s general objectives. Six projects were part of a series of projects included in the review. These project ICRs could be used to remeasure project results over time. External sources might include academic articles or other publications that are likely to contain data on project outcomes and results after the project’s official completion. For example, the Ecomarkets project (P061314 and P52009) in Costa Rica resulted in several formal evaluations after the project’s completion that could be used to remeasure results. Several other datasets on relevant outcomes are also now available, including from Buch, Buntaine, and Parks (2015) on institutional variables, the World Bank’s Living Standards Measurement Studies, and Global Forest Watch. A second phase of this work may include development of short a short guidance note, other tailored communications outputs, and trainings based on the report and the indicator menu, with the aim of helping World Bank and other donor project staff, country clients, and other relevant partners incorporate proxy indicators into project design and carry out other activities identified through portfolio review to build capacity to assess impacts in the forest sector. Discussions with World Bank TTLs and partner colleagues, including the Food and Agriculture Organization, the Center for International Forestry Research, and the International Union for Conservation of Nature, along with the new World Bank corporate strategy, the IEG forestry evaluation, and CODE report, suggest strong demand for such efforts. The results will be integrated into the forthcoming World Bank Forest Action Plan. They can also be incorporated into ongoing efforts to develop “typical” results chains for the forestry sector and more generally in the context of landscape approaches. Innovative use of technology should also be explored in efforts to identify, test, and implement PPIs (see, e.g., World Bank 2013). Within the context of the World Bank, the PPIs identified here might also inform future efforts to take stock of the performance of the CSIs as well as parallel indicator development efforts such as ongoing work on climate-smart agriculture indicators and the nascent activity on results monitoring and impact evaluation for resilience-building operations (P155632). Substantively, there is a particular need to develop additional PPIs focused on poverty reduction, economic growth, and shared prosperity in relation to forests. Such analysis should consider how such outcomes and their indicators relate to environmental sustainability indicators (on climate change and biodiversity, for example) and also indicators from other sectors relevant in a landscape perspective. Finally, further analytical work should explore the “portability” or external validity of PPIs, investigating specifically if and how context matters in shaping the effectiveness of the indicators. 56 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Annex A: Data and Methods Identifying World Bank Forestry Projects for Review Prior to beginning the review, a workshop was held with World Bank staff and other experts in June 2014 to share experience on previous and ongoing portfolio reviews of investments in agriculture and climate, biodiversity, and forests and to develop a data collection strategy and identify possible data sources for the review. Participants underscored the substantial time required to adequately review projects. Given resource availability, the review focused on a subset of projects from the World Bank forest portfolio rather than the entire portfolio. The review began by identifying the full universe of projects that could be considered for analysis. In total, the Bank has committed more than $6.2 billion (in current U.S. dollars) to 414 projects coded as forestry since 1950. This study examined projects that closed from 1991, the year the Global Environment Facility (GEF) was established, to 2013, the last year for which project evaluation data were available. This study period allows variation in lag times between project completion and the present, with the aim of forming a potential basis for analysis of the persistence of project outcomes as a potential next step in this work. Projects were excluded if forestry represented less than 5 percent of the sector focus. In addition, some projects could not be included in the review because the ICRs or the PCRs were not available. This process resulted in a universe of 204 World Bank forestry projects. Resources were sufficient to enable review of 80 of the 204 possible projects. Projects were selected for inclusion in the following manner. First, all projects in the top 11 countries receiving World Bank financing in terms of forest area and/or prominence of forest biodiversity were chosen. These countries are: Brazil, China, Colombia, Costa Rica, the Democratic Republic of the Congo, India, Indonesia, Madagascar, Mexico, Peru, and the Russian Federation. This focus ensured a full sample of the World Bank’s forest portfolio in these countries, or 48 projects. India (n=16) and China (n=10) had the largest number of projects included in the review (see Figure A1). Next, 32 projects were randomly selected from the remaining universe of projects in the portfolio. Project selection was weighted by region so that regions with a proportionately higher number of projects were proportionately represented within the review. In addition, at least two projects were selected from each region. This resulted in an additional 14 projects from Sub-Saharan Africa (AFR); 2 projects from the Middle East and North Africa (MENA); 5 projects from Europe and Central Asia (ECA); 4 projects from East Asia and the Pacific (EAP); 4 projects from Latin America and the Caribbean (LAC); and 3 projects from South Asia (SAR). UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 57 FIGURE A1: REPRESENTATION OF HIGH-FOREST COUNTRIES 20 16 15 10 10 6 5 5 3 2 2 1 1 1 1 0 Brazil China Colombia Costa Rica DRC India Indonesia Madagascar Mexico Peru Russian Federation Overall, including the projects from high-forest countries and the randomly selected projects, the majority of projects are from Latin America and the Caribbean, South Asia, Sub-Saharan Africa, and East Asia and the Pacific, with a smaller number of projects from the Europe and Central Asia and Middle East and North Africa regions (see Figure A2). FIGURE A2: REGIONAL REPRESENTATION OF PROJECTS REVIEWED 20 19 19 17 17 15 10 6 5 2 0 Africa East Asia Europe and LAC MENA South Asia and Pacific Central Asia Data Collection ICRs were used for all projects for which they were available. It is important to note that the information included in PCRs is not as comprehensive as the information included in present day ICRs; for instance, data on M&E were often missing, and some PCRs did not rate the project’s overall performance or only included qualitative descriptions of World Bank and Borrower performance rather than the quantitative ranking scale used in ICRs. Further, the more recent ICRs tend to include a higher focus on PDOs and indicators than earlier ICRs did. 58 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Descriptive Information on Projects Reviewed Project Closing Date Projects included in the review closed between 1991 and 2013. The most recent project included in the review is the Energy Access Project (P049395) in Ethiopia, which was approved on September 19, 2002, and closed on June 30, 2013, and is the only project that closed after 2010. Several projects closed in 1991, including two from India, one from Malaysia, and one from China. Several of the projects represent the World Bank’s first intervention in the forestry sector in that country, including projects in China, India, and Indonesia. These projects include the West Bengal Social Forestry Project (P010391) in India, the Sabah Forestry Technical Assistance Project (P004292) in Malaysia, and the Forestry Development Project (P003430) in China. The review also included some projects that represent the first intervention by the GEF in that country, such as the Cape Peninsula Biodiversity Conservation Project (P035923) in South Africa. The Table Mountain Fund, established by the project, is considered to be a model trust fund to support biodiversity and conservation in the area, while the development of the Cape Strategy is considered to be an international best practice. Project Budget A total of $5,884.28 million was disbursed across 60 projects, with project budgets ranging from $0.9 million for the Rural Environment Project (P066199) in Azerbaijan to $1,300 million for the First Programmatic Development Policy Loan for Sustainable Environmental Management (SEM DPL; P095205) in Brazil. It was not possible to calculate the disbursed amount in U.S. dollars for 20 projects. The majority of projects disbursed a lower amount than their original commitment amount: 47 of the project budgets did not disburse their full commitment, 22 projects disbursed their full commitment amount, and 5 projects increased their budget amounts.6 It is important to note that some projects appear to have had decreased budgets, although the actual project budget increased as a result of fluctuating exchange rates, which makes it appear as though project budgets decreased when they actually increased. For instance, the original commitment amount for the Third Andhra Pradesh Irrigation Project (AP Irrigation III; P035158) was $477.43 million and the final disbursed amount was $421.87 million. However, the ICR explains that the actual project costs increased by about 9.5 percent but that, because the rupee-to-dollar exchange rate changed during project implementation, the project costs just appear to be lower. Two other projects carried out in India, the National Social Forestry Project (P009848) and the Kerala Social Forestry Project (P009834), also appear to have experienced a decrease in their total budget amount because their PCRs report a lower disbursement rate in dollars; however, the overall project cost in rupees for both projects increased even though the projects spent less than predicted in dollars. Another challenge in estimating project disbursement amount is the inconsistency between reporting project costs in U.S. dollars and the Special Drawing Rate (SDR/XDR). In some cases, project documents do not report commitment and disbursed amounts in both US dollars and SDR, which makes it more difficult to compare project costs across the range of projects included in this review. Furthermore, some project documents, particularly the PCRs and earlier ICRs do not include disbursement amounts. Finally, budget amounts on the World Bank project website and the project ICRs were not always consistent; in cases where the amounts varied, the ICR was used. 6. It was not possible to calculate the difference between commitment and disbursed amounts for six projects because complete budget information was not available. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 59 Project Lending Type As shown in Figure A3, most projects included in the review were Specific Investment Loans (n=61). LIL 1 SAL 3 TAL 1 SIM 4 DPL 7 SIL 61 The majority of the projects were classified under environmental category B (n=43).7 (See Figure A4.) Older projects often did not include an environmental category, which is why 20 projects are listed as N/A. A-3 N/A-20 B-43 U-5 C-9 Project Outcome Rating The majority of projects included in the review received a satisfactory rating as described in the main text. Eight projects received highly satisfactory ratings: • Water Conservation Project in China (P056516), which closed on June 30, 2006 • Second Loess Plateau Watershed Rehabilitation Project in China (P056216), which closed on June 30, 2005 • Cape Peninsula Biodiversity Conservation Project in South Africa (P036062), which closed on June 20, 2005 7. The World Bank assigns projects a category of A, B, or C, in descending order of environmental and social sensitivity. 60 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS • Sustainable and Participatory Energy Management Project (PROGEDE) in Senegal (P046768), which closed on December 31, 2004 • Nature Reserves Management Project in China (P003402), which closed on June 30, 2002 • Andhra Pradesh Forestry project in India (P010449), which closed on September 20, 2002 • Tarim Basin Project in China (P003556), which closed on December 31, 1997 • National Afforestation Project (NAP) in China (P003463), which closed on December 31, 1997. Eight projects received unsatisfactory ratings: • Rural Environment Project in Azerbaijan (P066199), which closed on December 31, 2009 • Forest Concession Management and Control Pilot Project in Cambodia (P060003), which closed on December 31, 2005 • Forests and Parks Protection Technical Assistance Project in Haiti P007326, which closed on December 31, 2001 • Solomon Islands Structural Adjustment Credit (P061214), which closed on December 31, 2000 • Environmental Conservation and Rehabilitation Project (CVRD) in Brazil (P006512), which closed on June 30, 2000 • Forest Management and Protection Project In Madagascar (P001518), which closed on January 31, 1996 • Second Forestry Institutions and Conservation Project (FICP II) in Indonesia (P003942), which closed on June 20, 1995 • Second Forestry Project in Nepal (P010192), which closed on June 20, 1992. Project Type To gain an understanding of the main focus of projects included in the review, PDOs were coded as focusing primarily on biodiversity conservation, climate change adaptation and mitigation, good governance, poverty (including the concepts of economic growth and shared prosperity), or equally on biodiversity and poverty. Overall, the majority of PDOs (n=44, or 56 percent) focused on governance, followed by biodiversity (n=16, or 8 percent) and poverty (n=13, or 16 percent). 8 The overwhelming majority of GEOs (n=10, or 83 percent) focused on biodiversity compared with climate (n=1), governance (n=1), and poverty (n=0), as shown in Figure A5. Intitutions 8% Climate 9% Biodiversity 83% 8. One project only included a GEO and no PDO and so was not coded. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 61 The project’s area or sector focus was also coded to illustrate the additional range of issues addressed by projects, including biodiversity, climate, energy, forests, governance, land, poverty, and water/irrigation. These results are shown in Figure A6. Energy 3% Other 1% Climate 1% Biodiversity 15% Intitutions 30% Forest 24% Poverty 15% Water/irrigation 5% Land 6% In addition, we noted when projects focused on participation (n=30), protected areas (PAs) (n=15), or reforestation (n=4). Assessment and Validation of Indicators Following the data collection process, all project indicators were evaluated according to the SMART (specific, measurable, attributable, realistic, and time-bound) criteria, using a Likert scale from 1 to 5. Indicators were also rated for their potential as proxies using the same scale. Indicators were also coded into one of five categories based on their overall long-term outcome: poverty alleviation (including economic growth and shared prosperity), biodiversity conservation, climate change mitigation, governance, and other. The SMART criteria (World Bank 2014a: 7) used to evaluate indicators are as follows: • Specific means that the indicator measures only the design element (output or outcome), which is intended for measurement—not any other elements in the project. For example, if the target output is to construct 20 wells, the specific indicator to be measured will be the number of wells constructed. • Measurable means that there are practical ways of measuring the indicator, being clear and unambiguous in terms of what is being measured (for instance, the indicator should avoid words like successful unless it is possible to define exactly what successful would mean in the project context). For quantitative proportions or percentages, both the numerator and the denominator must be clearly defined. For quantitative whole numbers and qualitative data, a measurable indicator should define each term within the indicator such that there can be no misunderstanding as to the meaning of that indicator. This criterion is critical for ensuring that data collected by different people at different times are consistent and comparable. • Attributable means that the indicator is a valid measure of the targeted developmental issue and that the project can be credited for the changes in that developmental issue. • Realistic means that indicators selected must be realistic in terms of their ability to collect the data with the available resources. Some indicators present major problems for data collection owing to the cost or skills required (such as large-scale sample surveys). Being realistic in planning and identifying collectable information ensures that it will, in fact, be collected. This is an important factor to consider and may lead to compromises on other criteria. • Time-bound has several connotations. First, indicators must be time-bound in terms of the time spent in data collection. Second, indicators must reflect the timing of collection, being cognizant of seasonal differences. Third, the time lag between activities, outputs, and outcomes must also be reflected in the indicators that are chosen. 62 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Box A1 provides illustrative examples of how the SMART coding system was applied to sample indicators. BOX A1. APPLICATION OF SMART CRITERIA The following are illustrative examples of how the SMART coding system was applied to three sample indicators, including one of the highest and lowest scoring indicators and a relatively highly scoring indicator. EXAMPLE 1 Project: Forests and Rural Productivity Project in Honduras (P064914) Indicator: Tenure of at least 30 percent of the occupants in national lands in the project area regularized through formal, long- term usufruct agreements or title developed with the assistance of the project Target: 150,000 ha; revised to 7,700 Score: 29 = 5 for specific +5 for measurable + 5 for attributable +5 for realistic + 5 for timebound + 4 for proxy potential Notes: This indicator, one of three indicators that received a score of 29, the highest awarded, is specific in its aim and clearly links results to actions by the project in the project area. It is also able to be measured and is realistic. Although it does not include specific dates or timeframes, it still received a high score on the timebound criteria because it can be assumed that the indicator is expected to be achieved within the project timeframe because the title is expected to be developed with project assistance. All of these factors contributed to a high proxy potential. EXAMPLE 2 Project: Ecomarkets Project in Costa Rica (P061314 [GEF CR-Ecomarkets] and P52009 [Ecomarkets Projects]) Indicator: Establishment of a financial instrument to support easements targeting biodiversity conservation in Costa Rica by EOP Target: Development of revenue capture mechanisms, including establishment of a trust fund to finance contracts targeting biodiversity conservation beyond the life of the project Score: 24 = 5 for specific +4 for measurable + 4 for attributable + 4 for realistic + 3 for timebound + 4 for proxy potential Notes: This indicator is slightly lower on four criteria and is an example of a strong indicator that scored poorly on one criterion, resulting in a lower score. The indicator is specific in its aims, which makes it measurable and realistic, and is somewhat attributable to the project but it is not timebound, a common limitation among indicators. The concept of ensuring revenue beyond the timeframe of a project is an important one for long-term sustainability, which resulted in the indicator’s high score of 4 for proxy potential. EXAMPLE 3 Project: Madhya Pradesh Forestry Project in India (P010506) Indicator: Promote conservation of biodiversity: 1) Improved protected area management; 2) Policy framework for ecodevelopment Target: None, although it includes notes about the pre-project situation (Threatened biodiversity in protected area; No policy framework for eco-development) Score: 14 = 2 for specific + 2 for measurable + 3 for attributable + 2 for realistic + 3 for timebound + 2 for proxy potential Notes: This indicator, one of two indicators that received a score of 14, the lowest score awarded, is vague and lacks specificity on what would constitute improved management and what the policy framework should include, which also makes it hard to measure. The indicator does not include any project attribution, such as suggesting that the project would contribute to improved management through particular activities. Based on all of these limitations, the indicator is considered to be unrealistic. It also lacks information addressing the timebound criteria. It is important to note that many indicators lacked specific details on project attribution and timebound criteria and so a score of 3 was considered quite low. The indicator’s low score on all of these criteria also resulted in a low score for its proxy potential. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 63 After ranking each indicator based on the above criteria, the indicators were sorted from high to low scores to examine the distribution of scores across indicators. This distribution and related findings are included in the results section. Working criteria for scoring each indicator’s potential as a predictive proxy were developed. In addition to scoring highly on the SMART criteria, to be judged as having potential as a predictive proxy the indicator had to meet the following minimum requirement: • The indicator implied a plausible theory of change explaining why it is likely to accurately predict a desired future change or state resulting at least in part from a given intervention. A theory of change is a logical description of how a given intervention or change process is expected to lead to longer-term outcomes and impacts. Its distinguishing feature is an explicit articulation of assumptions thought to connect specific steps to achievement of longer-term goals (Schorr and Weiss 1995). For instance, a sustainable financing indicator may be a predictive proxy based on a theory of change that arrangements for funding a given intervention over time imply that the intervention will be implemented even after project closure, with the assumption that it will continue to generate positive impacts. This indicator implies that the necessary funding is secured for a given period of time, that institutional arrangements are in place to allocate the funds, and that the use of the funds is effective. Two additional criteria were also used: • Active stakeholder support—The indicator suggests “buy-in” by those whose behavior a given intervention wishes to influence such that desired behavior appears likely to persist after the intervention has finished (for example, incentives exist for a given action or behavior independent of project funding). Such indicators may relate to the strength of forest user groups, social capital or cohesion, and various forms of participation, among others. • Change in behavior or capacity—The indicator measures a change in capacity to implement actions related to a given intervention or a change in behavior to support the intervention’s actions. Examples include an indicator measuring an increase in capacity to carry out forest management by a forest department or forest user group or one that measures a reduction of community reliance on resources inside a protected area. Following our coding of indicators in the World Bank forestry portfolio based on these criteria and a discussion of results in various forums, a more comprehensive set of criteria for robust predictive proxy indicators was developed (see section 6). Twenty-seven projects included at least one PDO indicator. The number of PDO indicators included in projects ranged from 1 to 26, with an average of 6.41 PDO indicators per project. Five projects included 10 or more PDO indicators: • The MX Programmatic EnvSAL (First Programmatic Environment Structural Adjustment Loan (EnvSAL I; P074539) and Second Progammatic Environment Development Policy Loan (EnvDPL II; P079748) in Mexico had 26 PDO indicators. • The Water Conservation Project (P056516) in China included 16 PDO indicators. • The Daxinganling Forest Fire Rehabilitation Project (P003550) in China had 12 PDO indicators. • Forests and Rural Productivity Project (P064914) in Honduras included 11 PDO indicators. • The First Programmatic Development Policy Loan for Sustainable Environmental Management (SEM DPL; P095205) included 10 PDO indicators. Eleven projects included a GEO indicator, with a range from one to eight indicators per project. Four projects included one GEO indicator. The Consolidation of the Protected Areas System Project (SINAP II; P065988) in Mexico included eight GEO indicators. In the distribution of PDO indicators according to the SMART criteria, the majority of the indicators (n=80, or 45 percent) received scores between 21 and 23. 64 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Examples of High-Scoring Projects Out of the highly satisfactory projects included in the review, only the Water Conservation project (P056516) in China used PDOs and IOs. The remaining seven highly satisfactory projects all used outcome/impact indicators and output indicators. The Water Conservation project (P056516) in China included 16 PDOs and 28 IOs, a higher than average number of indicators that is in contrast with World Bank advice that fewer indicators are preferable. However, the PDOs were all similarly worded, with several of the PDOs only varying by the province in which they were measured, which means that the number of substantially different PDOs was much smaller. The PDOs focused on increased grain or cash crop production and increased annual farmer incomes in the project provinces. The PDOs received a score of 26 on the SMART and PPI criteria, scoring high on the specific and measurable quality. The poverty-focused PDO indicators were noted as potentially cost and resource-intensive to measure. The PDOs scored low for their proxy potential, receiving either a 2 or a 3, because it was not clear that an increase in grain crop production would lead to longer-term impacts and sustainability of achievements. The remainder of this section briefly highlights examples of indicators that scored high on the SMART criteria and illustrates limitations of some indicators. It also shows how some projects revised indicators to make them more measurable. Specific Under this criterion, indicators that used terms like increase without specifying the amount of desired increase or the desired percentage were rated lower. For instance, the Maharashtra Forestry Project (P010390) in India included the as indicators improved biodiversity conservation and improved forest sector management. The failure to specify what the desired increase is makes it more difficult to measure if the target is achieved. Similarly, words like improved, effective, or sustainable are qualitative in nature and will result in a qualitative evaluation. One such example is the indicator to improve the system of forestry education in research and academic institutions, from the Forestry Research Education and Extension project (P010448) in India. It aims for improvement without specifying how it will be achieved and without specifying metrics on which to evaluate improvement. One IO indicator illustrates a similar challenge with the term effective. This IO, creation of effective community organizations, with the target (formation of 4,393 area groups; 6,648 self-help groups; and 742 watershed societies) from the Karnataka Watershed Development project (P067216) in India does not define effective organizations, which makes it less specific and less measurable. The indicator otherwise would have been rated higher, especially because it includes a specific number of desired groups as the target. An indicator from the Madhya Pradesh Forestry project (P010506) in India shows how the term effective can be elaborated upon to be more specific by suggesting what an effective system would entail: area of forestland with effective protection system in place, reducing social erosion, improving moisture control, and reducing high pressure on biodiversity. By elaborating upon the term effective with specific actions, the indicator shows how the project envisions an effective protection system. Similarly, the term “success” resulted in a lower score for indicators that did not clearly define it. Examples include a GEO indicator from the Consolidation of the Protected Areas Project (SINAP II; P065988) in Mexico: number of projects successfully implemented, which does not suggest what the project envisions for “successful” implementation. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 65 The Maloti-Drakensberg Transfrontier Conservation and Development project (P052368) in South Africa includes as an IO the institutional development of the four conservation agencies that enables them to implement the transfrontier plan, which suggests how the concept of capacity building can be specific and attributable. Measurable Some indicators scored very high on the measurability criterion. One such indicator is a GEO indicator from the Consolidation of the Protected Areas System Project (SINAP II; P065988) in Mexico: trends in the rate of habitat conversion in PAs included in the project, with the target that 70 percent of the PAs had a reduction in habitat conversion from 2002 to 2009, as measured by remote sensing technology. This target is unique in saying it will be measured via remote sensing; by including the target on the desired rate of reduction, the indicator becomes more specific. Further, the indicator stands out for including a time frame. Another GEO indicator from this project (P065988) also has the potential to predict long-term impact: trends in the frequency of observations of indicator species selected for each area, with the target that the frequency of observation for the majority of indicator species monitored under the project 12 PAs has either increased or remained constant. This indicator is also an example of an indicator that local people may be able to collect. Two examples highlight how indicators can be revised to become more measurable. The Tanzania Forest Conservation and Management project in the Eastern Arc Forestry Conservation and Development project P057234 (and P058706) first included as an indicator that forest and woodland cover is brought under effective management by community and individuals in project areas. In 2007, at the mid-term review (MTR) the indicator wording was changed, with the aim of increasing the measurability of the project. It was revised to the area of forests on Tanzania Mainland managed according to approved forest management plans (including community-based forest management (CBFM) and JFM). This makes the indicator more clear and specific as well as increasing its similarity to a forest Core Sector Indicator (CSI). The project also reworded the indicator “private sector is involved in forest plantation management” to “area of forest plantations under private management agreement (hectares),” which is much clearer, more specific, and measurable. Attributable Overall, indicators tended to score lower on the attribution criterion. Indicators did not regularly make a link between project action and desired outcomes and failed to illustrate how the project could be credited for the change described in the indicator. Because few projects addressed this criterion, a score of 3 represented a low score. One exception is the indicator from the Madhya Pradesh Forestry Project (P010506) in India, in which PAs were assisted by project-supported eco-development committees, which scored highly on project attribution because it clearly aims to measure PAs that are assisted by the project. 66 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Realistic Most indicators scored relatively high on the realistic criterion, although a few were ambitious in their targets, making them less realistic. One such example is from the Natural Resources and Environmental Governance Program (DPO- 1: P102971; DPO-2: P113172; DPO-3: P118188) in Ghana, which included an indicator to strengthen institutions and governance in the forestry sector. The original target was 100 percent of timber exports verified as legal. Although this target was later revised to a 10 percent increase in legal wood supply to domestic markets and the first Forest Law Enforcement Governance and Trade license issued, the original aim of achieving 100 percent verification was likely too ambitious, so the indicator received a lower score on the realistic criterion. Time-bound Indicators generally scored low on the time-bound criterion, with the overwhelming majority of them typically not including a specific time frame. Consequently, most indicators received a 3 for time-bound, while those that did include a time frame received a score of 5. One such indicator, from the First Programmatic Development Policy Loan for Sustainable Environmental Management (SEM DPL) (P095205) in Brazil, is improved effectiveness of government agencies in implementing mandated Brazilian environmental and social management procedures, indicated by the number of environmental licenses issued per year at the federal level. By including the “per year” timeframe, the indicator received a high score on time-bound. The Ecomarkets Project (P061314 [GEF CR-Ecomarkets] and P52009 [Ecomarkets Projects]) in Costa Rica included several time-bound indicators, specifying that the indicator should be achieved by the end of the project period. These indicators included 100,000 hectares of land contracted as conservation easements in priority areas by end of program (EOP); a 30 percent increase in the participation of women land owners and women’s organizations in the Environmental Services Program (ESP) by EOP; and a 100 percent increase in the participation of indigenous communities in the ESP program by EOP. The inclusion of a time frame for these indicators resulted in high scores on the time-bound criterion. One project that incorporated a time frame included an indicator with a time frame after the project completion date. The Sustainable Forestry Development Project (P064729) and Sustainable Forestry Development Project (Natural Forest Protection; P060029) in China includes as the PDO indicator that the project generates 13.3 million cubic meters of timber and 2.73 million tons of bamboo by December 31, 2025, and RMB 1.1 billion net income from fruit tree crops by 2022. Although this indicator received a high score for including a time frame, the indicator is not an ideal example of a time-bound indicator because the results are expected after the project completion date, which makes it impossible to measure within the project time frame—and therefore unrealistic. At the same time, a good predictive proxy indicator aims to capture such forward thinking.   UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 67 Annex B. Exploring Potential PPIs in Knowledge Programs: The Case of PROFOR 9 Introduction Meeting the growing demand for rigorous evidence of what works and what does not in international development generally and in the forestry sector specifically is especially challenging when it comes to knowledge programs. The causal chain linking specific knowledge-related activities, including production of reports and other knowledge products and facilitating dialogue and exchange, to ultimate impacts on people and the environment is typically long, tortuous, and uncertain. In the vast majority of cases, attributing such impacts directly to knowledge-related activities will not be possible. For this reason, applied knowledge organizations in the forest sector such as CIFOR, IUCN, PROFOR, and others have increasingly sought to identify their contribution to outcomes rather than seeking attribution. PROFOR has thus focused its M&E efforts at the intermediate and end-of-activity outcome level while developing plausible narratives about how knowledge-related work contributes to broader development impacts. Given this, potential predictive proxy indicators for PROFOR and similar knowledge-related programs were not sought linking analytical outputs and processes such as those produced through PROFOR activities through to ultimate development impacts. Rather, we sought to identify indicators that have potential to predict knowledge uptake. Here we report briefly on our approach to trying to identify such indicators and our results. We also provide some guidance for PROFOR and comparable programs searching for useful indicators or qualities of indicators to look for during an activity’s design phase. Methodology To gain insight into indicators used to date and potentially useful as predictive proxies in forest knowledge–related efforts we reviewed a subset of activities supported by PROFOR. We focused on activities approved during the period 2008–14 given greater availability of necessary information on activity outputs, outcomes, and indicators. We randomly selected one or two activities for each of PROFOR’s four thematic areas (livelihoods, governance, financing sustainable forest management, and cross-sectoral coordination) per year. The total number of activities reviewed was 36. Once the sample of activities was selected, the indicators used by each activity were classified and analyzed using the SMART criteria as done for the review of World Bank projects. We also coded each indicator using two additional criteria: the extent to which it was broadly applicable and it may be a potential proxy for uptake. Each indicator was then given a score between 1 and 5 on each element of this “SMARTAP” framework, with 1 being the lowest and 5 the highest. Scores were then added to get the indicator’s total score, which was used to rank all indicators. The reasoning for scores was also given in each indicator’s Notes section to facilitate subsequent reviews of the dataset. Each indicator was also classified as an output indicator or an uptake indicator. As the name suggests, an output indicator refers to information on a given activity’s outputs whereas an uptake indicator supplies information on the use and application of outputs. 9. This portion of the study was prepared by Selene Castillo and Daniel Miller. 68 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Results Table B1 provides basic information on the indicators reviewed from the PROFOR portfolio. TABLE B1: SUMMARY INFORMATION ON PROFOR INDICATORS REVIEWED Total number of activities reviewed 36 Total number of indicators classified 112 Average number of indicators/activity 3 Percentage of output indicators 28 percent Percentage of uptake indicators 72 percent From the list of 112 indicators, the top 20 percent (25 in all) was extracted and analyzed in more detail (see Table B2). Broadly speaking, more than half of the top indicators were associated with uptake (10) and dissemination (3). Uptake- related indicators included website hits, number of report downloads, literature citations, collaboration and advice requests, and monitoring uptake through progress reports. The outcomes associated with these uptake-related indicators included increased awareness, enhanced policy, and increased quality of knowledge. Dissemination was mentioned in the form of workshops and trainings; these dissemination-related indicators had as outcomes increased knowledge and capacity. Key findings from this review highlight the absence of an explicit theory of change that links outputs with outcomes in the majority of surveyed projects, the use of a publication or study as an indicator of achieving the activity’s expected outcome of improving forest policy without clearly describing the relationship between the study and improved forest policy, an assumption that delivering outputs will result in achieving outcomes, and confusion between definitions of outputs and outcomes. Some top scoring indicators that may be worth further consideration include: • Collaboration with government (for example, through the creation of a joint product). Broad outcome: Strengthened policy and increased capacity • Design and prioritization of client-sustained tools (that is, the client helps in the design of the tool and then takes over implementation and upkeep). Broad outcome: Increased capacity • Written commitment to collaborate (that is, through a network). Broad outcome: Stronger network of practitioners with increased access to knowledge exchange • Meeting notes. Outcomes: Validation of the content and quality of a meeting; commitment to work together. Only two indicators emerged as strong potential candidates for providing credible information on post-project uptake: • Outputs or related dialogue from the project is included in another, independent piece of work (such as literature, policy, or external project design). Rationale and notes: This indicator provides a direct measure of uptake beyond the project itself. Although this appears to be an effective indicator, it provides a relatively high bar in that it will not always be possible for project results to be available during project implementation such that another project would use them before the end of the initial project. “Second-level” uptake of the original work beyond the initial other project (that is, use of the original work due to reference to it) will be more challenging to trace, particularly in the short time period of most applied knowledge activities. Citation and other search software may enable easier measure of this indicator at the end of a project and beyond, which can also help gauge broader uptake. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 69 • Written or unwritten agreement to do an activity proposed by the project (ssuch as work collaboratively, implement a tool, or share data) Rationale and notes: Such an agreement suggests a minimum level of uptake already with a commitment that should lead to deeper uptake and/or “second- level” uptake. Though other indicators of post-project uptake did not clearly emerge from this exercise, a theory-based approach, as taken in the analysis of operational investments, could be taken here. It is suggested that indicator clusters for project uptake might be identified using this approach. Special consideration should be given to considering how new technologies might better enable understanding of uptake—for example, looking at retweets on Twitter or citation analysis, among many possible others. Suggestions for Moving Forward In the majority of the activities reviewed, a theory of change was not explicitly articulated that links an indicator (for example, a study) to the expected outcome (such as enhanced policy). Further, many activities used terms the “output,” “outcome,” “impact,” and “indicator” interchangeably. Steps are needed to facilitate clearer understanding of the distinctions among output, outcome, and impacts and especially to encourage explicit articulation of theories of changes in PROFOR activities. A key step is to define these terms clearly and provide instruction for articulating theories of change in PROFOR’s M&E system. The 2015 update to PROFOR’s M&E system should accomplish these proposed improvements. Beyond terminology, one important finding was that a large percentage of all indicators (72 percent) and of the top indicators (40 percent) were linked to uptake. Having uptake indicators instead of output indicators provides a clearer picture of how a knowledge-based activity is planning on achieving its expected outcome after the output (a study or workshop, for instance) is completed. Output indicators fall short of providing that information. For example, if the expected outcome of an activity is improving policy and the given indicator is the number of studies published on the topic (an output of the activity), the person reviewing the activity will not know how the activity jumped from the studies to policy. For this reason, it is suggested that PROFOR activity indicators emphasize uptake rather than outputs as possible. To aid this process, the creation of a “menu” of indicators that could be adapted to meet the needs of specific activities may be useful, as has been done for operational investments. This review has been preliminary, but it provides a data source that can form the basis for more in-depth discussion within the PROFOR secretariat and PROFOR TTLs as well as partners in the KNOWFOR program to try to develop more effective indicators for forest-related knowledge uptake. It would be useful, for example, to convene a workshop to share experience and begin to try to develop some theory-based indicators of uptake. This may be especially appropriate to do within the context of KNOWFOR, building on the strong M&E-related work that has already been undertaken through the program. 70 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Table B2: Top 25 Scoring Indicators in PROFOR Indicators Review Output (0) or Uptake Project number10 Proxy Potential Attributable (1) indicator Time bound Measurable Applicable Realistic Specific Score Outcome Indicator 6 Project partners committed to joint Notes of meetings of project partners 1 5 5 5 5 5 5 4 34 approach and approve detailed reflect common understanding and methodology. Key data sets acquired. capture approved methodology. 8 The experts’ meeting proved valuable The opinions of the experts were 1 5 5 5 5 5 5 4 34 for debating the merits and risks of the recorded in detail (100+ pages of BACI evaluation method undertaken and formal notes on the meeting) and for getting constructive ideas on key were used in the elaboration and research questions and hypotheses, how to refinement of the methods and approach randomization, and how to build survey tools. a representative evidence base under tight time and financial constraints. 6 Increased recognition and understanding of Map launched at Nov 2009 high- 0 5 5 4 5 5 5 4 33 the restoration opportunity. level FLR round table. Refined map launched at UNFCCC COP. 14 All harvest timber from concession is Chain of custody reports should 1 5 5 5 5 4 4 5 33 legally taxed and exported. indicate production and revenues and show full compliance of logging companies. 14 FDA, government of Liberia, and private Monthly field performance report 1 5 5 5 4 4 5 5 33 sector capacity is sufficient to operate the will provide update on the status chain of custody properly and ensure the of training and capacity building. eventual transfer of the chain of custody Workshops and trainings should to FDA. Liberia Extractive Industries produce report and an evaluation of Transparency Initiative publishes full the related capacity building. forestry sector revenue reports. 29 Strengthened policy dialogue. Formulation and consensus with 1 5 5 5 5 3 5 5 33 government on SFE reform map. 29 Increase capacity of key policy research Methodology and capacity developed 1 5 5 5 5 3 5 5 33 institute. to independently carry out SFE performance assessments and scoring and formulation of reform steps. 32 The capacity of planners and analysts in Planners from the Republic of Congo 1 5 5 5 5 4 4 5 33 key Congo Basin countries to utilize land and other Congo Basin countries use modeling and other decision support participate in the study and take tools to mitigate impacts of large-scale an active role in writing the reports mining investments on deforestation and and leading some aspects of the forest degradation is improved. workshops. 35 Areas suitable for commercial reforestation A map showing the most appropriate 0 5 5 5 5 3 5 5 33 in Colombia have been defined in a areas for commercial reforestation participatory manner. has been prepared. 10. Note: Project number identifies projects in a database available separately, which includes project name, other descriptive information, and notes on coding. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 71 Output (0) or Uptake Project number10 Proxy Potential Attributable (1) indicator Time bound Measurable Applicable Realistic Specific Score Outcome Indicator 36 Increased company investment in Utilization of SCI plans will be 1 5 5 4 5 4 5 5 33 communities to increase development monitored by IFC consultants as impact. part of this project and subsequently by staff as part of their role in developing clients. 7 Awareness is raised that laws and Number of visits on the GLIN website. 1 5 5 3 5 5 4 5 32 regulations in Gabon are accessible through the GLIN system. 8 Practitioners of REDD+ development and There will be references to the 1 5 5 3 5 4 5 5 32 independent observers will categorize typology and the methods we various kinds of projects in terms of the developed in the literature. typology developed and will recognize the value of a counterfactual approach for measuring the performance of REDD+ in relation to the 3Es and co-benefits. Using this method of evaluation will improve the performance of REDD+. 8 The Guide will become recognized as a We will know the impact of the Guide 1 5 5 3 5 4 5 5 32 state-of-the-art approach for conducting through citations in the literature, research on the socioeconomic and downloads from the website, and governance dimensions of REDD+. It is proponents and researchers who expected that both BACI and other methods approach us to propose collaboration will be applied in the evaluation of REDD+ in the Global Comparative Study and that this will serve to improve the on REDD+ or to get advice and performance of REDD+ with respect to the guidance for their own independent 3Es and co-benefits. evaluation efforts. 8 CIFOR’s web platform on forests and Through CIFOR’s Information 1 5 5 3 5 4 5 5 32 climate change will become a “go to” Services Group we are able to get place for basic information on (at least) regular reports on downloads of the REDD+ projects in nine key countries information. We will keep track of encompassing the majority of REDD+ citations of our publications. We will sites in the world. The web platform will know of individuals and organizations also disseminate our other publications approaching us for advice and including the Guide, technical guidelines, collaboration in conducting research survey instruments, etc. on REDD+. 12 Stronger international network of SMFE Written commitments to continue 1 5 5 4 5 3 5 5 32 support practitioners with clearer vision for to work together as an alliance to how to do their work. pursue SMFE support and share tactics with each other on how best to do this. 19 Increased capacity to design and Percentage of REDD+ projects that 1 5 5 5 5 2 5 5 32 implement pro-poor REDD+. use Guidance Note in project design. 20 MFF would operate more efficiently and Key findings are taken into account in 1 5 5 5 5 3 5 4 32 would be able to accommodate novel the redesign of the MFF. financing sources. 72 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Output (0) or Uptake Project number10 Proxy Potential Attributable (1) indicator Time bound Measurable Applicable Realistic Specific Score Outcome Indicator 21 Knowledge generated and documented is Stakeholders and farmers in the 1 5 5 5 5 2 5 5 32 disseminated to the diverse stakeholder North West Region use the extension audience (wider development community). bulletins to guide their targeted grassroots agro-silvopastoral operations. 26 Better understanding on the part of the Inception report and detailed 0 5 5 5 5 5 5 2 32 study team on the true causes of forest research design and ongoing fires. progress reports; progress reports and supervision reports. 26 Models successfully tested, applying a Inception report and detailed 1 5 5 5 5 5 5 2 32 better understanding of the true causes of research design and ongoing forest fires. progress reports; Progress reports and supervision reports. 30 This product aims to improve development Bank projects will be monitored to 1 5 5 5 4 3 5 5 32 practitioners’ ability to gather, make sense collect information on the use of of, and distribute important information to ICTs; also, the “hits” on the online those involved in the forest and agriculture database will be monitored to follow sectors (including beneficiaries and how much the information collected policy makers) through the harnessing has been used. of ICT tools. The product will help Bank staff, external partners, and governments select appropriate technologies and applications for data collection and M&E; will improve the quality of knowledge on ICT applications; and will inspire the use of these tools in development projects. This will lead to better interventions and improvements in program and policy design, implementation, and output and outcome tracking. 34 The primary outcome by the end of this Tracking interest and participation 1 5 5 5 5 4 5 3 32 activity will be increased understanding in the expert workshop; uptake among policy makers, practitioners, and of the report on proxy indicators applied researchers of potential short-term (copies mailed and distributed, web proxy indicators for longer-term impacts downloads, etc.); survey of workshop of forest sector investments and how they participants and other potential may be used in practice. audiences before and after they receive the report. 35 The most promising timber value chains Timber value chains are assessed in 0 5 5 5 4 4 5 4 32 have been identified, taking into account terms of competiveness, job creation, Colombia’s comparative advantages, and revenue generation. Conditions production capacities, as well as objectives for their sustainable development in terms of rural development. analyzed. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 73 Output (0) or Uptake Project number10 Proxy Potential Attributable (1) indicator Time bound Measurable Applicable Realistic Specific Score Outcome Indicator 36 Improved capacity of local governments In-service training and educational 1 5 5 5 5 4 4 4 32 and indigenous groups to monitor wood workshops will be provided to production on transactional (community) communities and companies. These and regional (indigenous federation) levels. will result in the development Improved ability of companies to invest in and implementation of at least 3 communities with optimum development company strategies for long-term impacts. engagement with communities that will be monitored annually by involved parties. 36 Accurate calculation of operating costs Operational costs will be tracked 0 5 5 5 5 4 4 4 32 during harvest operations to determine and appropriate prices established price for wood produced. by communities and companies with constant in-service training by consultants.   74 Annex C. Menu of Highly Rated Forest Sector Indicators   POVERTY REDUCTION (INCLUDING ECONOMIC GROWTH) INDICATORS Indicator CSI SMART PPI Related PPI Indicators Use Context Outcome Notes Score Potential Rating People in targeted forest Yes N/A 5 1. People in targeted forest and Various projects CSI. Used in PPI cluster for sustainable forest-related income. and adjacent communities adjacent communities have with increased monetary or secure access and use rights. 2. nonmonetary benefits from Forest activities are aligned with forests biodiversity-friendly management practices. People in targeted forest and Yes N/A 5 1. People in targeted forest Various projects CSI. Used in PPI cluster for sustainable forest-related income adjacent communities have and adjacent communities secure access and use rights with increased monetary or nonmonetary benefits from forests. 2. Forest activities are aligned with UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS biodiversity-friendly management practices. Forest activities are aligned Yse N/A 5 1. People in targeted forest Various projects CSI. Used in PPI cluster for sustainable forest-related income with biodiversity-friendly and adjacent communities management practices with increased monetary or nonmonetary benefits from forests. 2. People in targeted forest and adjacent communities have secure access and use rights. Seedling survival rate after N/A 5 1. Thinning of seedling / young Various projects Used in PPI cluster Income from forest over the longer term. three years1* trees after three to five years* Thinning of seedling / young N/A 5 1. Seedling survival rate after Various projects Used in PPI cluster Income from forest over the longer term. trees after three to five years* three years* Indicator CSI SMART PPI Related PPI Indicators Use Context Outcome Notes Score Potential Rating At least a 55 percent 28 5 Forests Moderately The indicator measures an increase in beneficiaries' income and increase in the per and Rural satisfactory therefore also captures a contribution to poverty reduction. It is capita income of project Productivity specific, measurable, attributable, and realistic. The indicator participants Project also has high proxy potential. Limitations: The indicator may be (P064914) in expensive to collect, depending on availability of information in Honduras existing surveys, and the desired time frame is not specified. Average forest-based product 26 5 Andhra Pradesh Satisfactory This indicator measures an increase in beneficiaries' income and income (cash and kind at Community therefore also captures a contribution to poverty reduction. The 2009 real prices) realized by Forest indicator has high potential as a proxy because it measures livelihood VSS members resulting from Management benefits from improved forest management, which is likely to result improved forest productivity Project in long-term impacts and sustained interest by VSS members in (P073094) in ensuring improved forest productivity. Limitations: It may be difficult India to attribute changes to the project and may be time- and resource- intensive to measure. CSI potential: It could be interpreted as a CSI under "Number of people with increased benefits." Project generates 13.3 29 4 Sustainable Satisfactory This indicator could be used as an indirect measure of benefits to million m3 of timber and 2.73 Forestry beneficiaries in terms of income or contributions to poverty reduction. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS million tons of bamboo by Development The indicator is specific and measurable. Limitations: The indicator December 31, 2025. RMB 1.1 Project uses a time frame that extends beyond the lifetime of the project, billion net income from fruit (P064729) and making it unrealistic for use. The indicator is also not directly tree crops by year 2022. Sustainable attributable to the project. CSI potential: It could be interpreted as a Forestry CSI under "Number of people with increased benefits." Development Project (Natural Forest Protection; P060029) in China 75 76 Indicator CSI SMART PPI Related PPI Indicators Use Context Outcome Notes Score Potential Rating Sustainably finance and 27 4 Natural Moderately This indicator aims to measure sustainable financing, an important promote investment in Resources and satisfactory concept with the potential for ensuring long-term impacts. The forestry sector Environmental indicator has high proxy potential, particularly with the inclusion Governance of its target. The indicator is specific and measurable. Limitations: (NREG) The indicator is not directly attributable to the project and does not Program (DPO- include a time frame. 1: P102971; DPO-2: P113172; DPO- 3: P118188) in Ghana Increase in level of forest 25 4 Sustainable Moderately The indicator aims to measure sustainable financing, an important revenue collected Forestry satisfactory concept with the potential for ensuring long-term impacts. The Pilot Project indicator has high proxy potential. With the target, the indicator is (P053830) in specific and measurable. Limitations: The indicator is not directly the Russian attributable to the project and lacks a time frame. Federation Number of jobs created from Y 26 4 Forests Moderately This indicator represents one way of capturing the contribution UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS project investments and Rural satisfactory of a project to poverty and livelihoods. It is relatively specific, Productivity measurable, attributable, and realistic. Limitations: The indicator Project could be more clear in terms of what is meant by project (P064914) in investments and lacks a time frame. Honduras 20 percent increase in 26 4 Community Satisfactory This indicator measures the benefits from the project to net value of forest goods Forestry II beneficiaries in terms of income and contributions to poverty and services produced by (PROCYMAF reduction and is similar to a CSI. The indicator is specific and assisted communities and II; P035751) in clearly attributable to the project, as well as easily measured. ejidos Mexico The concept has potential as a proxy for long-term impacts; communities and ejidos that experience an increase in income and benefits are likely to continue to support sustainable forest management over time. Limitations: The indicator lacks a time frame and may be cost- and resource-intensive to measure. Indicator CSI SMART PPI Related PPI Indicators Use Context Outcome Notes Score Potential Rating 30 percent increase in 26 4 Community Satisfactory The indicator measures the benefits from the project to beneficiaries jobs available in assisted Forestry II in terms of income and contributions to poverty reduction. The communities vs control (PROCYMAF indicator is specific and clearly linked to the project, which makes it II; P035751) in easy to measure. It has some potential as a proxy indicator based on Mexico the assumption that increases in employment can reduce pressure on natural resources and contribute to support for biodiversity and other aims. Limitations: The indicator measures availability of jobs rather than an increase in employment or jobs gained; the availability of jobs does not guarantee that communities will gain employment or experience an increase in income. The indicator could be rephrased to capture employment, similar to how the CSOs measure it. The indicator is therefore less realistic and also lacks a time frame. CSI Potential: It could be interpreted as a CSI under "Number of people employed through forest products." Forest fire control: stumpage 26 4 Daxinganling Satisfactory The indicator measures the value of forest products saved through value saved (million yuan) Forest Fire proactive control of fire management. It has some potential as a proxy Rehabilitation because it links the concept of fire control with financial benefits. The Project indicator is specific and measurable. Limitations: The indicator is not UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS (P003550) in clearly attributed to the project and lacks a time frame. China Decline in seasonal 25 4 Andhra Pradesh Satisfactory The indicator measures a decrease in migration for employment, outmigration for employment Community which indirectly captures a potential increase in employment potential Forest in the project area and therefore increases in income and poverty Management reduction. The indicator has high proxy potential by using outmigration Project as a substitute for increased opportunities in the project area. (P073094) in Limitations: The indicator is not clearly attributable to the project and India lacks a time frame. If the indicator is not captured through an existing government survey, it may be time- and cost-intensive to collect. Poverty reduction: Per 25 4 1. People in targeted forest and Forestry Satisfactory This indicator measures poverty reduction among project capita income of project adjacent communities have Development beneficiaries. It is specific, measurable, and attributable to the beneficiaries increased by secure access and use rights. 2. in Poor Areas project. The concept of improved livelihoods and poverty reduction percentage Forest activities are aligned with Project among project beneficiaries has high proxy potential. Limitations: biodiversity-friendly management (P046952) in The indicator lacks a time frame. Outcome from PPI cluster: practices. China Sustainable forest-related income. 77 78 Indicator CSI SMART PPI Related PPI Indicators Use Context Outcome Notes Score Potential Rating Poverty reduction: Incidence 24 4 Forestry Satisfactory The indicator measures poverty reduction among project of poverty decreased from 40 Development beneficiaries. The indicator is specific, measurable, attributable percent in 1996 to [project in Poor Areas to the project, and includes a time frame. The concept of poverty end date] Project reduction has potential as a proxy. Limitations: "Incidence" of (P046952) in poverty may be harder to measure than other measures of poverty China and may be less likely to be collected through existing surveys. Cropping pattern diversified 24 4 Karnataka Satisfactory The indicator indirectly captures the concept of livelihood to high-value crops Watershed diversification, which can be important in increased incomes Development and poverty reduction. The indicator is specific and measurable. Project Limitations: The indicator is not attributable to the project and the (P067216) in time frame could be more specific (as the phases included in the India target are not defined within the target). It may also be challenging or time- and resource-intensive to measure. Annual incremental revenue 22 4 Sustainable and Highly The indicator measures an increase in revenue among project to villages Participatory satisfactory beneficiaries and can therefore be considered a proxy for poverty Energy reduction. The concept of benefits for project communities is an Management important one for proxy indicators. The indicator is specific and Project somewhat measurable and attributable. Limitations: "Incremental (PROGEDE) revenue" is a less common term that may not be included in existing surveys. The indicator is only vaguely attributable to the UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS project, at best, and lacks a time frame. BIODIVERSITY CONSERVATION INDICATORS Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Rating Predictable, sustainable financing Yes N/A 5 1. Forest area Various projects This indicator was judged to have strong potential as a PPI for brought under positive biodiversity (and other environmental) impacts. adaptive, biodiversity- friendly management plans. 2. Functioning institutions to enforce rules and resolve conflict. Functioning institutions to enforce rules Yes N/A 5 1. Forest area Various projects This indicator was judged to be a key element in an indicator and resolve conflict brought under cluster with strong potential as a PPI for positive biodiversity adaptive, impacts. biodiversity- friendly management UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS plans. 2. Predictable, sustainable financing. 79 80 Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Rating Forest and woodland cover is brought 26 5 1. Predictable, Forest Conservation Moderately This indicator measures the area of forests under sustainable under effective management by sustainable and Management unsatisfactory management, with improved management plans. The improved community and individuals in project financing. 2. Project and the wording after the MTR makes the indicator more specific and areas. [To measure the impact of the Functioning Eastern Arc Forest similar to a core forest sector indicator. The indicator has a high project, the indicator wording was institutions to Conservation proxy potential because of the assumption that the management changed in the 2007 restructuring enforce rules and and Development of an area according to an approved management plan, and by the to read: Area of forests on Tanzania resolve conflict. Project (P057234 community, will lead to sustainable impacts over the long term. Mainland managed according to and P058706) in Limitations: The indicator became less attributable to the project approved forest management plans Tanzania with the revision and lacks a time frame. Outcome: Positive (including CBFM and JFM).] environmental impacts (biodiversity-related). Private sector is involved in forest 26 5 Forest Conservation Moderately This indicator measures forest area under private management. plantation management [Reworded and Management unsatisfactory The indicator was revised at the MTR, which made it more specific in 2007 to enable measurability; area Project and the and measurable. The indicator has high proxy potential because of forest plantations under private Eastern Arc Forest of the assumption that a private management agreement will management agreement (hectares).] Conservation be sustainable and lead to long-term impacts. Limitations: The UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS and Development indicator is not attributable to the project and lacks a time frame. Project (P057234 and P058706) in Tanzania Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Rating 13 Nature Reserves management 28 5 Sustainable Forestry Satisfactory The indicator uses an external measure, the WCPA Framework, to improved by 1.5 points by end of Development Project measure improved management of nature reserves. The indicator project (Scores are qualitative self- (P064729) and is specific, time-bound, and uses a particular methodology for assessments. Project followed scoring Sustainable Forestry measuring improvement, which makes the term less subjective system of World Commission on Development Project and easier to be measured. The indicator has high proxy potential Protected Areas (WCPA) Framework) (Natural Forest because of the assumption that management will contribute to Protection; P060029) long-term impacts and because the measurement is likely to be in China more objective because it is an external measure. The indicator will also be able to be broadly compared with other areas, which makes it attractive as a proxy. Limitations: The indicator lacks attribution to the project. Reduction in average annual rate of 27 4 First Programmatic Satisfactory The indicator measures reduction in deforestation. The indicator deforestation in the Amazon for the Development Policy is specific, measurable, and time-bound. It has a high proxy period 2008–10 to 20 percent below Loan for Sustainable potential, which would have been higher if the indicator were 2005–07 average annual rate of Environmental clearly linked to project. The indicator is also somewhat similar to deforestation (14,800 km2) Management (SEM the core forest sector indicator on area reforested. Limitations: As DPL; P095205) in mentioned above, the indicator could be more clearly linked to the UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Brazil project. 81 82 Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Rating 1. Maintain Mulanje Mountain 25 4 Mulanje Mountain Moderately This indicator measures the management of the mountain ecosystem, including globally Biodiversity satisfactory ecosystem, increased awareness of the ecosystem's value, and significant biodiversity and vital Conservation Project improvements in sustainability. The indicator includes several ecological services. 2. Increase (MMBCP; P035917) components but each is specific and connected to the general aim awareness, understanding and in Malawi of maintaining the ecosystem's biodiversity and services. These appreciation of the value of the concepts are all strong ones for a proxy because of the potential Mulanje Mountain ecosystem at for longer-term impacts. It also has good proxy potential because local and national levels. 3. Improve it shows the potential of resources to be co-managed, which could sustainability of biological resource be important for long-term impacts. Limitations: The indicator use and enhance the value of the is not attributable to the project nor is it time-bound. Some of Mulanje Mountain ecosystem to the indicators, such as appreciation and trust, may need to be local communities. 4. Establish measured through a project-specific survey, which will be more long-term income stream and time- and cost-intensive. institutional capacity to ensure continuation of items 1–3; Mulanje Mountain Conservation Trust (MMCT) appreciated and respected by stakeholders at local, national, and international levels. Demonstrate the appropriateness of Conservation Trust Fund as financing mechanism for UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS biodiversity conservation. Forest fire control: Forest area saved 26 4 Daxinganling Forest Satisfactory This indicator measures the amount of forest area saved through (million m3) Fire Rehabilitation control of forest fires, which includes an implicit measure Project (P003550) in of improved capacity. The indicator is specific and therefore China measurable. It has relatively high potential as a proxy because saving forest areas could contribute to longer-term forest sustainability and longer-term impacts. Limitations: The indicator is not attributable to the project and lacks a time frame. Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Rating Annual deforestation reduction impact 25 4 Sustainable and Highly This indicator measures annual reduction in deforestation. The Participatory Energy satisfactory indicator could be considered a parallel to the forest core sector Management indicator on area reforested/afforested. The indicator has high Project (PROGEDE; proxy potential because a reduction in deforestation (and its P046768) in Senegal impacts) is likely to continue to produce long-term impacts over time. The indicator is specific in terms of the desired reduction and measurable. Limitations: The addition of the term "impact" is a little odd and makes the indicator slightly less clear, but it still remains a strong indicator. The indicator lacks attribution to the project. Replanting: Upkeep of about 55,000 24 4 Forestry Sector Satisfactory This indicator measures replanting and establishment of ha of existing forest plantations and (P001168) in Côte new plantations. The indicator is specific, measurable, and the establishment of 9,000 ha of new d'Ivoire attributable to the project since the project will be involved in the plantations to complete the planting planting. The indicator has relatively high proxy potential because program of the 1st and 2nd Forestry the establishment of plantations has the potential to contribute Projects to longer-term forestry outcomes and ensure forest sustainability over time. Limitations: The indicator lacks a time frame. Area under sustainable NRM 24 4 Sustainable and Highly The indicator measures the area under sustainable natural Participatory Energy satisfactory resource management, somewhat similar to the core forest sector UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Management indicator on area under management plans or area reforested. The Project (PROGEDE; indicator is relatively specific, although the term "sustainable" P046768) in Senegal could be more clearly defined. The indicator is easily measurable. The indicator has relatively high proxy potential because of the potential for an area under sustainable NRM to continue to contribute to long-term impacts over time. Limitations: The indicator is not attributable to the project and lacks a time frame. Improved forest cover profile in project 23 4 Andhra Pradesh Satisfactory The indicator measures improvements in forest cover profiles in VSS areas (sq. km.) Community Forest areas managed by forest user groups. The indicator's concept has Management Project high potential as a proxy because of the potential for forest cover (P073094) in India to be maintained over the long term and contribute to continued impacts. Limitations: The indicator is less specific because of the use of the term "improved" and the lack of a target to improve clarity on what is meant by improved, which makes the indicator more difficult to objectively measure. The indicator also lacks attribution to the project and a time frame. 83 84 Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Rating Strengthen and expand the 22 4 Central Asia Satisfactory The indicator measures the strengthening and expansion of zapovednik (strict nature reserves) Biodiversity GEF nature reserves with the aim of conserving plants and animals. network in the West Tien Shan to Project (P042573) The indicator has relatively high proxy potential because of the conserve unique plant and animal in Central Asia potential for strengthened and expanded nature reserves to communities, including wild relatives of (Kazakhstan, the continue to conserve plants and animals over time. Limitations: domesticated species Kyrgyz Republic, and The indicator uses the term "strengthened," which is subjective, Uzbekistan) particularly in the absence of a target, so scores low on specificity as well as measurability. It is also not clearly attributable to the project nor time-bound. Environmental management: Rate of 22 4 Forestry Development Satisfactory The indicator measures increased forest cover. The indicator is forest coverage increased, percent in Poor Areas Project specific because it includes a desired target rate for the increase, (P046952) in China making it measurable and realistic. The concept of increased forest cover has potential as a proxy because of the potential of increased forest cover to continue to generate impacts over time. Limitations: The indicator is not attributable to the project and lacks a time frame. Annual sustainable woodfuel 22 4 Sustainable and Highly The indicator measures the area of sustainable woodfuel production Participatory Energy satisfactory production. It is relatively specific, with the inclusion of the target, Management although the term "sustainable" is subjective and could be more Project (PROGEDE; clearly defined, which would increase the measurabilty of the UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS P046768) in Senegal indicator. By including the tons/per year target, the indicator is one of the few that includes a time frame. The indicator has relatively high proxy potential because annual sustainable woodfuel production is likely to continue to contribute to long-term results. In addition, although the indicator does not specify it, sustainable woodfuel production may involve behavioral change over woodfuel users; if so, the potential for long-term impacts is even higher. Limitations: In addition to the limitations mentioned above, the indicator does not indicate a desired change or increase, although it does specify a desired area. The indicator may also be time- intensive to measure and it is not attributable to the project. Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Rating Mechanisms to increase forest cover 21 4 Uttar Pradesh and Satisfactory The indicator measures a positive trend in forest cover area. established (Output 2): Upward trend in Uttaranchal Forestry The indicator has relatively high potential as a proxy because area of forest cover Project (P035169) of the potential for upward trends in forest cover to continue to in India generate impacts over time. Limitations: The indicator could be more specific on the desired amount or percentage of increase, particularly by including a target, although it still generally captures the idea of positive forest cover trends. The indicator is also not clearly linked to the project and lacks a time frame. Improved degree of biodiversity 19 4 Peruvian National Satisfactory This indicator measures improved biodiversity conservation in conservation in Project ANPs Trust Fund for protected areas. Although the indicator scores poorly on the PAs-Programme SMART criteria, it was still rated as having relatively high proxy for a Participatory potential because the concept of biodiversity conservation Management of PAs in project areas has potential for long-term impacts. Still, Project (P068250) the indicator would likely need to be revised before it could in Peru be recommended as a proxy. Limitations: The indicator lacks specificity; for instance, the term "improved" could be more specific and conservation could also be defined--more species UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS numbers? fewer threatened species? greater forest cover? The lack of specificity may also make the indicator difficult to measure. The indicator is somewhat linked to the project through the target on programs in national parks and activities but it may be hard to show how these activities contributed to improvements in biodiversity conservation. The indicator also lacks a time frame. Forest fire control: Fire damage area/ 25 3 Daxinganling Forest Satisfactory This indicator measures the control of forest fires by measuring total area (percent) Fire Rehabilitation the percent of damaged areas. The indicator has some limited Project (P003550) in potential for a proxy as currently written because it is not clear China how the percentage of area saved will contribute to long-term impacts and does not include variables that suggest the project is likely to be on track to achieve results over time. The indicator is specific and measurable. Limitations: The indicator is not attributable to the project and is not time-bound. 85 86 Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Rating Sustainable Natual Forest Management 23 3 First Programmatic Satisfactory The indicator measures the expansion of sustainable forest of private and public areas expanded Development Policy management areas. The concept of expanding sustainable forest from 27,000 km2 to 50,000 km2 Loan for Sustainable management areas has potential as a proxy because such areas Environmental may contribute to longer-term impacts. However, the expansion Management (SEM of areas alone may not be enough to ensure longer-term impacts; DPL; P095205) in because the indicator does not suggest the involvement of actors Brazil in maintaining this area or other factors, it was rated lower. Limitations: The indicator is specific, but the use of the term "sustainble" is somewhat subjective, which resulted in a slightly lower score. The indicator also lacks attribution to the project and a time frame. Improved habitat and wildlife 18 3 Transfrontier Satisfactory This indicator measures improved habitat and wildlife management in three TFCAs Conservation management in conservation areas. It has potential as a proxy Areas Pilot and because of the potential for improved habitat and wildlife Institutional management to generate long-term impacts. However, the Strengthening indicator's low score on the SMART criteria suggests that the Project (P001759) in indicator requires revision before it can be recommended. Mozambique Limitations: The indicator lacks specificity: what is meant by improved, particularly in the absence of a target? Consequently, the indicator is harder to measure and less realistic. It also lacks attribution to the project and a time frame. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Enhance biodiversity conservation 18 3 Forest Resource Satisfactory The indicator measures enhanced biodiversity conservation in through the protection of nature Development and nature reserves. The indicator has relatively low proxy potential reserves Protection Project as currently written because of its poor score on the SMART (P003557) in China criterion, although the general concept of enhanced conservation and protection has potential for long-term impacts. Limitations: The indicator appears to state an aim and then an indicator, which makes it less specific than if it just focused on protection of nature reserves. It lacks a target, which could have increased its specificity, making it less measurable and realistic. It also lacks attribution to the project and a time frame. Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Rating Develop improved models for watershed 18 3 Forest Resource Satisfactory This indicator measures the relationship between protected management through establishment of Development and forests and improved watershed management. The proxy potential protection forests Protection Project is low because of the indicator's poor score on the SMART criteria (P003557) in China and a weak articulation of the relationship between protection forests and watershed management models, which makes the indicator less clear. Limitations: In addition to the ones mentioned above, the indicator is not specific, using terms such as "improved models," which could be interpreted in different ways, and not including a target, which makes the indicator less measurable. The indicator also is not clearly attributed to the project and lacks a time frame. Production of nonarable lands improved 17 2 Karnataka Watershed Satisfactory The indicator measures the improvement in production of [PAD adds: Forest cover, and forest Development Project nonarable lands. The indicator has low proxy potential because of produce increased, output from (P067216) in India its low SMART score and because it is not clear how production revenue lands increased] will lead to impacts over time. Limitations: The indicator is not specific, not easy to measure, not attributable to the project, and lacks a time frame. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Promote conservation of biodiversity: 14 2 Madhya Pradesh Satisfactory The indicator measures the biodiversity conservation and related 1) Improved PA management; 2) Policy Forestry Project activities, such as management and policy frameworks. The framework for ecodevelopment (P010506) in India indicator has limited potential of a proxy because of its low SMART score; it is simply not clear enough to be recommended as an indicator that could suggest long-term impacts and measure change over time. Limitations: The indicator lacks specificity and will be hard to measure, particularly in the absence of a target. It is also not attributable to the project and lacks a time frame. Area currently under forest concession 22 1 Forest Concession Unsatisfactory The indicator measures area under forest concessions. The in Cambodia Management and indicator is specific and measurable. It has low proxy potential Control Pilot Project because information about the current area does not provide (P060003) in information on change over time as a result of the project. Cambodia Limitations: The indicator lacks attribution to the project and a time frame. 87 88 CLIMATE CHANGE MITIGATION AND ADAPTATION INDICATORS Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Rating Forest area brought under Yes NA 5 1. Functioning Various CSI. Indicator used for two different outcomes potentially: Positive adaptive, biodiversity-friendly institutions to projects environmental impacts (climate change–related) OR Increased carbon management plans enforce rules and stocks. resolve conflict. 2. Predictable, sustainable financing. OR 1. Predictable, sustainable financing. 2. Area restored or reforested (ha). Predictable, sustainable financing Yes NA 5 1. Forest area Various CSI. Indicator used for two different outcomes potentially: Positive brought under projects environmental impacts (climate change–related) OR Increased carbon adaptive, stocks. biodiversity-friendly management plans. 2. Functioning institutions to UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS enforce rules and resolve conflict. 1. Forest area brought under adaptive, biodiversity-friendly management plans. 2. Area restored or reforested (ha). Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Rating Functioning institutions to enforce NA 5 1. Forest area Various Part of PPI cluster with outcome of positive environmental impacts rules and resolve conflict brought under projects (climate change–related) adaptive, biodiversity-friendly management plans. 2. Predictable, sustainable financing. 60,000 terajoules per year to be 29 4 First Satisfactory This indicator measures the production of renewable energy and energy produced by renewable energy Programmatic efficiency. The indicator has relatively high proxy potential based on the sources or saved by energy Development assumption that renewable energy will be produced each year over the efficiency projects supported Policy Loan for long term, with the potential to contribute to longer-term impacts in by BNDES, once they are fully Sustainable energy savings. The indicator is specific, measurable, and attributable to operational Environmental the project. Management (SEM DPL; P095205) in Brazil UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Net CO2 emission reduction 24 4 Sustainable Highly The indicator measures reductions in carbon dioxide emissions. The and satisfactory indicator is specific. It has relatively high potential as a proxy indicator Participatory because of the potential for reduction each year, with the potential for Energy a long-term contribution to carbon dioxide emissions. Limitations: The Management indicator may be difficult to measure and attribute to project. Project (PROGEDE; P046768) in Senegal 89 90 Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Rating Increased planned signed 22 3 First Satisfactory The indicator measures reductions in emissions from CDM and other reduction of emissions from Programmatic projects. The indicator is specific in its aims, though could be more CDM projects, from other BNDES Development specific in terms of what is included in "other actions," and measurable. projects, and from other actions Policy Loan for The indicator has some potential as a proxy because the planned monitored under National Climate Sustainable emissions reductions per year could continue to contribute to emissions Change Action Plan. Environmental reductions, but the indicator could benefit from some revision in wording Management to make it more specific and linked to the project. Limitations: In addition (SEM DPL; to the limitations mentioned above, the achievement of the indicator may P095205) in be difficult to attribute to the project. Brazil Woodfuel stoves promotion 18 3 Sustainable Highly The indicator measures the promotion of woodfuel stoves. The indicator and satisfactory lacks specificity, which makes it difficult to measure. The indicator Participatory also has relatively low proxy potential because it simply captures the Energy promotion of stoves without considering their adoption or uptake by Management target groups or other actions that could contribute to ensuring long-term Project impacts. Limitations: In addition to the limitations mentioned above, (PROGEDE; the indicator will not be easy to attribute to the project as it is currently P046768) in worded and it lacks a time frame. Senegal Kerosene stoves promotion 18 3 Sustainable Highly The indicator measures the promotion of kerosene stoves. The indicator UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS and satisfactory lacks specificity, which makes it difficult to measure. The indicator Participatory also has relatively low proxy potential because it simply captures the Energy promotion of stoves without considering their adoption or uptake by Management target groups or other actions that could contribute to ensuring long-term Project impacts. Limitations: In addition to the limitations mentioned above, (PROGEDE; the indicator will not be easy to attribute to the project as it is currently P046768) in worded and it lacks a time frame. Senegal Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Rating Promote investment in climate 18 3 Natural Moderately The indicator measures the promotion of climate change adaptation change adaptation and mitigation Resources and satisfactory and mitigation. The indicator has relatively low proxy potential because Environmental it will be difficult to show how the promotion of investment will lead to Governance investment that will contribute to long-term impacts as the indicator is (NREG) currently worded. Limitations: In addition to the limitations mentioned Program (DPO- above, it will be hard to measure the promotion of something— 1: P102971; measuring a percentage or amount increase in investment would be DPO-2: more clear and would have improved the indicator. The indicator is also P113172; DPO- not clearly linked to the project. The targets are more specific but the two 3: P118188) in targets (original and revised) go in different directions. The indicator also Ghana lacks a time frame. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 91 92 GOVERNANCE INDICATORS Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Ratings Proportion of citizens who consider Yes N/A 5 1. Changes to Various projects Indicator can be used to meet World Bank Group citizen engagement that design and/or implementation project activities requirements. of project subject to consultation as a result of is responsive to their views consultations (yes/ (percent) no). 2. Grievances registered related to delivery of project benefits that are actually addressed (percent). Grievances registered related to Yes N/A 5 1. Proportion Various projects Indicator can be used to meet World Bank Group citizen engagement delivery of project benefits that of citizens who requirements. are actually addressed (percent) consider that design and/or implementation of project subject to consultation is responsive to their views (percent). 2. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Changes to project activities as a result of consultations (yes/no). Forest area brought under N/A 5 Various projects Key indicator in multiple PPI clusters. strengthened tenure or use rights Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Ratings Changes to project activities as a N/A 5 1. Proportion Various projects Indicator can be used to meet World Bank Group citizen engagement result of consultations (yes/no) of citizens who requirements. consider that design and/or implementation of project subject to consultation is responsive to their views (percent). 2. Grievances registered related to delivery of project benefits that are actually addressed (percent). Increase capacity to finance 29 5 Peruvian National Satisfactory The indicator measures increased capacity for financing, which has SINANPE recurrent costs with Trust Fund for high potential as a proxy indicator because financing is a critical local resources PAs-Programme component in ensuring sustainability of project activities and UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS for a Participatory long-term impacts. The indicator, with the inclusion of the target, is Management specific, measurable, attributable to the project, and includes a time of PAs Project frame, all of which contribute to a realistic and robust indicator. (P068250) in Peru Enactment of legislation designed 27 5 Economic Recovery Satisfactory The indicator measures the development of legislation as a first to reinvigorate the mining Credit (P057293) step in reforming the legal and regulatory framework for mining sector, including first steps for in DRC operations. The indicator is specific with the target information and restructuring Geacamines, the will be easy and cost-effective to measure. The indicator has high mining parastatal proxy potential because the development of legislation can play a key role in ensuring sustained long-term impacts. 93 94 Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Ratings Enactment of legislation Yes 27 5 Economic Recovery Satisfactory The indicator measures the development of legislation as a first supporting reform of the forestry Credit (P057293) step in reforming the legal and regulatory framework for forestry sector in DRC operations, with the aim of ensuring good governance in forest management operations. The indicator is specific with the target information and will be easy and cost-effective to measure. The indicator has high proxy potential because the development of legislation can play a key role in ensuring sustained long-term impacts. Transparency International Index 26 5 Economic Satisfactory This indicator uses an external measure to show improvement. It has Management a high potential as a proxy because it has the potential to capture a and Governance change in governance and related concepts that have the potential to Reform Grant contribute to sustaining long-term impacts. Other benefits of using (EMGRG; P106458) an external measurement is that the indicator will be cost-effective in Central African to measure and will provide a measurement that is not biased by the Republic project. The indicator is also specific and realistic. Limitations: The indicator is not clearly attributable to the project and lacks a time frame. Number of functioning forest user 26 5 Andhra Pradesh Satisfactory This indicator measures forest user groups that are implementing groups (VSS) undertaking forest Community Forest forest management plans. It has high proxy potential because of management tasks according Management the assumption that active inclusion and involvement of local forest UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS to microplans and operating Project (P073094) users in forest management will contribute to sustainable, long-term according to their covenants and in India impacts. The indicator is clear and specific and able to be measured. agreed procedures Limitations: Measurement could be time-consuming and resource- intensive, as the indicator is unlikely to be captured in government or existing surveys. The indicator also lacks attribution to the project and a time frame. Tenure of at least 30 percent of 29 4 Forests and Rural Moderately The indicator measures the formalization of land tenure in project the occupants in national lands Productivity satisfactory areas with the assistance of the project. The indicator has relatively in the project area regularized Project (P064914) high proxy potential because of its potential to capture long- through formal, long-term in Honduras term impacts. The indicator is specific, measurable, and clearly usufruct agreements or title attributable to the project, which makes it realistic. Limitations: The developed with the assistance of indicator lacks a time frame. the project Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Ratings 150,000 hectares under 27 4 Indigenous and Satisfactory The indicator measures the area under community and sustainable community conservation in Community use, which is somewhat similar to the core sector indicator on different ecozones in the project Biodiversity area brought under management plans. The indicator is clear, area and 150,000 hectares Conservation attributable to the project by noting the project areas, and should be of complementary area under Project (COINBIO; relatively cost-effective to measure. It has high potential for a proxy sustainable use PO66674) in indicator because of the assumption that community involvement in Mexico management can contribute to long-term impacts. Limitations: The indicator lacks a time frame and uses the term "sustainable," which can be defined and interpreted in different ways. Share of the projects 26 4 First Programmatic Satisfactory The indicator measures the submission, approval, and monitoring of submitted directly to BNDES Development projects submitted, therefore capturing more than just the existence screened, approved, and Policy Loan for or development of projects but the implementation as well. The monitored according to the Sustainable indicator has relatively high potential as a proxy with the potential new Environmental and Social Environmental to achieve long-term impacts, though it would be even higher if it Institutional Policy Management (SEM were more clear how the projects would address long-term impacts DPL; P095205) in and projects aims. The indicator is specific, measurable, and Brazil attributable. Limitations: Although the indicator is generally realistic, UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS achieving 100 percent seems like a difficult target and so it is rated slightly lower. It also lacks a time frame. Increased participation of 26 4 Peruvian National Satisfactory The indicator measures participation of women at different stages, women in planning, managing, Trust Fund for which is important. The indicator has high proxy potential because monitoring, and evaluating Project PAs-Programme of the assumption that participation will lead to sustainable, long- ANPs for a Participatory term impacts; further, because beneficiaries are participating in the Management planning, managing, monitoring, and evaluating of project ANPs, of PAs Project they are more likely to feel a sense of ownership and investment in (P068250) in Peru long-term outcomes. The indicator is specific, especially with the inclusion of the target, and measurable. Limitations: The indicator lacks a time frame. 95 96 Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Ratings 6900 villages benefiting from 26 4 Sustainable Satisfactory The indicator aims to capture the benefits to villages from project participation in project activities Forestry activities and is unique in including a time frame. The indicator is by 12/31/07 Development also attributable to the project. The indicator has relatively high Project (P064729) proxy potential because of the assumption that receiving benefits and Sustainable from the project will contribute to project support and longer- Forestry term impacts. Limitations: The indicator does not specify how the Development villages will benefit (financial? improved livelihoods? improved water Project (Natural sources? other?), which makes it less specific and measurable. CSI Forest Protection) - Potential: This could be interpreted to fulfill: "Number of people with P060029 in China increased monetary or nonmonetary benefits from forests." Establishment of a financial 24 4 Ecomarkets Project Satisfactory The indicator measures the establishment of a financial instrument instrument to support easements (P061314 [GEF to support biodiversity conservation, which is an important concept targeting biodiversity conservation CR-Ecomarkets] for ensuring sustainable and longer-term impacts, making this in Costa Rica by EOP and P52009 indicator have a high proxy potential. The indicator is specific and [Ecomarkets somewhat attributable to the project. The indicator also includes a Projects]) in Costa time frame, which is rare among the indicators. Limitations: The Rica indicator could be more specific in terms of the desired level of financing. Water quality monitoring and 25 4 First Programmatic Satisfactory This indicator measures the monitoring and evaluation of water evaluation for 90,000 km of main Development quality, with the aim of using results for improvement. The indicator UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS rivers being executed and publicly Policy Loan for also suggests government commitment to improving water quality released on a regular basis, with Sustainable and ensuring public access to information. The indicator has high results used for prioritization of Environmental proxy potential because it encompasses multiple actions necessary investments for improved water Management (SEM to ensure water quality over the long term, such as prioritization quality DPL; P095205) in of water quality activities and commitment to provide information. Brazil The indicator is specific and relatively measurable. Limitations: The indicator may be difficult to measure because it will be necessary to define how results have been used to prioritize improvements, or will at least involve specific measurements that will have to be carried out, which increases the cost and time of measurement. The indicator could also be more clearly attributable to the project and include a time frame. Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Ratings 70 organizationally advanced 24 4 Indigenous and Satisfactory The indicator measures community involvement in conservation, communities (Category 3 and 4) Community using defined categories. The indicator is specific, particularly with active conservation (and Biodiversity with the inclusion of a way to categorize communities, and is integrated resource use) on Conservation attributable to the project. It has high proxy potential because of the communally owned land of high Project (COINBIO; potential for communities who are involved in managing resources biodiversity in Oaxaca, Guerrero, PO66674) in to support their sustainable use and contribute to long-term and Michoacan. Mexico impacts. Limitations: The indicator lacks a time frame. It also uses the concept of "active" conservation, which could be more clearly defined. Technical capacity of staff in FD Yes 23 4 Uttar Pradesh Satisfactory The indicator measures improved technical capacity among two and local communities in forest and Uttaranchal target groups through the development and implementation of management improved Forestry Project a training plan. The indicator has relatively high proxy potential (P035169) in India because of the assumption that improved capacity of these target groups will contribute to improved forest management, which will contribute to longer-term impacts. Limitations: The indicator may be difficult to measure because it does not include a baseline of capacity or a way to evaluate improved capacity. The indicator can UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS be assumed to be attributable to the project, although it does not explicitly show the link. It also lacks a time frame. 97 98 Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Ratings Increased coordination at a 23 4 MX Programmatic Satisfactory The indicator measures increased coordination among sectors, multisectoral level, as evidenced EnvSAL (First which has relatively high potential as a proxy indicator because of by preparation of policy packages Programmatic the assumption that coordination and consensus can contribute to that are based on consensus Environment longer-term, sustainable impacts. The addition of "as evidenced among key stakeholders Structural by" is a nice example of how an indicator can be made to be Adjustment more specific and shows how something like coordination can be Loan (EnvSAL elaborated upon. The indicator is measurable through the policy I) and Second packages, although it could be more specific on the desired amount. Progammatic Limitations: The indicator lacks a target and a time frame. Environment Development Policy Loan (EnvDPL II) (P074539 for EnvSAL I and P079748 for EnvDPL II) in Mexico Management plans (ha): Yes 24 4 Forestry Sector Satisfactory The indicator measures both the preparation and implementation Preparation and implementation of (P001168) in Côte of forest management, including the desired area of forests under management plans for 700,000 ha d'Ivoire these plans, which is similar to the core sector indicator on forest UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS of these forests, of which 500,000 area brought under management plans. The indicator is specific and ha would be managed by private elaborates upon components of a management plan, which makes companies under long-term it measurable. The concept has relatively high potential as a proxy contracts; the preparation of a because of the assumption that management plans will lead to long- detailed land use and agricultural term management. Limitations: The indicator includes many different development plan for the buffer components, making it somewhat less specific and more difficult zone of 1 gazetted forest; and to measure. The indicator also lacks attribution to the project and a incremental agricultural inputs to time frame. help farmers intensify agriculture in the buffer zones Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Ratings Strengthen the operating 22 4 Forest Resource Satisfactory The indicator measures strengthened technical support to the forest efficiency of technical support Development and sector. It has relatively high proxy potential based on the assumption services in the forest sector, Protection Project that increased support will contribute to longer-term impacts. particularly planting materials, (P003557) in Limitations: The indicator could be more specific; "strengthened" is research, and extension programs China not specific, especially in the absence of a target, which makes it more difficult to measure the indicator. It is not clearly linked to the project and lacks a time frame. Sustainable funding mechanisms 22 4 Uttar Pradesh Satisfactory The indicator measures the establishment of sustainable funding established and Uttaranchal mechanisms. The concept is important for proxies, which resulted in Forestry Project a relatively high proxy rating, but the indicator could be more specific (P035169) in India in terms of what the desired mechanism is, amounts of percentages that would ensure sustainability, etc. Limitations: As noted above, the indicator could be more specific, which would make it more measurable. It also lacks attribution to the project and a time frame. Number of hectares of forests 22 4 Energy Access Moderately This indicator measures the area under participatory management, brought under participatory Project (P049395) unsatisfactory which is somewhat similar to the core sector indicator on areas under management in Ethiopia management plans. The indicator has relatively high proxy potential because participatory management is considered important in UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS ensuring sustainable, long-term impacts. Limitations: The indicator could be more clear; participatory management is somewhat subjective and could be defined in different ways by different users. The indicator is also not attributable to the project and lacks a time frame. CSI potential: This could be interpreted as: "Forest area brought under management plan." 99 100 Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Ratings Strengthen institutions and 22 4 Natural Resources Moderately This indicator measures strengthened forest governance and governance [in forestry sector] and Environmental satisfactory suggests a commitment to increased legality in the forest sector. Governance (NREG) These concepts have potential as proxies because of the assumption Program (DPO-1: that stronger institutions, governance, and legal supplies of timber P102971; DPO-2: will lead to sustainable, long-term impacts. It is important to P113172; DPO-3: note, however, that the target was revised, resulting in a different P118188) in Ghana direction, though still generally on forest legality issues. Limitations: The indicator uses the term "strengthened," which could be more specific. The indicator is somewhat unrealistic, because the original target aims to achieve 100 percent legality, which is a high bar that may not be achievable. The indicator is not clearly attributable to the project and lacks a time frame. Reduce social conflict issues in 22 4 Natural Resources Moderately The indicator measures reduction in social conflicts in mining mining communities and improve and Environmental satisfactory communities. The target helps to elaborate on the concept of social support to small scale miners Governance (NREG) conflict. The reduction of social conflict in general has potential (SSM) Program (DPO-1: as a proxy indicator because of the assumption that reduced P102971; DPO-2: social conflict can contribute to consensus on decision making or P113172; DPO-3: management and a generally stronger community environment in P118188) in Ghana which to realize project achievements. Limitations: The revised target does not seem to link as clearly to the indicator as the original UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS target; for instance, it is not clear what the survey aims to do and how a survey will reduce social conflict. The indicator is not specific in how social conflict will be reduced with the revised target and lacks attribution to the project and a time frame. The indicator may also be difficult to measure, as a reduction in social conflict could be a subjective measurement without clear baselines and targets. Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Ratings Number of firms in breach of 22 4 Economic Satisfactory This indicator measures compliance with forest regulations, a national regulations in forestry Management concept that can substitute for improved forest governance. The and Governance indicator has potential as a proxy for institutional aspects of Reform Grant forestry because of the potential for firm compliance on regulations (EMGRG; P106458) to contribute to long-term impacts. The indicator is very clear and in Central African specific. Limitations: The indicator may be difficult to measure if Republic transparency or reporting are limited and it could be time-consuming to measure. The indicator is also somewhat unrealistic, because achieving 100 percent compliance may be too ambitious. The indicator also lacks attribution to the project and a time frame. Number of firms in compliance 22 4 Economic Satisfactory This indicator measures compliance with oil regulations, a concept with new regulations in the oil Management that can substitute for improved oil governance. The indicator has sector and Governance potential as a proxy because of the potential for firm compliance Reform Grant on regulations to contribute to long-term impacts. The indicator is (EMGRG; P106458) very clear and specific. Limitations: The indicator may be difficult in Central African to measure if transparency or reporting are limited and it could Republic be time-consuming to measure. The indicator is also somewhat UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS unrealistic, because achieving 100 percent compliance may be too ambitious (although the small number of firms makes it more realistic). The indicator also lacks attribution to the project and a time frame. Significant increase in social 22 4 Community Satisfactory The indicator measures increased social capital among project capital in assisted communities Forestry II communities. Social capital is considered an important factor in (PROCYMAF II; sustainable community management of resources, so the concept P035751) in Mexico in general has strong proxy potential, although the indicator would likely need to be reworded, especially because "significant" is not specific. By including the term "assisted" communities, the indicator includes a link to the project. Limitations: As noted above, the term "significant" could be more clearly defined with a clear amount, desired increase or baseline. This limitation makes the indicator harder to measure. The indicator also lacks a time frame. 101 102 Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Ratings Increase forest cover and 22 4 Madhya Pradesh Satisfactory The indicator measures multiple components of forest management, productivity through development Forestry Project including increased forest cover and productivity. Although the of participatory processes for (P010506) in India indicator is complex, the components fit together as steps of a management and use of forest whole. Several of the concepts in the indicator, including JFM and resources, taking special account participatory management processes, contribute to the indicator's of the interests of tribals and relatively high proxy potential. Limitations: The indicator aims to other disadvantaged groups 1) measure too many issues through one indicator, which makes it less JFM approach established; 2) PRA specific and clear and more difficult to measure. The indicator also based microplanning methods lacks attribution to the project and a time frame. established; 3) silvicultural practices adapted to multiple objectives of JFM; 4) restoration techniques for degraded areas based on natural regeneration tested; 5) measure tribal interests in planning; 6) measure interest of other disadvantaged groups, scheduled castes and women, landless Forest fire control: accuracy of fire 21 4 Daxinganling Satisfactory The indicator indirectly measures improved capacity to forecast fires UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS forecast (percent): a) developed Forest Fire by measuring the percentage increase in accuracy of forecasts. This areas; b) undeveloped areas Rehabilitation indicator has potential as a proxy because it implicitly shows how Project (P003550) human resource investments in improved capacity can contribute in China to a long-term outcome and therefore a measure of "improved" institutional effectiveness. Limitations: The indicator will depend on having good data for measuring, which may be difficult. Although the indicator is implicitly attributable to the project, the link is not explicit. The indicator also lacks a time frame. Indicator CSI SMART PPI Related PPI Use Context Outcome Notes Score Potential Indicators Ratings Forest fire control: time for 21 4 Daxinganling Satisfactory The indicator indirectly measures improved capacity to discover discovering fire, a) developed Forest Fire forest fires by measuring the time to discover fires. This indicator areas (minutes); b) undeveloped Rehabilitation has potential as a proxy because it implicitly shows how human areas (minutes) Project (P003550) resource investments in improved capacity can contribute to a long- in China term outcome and therefore a measure of "improved" institutional effectiveness. Limitations: The indicator will depend on having good data for measuring, which may be difficult. Although the indicator is implicitly attributable to the project, the link is not explicit. The indicator also lacks a time frame. Reinvestment by VSS in forest 21 4 Andhra Pradesh Satisfactory This indicator measures local investment in forest management. areas from generated funds Community Forest The indicator concept has relatively high potential as a proxy based (number of VSS) Management on the assumption that investment by forest user groups in forest Project (P073094) areas signifies interest in the long-term sustainable management in India of forests. Limitations: The indicator could be more specific in terms of what reinvestment means and the desired percent increased. The absence of a target also makes this indicator more difficult to measure. The indicator further lacks attribution to the project and a time frame. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Increase in local government's Yes 21 4 Sustainable Satisfactory This indicator measures local government capacity to conduct capacity to supervise, monitor, Forestry conservation activities. The indicator has relatively high potential and implement conservation and Development as a proxy because of the assumption that improved capacity in the sustainable resource use activities Project (P064729) specified areas will contribute to sustainable impacts over time. and Sustainable The indicator captures different components of increased capacity, Forestry rather than a single component, making it a more robust indicator. Development Limitations: The indicator could be more specific on the desired Project (Natural increase in capacity, such as by including a baseline and target. This Forest Protection; limitation also makes the indicator more difficult to measure. The P060029) in China indicator lacks attribution to the project and a time frame. 103 104 OTHER INDICATORS Indicator CSI SMART PPI Related PPI Indicators Use Outcome Notes Score Potential Context Rating Government officials and relevant project Yes N/A 5 1. Government officials and project staff regularly conduct Likely key PPI indicator useful in staff provided with capacity building high quality M&E. 2. Predictable, sustainable financing for different contexts. support to improve management practices M&E activities (including after project completion). to design and implement M&E Government officials and project staff Yes N/A 5 1. Government officials and relevant project staff provided Likely key PPI indicator useful in regularly conduct high-quality M&E with capacity building support to improve management different contexts. practices to design and implement M&E. 2. Predictable, sustainable financing for M&E activities (including after project completion). Predictable, sustainable financing for Yes N/A 5 1. Government officials and relevant project staff provided with Likely key PPI indicator useful in M&E activities (including after project capacity building support to improve management practices different contexts. completion) to design and implement M&E. 2. Government officials and project staff regularly conduct high-quality M&E. Development, establishment, and Yes N/A 5 Likely key PPI indicator useful in implementation of a financial mechanism different contexts. or trust fund to support activities or efforts identified as critical for the continued UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS achievement of forest investment objectives, including capacity building and training to ensure that local actors can manage and disburse funding (Footnotes) * Denotes Intermediate Outcome (IO) indicator. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 105 A n n e x D. G l o s s a r y Activity. Actions taken or work performed through which inputs, such as funds, technical assistance, and other types of resources are mobilized to produce specific outputs (DAC 2002). Related term: development intervention. Attribution. The ascription of a causal link between observed (or expected to be observed) changes and a specific intervention. Attribution refers to that which is to be credited for the observed changes or results achieved. It represents the extent to which observed development effects can be attributed to a specific intervention or to the performance of one or more partners taking account of other interventions, (anticipated or unanticipated) confounding factors, or external shocks (DAC 2002). Counterfactual. The situation or condition that hypothetically may prevail for individuals, organizations, or groups were there no development intervention (DAC 2002). By definition, a counterfactual cannot be observed. Therefore it must be estimated using comparison groups. Development intervention. An instrument for partner (donor and non-donor) support aimed to promote development. Examples include policy advice, projects, and programs (DAC 2002). Related term: activity. Impacts. Positive and negative, primary and secondary long-term effects produced by a development intervention, directly or indirectly, intended or unintended (DAC 2002). Impacts represent the ultimate result of an outcome, which may often only become evident several years or more after project completion (World Bank 2014a). Related term: longer-term outcome. Indicator. Quantitative or qualitative factor or variable that provides a simple and reliable means to measure achievement, to reflect the changes connected to an intervention, or to help assess the performance of a development actor (DAC 2002). An indicator is a variable that measures a phenomenon of interest. The phenomenon can be an input, an output, an outcome, a characteristic, or an attribute. Inputs. The financial, human, and material resources used for the development intervention (DAC 2002). Intermediate outcome. A result that is proximate to an intended final outcome but often more achievable and measurable during a project’s lifetime than an intended final outcome (World Bank 2014a). Longer-term outcome. Outcomes that are typically not visible at project closure and may not be apparent until 5–25 years after a project closes (World Bank 2014a). Related term: impacts. Proxy indicator. An indirect measurement of a variable lacking direct information (IEG 2012) and a substitute for an indicator that is hard to measure directly and may reveal performance trends, potential problems or areas of success (World Bank 2014a). 106 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Predictive proxy indicator. Or predictive proxy; a specific type of proxy indicator, which seeks to provide information about the future. This study has developed this term to refer to a measure taken during implementation of a project, program, or policy that stands in for impacts, often ones that take a long time to materialize. Outputs. The products, capital goods, and services that result from a development intervention; may also include changes resulting from the intervention that are relevant to the achievement of outcomes (DAC 2002). Outcome. The likely or achieved short-term and medium-term effects of an intervention’s outputs (DAC 2002). Related terms: results, outputs, impacts. Results. The output, outcome, or impact (intended or unintended, positive and/or negative) of a development intervention (DAC 2002). Results chain. The causal sequence for a development intervention that stipulates the necessary sequence to achieve desired objectives, beginning with inputs, moving through activities and outputs, and culminating in outcomes, impacts, and feedback (DAC 2002). It describes how particular inputs will likely lead to intended outcomes (World Bank 2014a). Results framework. The program logic that explains how the development objective is to be achieved, including causal relationships and underlying assumptions (DAC 2002). Sustainability. In the context of development evaluation, sustainability refers to the continuation of benefits from a development intervention after major development assistance has been completed. It also denotes the probability of continued long-term benefits and the resilience to risk of the net benefit flows over time.   UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 107 References Agrawal, A. 2001. “Common Property Institutions and Sustainable Governance of Resources.” World Development 29 (10): 1649–72. Blundell, R., L. Dearden, and B. Sianesi. 2005. “Evaluating the Effect of Education on Earnings: Models, Methods and Results from the National Child Development Survey.” Journal of the Royal Statistical Society: Series A 168 (3): 473–512. Buch, B., M. Buntaine, and B. Parks. 2015. Aiming at the Wrong Targets: The Difficulty of Improving Domestic Institutions with International Aid. AidData Working Paper No. 4. Cawley, J. 2004. “The Impact of Obesity on Wages.” The Journal of Human Resources 39 (2): 451–74. CIFOR (Center for International Forestry Research). 2015. “Evidence-Based Forestry.” Available at http://www1.cifor. org/ebf/about.html. Accessed 14 April 2015. CODE (Committee on Development Effectiveness). 2013. Report to the Board from the Committee of Development Effectiveness: Managing Forest Resources for Sustainable Development: An Evaluation of World Bank Group Experience and Draft Management Response. Washington, DC: World Bank. DAC (Development Assistance Committee). 2002. Glossary of Terms in Evaluation and Results-Based Management. Paris: Organisation for Economic Co-operation and Development. Deforce, K., A. Storme, and J. Bastiaens. 2014. “Middle-Holocene Alluviual Forests and Associated Fluvial Environments: A Multi-proxy Reconstruction from the lower Scheldt, N. Belgium.” The Holocene 24: 1550–64. FAO (Food and Agriculture Organization). 2014. State of the World’s Forests 2014: Enhancing the Socioeconomic Benefits from Forests. Rome: FAO. Global Environment Facility (GEF). 2014.  Combating Land Degradation in Production Landscapes: Learning from GEF Projects Applying Integrated Approaches. Washington, DC: World Bank. Huselid, M. 1995. “The Impact of Human Resource Management Practices on Turnover, Productivity, and Corporate Financial Performance.” The Academy of Management Journal 38 (3): 635–72. IEG (Independent Evaluation Group). 2012. Designing a Results Framework for Achieving Results: A How-To Guide. Washington, DC: World Bank. ———. 2013. Managing Forest Resources for Sustainable Development: An Evaluation of World Bank Group Experience. Washington, DC: World Bank. 108 UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS Kishor, N., and M. A. de Rijk. 2014. Assessing Impacts of Forest Governance Interventions: Learning from World Bank Experience. Washington, DC: World Bank. Kishor, N., D. C. Miller, J. Virdin, and C. Wahlén. Forthcoming. “The Contribution of Tenure Reforms to Sustainable Management of Forests and Fisheries.” In Looking Back and Looking Ahead: The World Bank’s Experience with Land Tenure Projects, ed. J. Munoz et al. Washington, DC: World Bank. Lawry, S., C. Samii, R. Hall, A. Leopold, D. Hornby, and F. Mtero. 2014. “The Impact of Land Property Rights Interventions on Investment and Agricultural Productivity in Developing Countries: A Systematic Review.” Campbell Systematic Reviews 10.1. Levin, K., B. Cashore, S. Bernstein, and G. Auld. 2012. “Overcoming the Tragedy of Super Wicked Problems: Constraining Our Future Selves to Ameliorate Global Climate Change.” Policy Sciences 45(2): 123–52. Mann, M., R. Bradley, and M. Hughes. 1998. “Global-Scale Temperature Patterns and Climate Forcing over the Past Six Centuries.” Nature 392: 779–87. Mendelsohn, R. 1994. “Property Rights and Tropical Deforestation.” Oxford Economic Papers 46: 750–56. Miteva, Daniela A., S. K. Pattanayak, and P. J. Ferraro. 2012. “Evaluation of Biodiversity Policy Instruments: What Works and What Doesn’t?” Oxford Review of Economic Policy 28.1: 69–92. Morrone, Adolfo, Noemi Tontoranelli, and Guilia Ranuzzi. 2009. How Good Is Trust? Measuring Trust and Its Role for the Progress of Societies. Paris: OECD Statistics Working Paper. OECD (Organisation for Economic Co-operation and Development). 1993. OECD Core Set of Indicators for Environmental Performance Reviews. Paris: OECD. ———. 1994. Environmental Indicators. Paris: OECD. Ostrom, E. 1990. Governing the Commons: The Evolution of Iinstitutions for Collective Action. Cambridge, U.K.: Cambridge University Press. Ostrom, E., and E. Schlager. 1996. “The Formation of Property Rights.” In Rights to Nature: Ecological, Economic, Cultural, and Political Principles of Institutions for the Environment, ed. S. Hanna and C. Folke, 127–56. Washington, DC: Island Press. Psacharopoulous, G. 1994. “Returns to Investment in Education: A Global Update.” World Development 22 (9): 1,325–43. Sayer, J., T. Sunderland, J. Ghazoul, J. Pfund, D. Sheil, E. Meijaard, M. Venter et al. 2013. “Ten Principles for a Landscape Approach to Reconciling Agriculture, Conservation and Other Competing Land Uses.” Proceedings of the National Academy of Sciences 110 (21): 8349–56. Schorr, L., and C. Weiss. eds. 1995. New Approaches to Evaluating Community Initiatives: Concepts, Methods, and Contexts. Washington, DC: Aspen Institute. UNDERSTANDING LONG-TERM IMPACTS IN THE FOREST SECTOR: PREDICTIVE PROXY INDICATORS 109 Scriven, Michael. 1991. Evaluation Thesaurus. Newbury Park: Sage. Shyamsundar, Priya. 2002. Poverty-Environment Indicators. Washington, DC: World Bank. Smith, D., B. Larson, M. Kelty, and P.M. S. Ashton, eds. 1997.  The Practice of Silviculture: Applied Forest Ecology. New York: John Wiley and Sons, Inc. Stem, C., R. Margoluis, N. Salafsky, and M. Brown. 2005. “Monitoring and Evaluation in Conservation: A Review of Trends and Approaches.” Conservation Biology 19 (2): 295–309. Stuart, S., and B. Collen. 2013. “Conserving Biodiversity in a Target-Driven World.” In Biodiversity Monitoring and Conservation: Bridging the Gap between Global Commitment and Local Action, ed. B. Collen, N. Pettorelli, J. Baillie, and S. Durant. New York: John Wiley and Sons, Ltd. Thomson, V., and X. Luo. 2011. “Overlooked Links in the Results Chain.” IEG Evaluation Brief 14. Washington, DC: World Bank. Weiss, C. H. 1995. “Nothing as Practical as Good Theory: Exploring Theory-Based Evaluation for Comprehensive Community Initiatives for Children and Families” In New Approaches to Evaluating Community Initiatives, ed. J. Connell and A. Kubisch. Washington, DC: Aspen Institute. ——. 2000. “Theory-Based Evaluation: Theories of Change for Poverty Reduction Programs.” In Evaluation and Poverty Reduction, ed. O. Feinstein and R. Picciotto. Washington, DC: Operations Evaluation Department, World Bank. Woolcock, M. 2013. “Using Case Studies to Explore the External Validity of ‘Complex’ Development Interventions.” Evaluation 19 (3): 229–48. World Bank. 2009. 2009 Annual Review of Development Effectiveness: Achieving Sustainable Development. Washington, DC: World Bank. ———. 2013. ICT FOR DATA COLLECTION AND MONITORING & EVALUATION: Opportunities and Guidance on Mobile Applications for Forest and Agricultural Sectors. Washington, DC: The World Bank. ———. 2014a. Results Framework and M&E Guidance Note. Washington, DC: World Bank. ———.2014b. Core Sector Indicators and Definitions. Washington, DC: World Bank. Wunder, S., A. Angelsen, and B. Belcher. 2014. “Forests, Livelihoods, and Conservation: Broadening the Empirical Base.” World Development 64: S1–S11.