Independent Evaluation Group designing a results framework for achieving results: a how-to guide 4 1 designing a results framework for achieving results: a how-to guide The World Bank I ndependent E valuation G roup Washington, D.C. 2 Acknowledgements This booklet was written by Dawn Roberts (independent consultant) and Nidhi Khattri (IEG). Peer review comments from Susan Stout (consultant) and Maurya West Meiers (IEG) are gratefully acknowledged. The task manager for the work was Nidhi Khattri. The IEG Blue Booklet series disseminates practical information on various aspects of monitoring and evaluation. Objectives of the series are to focus on implementation aspects of monitoring and evaluation and to disseminate information on other sources of knowledge. The series is prepared under the overall guidance of Hans-Martin Boehmer, Senior Manager, IEGCS. © 2012 Independent Evaluation Group Strategy, Learning, and Communication International Bank for Reconstruction and Development/World Bank 1818 H Street, N.W. Washington, DC 20433 Email: ieg@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 Internet: http://ieg.worldbankgroup.org All rights reserved ISBN-13: 978-1-60244-220-7 ISBN-10: 1-60244-220-7 The opinions expressed in the report do not necessarily represent the views of IEG/ the World Bank Group or its member governments. IEG/the World Bank Group do not guarantee the accuracy of the data included in this publication and accept no responsibility whatsoever for any consequence of their use. 2 Table of Contents Introduction................................................................................................ 5 What Is a Results Framework?..................................................................... 7 Uses of Results Frameworks....................................................................... 14 What Is Required to Design a Results Framework? ................................... 20 Understanding of Problem and Needs Assessment............................ 20 Specifying the Theory of Change...................................................... 21 Step-by-Step Guide................................................................................... 26 Step 1. Establish Strategic Objective(s) for the Problem(s) to Be Addressed......................................................................................... 26 Step 2. Identify and Work with Stakeholders.................................... 27 Step 3. Define Results (Outputs and Outcomes).............................. 28 Step 4. Identify Critical Assumptions and Risks................................ 29 Step 5. Review Available Data Sources and Specify Indicators........... 29 Step 6. Assign Indicators and Data Sources for Each Level of Result.33 Step 7. Establish the Performance Monitoring Plan.......................... 38 Step 8. Establish a Communication and Dissemination Plan............ 40 Challenges................................................................................................. 41 Bibliography.............................................................................................. 43 3 4 5 Introduction It is difficult to know if programs have succeeded or failed if the expected results are not clearly articulated. An explicit definition of results— precisely what is to be achieved through the project or program and by when—keeps measurable objectives in sight, helps monitor progress toward those objectives, and assists with adjustment and management of program implementation. Results-based management is a key tool for development effectiveness. Recent years have witnessed a trend in explicitly specifying the results (outcomes and impacts) of both broader country strategies and more specific programs and projects. Internationally agreed principles have underpinned this push for results, most notably with the adoption of the United Nations Millennium Declaration (2000), which established the Millennium Development Goals with targets and indicators to provide the basis for measuring progress and the effectiveness of aid. Landmarks in this emerging results orientation include the Monterrey Consensus (2002), the Rome Declaration on Harmonization (2003), the Paris Declaration (2005), the Hanoi Conference on Managing for Development Results (2007), the Accra Agenda for Action (2008), and the Busan Partnership for Effective Development Cooperation (2011). Each of these agreements underscored the importance of increased accountability of governments, donor agencies, and other partners toward the achievement of results. A results framework serves as a key tool in the development landscape, enabling practitioners to discuss and establish strategic development objectives and then link interventions to intermediate outcomes and results that directly relate to those objectives. This publication provides how-to guidance for developing results frameworks by discussing the following: ■ The definition of a results framework. What is it? How does a results framework complement and differ from a traditional monitoring and evaluation logical framework? 6 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s ■ Uses for results frameworks. What are the functions of a results framework? At what levels can one be developed and used effectively? ■ Requirements to design a results framework. Is there an assessment and diagnosis process to understand the problem and desired results before the design and implementation of the intervention is developed? Does the team adequately understand the problem that a development intervention is designed to address? Has the program or project logic been defined? ■ Designing a results framework step by step. What are the steps in formulating a results framework? How should practitioners establish strategic objectives and articulate the expected results? What is the process through which results, indicators, and data sources can be assigned for each level of desired result (output, outcome, and impact)? What are the criteria for designing a useful results framework? Who should be involved in developing and using the framework? ■ Challenges. What are the potential pitfalls in developing results frameworks? What strategies help in avoiding these? This publication also provides various examples or excerpts of results frameworks used at various levels (for example, country, project, and organization) and offers references for further support to practitioners in designing and using results frameworks for development effectiveness. For more on context, see Binnendijk (2000) and OECD-DAC (2008). 7 What Is a Results Framework? A results framework is an explicit articulation (graphic display, matrix, or summary) of the different levels, or chains, of results expected from a particular intervention—project, program, or development strategy. The results specified typically comprise the longer-term objectives (often referred to as “outcomes” or “impact”) and the intermediate outcomes and outputs that precede, and lead to, those desired longer-term objectives. Although the World Bank has used the term “results framework” over the last decade, similar conceptual tools, also designed to organize information regarding intended outcomes and results, are used across different agencies: logical frameworks, logic models, theories of change, results chains, and outcome mapping. Thus, the results framework captures the essential elements of the logical and expected cause-effect relationships among inputs, outputs, intermediate results or outcomes, and impact. There are many debates, and considerable controversy, on the distinctions among outputs, outcomes, and impact. A generally useful approach is to consider outputs as the particular goods or services provided by an intervention (for example, nutrition supplements), whereas an outcome is usefully thought of as benefits of that particular good or service to the target population (such as improved nutrition intake), and impact refers to evidence on whether outcomes are actually changing beneficiary behavior or longer-term conditions of interest (for example, improved eating habits, a healthier population). The key is to distinguish between the provision of goods and services (which involves supply-side activities) and actual demand for and/or utilization of those goods and services (demand-side response). Defining cause-effect linkages for one or more interventions lays the groundwork for a results framework. Thus, the development of a good results framework requires clarity with respect to the theory of change—the reasons why the project, program, or strategy will lead to the outputs; why those outputs are likely to lead to the immediate or intermediate outcomes; and how those outcomes are (at least hypothetically) linked with longer-term outcomes or impact. The 8 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s theory of change also requires knowing or estimating how long it will take to achieve each stage of the program and how much of the outcome is likely to be achieved. Thus, defining cause-effect linkages for one or more development interventions lays the groundwork for a results framework. Outcomes and impacts are the main focus of a results framework; project inputs and implementation processes are generally not emphasized, although outputs are often noted. This conceptual presentation of a results chain (outputs, outcomes, and impacts) is often accompanied by a more detailed plan for monitoring progress toward the ultimate objectives through measuring the achievement of outputs, outcomes, and impacts at different intervals of time. Results are typically defined through indicators, which are often, but not always, quantifiable and measurable or observable.1 (Some indicators are qualitative.) The monitoring plan typically includes baseline values and targets expected for outputs and outcomes, and it specifies the measures that will be used for data gathering to ensure that the results framework is actually populated with data, updated with information at key points during program/project implementation, and used in decision making. A results framework also often identifies any underlying critical assumptions that must be in place for the intervention to be successful, that is, to lead to achieving the targeted outcomes and impacts. Table 1 presents a basic outline of a country-level results framework that incorporates indicators for each level of result expected. A well-constructed results framework is beneficial for monitoring, management, and evaluation in several ways: n It helps focus on specific outcomes. A well-conceived results framework clearly outlines the ultimate objectives of the project, program, or strategy, rather than simply listing implementation activities, processes, and inputs. It facilitates a focus on specific expected outcomes. 1. In some settings, desired outcomes may include changes in organizational or institutional behaviors, which may best be tracked through qualitative data. W h at I s a R e s u lt s F r a m e w o r k ? 9 Table 1. Basic Outline of a Results Framework Issues/ Country obstacles/ Outcomes Outputs/ Use of development critical expected milestones monitoring goals assumptions Statement of first [critical issues Statement of first Statement of first [short descriptive country goal and obstacles to outcome output/milestone to text highlighting Indicator achieving country Indicator be realized within the how the Baseline: xxxx (2005) development goals] Baseline: xxxx (2005) time of the results information will Target: xxxx (2010) Midline: xxxx (2007) framework be used] Additional/alternative Target: xxxx (2010) Indicator (if quantitative indicator Additional/alternative milestone) Baseline: xxxx (2005) indicator Baseline: xxxx (2005) Target: xxxx (2010) Baseline: xxxx (2005) Target: xxxx (2006) [continue with Midline: xxxx (2007) xxxx (2007) additional indicators Target: xxxx (2010) xxxx (2008) or move to next goal] [continue with xxxx (2009) additional indicators xxxx (2010) or move to next Additional/alternative outcome] indicator (if quantitative) Statement of second [continue with country goal additional indicators or Indicator Statement of second move to next milestone] Baseline: xxxx (2005) outcome Target: xxxx (2010) Indicator Statement of second Additional/alternative Baseline: xxxx (2005) outputs/ milestone indicator Midline: xxxx (2007) [continue as above] Baseline: xxxx (2005) Target: xxxx (2010) Target: xxxx (2010) Additional/alternative [continue with indicator additional indicators Baseline: xxxx (2005) or move to next goal] Midline: xxxx (2007) Target: xxxx (2010) [continue with additional indicators or move to next outcome] n It highlights the key linkages in the theory of change that underpin the intervention. A simple but clear results framework engages constituents in thinking through the theory of change underpinning the intervention. Discussion of a results framework often requires program staff and other constituents to identify the development hypothesis—Why would a particular intervention lead to the outputs identified and the outcomes expected? How does it link with the ultimate objective? This participatory discussion serves a critical role in building consensus and ownership around shared 10 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s objectives and clarifying different interpretations of the elements of the hypothesis. n It helps establish an evidence-based approach to monitoring and evaluation. By including specific indictors of outcomes and impacts and identifying baselines and targets to be achieved, results frameworks help answer the question, How will we know that the intervention has succeeded? n It helps measure progress toward strategic objectives. The emphasis on concrete outcomes rather than on the completion of activities requires that program implementers monitor key outcome variables and make midstream corrections as necessary. A results framework is therefore a useful management tool, with program implementation assessed in direct relationship to progress in achieving results, at the outputs, outcomes, and impact levels. n It helps achieve strategic objectives. The strategic objective is the ultimate driver of a program. Interventions might range in complexity from a simple intervention in a community to a number of interrelated interventions at a national level. A results framework can include outcomes of many related projects or of nonproject activities, if they arerelevant to the strategic objective, rather than simply charting the expected achievements of an isolated development initiative. All intermediate results needed to achieve the strategic objective are specified, allowing partners to harmonize their efforts or to identify areas where additional program activities will be needed. The preferred format and level of detail for results frameworks vary by organization and by the scope and scale of the intervention, but all include the same basic components to guide implementers in achieving, and evaluators in assessing, results. Examples of project-level results frameworks from one World Bank project are shown in Tables 2 and 3. For more information on what a results framework is, see Imas and Rist (2009) and USAID (2000). W h at I s a R e s u lt s F r a m e w o r k ? 11 Table 2. Sample Project-Level Results Framework Project development Project development Use of outcome monitoring objective objective indicators Increase small producers’ Average yield of commodities in Project development objective indicators would productivity and market targeted value chains (bananas, show the efficiency and effectiveness of sub- access for targeted irrigated rice, coffee and milk) projects and other project investments in boosting commodities in the project (tons/ha or liters per cow, agricultural productivity (increase in yields) area respectively). and generating market surpluses (fraction of production marketed) for targeted value chains. Intermediate outcome Outcome indicators Use of outcome monitoring Component 1: Agricultural Percentage of participating This indicator would show how effective project technology transfer and farmers (male/female) adopting services are in assisting farmers with technology linkage to market new technology packages (for change. production, post-harvest, processing, etc.) This indicator would show the effectiveness of the Percentage of producers adopting transfer of knowledge and advisory services. animal breeds and husbandry practices for milk production This indicator would show to what degree Percent participating producer producer groups/associations/cooperatives are groups/associations/cooperatives embarking on commercial agriculture. having contractual arrangements with marketing agents This indicator would show how the project would affect other people living in the project area. Number of direct project beneficiaries of the new technological packages and This indicator would show the overall effectiveness market linkages of subproject completion. Number of indirect project beneficiaries of the new technological packages and market linkages Source: World Bank, adapted from the Agro Pastoral Productivity and Markets Development Project in Burundi (2010). 12 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s Table 3. Sample Project-Level Results Framework, Including Target Outcome Values Target values Project outcomes YR5 indicators Baseline YR1 YR2 YR3 YR4 (Target) Average yield of targeted commodities Rice (t/ha) 2.5 2.8 3.3 3.8 4.0 4.0 Banana (t/ha) 9.0 NA 12 14 16 16 Coffee (t cherries/ha) 0.4 0.5 0.6 0.7 0.8 0.8 Milk (l/yr) 360 400 500 650 800 950 Percent of production of commodity in targeted value 10 12 18 22 26 30 chains marketed by participating producers Target values (cumulative) Support to agricultural productivity and market process Percentage of participating farmers (male/female) adopting new technology packages (for 10 15 30 45 60 70 production, post-harvest, processing, and so forth) Percentage of producers adopting animal breeds and husbandry 10 15 30 45 60 70 practices for milk production Percentage of participating producer groups/associations/ cooperatives having contractual 0 2 5 10 15 20 arrangements with marketing agents Source: World Bank, adapted from the Agro Pastoral Productivity and Markets Development Project in Burundi (2010). Note. PCU = Project Coordination Office; IPC = Interprovincial Coordinating Unit; PS = Private Sector Providers W h at I s a R e s u lt s F r a m e w o r k ? 13 Data collection and reporting Responsibility for data Frequency and reports Data collection instruments collection End of cropping season Annual producer survey PCU End of annual marketing period Annual producer survey PCU End of annual marketing period Annual producer survey PCU End of annual marketing period Annual producer survey PCU End of annual marketing period Annual producer survey PCU Cropping season/ Cropping season/annually IPCU/PSP annual sample survey Annually Annual sample survey IPCU/PSP Annually Annual sample survey IPCU/PSP 14 Uses of Results Frameworks The advantages of using results frameworks translate to an array of applications for practitioners or others pursuing development objectives: n Planning. Working backwards from the strategic objectives, a results framework offers a systematic approach for practitioners to plan their strategies and to select interventions that are most likely to address targeted problems. The process of designing a results framework guides a program team in establishing a valid development objective, assessing what intermediate outcomes and outputs are needed to achieve that objective, and designing or prescribing appropriate interventions aligned with the desired cause-and-effect linkages. (See the “Step-by Step Guide” section in this booklet.) n Consensus, coordination, and ownership. A results framework provides the opportunity for the primary implementers (an agency, operating unit, project team, and so forth) to work with key stakeholders to agree on coordinating the implementation approach, agreeing on the expected results, highlighting and checking the underlying assumptions, and specifying needed resources. n Management. Performance data can and should inform programming decisions. A results framework is a tool for guiding corrective adjustments to activities, reallocating resources, and reevaluating targeted objectives or underlying assumptions. It provides a way to understand and make decisions related to unintended (especially negative) effects of the program (for example, increased traffic accidents in the local community associated with a new road). It is therefore very useful for those involved in constructing a results framework to make explicit their assumptions about who will be using indicators at each link in the chain, as well as what form of and the frequency of decisions each user will be encouraged to make in response to changes in the indicator(s). n Communication and reporting. In defining a program’s causal relationships, a results framework acts as a vehicle for Uses of R e s u lt s F r a m e w o r k s 15 communicating about the resources, activities, and outcomes to program staff, development partners, or other stakeholders. These frameworks can be an important tool in illustrating to the beneficiaries or community what a project is meant to achieve; steps should be taken to share this information in posters at community centers, on program websites, and through other means. n Evaluation. The specification of each level of results with associated indicators, measures, and targets establishes an effective framework for ongoing monitoring and evaluation, including early on in the project or program cycle. A results framework clearly identifies how progress toward the targeted objective(s) will be measured and thus provides the basis for the development and use of the implementing unit’s performance monitoring system. It also serves as the basic accountability tool for developing an evaluation approach to the intervention. n Harmonization. The planning, consensus building, and communication functions of a results framework all help development partners (for example, government teams, donors, nongovernmental organizations, and so forth) clarify efficient divisions of responsibilities for achieving specific results. Different operating units can coordinate efforts to design related activities (project or nonproject) to achieve the intermediate outcomes for which they are responsible. Figure 1 provides an example of this approach. n Learning from experience. Over time, the systematic use of results frameworks allows practitioners to assess what approaches or interventions contribute most effectively to achieving specific development objectives, a process that helps identify good practices for replication. A body of knowledge also forms regarding which indicators, measures, and data sources are best suited to monitoring progress in similar contexts. In summary, a results framework underpins a strategic planning process and serves as a living management tool—fostering ownership and consensus, guiding corrective actions, facilitating the coordination of development efforts, charting the course for achieving a strategic objective, and ultimately serving as key accountability tool for evaluation. 16 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s Figure 1. Sample Results Framework: Harmonization of Development Efforts Agency objective More rapid and enhanced agricultural development and food security encouraged Strategic objective Increased use of improved production practices by farmers in the Upper River Zone (6 years) Farmers’ access to Community control over Farmers’ knowledge Farmers’ transport costs commercial capital local resources increased about production options decreased (5 years) increased (5 years) (5 years) increased (4 years) Farmers’ Role of Banks’ loan Village Input/ Village Farmers’ capacity forestry policies more associations’ output associations’ exposure to develop agents New favorable for contract markets control to on-farm bankable changed technologies rural sector negotiation liberalized over local experiences loan from available (3 years) capacity (3 years) resources of peers applications regulatory to (4 years) increased increased increased increased outreach (4 years) (4 years) (3 years) (4 years) (2 years) Farmers’ capacity to make enterprise management USAID solely decisions increased Adult literacy increased materially Key (3 years) (2 years) responsible USAID plus Partner(s) partner solely materially materially responsible responsible Source: Adapted from USAID 2000. Note: Critical assumptions: (i) Market prices for framers’ product remain stable or increase. (ii) Prices of agricultural inputs (for example, fertilizer or seeds) remain stable or decrease. (iii) Roads needed to get produce to market are maintained. (iv) Rainfall and other critical weather conditions remain stable. Uses of R e s u lt s F r a m e w o r k s 17 Results frameworks can serve these functions at various levels. n They can be used to chart the contributions of an individual project to a development objective. n They help take a results-oriented approach to addressing objectives for a particular sector or subsector. n An organization can use a results framework to guide and gauge progress in pursuing its mission. n Country-level results frameworks are also commonly used, with national governments and development partners including them as part of their national development plans, country assistance strategies, joint assistance strategies, country development programming frameworks, or other official strategies. Table 4 provides an example of the World Bank Group’s contributions to country-level goals. For more information on the uses of results frameworks, see OECD-DAC (2008). 18 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s Table 4. Sample Results Matrix Country dev’t. Outcomes the World Bank Group goals Issues and obstacles program is expected to influence (from the NSDI) Strategic objective 1: Accelerating the recovery to Albania’s economic growth through improved competitiveness (i) Improving business regulations and reducing compliance costs for the private sector “A favorable business The cost of doing business in Albania Reduce the cost of business, as evidenced by: climate will be created remains high in part due to administrative • Two full Regulatory Impact Assessments for the dynamic corruption and a high “time tax.” Albania’s conduced per year by 2013. development of investment climate is undermined by • Decrease in the number of inspections per private enterprise inadequate property administration and business per year (status: 32 in 2009; target: and the attraction of weaknesses in urban land administration. 12 in 2011) foreign investment.” • Decrease in time to receive a building permit (status: 180 days in 2007; target: 100 days at end-2011) Strategic objective 2: Broadening and sustaining Albania’s social gains (ii) Broader access to education, particularly secondary and higher education, and improved quality “Establish a good Albania needs to broaden its sources of Improved access to and quality of secondary quality, inclusive and productivity and employment growth education, as indicated by: flexible education through more effective investment in • Increase in secondary enrollment (status: 60% system that will human capital – secondary enrollment is in 2009; target: 100% at end 2014) respond to the low in part because of poor quality and • Triple shifts eliminated, double shifts reduced demands of the learning outcomes, and tertiary education in secondary schools by 2014 market.” needs to be more market responsive. Reform of higher education institutions as indicated by: • Fully functioning quality assurance system for higher education Source: World Bank. Note: NSDI = National Strategy for Development and Integration; IFC = International Finance Corporation; TA = Technical Assistance. Uses of R e s u lt s F r a m e w o r k s 19 Milestones World Bank Group instrument Increase in government officials trained in regulatory review Business Environment Reform and Inst techniques (status: 3 in 2008; target: 375 in 2011). Strengthening Project Regulatory framework for inspections revised and adopted Land Administration and Management Project Increase in number of titles issued in urban areas (status: 130,000 Governance Partnership Facility Expropriations $ in 2009; target: 320,000 at end 2011) Safeguards TA Complete computerization if Immovable Property Registration IFC Trade Logistics Program Office. IFC Tax Simplification Project IFC Subnational Competitiveness Program SEE Doing Business initiative Complete reform of all three grades of secondary curricula Education, Excellence, and Equity Project Increase in teachers participating in continuous professional Education Project (FY14) development (status: 25% in 2009; target: 70% at end 2010) Construction of 12 new schools; extension of 8 existing schools; and rehabilitation of 10 schools completed by end 2013 Performance-based financing in place for all (11) public universities by 2014 20 What Is Required to Design a Results Framework? Designing a results framework is an iterative process, with objectives and interventions providing the base for its design, and draft results frameworks in turn helping to clarify specific objectives and interventions. A results framework builds on, and helps articulate, a project’s or program’s theory of change—the causal pathways from the planned interventions to the intended outcomes. Actions for developing a results framework therefore start with understanding both the problem to be addressed and the desired outcomes, specifying the program logic, and building stakeholder consensus related to this theory of change. Once this agreement is in place, stakeholders can focus on selecting appropriate indicators to measure intended outputs and outcomes, setting baseline and target values, and exploring the relevance of available data and data collection methods. All these steps in the results framework process require that practitioners come prepared with four types of information: n An understanding of the problem or assessment of needs that the development intervention is intended to address n An initial theory of change for the project or program, even as it is being designed n A working knowledge of evidence required for measuring and assessing desired outcomes and impacts n Available data sources and proven data collection approaches relevant for the project or program context. These components provide a solid foundation on which to base a results framework. Understanding of Problem and Needs Assessment The needs assessment or problem analysis stage is critical for informing the pathways and outputs and intermediate outcomes needed to achieve each objective. Practitioners should consider the following questions: W h a t I s R e q u ir e d to Design a R e s u lt s F r a m e w o r k ? 21 n What are the current results being achieved and the hoped-for results related to the issue? In other words, what is the gap between these results, and what are the expectations for minimizing the gap? n What are the main challenges or barriers that have constrained stakeholders from reaching the development objectives? For example, is the limited access to markets for farmers and manufacturers caused by poor roads or restrictive trade policies? n What current conditions in the area or population of interest might hinder or facilitate progress toward the development objective? What else is happening (for example, projects in the country, community efforts, and so forth) that might also be working toward addressing the gap in results? n What are the stepping stones that will lead from the current status to the desired results? In other words, what outputs and intermediate outcomes are most likely to contribute to and mark progress toward the targeted objective? n What are the risks that the program might face from factors outside the program? n What other relevant experiences in the region or the sector could help the implementation team better understand how to achieve the strategic objective? n Who is potentially affected by the interventions, and how can their opinions inform the needs assessment process? The answers to these questions help determine which stakeholders should be engaged in program planning, implementation, and assessment, as well as which intermediate objectives are critical to the success of the intervention. Specifying the Theory of Change A results framework must be based on a clear understanding and specification of how any planned interventions are expected to lead to desired outcomes. The theory of change model allows stakeholders to visualize the logic of an intervention and identify the proposed causal links among inputs, activities, outputs, and outcomes. An example 22 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s of this logical sequence for achieving one national development goal is shown in Figure 2. Various formats and approaches can be used to depict this causal chain, with each designed to help practitioners understand not only the intended outcomes of a program but also the inputs and activities needed to achieve them. Four main areas should be considered for accurately predicting and explaining the conversion process from inputs to results: n Broader context. Development activities occur within a complex system of factors, all of which might facilitate or hinder intended outcomes. These include the macroeconomic environment, social Figure 2. Sample Program Logic to Reduce Childhood Morbidity through the Use of Oral Rehydration Therapy Goal Reduce mortality rates for children under 5 years old Results Outcome Improved ORT for managing childhood diarrhea • 15 media campaigns completed Outputs • 100 health professionals trained • Increased maternal knowledge of ORT services • Increased access to ORT Implementation • Launch media campaign to educate mothers Activities • Train health professionals in ORT • Trainers Inputs • ORT supplies • Funds • Participants Sources: Kusek and Rist 2004; also adapted from Binnendijk 2000. Note: ORT = Oral rehydration therapy. W h a t I s R e q u ir e d to Design a R e s u lt s F r a m e w o r k ? 23 norms, policy context, initiatives of government and development organizations, the political climate, and even the weather. n Prior research and evaluation. Existing literature on prior studies can shed valuable light on how effective proposed interventions are likely to be. Theories that are based on lessons learned and existing data will be stronger than unsubstantiated cause-and- effect linkages. n Level of risk of assumptions. Although graphic presentations of a program’s change theory often show a linear logic chain leading to outcomes, cause-effect relationships are rarely simple and linear. The many factors that affect each stage of the change process must be assessed—by reviewing both the broader context and prior research—to identify which underlying assumptions would facilitate and which ones could endanger the success of the proposed intervention. n Change agents. The success of most development interventions is predicated on the changed behaviors of stakeholders. Depending on the intended outcomes, these stakeholders might, for example, include targeted beneficiaries making use of the intervention’s products or services, government authorities delivering services better or differently, or donors harmonizing their joint support efforts more systematically so as to unburden the government aid recipients. Once practitioners have considered the broader context, reviewed prior research, and assessed underlying assumptions, a visual representation of the causal chain can take various forms. The best foundation for a comprehensive results framework will be established using a variation that explicitly shows the assumptions that are behind the design of the development initiative. Figure 3 shows an example of this approach. For more information on change theory, see W.K. Kellog Foundation (2004), Imas and Rist (2009), and Weiss (1997). 24 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s Figure 3. Assumptions Underlying Expected Change Assumptions Inputs Activities Health is a community issue; communities will form Consumers partnerships to resolve health Active care problems. participation in the reform Communities can influence and process Providers shape policy at the local, state, and national levels. External agents as community partners can serve as catalysts for Payers change. Shifting revenues and incentives Inclusive to prevention and primary care Staff community will improve overall health decision status. making Information on health and External health systems is necessary to technical make informed decisions. assistance Beginnings Planned work Source: Adapted from W.K. Kellogg Foundation 2004. W h a t I s R e q u ir e d to Design a R e s u lt s F r a m e w o r k ? 25 Outputs Outcomes Impact More effective Communitywide coverage distribution and access Improved health status of community health care resources Comprehensive, integrated delivery system Community health assessment Administrative processes for health data, Community-based health Increased health care policy, and information systems system efficiency advocacy Intended results 26 Step-by-Step Guide Ideally, results frameworks are designed at the start of new initiatives as part of the strategic planning process. A common practice, however, is to retrofit a results framework to an operation already in progress or to design one for a new project or portfolio that builds on programs recently completed or still under way. Either way, the following eight steps can guide practitioners in developing an effective results framework, as long as the prerequisite components are in place to support this exercise. Step 1. Establish Strategic Objective(s) for the Problem(s) to Be Addressed A results framework is centered on one or more strategic objectives, the achievement of which represents the ultimate impact envisioned for a set of activities. An appropriate strategic objective will likely n Represent higher-order changes in systems, communities, or organizations n Reflect “an operating unit’s best assessment of what can realistically be achieved… within a given timeframe and set of resources” (USAID 2000) n Be stated clearly and precisely to lend itself easily to objective measurement. In short, a strategic objective is a calculated response to a known problem. That calculated response should be rooted in evidence (ranging from prior research to needs assessments), and the quality and quantity of that evidence should be commensurate with the importance of the problem to be addressed. Typically, simpler or low-stakes problems will require less evidence, and complex or new problems will require a higher level of evidence for decision making. The wording and intent of the objective should be clear and specific enough that practitioners will be able to identify when it has been achieved, as described in Box 1. S t e p - b y- S t e p G u i d e 27 Box 1. Tips for Effectively Stating Strategic Objectives n Emphasize the results of actions, not the actions themselves. For example, instead of “reduce the transmission of HIV/AIDS,” use “reduced transmission of HIV/AIDS.” Instead of “promote credit opportunities for farmers,” use “increased credit available for farmers.” n Maintain a single focus. Multiple objectives with multiple components are challenging to manage and measure. n Test wording to avoid ambiguity. Test the wording with various stakeholders to ensure that the objective is consistently understood and not interpreted differently by different constituents. n Specify the time frame. The amount of time available helps determine what is realistic and feasible for a strategic objective to be achieved. Step 2. Identify and Work with Stakeholders In practice, various key parties (for example, government authorities or development partners) are already involved with the planning and/or implementation at this stage. However, the team designing a results framework should revisit whether all the main stakeholders have been engaged to facilitate consensus and ownership of the initiative. Wherever possible, the views and understanding of expected beneficiaries or target population should be considered in constructing the results framework. The needs assessment process has often highlighted additional players that could influence the outcomes of an intervention. In particular, efforts should be directed at identifying and including the following groups: n Targeted beneficiaries or others from whom a behavior change is expected. If new roads are constructed, are farmers or manufacturers likely to use them to bring their goods to market? If health benefits are provided, will those who are eligible avail themselves of needed services? 28 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s n Those who may realize negative consequences from an intervention. In addition to bringing new jobs to a community, will the manufacturing plant contribute to air, water, and noise pollution that will negatively affect those living nearby? What will the new dam mean for those potentially displaced or who are reliant on the downstream river for income generation? n Those who are expected to allocate additional resources for or related the intervention. Is the planned awareness campaign relying on an announcement that radios will be distributed by another organization? Is the proposed distribution of textbooks and teaching materials expected to be complemented by teacher training initiatives supported by other donors? n Implementers of potentially conflicting development interventions. Are a project’s efforts to reduce the HIV/AIDS infection rate at risk of being compromised by transport corridor-related interventions that give community members opportunities farther from home? n Those with needed decision-making authority. Will achieving the strategic objective require policy changes during the project implementation period? Having representatives of key targeted stakeholder groups engaged early in the planning process will increase the likelihood that intermediate outcomes can be defined realistically and then achieved as planned. Step 3. Define Results (Outputs and Outcomes) The needs assessment or problem analysis process, combined with a review of critical stakeholders, helps the project team or operational unit to identify the outputs and outcomes that must be achieved in order to reach the intervention’s ultimate strategic objective. Typically, a team can identify a large number of relevant intermediate results (outputs and outcomes). An important task in developing a results framework is to reach consensus on a small core set of critical outcomes. Outputs and outcomes represent those causal links in the results chain that bridge the gap between the current status and the desired high-level results. Starting with the end strategic objective(s), practitioners can backtrack to outline a program logic with immediate and intermediate S t e p - b y- S t e p G u i d e 29 outcomes. Figures 4 and 5 provide examples of how specifying outcomes along the results chain establishes a framework for monitoring and evaluation to which appropriate indicators can be assigned. To ensure the accuracy of assigned intermediate outcomes, the consideration of each proposed outcome should include reviewing who is best situated to achieve the outcome (that is, is this within or outside the scope of this intervention?) and how the outcome might be effectively measured. Step 4. Identify Critical Assumptions and Risks Development interventions inevitably rely on some assumptions about factors that are beyond the control of the planners and implementers. Results frameworks should not be based on critical assumptions that are perceived to have a low probability of holding true over the implementation period. If the risks are high, the intervention needs to be reconsidered. For each output and outcome considered critical in the results chain, the framework developers should explicitly note assumptions related to external factors (for example, political environment, economy, climate change, and so forth) that could carry risks. In cases where the assumption is seen to represent a more substantial risk, practitioners commonly adjust the development strategy, develop a contingency plan, and/or establish a risk management plan to monitor and address conditions as needed. Step 5. Review Available Data Sources and Specify Indicators Where possible, measurement strategies should be based on existing data sources or tested data collection methodologies. Relevant information for analysis and aggregation could already be available through administrative databases or through sample or census-based surveys. Before specific indicators are defined for desired outcomes, practitioners should identify data sources that could be used to measure desired changes. In many cases, governments are in the process of institutionalizing monitoring and evaluation systems that rely on ministry data systems. However, a review of the data systems and diagnosis of data capacity relevant for the particular objectives is likely warranted. Beyond verifying 30 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s Figure 4. Summary Results for the International Development Association’s Program in Burkina Faso Accelerated and shared growth Reduced poverty and improved social welfare Overarching objectives of poverty reduction strategy Strategic objective 1: Strategic objective 2: Strategic objective 3: Strategic objective 4: Accelerated and shared Improved access to Increased employment Better governance with growth basic social services and income greater decentralization Priority opportunities for the country poor outcomes 1.1 Increased 2.1 Improved access 3.1 Enhanced labor 4.1 A more efficient regional to and quality of market flexibility judiciary for integration basic education 3.2 Increased commercial cases Country 1.2 Expanded and 2.2 Improved access economic 4.2 Improved outcomes diversified export of the poor to opportunities for public resource to which CAS earnings health and HIV/ women management in contributes 1.3 Improved AIDs services 3.3 Reduced risk a decentralized investment 2.3 Increased access and increased context climate to potable water revenues for rural 4.3 Increased local 1.4 Efficient and sanitation households capacity and economic services 3.4 Community- participation in infrastructure 2.4 Better targeted based natural public policy social protection resource decisions management 3.5 Expanded access to rural infrastructure Source: World Bank. Note: CAS = Country Assistance Strategy. S t e p - b y- S t e p G u i d e 31 Figure 5. Sample Monitoring and Evaluation Indicators Associated with Strategic Objectives • Real per capita income increases by 4% per annum from 2004 onward. • Real per capita income of the poorest quintile rises at least as fast as average income. Sustained pro-poor growth • Poverty headcount declines from High-level country 46% in 2003 to 41% in 2006 and Reduced poverty and 35% in 2015. outcomes improved social welfare • Under-5 mortality declines from 184 in 2003 to 150 in 2009. • Under-5 malnutrition (underweight) declines from 38% in 2003 to less than 30% in 2009. Outcomes to Accelerated and Improved access to Increased income Better governance which CAS shared growth basic social services and employment with increased contributes opportunities decentralization • Noncotton export • Primary school • Average annual • Steady increase in earnings increase enrollment rate production increase annual number of by at least 8% per rises from 52% in of at least 5% for decisions made by annum from 2004 2003 to 60% in key noncotton commercial courts. onward. 2007 and 70% in crops. • Budgetary • Days to register a 2009. • Cotton yield execution fully business reduced • Primary increases by at least deconcentrated in from 45 in 2004 completion rate 10% relative to 13 regions. to 20 or fewer in rises from 26% in 2004 level. • Full 2009. 2003 to 35% in • Number of implementation of • Rigidity of 2009. property titles human resources employment index • Births attended by issued in six pilot information drops from 90 qualified health rural areas increases system. to 50 or lower in staff increase from steadily. • Regulatory 2009. 36% in 2002 to • Female literacy framework defining • Transfer of funds 50% in 2009. increases to at responsibilities and reduced from 10 • Immunization rate least 50% in 26 resource transfers days in 2004 to 2 for DTC3 rises target provinces for to rural communes days. from 57% in 2003 community-driven is in effect. • Increased annual to 87% in 2009. development. • Independent electricity • Condom use • Share of rural observers publish generation of about during last risky population within reports on 50% of 86 GWh at an sexual encounter 2 km of all-season large-scale public availability factor of rises from 69% in road increases. contracts. 70%. 2003 to 80% in • Maintenance 2009 for men. carried out on • Access to safe 100% of classified drinking water road network. in rural and semi-urban areas increases from 88% in 2003 to 92% in 2009. Source: World Bank. Note: CAS = Country Assistance Strategy. 32 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s the existence of data sources and systems, program implementers or evaluators will want to ensure that they would be given regular access to data or data reports by the relevant government agency. Construction of a results framework will often lead to the identification of gaps or missing elements in the availability of information necessary to populate the results framework. It is useful to recognize that developing interventions to improve the availability, validity, and reliability of information can be a useful byproduct of work on the initial design of a results framework. Data can be obtained two main ways: by using existing data available from others or the program or by collecting new data (typically through surveys, focus groups or observations). There are advantages and disadvantages to each. Resources to consult for finding or collecting useful data include the following: Existing Data n Sector or region-specific data sources. Household budget surveys, demographic health surveys, and a host of other routine data collections populate sectoral databases. Sectoral and regional specialists on the team should have insights and expertise to locate data for valid measurement. n Other development data. This includes the World Development Indicators database (see www.worldbank.org/data) and other databases administrated by international development agencies. Collecting New Data n Proven approaches to collecting data. Where existing data cannot effectively measure the desired changes, the lessons of other similar interventions should inform plans for data collection. For example, focus groups with community leaders or surveys of truckers are best only proposed and planned if practitioners are able to draw on experiences where these have been conducted before under similar circumstances. Newly designed data collection strategies, proposed specifically for the intervention, add an additional burden and risk for the project or evaluation team and should be relied on only as a last resort. S t e p - b y- S t e p G u i d e 33 Step 6. Assign Indicators and Data Sources for Each Level of Result Strategic objective(s) and intermediate outcomes reflect constructs that need further definition to be measured. These outcomes need to be translated into a set of measurable indicators to establish whether progress is being achieved. Indicators are tied to results by focusing on one or more characteristics of the outcome. A measure then expresses an indicator’s value quantitatively or qualitatively using SMART criteria, as described in Box 2. There can be indicators of inputs, activities, outputs, outcomes, and impacts. Results frameworks focus on the effects of development interventions and therefore need measures of short- and long-term outcomes and impact. Indicators and measures should be agreed on by key stakeholders for each level in the results chain, making it clear how progress toward strategic objectives will be assessed. Tables 5 and 6 provide examples. Box 2. Characteristics of Effective—SMART—Indicators n Specific. Indicators should reflect simple information that is communicable and easily understood. n Measurable. Are changes objectively verifiable? – Students’ learning achievement – Value of land (number of hectares, multiplied by price per hectare) – Percentage of customers who are satisfied with the availability of potable water or electricity n Achievable. Indicators and their measurement units must be achievable and sensitive to change during the life of the project. n Relevant. Indicators should reflect information that is important and likely to be used for management or immediate analytical purposes. n Time bound. Progress can be tracked at a desired frequency for a set period of time. Source: World Bank Group. 34 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s Table 5. Sample Project-Level Results Indicators Project Development Objective: The development objective of the proposed Second Rural Electrification Project is to increase access to electricity in rural areas of Peru on an efficient and sustainable basis. PDO-level results indicators Unit of measure Baseline Indicator One: Infrastructure established with the capacity Number of potential 0 to provide service to rural households in the long term connections Indicator Two: Actual number of households and Number of new connections 0 businesses connected during the project Indicator Three: Number of households and businesses Number of new renewable connected using individual solar photovoltaic systems (to 0 connections be tracked, no target) Indicator Four: Number of household productive units Number of units adopting adopting electricity using equipment 0 electrical equipment Source: World Bank 2011b. Note: PDO = project development objective; PU = project unit. Table 6. Sample Project Development Objective with Target Values Project Development Objective: To enable the government of Bangladesh to strengthen health systems and improve health services, particularly for the poor PDO-level results indicators Unit of measure Baseline Proportion of delivery by skilled birth attendant among the 11.5% lowest two wealth quintile group Percent UESD 2010 Coverage of modern contraceptives in the low-performing Sylhet: areas of Sylhet and Chittagong 35.7% Percent Chittagong: 46.8% UESD 2010 Prevalence of underweight among children under 5 years of age among the lowest two wealth quintile groups 48.3% Percent BDHS 2007 S t e p - b y- S t e p G u i d e 35 Cumulative target values Responsibility Data sources/ Frequency for data YR 1 YR 2 YR 3 YR 4 methodology collection Electricity service 0 20,000 42,500 Semestral PU providers Electricity service 0 15,500 34,000 Semestral PU providers Electricity service 0 Semestral PU providers Electricity service 0 1,000 3,000 5,000 Semestral providers and PU contractors Cumulative target values Data Respon- Description Fre- source/ sibility (indicator YR1 YR2 YR3 YR4 YR5 quency method- for data definition, ology collection and so on) BDHS every SBA defined as BDHS, 15% 3 yrs; UESD NIPORT medically trained UESD every 2 yrs providers Sylhet & BDHS every BDHS, Chitta- 3 yrs; UESD NIPORT UESD gong: every 2 yrs 50% Percent of children in the two lowest BDHS every quintiles having 43.3% BDHS NIPORT 3 yrs weight-for-age ≤2SD from the median group (Table continues on the following page.) 36 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s Table 6. Sample Project Development Objective with Target Values (cont.) Project Development Objective: To enable the government of Bangladesh to strengthen health systems and improve health services, particularly for the poor PDO-level results indicators Unit of measure Baseline Intermediate result (component one): Service delivery improved Proportion of births in health facilities 23.7% Percent UESD 2012 Number of functional community clinics 10,323 CC Number Project 2011 Coverage of measles immunization for children under 12 82.4% Percent months of age CES 2009 Proportion of infants exclusively breastfed up to 6 months 43% Percent of age BDHS 2007 Proportion of postnatal care for women within 48 hours 20.9% (at least 1 visit) Percent UESD 2010 Source: World Bank 2011a. Note: CC = community clinic; CES = Coverage Evaluation Survey; BDHS = Bangladesh Demographic and Health Survey; EPI = Expanded Program for Immunization; MOHFW = Ministry of Health and Family Welfare; NIPORT = National Institute of Population Research and Training; PDO = project development objective; UESD = Utilization of Essential Services Delivery survey. Definition of indicators at the PDO level: Proportion of delivery by skilled birth attendant among the lowest two wealth quintile groups: The percentage of women age 15–49 from two lowest wealth quintiles, giving live birth in the five years preceding the survey, attended by a medically trained provider. Numerators: Number of live births with a medically trained person providing delivery assistance, that is, a qualified doctor, nurse, midwife, paramedic, family welfare visitor, or community skilled birth attendant. Denominator: Number of live births in the last five years. Coverage of modern contraceptives in the low performing areas of Sylhet and Chittagong: The percentage of currently married women who use any modern method of contraception. Numerator: The number of women who say they use one of the following methods at the time of the survey: female sterilization, male sterilization, contraceptive pill, male condom, IUD, injectables, implants (does not include abortions and menstrual regulation). Denominator: All women between ages 15 and 49 years who are currently married. Prevalence of underweight among under-5 children from the lowest two wealth quintile groups: The percentage of children under 5 years of age underweight from the two lowest wealth quintiles at the time of the survey. Numerator: Number of children with weight-for-age z-score is less than –2 SD below the median of the WHO Child Growth Standards. Denominator: Number of living children younger than 5 years. S t e p - b y- S t e p G u i d e 37 Cumulative target values Data Respon- Description Fre- source/ sibility (indicator YR1 YR2 YR3 YR4 YR5 quency method- for data definition, ology collection and so on) BDHS every BDHS, 40% 3 yrs; UESD NIPORT UESD every 2 yrs 11,000 12,000 13,500 CC Project/ Administra- Every year MIS/ tive Record MOHFW 90% Every year CES EPI 50% Every 3 yrs BDHS NIPORT BDHS every BDHS, 50% 3 yrs; UESD NIPORT UESD every 2 yrs 38 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s The following considerations will help guide this process: n An objective or intermediate outcome could need more than one indicator to measure both its qualitative and quantitative aspects. n A minimal number of indicators should be selected. More information is not necessarily better. A common problem with results frameworks and performance measurement systems is that they are tied to a large set of indicators that are burdensome to track. There is no correct number of indicators to assign per outcome, but the following are useful questions to ask: Is this indicator absolutely necessary to measure whether progress toward the strategic objective is being achieved? Will it create additional burdens on the respondents or on the staff collecting the data? How will this indicator help with monitoring, management, and evaluation? Having multiple stakeholder opinions on the design of the results framework provides an opportunity for discussion and eventual consensus on what good indicators and measures are and what number of indicators will suffice. n Proxy indicators can be used as indirect measures of achievement when direct measures are difficult to assign. For example, where tests of student achievement have not been systematically applied to assess education quality, a proxy measure might focus on student completion rates. Such proxy indicators must be assigned carefully; a common threat to the validity of results measurement is the use of inadequate proxy indicators (for example, students’ self-reported levels of achievement in mathematics). Indicators assigned in the design of a results framework might be modified during implementation, as data capacity increases and new data elements become available. Step 7. Establish the Performance Monitoring Plan The next step in designing a results framework is to plan how it will be operationalized to monitor progress and assess the effects of interventions. The plan for monitoring performance typically lists the following elements in a complementary tool, the monitoring plan: S t e p - b y- S t e p G u i d e 39 n Baseline and target values for selected measures to provide the means for verification to measure changes in the indicators. It is important to remember that the construction of baseline values can be based on available secondary data sources, which can be improved or sharpened through subsequent work. Delaying the preparation of an intervention until a “perfect” baseline is available is rarely feasible or desirable. At the same time, the absence of a baseline too often indicates that the problem definition stage of intervention design is being conducted on the basis of either conventional wisdom or untested assumptions. n Data sources or methods for data collection. n The agent(s) responsible for collecting or providing the data (for example, independent evaluation team, project staff, and so forth). n Designated intervals at which the data will be collected or provided. n Assumptions and risks associated with the indictors or information being collected (such as the assumption that data will be available from a second party). Areas of importance that are often overlooked in the planning process involve the development of the management information system, data entry, data quality, the efficient transmittal of data to a central database, and the development of data utilization guidelines. When multiple agents are involved with providing data or reports (for example, with field-based employees reporting to a central body), consideration should be given to how that data will be transferred to the person(s) or group maintaining the plan and how the users will or should be able to use the information in making decisions. This is particularly important when a program is aggregating information from multiple projects. Web-based monitoring plans are becoming more common. Where Internet access and high connectivity speed are available, and when proper data entry and transmittal guidelines are in place, Web-based monitoring plans allow for quicker reporting and use, contributing to improvements in management and decision making. Although the Internet and new information and communication technologies can significantly reduce the costs and difficulty of collecting information, they are only useful when developed on the basis of a sound 40 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s understanding of the processes of decision making which are part of the implementation of the intervention. Where Web-based data entry and reporting are not practical, provisions should be made for the next best means of transmission, such as the use of Excel templates, which can be transferred by email or hard copy. A basic results framework presenting the strategic development objective and intermediate outcomes is most useful if it is directly associated with a detailed results matrix that provides guidance to the implementation team about how to consistently and systematically track progress during implementation and adjust the design of the intervention as needed. Table 6 illustrates how this progression from high-level results to performance indicators for monitoring works in practice. Step 8. Establish a Communication and Dissemination Plan The final step is to plan how the results framework will be used to communicate the progress and results of the intervention and how the results will be disseminated. Some common approaches are to include results in a “dashboard,” highlighting only the key high-level objectives and outcomes/outputs achieved, using the framework for planning and review meetings (with the current status of the indicators highlighted), and using the change in the indicators from baseline to highlight the results. Thus, choosing the correct outcome indicator (for example, change in rates of HIV) and connecting it to key intervention outputs (number of education campaigns about how HIV is transmitted) can provide a powerful communication and dissemination tool to inform and gather support from key stakeholders. For more information on steps to develop a results framework, see USAID (2000), AusAid (2005), and Gorgens and Kusek (2009). 41 Challenges Results frameworks offer clear benefits to practitioners and others working to achieve development results, but the approach is potentially challenging: n An up-front investment of time and resources is needed at the start of an intervention. The process of establishing an agreed-on set of results and identifying relevant indicators and data sources can be logistically intensive. However, this process is likely to yield greater engagement and ownership among stakeholders and could reduce the resources traditionally required to complete midterm and final evaluations of the project or program. n The effects of interventions can be difficult to measure fully. Translating complex processes into a core set of indicators carries the risk of not fully capturing achievements and progress and of missing unintended consequences, because they are generally not noted in the results framework. This reality underscores the importance of having key stakeholders collaborate in developing and approving the results framework by which implementers will be held accountable. n Results frameworks can become overly complicated. Attempts to apply quantitative and qualitative measures to complex development processes often end up establishing an unwieldy set of indicators to monitor. Practitioners should be mindful that each indicator carries a cost. Thus importance, relevance, cost, timeliness, and utility are key considerations for determining which set of indicators should be included. n Involving program staff in the evaluation process could bias results measurement. Implementers are motivated to collect data that reflect positively on an intervention and its results. Even in cases where a results orientation is integrated throughout the project cycle, evaluators should still be engaged adequately to supervise the monitoring process and to disaggregate the data and conduct analysis that goes beyond what is presented in the results framework. 42 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s For more information on challenges, see Toffolon-Weiss, Bertrand, and Terrell (1999) and OECD-DAC (2008). Ultimately, a results framework provides an important foundation for the results-based management of a development initiative and helps teams keep an eye the achievement of strategic objectives. Investing the time and effort early on to define targeted outcomes and using a disciplined approach to assigning indicators, defining data collection sources and strategies, and establishing a monitoring plan can yield a comprehensive and powerful tool to promote and assess development results. 43 Bibliography Publications AusAid (Australian Agency for International Development). 2005. “Using the Results Framework Approach.” AusGuideline. http://www. ausaid.gov/au/ausguide/pdf/ausguideline2.2.pdf. Binnendijk, Annette. 2000. “Results Based Management in the Development Cooperation Agencies: A Review of Experience.” Paper prepared for OECD-DAC Working Party on Aid Evaluation, February 10–11. Paris. Canadian Department of Foreign Affairs and International Trade. 2005. “Six Easy Steps to Managing for Results: A Guide for Managers.” Evaluation Division, Ottawa. Canadian International Development Agency. 2004. CIDA Evaluation Guide. Evaluation Division, Performance and Knowledge Management Branch, Ottawa. Department for International Development. 2002. DFID Tools for Development—Log Frame. London. Gorgens, Marelize, and Jody Zall Kusek. 2010. Making Monitoring and Evaluation Systems Work: A Toolkit for Development Practitioners. Washington, DC: World Bank. IEG (Independent Evaluation Group). 2007. How to Build M&E Systems to Support Better Government. Washington, DC: World Bank. Imas, Linda G. Morra, and Ray Rist. 2009. The Road to Results: Designing and Conducting Effective Development Evaluations. Washington, DC: World Bank. Khan, M. Adil. 2001. A Guidebook on Results-Based Monitoring and Evaluation: Key Concepts, Issues and Applications. Monitoring and Progress Review Division, Ministry of Plan Implementation. Colombo, Sri Lanka: Government of Sri Lanka. Kusek, Jody Zall, and Ray C. Rist. 2004. Ten Steps to a Results-Based Monitoring and Evaluation System. Washington, DC: World Bank. 44 Designing a R e s u lt s F r a m e w o r k for A c h i e v i n g R e s u lt s OECD-DAC Joint Venture on Managing for Development Results. 2008. Managing for Development Results Principles in Action: Sourcebook on Emerging Good Practice, 3rd ed. Paris: OECD. Otoo, Samuel, Natalia Agapitova, and Joy Behrens. 2009. The Capacity Development Results Framework: A Strategic and Results-Oriented Approach to Learning for Capacity Development. Washington, DC: World Bank Institute. Scott, Alison. 2006. “Emerging Practices of Results-Based Country Programming among Aid Agencies.” Managing for Development Results Principles in Action: Sourcebook on Emerging Good Practice, 1st ed. Paris: OECD-DAC Joint Venture on Managing for Development Results. Toffolon-Weiss, Melissa M., Jane T. Bertrand, and Stanley S. Terrell. 1999. “The Results Framework—An Innovative Tool for Program Planning and Evaluation.” Evaluation Review 23 (3): 336–59. USAID (United States Agency for International Development). 1996. “Selecting Performance Indicators.” Performance Monitoring and Evaluation Tips. Washington, DC: Center for Development Information and Evaluation. ———. 2000. “Building a Results Framework.” Performance Monitoring and Evaluation Tips. Washington, DC: Center for Development Information and Evaluation. Valdez, Joseph, and Michael Bamberger. 1994. Monitoring and Evaluation Social Programs in Developing Countries: A Handbook for Policymakers, Managers, and Researchers. Washington, DC: World Bank. W.K. Kellogg Foundation. 2004. Logic Model Development Guide. Battle Creek, MI. Weiss, Carol H. 1007. Evaluation, 2nd ed. Upper Saddle River, NJ: Prentice Hall. World Bank. 2005a. “Burkina Faso—Country Assistance Strategy.” Report No. 32187, Washington, DC. ———. 2005b. “The LogFrame Handbook: A Logical Framework Approach to Project Cycle Management.” Working Paper No. 31240, World Bank, Washington, DC. ———. 2006. “Lesotho—Country Assistance Strategy.” Report No. 35621, Washington, DC. Bibliography 45 ———. 2007. Healthy Development: World Bank Strategy for Health, Nutrition, and Population Results. Washington, DC: World Bank. ———. 2011a. “Proposed Credit in the Amount of SDR226.40 Million to the People’s Republic of Bangladesh for a Health Sector Development Program.” Project Appraisal Document No. 59979-BD, World Bank, Washington, DC. ———. 2011b. “Proposed Loan in the Amount of US$50 Million to the Republic of Peru for a Second Rural Electrification Project.” Project Appraisal Document No. 60154-PE, World Bank, Washington, DC. Websites Asian Development Bank. Results Framework. http://www.adb.org/ Documents/Policies/ADB-Results-Framework/r166-08.pdf AusAid (Australian Agency for International Development). 2005. Using the Results Framework Approach. AusGuideline. http://www.ausaid.gov. au/ausguide/pdf/ausguideline2.2.pdf Independent Evaluation Group Monitoring and Evaluation Capacity Development. http://www.worldbank.org/ieg/ecd/ OECD Development Assistance Committee Working Party on Aid Evaluation. http://www.oecd.org/home/ OECD/DAC Joint Venture on Managing for Results. Sourcebook on Managing for Development Results. http://www.mfdr.org University of Wisconsin-Extension. Logic Model. http://www.uwex.edu/ces/ pdande/evaluation/evallogicmodel.html USAID Center for Development Information and Evaluation. http://www. usaid.gov/pubs/usaid_eval/ W.K. Kellogg Foundation. Evaluation Resources. http://www.wkkf.org/ World Food Programme. Strategic Results Framework. http://www.wfp.org/ content/strategic-results-framework 4 The World Bank Group 1818 H Street, N.W. Washington, D.C. 20433, U.S.A. Telephone: 202-477-1234 Facsimile: 202-477-6391 Internet: www.worldbank.org Independent Evaluation Group Strategy, Learning, and Communication E-mail: ieg@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125