90176 Outcome-Based Learning Field Guide Tools to harvest and monitor outcomes and systematically learn from complex projects Based on pilot experiences with World Bank teams and clients 1 Copyright © June 2014 The World Bank 1818 H Street, N.W. Washington, D.C. 20433, USA All rights reserved Cover photo: Curt Carnemark, World Bank Photo Collection 2 Outcome-Based Learning Field Guide Tools to harvest and monitor outcomes and systematically learn from complex projects ________________________________________________ Based on pilot experiences with World Bank teams and clients 3 4 Table of Contents Acknowledgments 7 Summary 8 Introduction to Outcome-Based Learning 9 Harvest Outcomes with Stakeholders Identify outcomes during implementation or at review stage Guide for Harvesting Outcomes 13 Tools for Harvesting Outcomes 18 A. Sample Template to Document Outcomes Retrospectively 18 B. Sample Template to Report Outcomes 20 C. Visually Map Outcomes to Date or as They Happen 22 D. Checklist to Review Outcome Information 23 E. Sample Open Questionnaire for Harvesting Outcomes from Stakeholders 24 F. Sample Questions for an Open Interview with Stakeholders 25 G. Sample Template for Terms of Reference: Consultant to Harvest Outcomes 26 Substantiate Outcomes with Stakeholders Validate the significance of and contribution toward the outcomes Guide for Substantiating Outcomes 30 Tools for Substantiating Outcomes 35 A. Sample Tables to Organize Outcome Substantiation 35 B. Sample Communication for Email Outreach to Substantiators 36 C. Sample Interview for Outreach to Substantiators 37 D. Sample Form to Collect Stakeholder Feedback 40 E. Sample Table to Track Substantiation Responses 42 Make Sense of Outcomes with Stakeholders Interpret outcomes to draw lessons and communicate progress Guide for Interpreting Outcomes 44  Is the quality and quantity of the outcome information sufficient?  What is the sustainability of the changes to institutionalize results?  How are the outcomes advancing impact?  How is the program design advancing what needs to be achieved?  Which interventions have been most beneficial?  What are essential “how to” lessons to adapt or inform scale-up?  What are lessons to overcome critical development challenges?  How can a results framework be developed to assess achievement of outcomes?  What progress has been achieved?  Which outcomes reflect setbacks or failures? Tools for Interpreting and Monitoring Outcomes 60 A. Quality Checklist for an Outcome Statement 60 B. Decision Tree for Identifying Institutionalized Results 61 C. Sample Change Strategy Maps and Template for Organizing Outcome Areas 62 D. Sample Results Framework Template 64 E. Sample Monitoring Plan Template 65 F. Sample Stakeholder Questionnaire for Tracking Progress Toward Targets 66 5 Guide for Identifying Lessons from Mapped Outcomes 67 Guide for Communicating Outcomes 73 Snapshot of Outcomes from External Training Knowledge Services Projects 76 Resources 80 Case Study in Outcomes Evaluation: Mongolia 81 Example of how and why a World Bank program used outcome harvesting to learn about social change impacts from capacity building interventions 6 Acknowledgments This toolkit was developed based on first pilot experiences with about 20 World Bank knowledge projects in strategic thematic areas. Since 2012, the World Bank has used the experiences of projects applying these tools to customize this evolving toolkit of resources for learning across the program cycle. Examples of knowledge projects that used the tools:  Constitutional mandates in health initiative (online community, country-based coalitions)  Leadership public sector reform initiatives (in education, civil service, financial management information systems, sewerage services, energy)  Open contracting program at global and country levels  Access to Information network in Latin America  Parliamentary oversight of national budgets in Africa  Local governance capacity building in South East Europe  eLearning for capacity development of city officials in China  Solid waste management in Bosnia  Governance in pharmaceutical procurement in African countries  Social accountability in education in the Philippines  Brazil drought preparedness and climate program Jenny Gold coordinated the development of the tools with Dawn Roberts, Dale Hill, and the project teams. Sharon Fisher provided analysis, writing, editing, and design for the guides and tools. Samuel Otoo and Ricardo Wilson-Grau provided overall guidance. 7 Summary Documenting, understanding, and making sense of outcomes—achieved or not achieved, planned or unexpected—can facilitate strategic, systematic knowledge sharing among task teams, partners, and clients. However, this can be a difficult undertaking, especially with complex development processes that involve profound development challenges, experimentation with new innovations, or country-led behavioral and institutional changes by multiple social actors. Methods such as outcome harvesting (from the Outcome Mapping Learning Community) offer approaches to manage knowledge and learn from complex and difficult-to-monitor development processes. Collecting—also known as harvesting—bites of detailed outcome information with colleagues, partners, and stakeholders can allow teams to identify, monitor, and learn from changes in social actors. The information collected describes who changed what, when and where, why it matters to the development objective—the significance of the change to the development challenge and context—and how the program contributed to the change. Harvesting program information for outcome-based learning is a stakeholder-centered process that captures qualitative, tacit knowledge. The process includes substantiating and analyzing harvested knowledge collaboratively to communicate progress toward impact to clients, management, and partners. Tools are flexible to adapt to a program’s design and can provide useful details to inform the theory of change, implementation lessons, outcomes, and indicators. From 2012–2014, the World Bank Capacity Development and Results team undertook pilot experiences to identify how outcome harvesting could be integrated with the World Bank’s results management approach. Teams explored how customized outcome harvesting concepts and tools could lend themselves to learning about how change happens in complex aspects of about 20 World Bank knowledge projects. Based on feedback from the task teams and clients involved in the pilot experiences, suggestions on ways to use the outcome harvesting tools going forward include: capture tacit and contextual knowledge from program implementation; use for systematic learning to maximize benefits of interventions; seek evidence and lessons from an implementation process; use for client reporting on results and promoting learning by doing; and triangulate outcomes with other data to validate results. To move forward on these suggestions and advance project learning, the guides and tools customized in the pilot experiences were brought together in this field guide. The field guide is intended as a collaborative vehicle for World Bank staff, clients, partners, and other stakeholders to harvest, substantiate, interpret, monitor, and learn from outcomes across the project cycle to improve effectiveness and results. Key uses identified include:  Reviewing results from complex program/project components to improve the benefit of interventions, sustainability and mix of actors involved  Gathering evidence and lessons from the change process that can complement other M&E tools  Rigorous knowledge management of complex, multi-actor processes It is hoped that these first outputs of the pilot experiences will continue to be adapted in different ways and improved upon through ongoing application and learning. 8 Introduction to Outcome-Based Learning What is it Outcome harvesting—a practical tool for outcome-based learning—is a participative method used to identify, monitor and learn from the changes a program influenced or produced in social actors. Implementation teams and stakeholders collect outcome information to define who changed what, when and where, why it matters, and how the program contributed to the change. This process provides the foundation of outcome-based learning: information on the program’s outcomes allows teams and stakeholders to understand the progression of change—what worked and did not work—to adaptively learn from results. The outcome harvesting approach is adapted from the outcome mapping community, with an outcome defined as a significant change in the behavior, relationships, activities, or actions of the people, groups, communities or organizations with whom a program works directly. By identifying and monitoring those changes that fall within the “sphere of influence” of a program (figure 1), practitioners can understand the many often non-linear streams of outcomes that characterize that program’s contribution toward a development goal. Figure 1. Sphere of influence of a program Source: Simon Hearn. Overseas Development Institute. At www.outcomemapping.ca Civil society organizations, research institutes, and government agencies globally have adapted outcome harvesting and related outcome mapping concepts to understand complex programs. The Outcome Mapping Learning Community (www.outcomelearning.ca) has developed a hub of information and experiences on how, why, and when to use related tools. Other development partners have also used outcome harvesting and/or outcome mapping, such as the Ford Foundation, UK Department for International Development, Canadian International Development Agency, Oxfam, and Open Society Institute. The Overseas Development Institute supports the Outcome Mapping Learning Community, and the US Agency for International Development identifies outcome harvesting tools as useful for complex aspects of programs and strategies as a complement to performance monitoring (http://usaidlearninglab.org/library/discussion-note-complexity-aware- monitoring). An August 2013 discussion paper from the UN Development Programme evaluation office selected Outcome Harvesting as one of 11 promising innovations in monitoring and evaluation practice (https://undp.unteamworks.org/node/370238). 9 When to use it Outcome harvesting can be useful for understanding complex aspects of development programs. What do we mean by complexity? Complexity comes in part from the number and nature of the social actors, the challenging development problems addressed and the uncertain solutions to these problems. There are often difficult to monitor processes because of the multiple actors involved and unclear results chains, but they offer an opportunity for experimentation to gather new lessons. The Cynefin Framework (figure 2) shown below provides a typology of contexts to guide explanations and solutions to address development challenges. When a program is complex, there tends to be considerable uncertainty about whether the planned activities will have the intended results. Outcome- based learning—documenting, understanding, and interpreting outcomes—can help teams and stakeholders learn how a program has (or has not) addressed development problems with uncertain solutions and to understand the context of the changes. Figure 2. Cynefin Framework Source: Dave Snowden at http://cognitive-edge.com/ How it promotes learning Outcome harvesting offers tools for knowledge capture and results learning across the stages of substantially complex development programs where it is difficult to show impact (figure 3). The information gathered can be used to maximize the benefits of interventions and offer a context-specific lens to inform and complement learning from other M&E data. Outcomes provide qualitative learning on key interventions and identify essential lessons, such as how best to adapt successful efforts to different contexts and how to choose the best mix of actors to involve. Learning could be used to inform program design and delivery, as well as defined areas for further operational research and evaluation. 10 Figure 3. Results learning across the stages of development programs This field guide focuses mainly on the implementation stage, offering guides and tools for systematic qualitative learning of what has been achieved and then working backwards to determine how the change happened. It is a paradigm shift in thinking: first, identify the outcomes (changes in social actors), then discover the activities and outputs that contributed to the outcomes. 11 Harvest Outcomes with Stakeholders g Outcomes 12 Guide to Harvesting Outcomes Outcome harvesting is a participative method for What Is Meant by Outcomes identifying outcome information with colleagues, In outcome harvesting, an outcome is what each partners, and stakeholders. It is used to learn about social actor (or change agent) did, or is doing, that who changed what, when and where, why it matters, reflects a significant change in their behavior, and how the program contributed. relationships, activities, actions, policies or practice. Why Outcome Information Is Important In a World Bank program, an outcome might describe The outcome information collected can capture an action that reflects a demonstrated change in unrecorded knowledge on a program to provide a awareness, knowledge or skills, collaborative action, detailed understanding of results and how the actors or the use of knowledge or innovative solutions. Outcomes might also describe deeper institutional achieved them on the ground. This information can changes relating to policy, citizen engagement or inform evidence-based adjustments during delivery, government accountability and organizational leading to continuous adaptive management of arrangements. The outcome always refers to the complex change processes. The information can also changes influenced in change agents. promote dialogue and inform strategic decisions on next steps. Finally, the information can help answer Discovering what a program’s outcomes are is a questions about a program’s results, and be a starting process—you may discover that some outcomes point for more comprehensive evaluation. aren’t outcomes at all or that some matter more than others. The focus of the process is looking at actions Process for Learning from Outcomes by social actors, asking why they matter and what they represent within the change process. This guide summarizes the basic steps to systematically learn from outcomes. Other learning materials will explain specific steps in more detail. Strategize evaluative learning questions Adapt Harvest information Reflect on the project, within teams and outcomes inform next from clients or steps partners Gather evidence and substantiate key outcomes 13 Strategize Evaluative Learning Questions Tools for harvesting outcomes can be used in a variety of ways to gather the information that best allows programs to address strategic questions of most concern. A team may decide to:  Look at outcomes across the entire program to learn from the whole change process.  Examine part of a complex reform to understand what problems were addressed and how.  Collect outcome information to help answer specific questions, such as: o How has the change process differed across countries? o What is the sustainability of the changes to institutionalize results? o How is the program design advancing what needs to be achieved? o What are lessons to overcome critical development challenges? The aim could be continuous learning from outcome information, gathered frequently from the design stage of a program through implementation and completion. Or the aim could be to record change in more detail periodically—collecting it at key stages to learn from cumulative results achieved over several years. Harvest Information within Teams and from Clients or Partners For a 2–3 year program, 20–30 outcomes might be reported. An attempt should be made to “unpack” significant outcomes to show the various change processes that contributed to the outcome. All outcomes—positive and negative, intended and unintended—hold the potential of enhancing learning about the change process and should be recorded. Think like a detective to first identify outcomes and evidence of changes, and then work backwards to discover how the change happened. Activities to identify outcomes with team members and stakeholders might include: • Facilitated group sessions to identify outcomes • Periodic documentation or reporting of outcomes in a template, database, or online form by the project team or clients • A client survey which is designed to collect information on outcomes relevant to the project and key evaluation questions • Structured interviews by a trained third party to identify outcomes. If methods are used for the initial outcome harvesting that do not involve the entire team, a later team discussion would allow for dialogue, bring out multiple perspectives, and provide richer content for learning together. Outcome information can be monitored in a table or spreadsheet, to include the outcome statement, outcome significance and program contribution. OUTCOME STATEMENT SIGNIFICANCE CONTRIBUTION Actions by government, civil Significance to objective and Program activities and outputs society or private sector particular problem plausibly linked to change agents the outcomes Change in a change agent’s A description of the relationship of Processes, goods or services behavior, relationships, activities, the outcome to the specific problem, produced by the program that actions, policies or practice that need or situation it influences, which influence change in some way. the program has influenced in brings out its significance in moving some way. the program closer to meeting its development objectives 14 IN MORE DETAIL: Gather information on outcomes Each description should:  State who is the change agent as the subject of the sentence.  Describe what they are doing differently from before.  Specify the timeframe for when the change occurred—at least the year and month, if relevant.  Identify the geographical location where the change took place. As you brainstorm on outcomes, use the following checklist:  Identify outcomes (or lack of expected outcomes) for all the major agents involved.  Identify changes across the relevant time line of the program.  List both small and big changes observed in each agents to capture milestones in the change process.  Identify key setbacks or turning points that shifted the direction of the program.  Where possible, specify quantitative and qualitative aspects of each change. Describe the significance of the outcomes Explain why the outcome matters. Put the meaning of each outcome into detailed context so a reader who does not have country or topic expertise will understand its importance. Include enough information such that someone not familiar with the program can understand the baseline problem or situation that changed. Significance to objective  Describe how the outcome is a relevant step toward a development objective, or a key milestone in the change process. Projecting forward potential impact can be done here if it can be substantiated by external stakeholders and if conditional wording is chosen carefully (not should or will happen but could or may be expected to happen) 15  Explain if the change represents positive progress toward the objective, if it set back or blocked progress and/or how it shifted the direction of the program. Significance to particular problem(s) and solution(s)  Describe how the outcome addresses a problem, need or changes a situation. o Explain significance of changes in policy, citizen engagement or government accountability, and organizational arrangements. o Explain significance of changes in awareness, knowledge or skills, collaborative action, or the use of knowledge or innovative solutions. o Explain any changes in beneficiaries’ welfare that resulted from the outcome. Describe how the project or program contributed Describe how program activities plausibly influenced the outcome(s).  Describe what the program did, when and where. The description should describe a contribution that goes beyond financial support.  Report both direct and indirect contributions, explaining in precise terms the form of the contribution. The contribution may have been cumulative over time, or may have been shared with other partners.  If the program’s contribution is part of other organizational or external partner support, explain the program’s role as a part of that support. Be as precise as possible.  Quantify the program’s contribution as much as possible, such as 3 advisors, 4 meetings, 100 hours of advisory services, 3 guidance documents translated, etc. As with the outcome, the formulation of the program’s contribution must be specific enough to be verifiable. Gather Evidence and Substantiate Key Outcomes We can deepen our understanding of the changes reflected by select outcomes through stakeholder feedback. This feedback loop serves to verify our understanding of the outcomes, and provides a medium for stakeholders to raise issues to improve the program. Substantiation can also collect evidence on the select outcomes, such as documents, reports, photos, videos, or media clips. A separate guide provides more detail on the substantiation process. Reflect on the Outcomes to Adapt the Project or Program and Inform Next Steps Interpret what types of changes the outcomes reflect, and what has worked to address critical problems. Understand the overall picture, that is, how the outcomes come together to advance change toward the development objective. Use the outcomes to identify next steps:  What to continue and what to adjust to make the project or program more effective  Emerging practices or lessons that could be applied to other projects or programs  How to scale up the influence of the project or program. Discuss with all involved the benefits of using outcome information. Consider how and whether your unit might continue to monitor and learn from outcome information. 16 Do’s & Don’ts DON’T DO Don’t describe a project activity as Describe not what you did, but what the change agent did because of the an outcome, such as, “We organized influence of your activities. A change agent may be an individual leader, a conference or prepared tool X.” coalition, civil society organization network, mayor or elected official, or government team. Don’t force yourself to find direct Realize that some changes may not be directly linked to an activity. Also, an relationships between all activities activity might have been completed last year but the outcome may only and an outcome. have become visible this year. And finally, some things carried out may not lead to the expected or desired or unintended outcomes. Don’t report only positive outcomes Report outcomes that are negative, unintended or unexpected. Report or intended outcomes. expected outcomes that did not occur as planned. These can be identified as pivot points and provide opportunities for learning to help explain why a process of change took a new direction. Don’t give vague descriptions of the Describe the outcome information in simple language and in sufficient change that happened. detail so third parties are able to understand the change and verify it. Quantify the change to the degree possible. Don’t use words that embellish the Explain the reasoning as to why the outcome is valuable, worthy, or potential impact of an outcome successful at that point in time, providing sources or illustrative examples. 17 Tools for Harvesting Outcomes The following templates, exercises, questions, and other support are assembled here to help teams harvest outcomes: A. Sample Template to Document Outcomes Retrospectively B. Sample Template to Report Outcomes C. Exercise to Visually Map Outcomes to Date or as They Happen D. Checklist to Review Outcome Information E. Sample Open Questionnaire for Harvesting Outcomes from Stakeholders F. Sample Questions for an Open Interview with Stakeholders G. Sample Template for Terms of Reference: Consultant to Harvest Outcomes A. Sample Template to Document Outcomes Retrospectively It is helpful to record information on significant milestones or outcomes as you proceed with a project. This makes the gathering of evidence for review at key points or at completion of a project or program less time-intensive. You can record information quarterly or monthly, depending on the pace of the project. The option here is to divide your entries by the expected outcome/results areas of the project. These areas may change during implementation, so you may add any new area to track those results. The template may be modified based upon your needs. Last updated Date of last outcome entry PROJECT TITLE – GOAL – OBJECTIVE – OUTCOME AREA 1 – OUTCOME AREA 2 – OUTCOME AREA 3 – *Note: The outcome areas could change during implementation or a new area of unforeseen outcomes could be identified. Outcome Area 1: Timing and Milestone Significance Contribution Progress Location WHEN and WHERE WHAT happened and WHO WHY is the change HOW did the project DEGREE OF was it? was involved? relevant? contribute? PROGRESS • WHEN did the • WHO are the actor(s) • Relevance to the  HOW did project made toward a change happen • WHAT changed in objective activities/outputs development • WHERE did the their behaviors, • Relevance to support the change objective change take relationships, address problem  HOW did other (indicate 1 for place activities, actions, or need in local partners support the low, 2 for policies or practices context change, if relevant medium, 3 for high) 18 Outcome Area 2: Timing and Milestone Significance Contribution Progress Location WHEN and WHERE WHAT happened and WHO WHY is the change HOW did the project DEGREE OF was it? was involved? relevant? contribute? PROGRESS • WHEN did the • WHO are the actor(s) • Relevance to the  HOW did project made toward a change happen • WHAT changed in objective activities/outputs development • WHERE did the their behaviors, • Relevance to support the change objective change take relationships, address problem  HOW did other (indicate 1 for place activities, actions, or need in local partners support the low, 2 for policies or practices context change, if relevant medium, 3 for high) Outcome Area 3: Timing and Milestone Significance Contribution Progress Location WHEN and WHERE WHAT happened and WHO WHY is the change HOW did the project DEGREE OF was it? was involved? relevant? contribute? PROGRESS • WHEN did the • WHO are the actor(s) • Relevance to the  HOW did project made toward a change happen • WHAT changed in objective activities/outputs development • WHERE did the their behaviors, • Relevance to support the change objective change take relationships, address problem  HOW did other (indicate 1 for place activities, actions, or need in local partners support the low, 2 for policies or practices context change, if relevant medium, 3 for high) 19 B. Sample Template to Report Outcomes Program (name and P number) Start date: Country(ies) or region(s): Names of team members: Development objective: Change agents: Development problem: (What specific development problem(s) does the program address) Also cite background materials on the program’s results, such as concept notes, videos, reports, other You can organize outcomes by actor (change agent) and date. Outcome Statement Outcome Significance Program Contribution Who did what, when and where?  Significance for the development How did the program contribute to the objective outcome?  Specific problem addressed  Note if outcome describes a positive change, a setback or turning point Example: Example: Example: In early 2012, a municipal reform team in Significance to objective The municipality was one of the urban teams Bijelijna, Bosnia (including the municipality The creation of the new team allowed the that received project support in 2011. The and 2 utilities) formed a new implementation reform team to move forward when it was support began with a regional workshop in team in the waste collection utility to find a stuck. Basing the team in the collection utility January 2012, which provided learning solution to the difficulties in setting up a was important to improve its collection opportunities on leadership, coalition- household database in order to reach its goal processes. This structure allowed senior building diagnostics, strategic of increasing solid waste collection coverage leaders from the municipality and implicated communication and the Rapid Results and fees. utilities to provide joint oversight to a team Approach to policy reform and change. This that could find solutions to the problems. was followed by an 11-month laboratory phase to explore innovative solutions to their Problem addressed chosen urban problem, which was solid There was no information on household waste collection in the case of this team. The locations, following the civil war, which was project provided technical support to the required to increase solid waste coverage to reform team to help them re-think the all households and create a database. There nature of their problem, goal and was also a requirement for the municipality stakeholders. to approve any service fee increase. The collection utility had shifted its staff to work on more profitable services, such as street cleaning, since there was not an effective system to collect waste from households. The dump utility was operating on a deficit since it did not have information to charge accurate service fees for the amount of waste collect. . Shift in direction of the program, since the team was stuck and a new team was formed. 20 Outcome Statement Outcome Significance Program Contribution Who did what, when and where?  Significance for the development How did the program contribute to the objective outcome?  Specific problem addressed  Note if outcome describes a positive change, a setback or turning point Between April and June 2012, the new Significance to objective The project provided rapid results coaching implementation team conducted a The survey became a means to increase support to facilitate the team to launch a community survey in one village and one support and participation from the broader rapid cycle learning process to find their own urban area to find out under which citizenry, and allow for data collection to test solutions to the problems and to advance the conditions citizens would pay fees and demand for services at alternative price goal of improved waste collection in 11 identify household location. levels and to document household locations months. This support was part of a in a database. Citizens began to express laboratory phase. specific complaints. Problem addressed Citizens were not willing to pay for waste collection and did not demand service improvements. Households were not documented. Positive change toward objective In October 2012, after receiving input from Significance to objective The project encouraged the inclusion of a many local stakeholders, one municipality in The policy provided guidance that previously diverse set of stakeholders in the reform Bosnia drafted a new policy to guide solid was lacking and which was necessary to team structure, as well as an oversight waste utility operations. regulate the utilities and improve services to relationship of the reform team with the citizens. The pilot provided motivation and implementation team. This encouraged lessons of experience to help inform this better communication and the development policy guidance for the municipality, since it of relationships to inform policy, and for the gave examples of solutions to the same learning from the rapid results process to be problems. It provided a new consensus and institutionalized in the new policy. The clarity between the municipality and utilities lessons could be replicated or reinforced to meet the objective to improve solid waste once the laboratory phase was complete. management coverage while improving financial sustainability through gradually increased fees. Problem addressed There was previously a lack of municipal level guidance to regulate utility services and payments. There was also a lack of dialogue among the municipality and waste management utilities (including consultation with the utilities and citizens) on how to clarify policy to guide household services and fees. Positive change towards objective - Add rows for more outcomes - 21 C. Exercise to Visually Map Outcomes to Date or as They Happen On a flip chart or in a table, list the relevant actors (or change agents) along the left and the program timeline across the top. Then, document outcome statements for each actor across the timeline in the table cells or on sticky notes. Timeframe Month, Year Month, Year Month, Year Month, Year Social actor OUTCOME (Name the change agent) • What did the actor do, where? • Significance to objective • Problem addressed • Contribution of initiative Social actor Social actor - Add more actors and outcomes - Source: Overseas Development Institute and International Livestock Research Institute, 2006 22 D. Checklist to Review Outcome Information Each outcome should describe detailed information listed below in the program context. Outcome Statement Outcome Significance Program Contribution  Names the change agent as  Explains how outcome is a  Describes what the program the subject relevant step toward did objective or key milestone in  Explains what the agent is  Includes both direct and change process doing differently indirect contributions, as  Describes the specific relevant  Specifies the timeframe problem or need the  Explains the program’s role as  Identifies what change took outcome addressed part of other organizational or place  Notes whether change external partner support  Describes how much/how positively advanced the  Gives precise details—how many objective, set back progress much, where, and when and/or shifted direction of  Is plausible program 23 E. Sample Open Questionnaire for Harvesting Outcomes from Stakeholders An outcome is what each social actor did or experienced, differently from before, intended or not. Outcome information may be collected through a blend of different approaches, such as a survey questionnaire of stakeholders. This sample questionnaire was developed and used by the Constitutional Mandates for Right to Health program to collect outcomes from members of an online community of practice. Introduction In 2011, the Constitutional Mandates for Right to Health Program supported Latin American regional stakeholders in launching the online SaluDerecho network. The network aimed to engage a broader group of practitioners from countries, regionally and globally, to discuss, analyze and inform solutions to health and judiciary rights and policy issues. The program is now exploring the outcomes of the SaluDerecho Initiative to date and would like to hear from you. Changes Changes influenced by SaluDerecho network. Please reflect on your experience with the SaluDerecho network. To date, how has the network influenced new actions or changes in behaviors, relationships, activities, policies or practices on the right to health? These changes could be observed in stakeholders in your country, regionally, or globally. These changes could be positive transformational changes or negative changes for the right of citizens to health. Also, if you have influenced a social actor not to take action—e.g., you helped prevent a law from being passed—that too is as an outcome. For each important change, please describe the following: a. WHAT happened and WHO was involved? Please identify the main actor(s) by organization(s) and/or titles and describe what changed in their behaviors, relationships, activities, policies or practices. b. WHEN and WHERE did the change take place? c. WHY is the change relevant? Explain why you consider this to be a substantial change related to the right to health. How is it different from the previous situation of right to health? d. HOW did the SaluDerecho Coalition contribute to making this change happen? Describe which activities or inputs specifically supported the change. e. WHAT EVIDENCE could be used to demonstrate this change? Identify any documents, online content, data sources or methods that could be used to verify this information. f. Would you like to describe additional important changes?  Yes  No [If YES, then questions would repeat] Contact Information (Optional) Please provide the following information to indicate how you have been involved with the SaluDerecho Coalition and how we can contact you if any clarification or additional details are needed. Name: Organization: Title/Position: Email address: Brief description of your engagement or role with the SaluDerecho Coalition: 24 F. Sample Questions for an Open Interview with Stakeholders The following is a sample script and questions for talking with key actors to collect outcomes. In this case, it involves a coalition process in Costa Rica supported by the Constitutional Mandates in Health program. Kemly Camacho developed the questions. Introduction As you are aware, an initiative aimed at promoting the right to health is being carried out in Costa Rica. As part of this process, we are evaluating the program using a methodology known as outcome harvesting. The discussion that you have kindly agreed to engage in today will contribute to harvesting the outcomes that have been achieved to date in our country. Once collected, all of the outcomes will be organized into a document to facilitate an evaluation of the process thus far. Thank you for taking the time to participate in this interview, which will last approximately 40 minutes. We will send you the results of the outcome harvesting exercise at the end of the process. Questions A. Involvement in the process 1. Where are you currently working? 2. What is the nature of your involvement with the right to health issue? 3. What do you think about the issue of right to health? 4. In what way have you participated in the process promoted by the SaluDerecho Coalition and supported by the initiative? B. Outcomes When we speak about outcomes, we are referring to changes that have been influenced during the process promoted by the SaluDerecho Coalition. 1. What are the most significant changes that have resulted from the activities and actions being promoted by the SaluDerecho Coalition? 2. Which individual, institution, or unit has been responsible for these changes? 3. When and where were these changes produced? 4. Is there any evidence to substantiate these changes? 5. Why do you think that these are major changes regarding the issue of right to health? 6. How are these changes relevant to citizens’ right to health? 7. In what way has the process promoted by the SaluDerecho Coalition and supported by the initiative contributed to these changes? 8. Are there any changes that are crucial and should be highlighted? If the answer to question 8 is yes, return to question B.2. If the answer to question 8 is no, end the interview, thank the respondent and provide my contact information in case of questions or additional information. Inquire if there is another suitable individual who could be interviewed to provide further input about the outcome harvesting process. 25 G. Sample Template for Terms of Reference: Consultant to Harvest Outcomes A. Background Outcome harvesting is a tool for learning on implementation science. It is used to identify, monitor, and learn from changes in social actors, through collecting outcome information with colleagues, partners, and stakeholders.* The information defines what changed, for whom, when and where, why it matters, and how the program contributed to the change. Outcome harvesting is useful for complex projects that involve behavioral changes, challenges with unknown solutions and multiple actors with different roles and capacities. The harvesting process is stakeholder-centered and captures qualitative, tacit knowledge. It includes tools to substantiate and analyze this knowledge collaboratively and communicate progress toward impact to clients, management, and partners. The method is flexible to adapt to a project’s design and combine with other M&E tools. B. Objectives The consultant will harvest outcomes to support real-time learning, inform program adjustments and next steps and communicate progress and results. To fulfill these objectives, the consultant will engage with the team and relevant stakeholders on the following tasks:  Strategize the outcome harvest process based on learning questions  Harvest outcome information from a specified timeframe  Review the outcomes found in documentation and identify additional outcomes by engaging with various stakeholders and team members  Substantiate selected outcomes for additional credibility  Review and make sense of the outcome information  Write up the outcomes and analysis in a report Guides, tools and case examples are available to support the consultant in these tasks. C. Duties and Responsibilities A focal person from the local team will be the counterpart to the consultant. Joint work with the team and local stakeholders ensures accurate outcome information and a locally owned process. The steps for the consultant are as follows.  Step 1. Orientation: The consultant will participate in an orientation with the project team.  Step 2. Outcome Harvesting workshop: The consultant will support a workshop with the relevant stakeholders to launch the harvesting process. Following the workshop the consultant will prepare a 1-page work plan with the country counterpart to guide the harvesting activities.  Step 3. Identification, formulation and review of outcomes: o The consultant will work with relevant stakeholders to identify and formulate outcomes (about 20-40, depending on the scope and magnitude, or about 3-5 per informant). The number of outcomes should be sufficient for a critical mass of evidence to answer the learning questions. Identifying outcomes should include a review of existing documentation to extract changes in social actors. It may also include meetings or workshops with relevant stakeholders to review these outcome descriptions and identify additional outcomes. o The consultant will review the outcomes with the country counterpart and project team and update as needed, to ensure the information: is detailed and accurately describes local changes and context; includes outcomes at different levels (major outcomes and 26 process related), for all key actors and across the timeframe; describes significant change in behaviors, relationships, activities, actions, policies or practice; and links to tell a story of change from the start to the current situation (if not, outcomes are likely missing).  Step 4. Substantiation: The consultant will engage with relevant stakeholders to identify about 20% of the outcomes for substantiation through client feedback and evidence collection.  Step 5. Outcome analysis and mapping: o The consultant will engage with the team and relevant stakeholders to identify outcomes that reflect major changes: solutions to overcome critical problems or bottlenecks; major achievements in a change process; direct results to further a development objective; or more permanent/sustainable solutions or deeper institutional changes. o The consultant will identify 2-3 outcomes that reflect key setbacks, constraints or turning points and explain why. o The consultant will visually map the outcomes chronologically and along the results chain of the project. .  Step 6. Report: The consultant will write a report on the outcomes, and organize meetings to review and discuss the draft report with country counterparts and the project team, making changes and finalizing content. D. Deliverables The consultant is given a 30-day timeframe to deliver the following:  Work plan outlining key activities and timelines to harvest outcomes, provide progress updates, substantiate selected outcomes (estimated 20%), make sense of the outcome information and write the report.  Table of formulated outcomes, with the following information on each: o Where –where the outcome took place. o Who –the social actor(s) [individual(s) or group(s)] that showed a significant change in their behaviors, relationships, activities, actions, policies or practice. o When –month and year in which the outcome was achieved (ex. January 2013). o What changed – 2-3 sentence plausible outcome statements explaining what the actor is doing differently, identifying what change took place and quantifying the change in terms of how much/how many. o Problem the change addressed: 3-5 sentences that detail the local problem and/or situation that is addressed or changed by the outcome. o Significance of outcome to further an objective: 2-3 sentences that explain how the outcome is a step toward objective or a key milestone in a process of change. o Supporting activities – Activities and how they contributed to the change. o Classification of outcome – whether the outcome represents a major change and which outcomes represent key setbacks or turning points.  Table of substantiated outcomes, gathered through the opinions of knowledgeable independent third parties.  5-10 page report that includes: o Introduction and background: Introduce the case and summarize the goal, problems and key objectives being addressed as well as local projects and activities. o Outcome stories on each identified major change area. o Substantiation findings: Describe ratings and selected quotes from individual feedback providers. 27 o Lessons: Describe how the outcome stories and setbacks provide key learning on the project and inform next steps. E. Qualifications The candidate must have experience working in the specified location and have: 1. Experience in the use and facilitation of Outcome Mapping or Outcome Harvesting 2. Experience in evaluation and strong analytical thinking 3. Willingness to learn how to apply a new tool for monitoring and evaluation 4. Demonstrated capacity to support medium and high range professionals to learn to use a new tool 5. Demonstrated working ability to write synthetically in the local language and English. Experience working with multi-sectoral groups and coalition-building processes is not required but will be strongly considered. 28 Substantiate Outcomes with Stakeholders 29 Guide for Substantiating Outcomes Once your information on outcomes—and related outputs and milestones—is harvested within your team and from stakeholders, you can collect feedback from third parties and evidence sources to further verify the accuracy of the information and deepen understanding of what has been achieved and how. Decide on the Purpose of the Substantiation The team should decide the purpose of substantiating the outcome information since it affects the way the process is conceived and managed. Possible purposes are:  Encourage strategic learning within the team. The discovery and agreement on outcomes, their relationships, and their significance provides new information that can advance team understanding and influence decisions on next steps. The new information may lead to a modification in the change strategy, or it may inform a strategy for scale up in the next phase. This learning could be partially achieved without substantiation, but is likely enhanced by selective substantiation.  Enhance the credibility or appreciation of the findings for an external audience. A side benefit of substantiation is that quotations are often suitable for published case studies or other communications products on a program’s results.  Ensure outcome information is consistent and concretely and correctly presented to stakeholders and beneficiaries.  Elicit different perspectives of the various stakeholders, which may have implications for the team’s decisions on next steps.  Further understand the influence of context on the change processes, which may yield important information on potential for replication or scale up. Choose a Subset of Outcomes to Substantiate Given that substantiation takes time and it may not add value to substantiate all of the outcomes, in the past a subset of information (about 20% of outcomes) for substantiation has been the aim. The information is usually summarized as a table with columns for each outcome, its significance and the program contribution. Once the choice of stakeholders to be contacted for substantiation is made, columns can be added to record their degree of agreement with the outcome information, and to list other evidence sources. Suggestions to help a team choose a subset of outcomes for substantiation by stakeholder feedback:  Outcomes over milestones: “True outcomes” represent a key behavioral, institutional or policy change. An outcome harvesting exercise might also unpack key steps and milestones important to the change processes.  Pivotal outcomes: ”Pivotal” outcomes have affected follow-up decisions in major ways, such as resulting in scaling up, new networks, or replication, or in determining steps that followed. A deeper understanding of the context of these outcomes is recommended for its learning value.  Setback outcomes: “Setbacks” are outcomes that are unintended, negative or reflect an expected outcome that did not occur. A deeper understanding of the context of these outcomes is recommended for its learning value.  Non-action outcomes: Sometimes an important change is a proposed policy that is not approved or an undesirable practice that is stopped.  No easy access to evidence: Do not choose outcomes where the behavioral, institutional or policy change is easily verified by means other than stakeholder consultation, such as inspection 30 of documentation. For example, commitment of an external partner to finance subsequent steps, or a change in legislation, are presumably credibly supported by documentation.  Most recent outcome in a causal series: If there is a series of outcomes that are causally related, choose the most recent to be substantiated. The earlier outcomes are less likely to be questionable.  Potential to strengthen understanding of program contribution: Choose an outcome where the program contribution may be questioned, or where there may be differences of opinion on its significance. For example, a contribution of advisory support may require further substantiation that such support was relevant to the context and/or linked to the outcome. Outcome Information to Substantiate All parts of the information on a given outcome can be substantiated. The information consists of:  Outcome statement, describing the action taken by a social actor or change agent—consisting of a description of who changed what, when, and where.  Significance of the outcome, elaborating on the type of change the action reflects, and describing why the action mattered—how it addressed a given problem, need or situation.  Program’s contribution to the outcome, and its extent and importance. Verification of the outcome statement is likely to be the least controversial part of the substantiation, whereas agreement on significance of the outcome, and/or the extent to which the program contributed might be subject to more variation depending on stakeholders’ roles, interests and views. Identify Sources for Feedback Consider who can best attest to the outcome information. For each possible substantiator, describe the individual’s title, contact information and role s/he played in the program. Consider:  Substantiators within a department outside the program unit who are knowledgeable about your initiative. You may also consider others you know who are in contact with country-level stakeholders associated with your initiative.  Substantiators that are stakeholders outside of your organization. It is best to have at least one external substantiator who is perceived as an objective observer and is knowledgeable about the initiative. The suggestion is to identify three potential substantiators to provide feedback per selected outcome. Possible sources are described in table 1. Table 1: Possible Sources of Substantiation What is in need of substantiation? Source of substantiation Whether an event or milestone occurred Usually evidence is available to collect, such as in written documents. Whether an agent’s behavior changed in the The substantiator should be someone other than the change agent, unless the way described interest is to understand the motivation of the change. Whether a change in organizational The substantiator should be a stakeholder who can observe the effectiveness effectiveness occurred of the organization, and/or is affected by it. Details about the information on an Choose substantiators who have been sufficiently engaged to know details. outcome Some details may be available in documentation. Significance of the outcome—for example, in Choose a knowledgeable, engaged stakeholder who has been associated with terms of its relationship to program the change agent or organization long enough to observe the change. objectives, or the degree to which it addresses a relevant development problem Link of the program contribution to the Choose a knowledgeable, engaged stakeholder who has been associated with outcome (causality) the change agent or organization long enough to be aware of the program contribution and how it influenced the change, but who is not the main counterpart. 31 Significance of the program contribution— Choose a substantiator from the client country organization benefitting most relative to other external partners from the contributions. A second substantiator may be from another external partner organization, but preferably one who works directly with the client, not one who is exclusively an external financing partner. Select the Outreach Method for Stakeholder Feedback You will need to decide on an outreach method to conduct the substantiation. Table 2 provides information on possible outreach methods. Sample communications and questions for outreach by email or interview are found in the section, “Tools for Substantiating Outcomes.” Table 2: Advantages and Disadvantages of Possible Outreach Methods for Substantiation Email alone Telephone + email Conference call + Face-to-face meeting (individual) email + email Advantages Can reach out faster Once scheduled, can Presence of several peers Where possible, this has to more respondents. motivate real-time may motivate more all the advantages of the response, answer any detailed response; previous two methods, questions, obtain richer collaborative approach with a promise of higher detail in some cases, can motivate motivation to respond; and ask for substantiators to involve documentation may be substantiation of more other stakeholders for readily accessed; if a than one outcome further verification or group is involved, while substantiator is detail personal contact may available. make it easier to deal with disagreements Disadvantages Explanatory email is An additional step of Additional step of finding This method may long; respondents can scheduling the call is convenient time for all require extra time and postpone response; involved; during call, it may impose costs; cost in some cases, but considering may be harder to stay experience and roles of may be preferred for connectivity and focused on descriptions substantiators may differ; stakeholders available at communication of specific outcomes. and disagreements organization habits, this method among different headquarters. may work better in participants may be some contexts or awkward. countries When contacting the substantiators, inform them that their role will be part of the published record. Ask if there is anything they have said they do not want quoted by name and title in the evaluation report. Or, if there is anything that they do not want quoted in an external publication. Explain that at the least they will be listed among all of the substantiators. Assign Roles in the Substantiation Process  One person needs to be accountable for overseeing the process of substantiation, such as approving external communications, assuring follow through, enlisting optimal participation for objectivity and quality, and ensuring full reporting and transparency of the substantiation. This person may be the team leader or another assigned person.  Other tasks to conduct the substantiation may be delegated, for example: o An assigned person may be responsible for sending emails to substantiators, following up with reminders and keeping track of results. o Another person may be enlisted to write the findings for an external audience.  To conduct the substantiation—such as questionnaires, interviews or meetings—a team of two persons is recommended to assure objectivity. It is suggested that one of the two persons is not engaged in the implementation of the program. 32 Handling Disagreement Among Substantiators A challenging aspect is making decisions when the substantiators disagree on the accuracy of information submitted to them. If there is disagreement, consider the following options:  Revise the outcome information descriptions to incorporate any comments Tips on Implementing the Substantiation provided; this is suitable if most responses  Share the proposed wording of the outcome are positive, but some useful additional information with the substantiators in comments are received that may expand the advance, so they can verify the accuracy of description. the descriptions and provide comment.  Drop the outcome from the substantiation,  Keep track of responses in a matrix or a similar format. A template is in the Annex. while keeping it in the outcome map; in this  Once there is sufficient response on all case, it would still be required to report that chosen outcomes, the feedback can be substantiation was sought and full reviewed to glean lessons and decide next agreement not received. This option may be steps for the program. If insufficient appropriate when the team agrees the responses are received (not all “fully agree”), outcome is significant and/or pivotal. a decision should be made whether to reach  Drop the outcome from the outcome map out to a second set of substantiators or to completely; this may be considered in the follow up with the original ones using face of disagreement on any part of the another method, such as by meeting, phone, or email from a more influential outcome information. If the team agrees intermediary, such as a country manager or with this decision, this adjustment may be field officer. left out of public reporting.  Follow up with each substantiator who  Follow up with the substantiators to clarify responded to acknowledge responses, and the different perspectives and explain briefly how the feedback will be disagreements. acted upon, such as whether it is for internal  Conduct another round of substantiation learning or external publication. using a different outreach method or with  If significant disagreements occur in the additional substantiators. feedback, or if a substantiator’s comments appear useful but are hard to understand, a follow-up conversation is recommended. Reporting Substantiation Responses  If you change any part of the outcome Feedback from substantiators typically includes information in response to feedback, you qualitative scores (such as “fully agree” and may need to re-consult with other “partially agree”) and comments. To assure the substantiators on the new description(s). credibility sought, transparency is desirable and  Include the substantiation results in any statistics on responses should be presented in the reporting, disclosing the names and titles of process manager’s documentation, whether good substantiators and the reason they were news or bad news. Tips for sharing substantiation chosen for the given outcome information. results in a report:  Include a description of the substantiation methodology used.  Number the outcomes so they can be easily identified. Indicate those outcomes where substantiation took place. Examples Asterisk the substantiated outcome within the narrative: ….” In November 2012, the Chinese Academy of Governance and Urban Planning Society of China piloted the course among 120 professionals and officials in charge of urban planning, land management and infrastructure investment from 18 provinces across China. [10] The Chinese Academy of Governance later incorporated the course into its eLearning curriculum and currently offers it twice a year. [11*] 33 Present the results of the substantiation within the main body of the narrative or in a sidebar: “To verify the accuracy of the outcomes mapped and enrich our understanding of them, the team selected four outcomes [6*, 7*, 11* and 17*] and asked 9 people independent of the program unit but knowledgeable about the change to review the outcomes and record whether they agreed with the outcome description as formulated. 7 people responded. 6 fully agreed and 1 partially agreed. 5 respondents provided additional comments to clarify the descriptions.” Outcomes selected for collection of evidence can be similarly presented in the body of a report. Comments within a section of the report or a sidebar: In selecting comments, it is useful to bring out different perspectives held by different stakeholders for learning purposes. Consider those that provide significant or new details on a program’s results that were not captured in the original outcome information. If the comments by substantiators require further information on the outcome or context to be fully appreciated, record them within the body of the report. If they can be self-standing, such as about next steps or the potential for replication, they may be suitable for a sidebar, alongside similar comments from other substantiators. 34 Tools for Substantiating Outcomes The following tables, steps, and other support are assembled here to help teams substantiate outcomes: A. Sample Tables to Organize Outcome Substantiation B. Sample Communication for Email Outreach to Substantiators C. Sample Interview for Outreach to Substantiators D. Sample Substantiation Form to Collect Stakeholder Feedback E. Sample Table to Track Substantiation Responses A. Sample Tables to Organize Outcome Substantiation The following are sample tables that could be used to select outcomes for substantiation by stakeholder feedback and/or the collection of evidence sources. Outcomes to be substantiated by stakeholder feedback Suggestions on choosing a subset of outcomes for substantiation by stakeholder feedback:  Select outcomes that represent a key behavioral, institutional or policy change.  Select outcomes that have affected follow-up decisions in major ways.  Select outcomes that reflect a setback or stall in progress or shift in direction of the program.  Select outcomes with no easy access to evidence.  Select the most recent outcome in a causal series.  Select outcomes that may be questioned, or where there may be differences of opinion on its significance. No. Outcome Substantiator name Substantiator Substantiator Role s/he played Comments title contact information in the program 1. “......” “.....” “.....” “.....” “.....” “.....” 2. 3. Outcomes to be substantiated by evidence collection: Select outcomes where there is observable evidence, such as a policy document or public statement. No. Outcome Source of evidence Links to evidence source Comments 1. “......” “.....” “.....” “.....” 2. 3. 35 B. Sample Communication for Email Outreach to Substantiators SUBJECT: Documenting results of the [name of program]: Requesting your assistance Dear [name of substantiator], We are seeking your input to learn from and document the changes influenced by the program [name of activity]. This is to inform decisions on program improvement and next steps. For this purpose, we are using a tool called “outcome harvesting,” which has been developed and used successfully by professional evaluators. The tool calls for feedback from diverse stakeholders who are knowledgeable about aspects of our program, specifically on the description of selected significant outcomes or milestones identified by the program team. For your response to be most helpful, we ask that you send your answers within one week. The questions should only take 10 minutes of your time. This is not an anonymous process, since we want to use your feedback to strengthen the credibility and our understanding of the project. However, we provide the option not to quote your responses by name or not quote you in any external publication on the project. We are grateful for your prompt response. You, along with other collaborators who have responded to similar questions, are contributing valuable perspectives to our team learning. Sincerely yours, [program lead] 36 C. Sample Interview for Outreach to Substantiators: individual/group interview in a phone call/Skype exchange/videoconference/meeting A. Preparation for the interview The following should be completed before the interview:  Send preliminary emails or have a pre-exchange with the substantiators, explaining the process, the primary purpose (such as learning or publication), providing the outcome information to be substantiated, including the feedback form for responses, comments and questions. The feedback form should be similar to the questionnaire used in the sample email outreach tool.  Agree on a date and time (and time zone) for the meeting.  Assign an interviewer (typically a team of two) to conduct the substantiation interviews. Assign roles and possibly questions for each team member to ask during the interview.  Decide whether to audio record the substantiators’ responses, in which case their permission must be asked in advance. B. Beginning of the interview  Thank the person for participating, and identify any people on the call/ videoconference/meeting on your side who may be unknown to the interviewee(s).  Ensure that you have their correct name and title.  Begin by referring to the previous email and stating the purpose of the interview. “We are using a technique called Outcome Harvesting to document and learn from the results of the program [activity name] on which we are collaborating. An important part of this process involves feedback from stakeholders to learn from their perspective on the program results.”  Ask if there are any questions before you begin. Be prepared for a range of questions, particularly if the program is still ongoing, but try to keep the conversation focused on the substantiation at the beginning. You can mention that there will be time at the end for more general questions or to share any background information, and that further follow up through other means (e-mail, a second call, etc.) is possible. C. Explain the substantiation  Explain that we have identified key outcomes for verification by stakeholders. The description of these outcomes or milestones is provided to aid understanding of the changes that have been influenced by the program. We have identified one key outcome to verify with each stakeholder, and we have pre-drafted the description.  Explain there are three pieces of information being verified for each outcome: Description of the outcome (who changed what, where, when); description of the significance of the outcome (why does it matter); and description of the program contribution (what, where, when, who). There will also be an opportunity for the substantiator to answer a question on the extent and/or significance of the program contribution.  Explain that the purpose of the substantiation is to better understand the outcome, by asking the extent to which they agree with the pre-prepared description. They may also give an explanation or provide an alternative description. Explain that any comments are welcome to learn from and improve the described information. Explain that if they do not wish to be quoted by name in the report or in an external publication, they will have that option. They will be listed by name and title amongst all the substantiators. 37 D. Step-by-step through the form Overall description of the information on the outcome:  Say “Let’s start with the overall description of the information on the outcome [this response is for the overall accuracy of the three described pieces].” You may remind them that this is only one outcome out of many [such as 30]. You may wish to explain why this outcome was chosen for their specific feedback.  Read the description of each piece of the outcome. Ask “Do you fully agree?” Pause for the answer. Whether yes, or some other answer, record it. If they do not fully agree, ask for an explanation. Explain to the interviewee that the following questions will provide the opportunity to explain any disagreement with the accuracy of the information.  Ask, “Do you wish to comment further on the overall description of the outcome information?” Outcome statement:  Say “Now let’s consider the outcome statement, specifically.”  Read the description again if helpful. Ask “Does the description accurately explain what happened, where, by whom, and when? Are there any other relevant missing elements? Would you like to suggest an alternative description of the statement?” Significance of outcome:  Say “Now let’s consider the significance of the outcome, for the progression toward the objective of the program.” You may wish to remind the person of the objective of the program.  Read the description again if helpful. Ask “Is the description accurate? Are there any other relevant missing elements (such as the description of the problem addressed by the outcome or how it relates to the objective)?” As suitable, ask probing questions, such as “Would you like to suggest an alternative description of the significance?” Program contribution:  Say “Now let’s consider the program’s contribution.” You may wish to describe in a broad sense the various partners for this program.  Read the description. Ask “Is the description accurate?” Pause for the answer. Ask for any additional details they can provide on the program contribution. Extent of program contribution:  Say “Now let’s consider the extent of significance of the program contribution. The format of this question is a little different than those before.”  Read all four potential answers (negligible contribution, indirect or lagged contribution, some contribution, major contribution). Then invite the person to select the response that s/he believes is most accurate. Invite them to add comments to explain their answer. For the interviewer, note the explanations of these ratings, in case questions arise: o Negligible contribution: The program is associated with the whole initiative, but the progression to this outcome step and its significance was not influenced by program input at the stage of achievement of the outcome. o Indirect or lagged contribution: progression to this outcome step or its significance may not be directly identified with the program by all stakeholders, but according to the knowledge of this substantiator, it would not have occurred without program input, which may be identified with present or past contributions. o Some contribution to this outcome: This substantiator believes, and states some evidence, that the program directly contributed to progression to this outcome step or its significance, but that impact was not solely due to the program. The substantiator may wish to comment on others that contributed. The others could be external partners or the stakeholder organizations and networks themselves, who, given a certain momentum, were mainly responsible for progression to this step or its significance. 38 o Major contribution to this outcome: This substantiator believes and states some evidence that the program directly contributed to progression to this outcome step, was the main contributor, and in all likelihood, the outcome would not have happened without the program contribution. Others may still have contributed, but the program was the main contributor, according to this substantiator. E. Invite further questions and bringing closure  Thank the interviewee for their contribution and ask if they have any further questions.  Ask if there is anything they have said they do not want quoted by name and title in the evaluation report. Or, if there is anything that they do not want quoted in an external publication. If necessary, explain that at the least they will be listed among all of the substantiators.  Invite the interviewee to still fill out the form and return it within one week. Tell them that they will favor the information in the form, if received promptly, but plan to use the oral information just collected if the form is not received.  Ask, if appropriate, if they can suggest colleagues or other stakeholders that could contribute to understanding the outcome. If so, obtain the names, titles, and contact information, and/or ask for that information to be supplied by email. Also, if appropriate, ask them to send any written documentation that they may have cited in the interview.  Acknowledge any questions that need to be answered in a separate communication. If appropriate, ask if they would like a copy of the final report. Before offering this, be clear on what form the final report will likely take—will it be an internal set of notes for the team, or will it be a published case study. F. In case of disagreement In case of significant disagreement, remind the interviewee(s) that Outcome Harvesting is a learning exercise. Thus, accuracy is important but there may be different views on what happened, its significance and how the program contributed. Remind the interviewee(s) again that they may further discuss their views with the program team at a future time and date or follow up to complete the form in writing, including any comments. 39 D. Sample Substantiation Form to Collect Stakeholder Feedback Below is the description of one outcome selected for your review, including information on the outcome statement, outcome significance and program contribution. Note that, this selected outcome is just one change influenced by the program. A. Outcome Statement B. Outcome Significance C. Program Contribution In October 2012, after receiving The policy demonstrated a The municipality was one of many urban input from many local consensus built between the teams that applied for and received stakeholders, a large municipality and utilities to meet project support in 2011. The support municipality in Bijeljina, Bosnia an 11-month goal set by local city began with a regional workshop drafts a new policy to guide solid stakeholders to improve solid followed by a laboratory phase to waste utility operations. waste management coverage explore innovative solutions to their while improving financial chosen urban problem. The project sustainability through gradually provided Rapid Results Coaching support increased fees. The policy to facilitate the team to find solutions to provided guidance that previously achieve their 11-month goal. The project was lacking and necessary to encouraged the inclusion of a diverse set regulate the utilities and improve of stakeholders in the team structure to services to citizens. strengthen ownership of the solutions that the team identified. The following are questions for you to answer about this outcome. 1. On the description of the overall outcome information (all three boxes above): To what degree do you agree that the information is accurate? Please choose only one answer. [ ] Fully agree [ ] Partially agree [ ] Disagree [ ] No opinion – not knowledgeable enough to answer [ ] No opinion – prefer not to answer 2. On the outcome statement (A above): Please explain any disagreement you may have with the accuracy of the description of the outcome, or present an alternative description. _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ 3. On the outcome significance (B above): Please explain any disagreement you may have with the significance of the outcome, or present an alternative description. _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ 4. On the program’s contribution (C above): Please explain any disagreement you may have with how the program contributed to the outcome, or present an alternative description. _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ 40 5. On the extent of the program’s contribution: [ ] Program unit contributed negligibly to this result [ ] Program unit’s contribution to this result was indirect, or the result of past actions of the program unit [ ] Program unit made some direct, real-time contribution to this result [ ] Program unit made major direct, real-time contributions to this result. [ ] No opinion – not knowledgeable [ ] No opinion – prefer not to answer Explanation, and/or description of program contribution and its significance _____________________________________________________________________________________ _____________________________________________________________________________________ 6. Other overall comments or questions: _____________________________________________________________________________________ _____________________________________________________________________________________ _____________________________________________________________________________________ Next Step As you review your responses to each question and comments, please mark with a * if you wish: Not to be quoted by name, although your name and title will be included among the list of substantiators for all the outcomes we are substantiating. You may also indicate in the final question #6 any names and contact information for others you think could provide helpful comments, and if you would like to receive a copy of the final report. If you would prefer to convey comments or questions by phone, please use the same section (“6. Overall Comments or Questions” ) to suggest a date, time and best contact information. 41 E. Sample Table to Track Substantiation Responses The following is a sample table that could be used to track substantiation feedback from stakeholders. It would document the feedback on the questions asked of each substantiator in the substantiation form. No. Outcome Signi- Program Substantiator Agreement with Outcome Outcome Program Significance Overall ficance contri- name, title and overall outcome statement significance contri- of program comments/ bution contact information comment comment bution contribution questions information comment 1. “......” “.....” “.....” [Name, title, [ ] Fully agree “.....” “.....” “....” [ ] Program “....” contact [ ] Partially unit information] agree contributed [ ] Disagree negligibly to [ ] No opinion – this result not [ ] Program knowledgeable unit’s enough to contribution answer to this [ ] No opinion – result was prefer not to indirect, or answer the result of past actions of the program unit [ ] Program unit made some direct, real-time contribution to this result [ ] Program unit made major direct, real- time contribution s to this result. [ ] No opinion – not knowledgea ble [ ] No opinion – prefer not to answer 2. 3. 42 Make Sense of Outcomes with Stakeholders 43 Guide for Interpreting Outcomes Making sense of collected outcome information to improve development results Development processes are complex—multiple actors and agendas influence the paths from inputs to impact. It is often difficult to identify the changes a program influenced or produced in social actors— what was achieved and how did it happen. By identifying and monitoring outcomes, practitioners can understand a program’s contribution toward a development objective and take corrective actions to strengthen results. This guide is to help teams make sense of the information they have harvested about the actions, relationships, practices and other behavioral changes in social actors (also known as change agents) that have flowed from program activities. Teams can select questions relevant to their learning needs to analyze the effectiveness and sustainability of outcomes and facilitate a collaborative adaptive learning process. One or more questions can be selected from the list below for customized support based on the program needs.  Is the quality and quantity of the outcome information sufficient?  What is the sustainability of the changes to institutionalize results?  How are the outcomes advancing impact?  How is the program design advancing what needs to be achieved?  Which interventions have been most beneficial?  What are essential “how to” lessons to adapt or inform scale-up?  What are lessons to overcome critical development challenges?  How can a results framework be developed to assess the achievement of outcomes?  What progress has been achieved?  Which outcomes reflect setbacks or failures? 44 (Q) Is the quality and quantity of the outcome information sufficient? The extent to which the harvested information is useful for understanding and communicating results and guiding program improvements will depend on the completeness and accuracy of the data collected. Quality Checklist 1. Apply the quality checklist to ensure the information An effective outcome statement: collected is complete. Outcome statements that fall short  Names the change agent (individual or on any component(s) should be revisited for accuracy, group) as the subject of the sentence significance, and relevance for the development process.  Explains in detail what the change agent Outcomes that are ambiguous should be removed from is doing differently the harvested information.  Specifies the timeframe for the change 2. Review the completeness of the change story. As a  Identifies where the change took place whole the outcomes should describe key results and  Describes the change in qualitative represent a story of the program. and/or quantitative terms (How much? How many?)  Do the outcomes explain the major changes  Is based on clear evidence or is widely influenced by the program to date? agreed upon by stakeholders  Do outcomes describe who did what during each timeframe of the program?  Are outcomes listed for all key actors, as well as beneficiaries where influenced (such as citizen groups)? 3. Map the outcomes. A visual map serves as an effective tool for reflecting on how program outcomes link to influence change over time or themes. If a map has not yet been completed:  Make a grid with change agents along the left and timeline across the top.  Plot the outcome statements from your harvested information.  Identify which outcomes influenced other outcomes or came first and mark these on a map with numbers and arrows (see example).  Review the map with stakeholders for wide agreement on the links. Example Outcome Map: Learning from Utility Reform in Bosnia 45 (Q) What is the sustainability of the changes to institutionalize results? Outcomes that are most likely to be sustained beyond a project’s implementation period are those that reflect changes in local capacity. These results go beyond improvements in the behavior or disposition of specific individuals or groups and are therefore considered “institutional” in nature. Typically these results are major societal, policy or organizational changes that have significance for the development objective of a program. Teams can assess each of their outcome statements by using an established capacity development or knowledge results framework such as the World Bank’s Capacity Development and Results Framework (CDRF) to explore whether the reported change reflects a behavioral or learning outcome for individuals or groups or whether it reflects a higher-level institutional capacity change. For example, the decision tree based on the CDRF below could be used to determine which changes are key institutional outcomes. Any institutional changes identified should be reviewed with stakeholders for broad understanding and agreement. Decision Tree to Identify Institutional Changes in the Development Process Does the outcome statement reflect… The decision tree is a checklist against which a team member can assess whether an outcome represents an institutional change—as long as that team member has a good understanding of the local context in which the program is being or has been implemented. For example, the signing of a decree or the approval of a new budget could be used to signify increased commitment by leaders if these actions are not typical in the context and reflect the overcoming of a key challenge or barrier to the development objective. Examples showing the interpretation of outcome statements by type of result are listed in the next table. 46 Identifying Institutionalized Results: Examples from Mapped Cases Outcome statement describes… Type Example 1 Example 2 A broad change in the engagement of Institutional Colombian Court set Vice Presidents signed local stakeholders new precedent to decree to expand use of clarify health rights results-based management EX: More participation of citizens/CSOs in to citizens by to implement and scale up decision-making, better access to broadcasting hearing reforms across information, commitment from leaders, of a judicial ruling. government. shifts in local customs or social norms… New/revised rules, standards, regulations, Institutional Municipality drafts Chinese Academy of processes or policies to facilitate new policy to guide Governance issues first achievement of objective utility operations guidelines for e-learning to with input from network of 451 institutions. EX: Guidance to clarify roles and collection utility and responsibilities across organizations, dump. procedures that are less costly to administer, new legislation backed by broad public support… The more effective or efficient operation Institutional Collection utility Ministry of Education of an organization used survey findings network of volunteers to set realistic price delivered 25,000 textbooks EX: Delivering better results, operating at for citizens to pay for within 60 days instead of 1 lower cost, communicating better with waste collection. year. stakeholders, adapting better to changing demands… A change in behavior by individual(s) or Progress Regional coalition Nigeria coalition built group(s) reflecting new knowledge, skills, marker formed online knowledge of observers on and/or relationships community of how to use Public practice with Procurement Act, Freedom EX: Collaboration, performing new job broader group of of Information Act, and functions, adaptation of knowledge or practitioners. procurement monitoring innovative solutions, functioning as part of tools. network… 47 (Q) How are the outcomes advancing impact? Outcomes that have been identified through a mapping process can be placed along a results chain to identify how the program is contributing to higher-level results over time. Examples are provided for two projects to show how the levels of mapped outcomes can be interpreted. In each case, the lightest levels are furthest from the project’s “sphere of influence” and closest to the impact. The darkest levels are more directly under project control.  Impact is the goal in the line of sight of the project.  Institutional changes are outcomes that reflect institutionalization or sustainability. Often these are more permanent changes in stakeholder engagement/responsiveness, formal policy/rules/guidance and organizational effectiveness. Every outcome at the institutional change level has an “outcome story” that can be told separately to outline the project’s theory of change and inform sustainability.  Changes in behavior are outcomes that reflect new knowledge, skills, relationships or implementation know-how after any intervention or with a different group than the direct participants.  Early outcomes are changes observed during an intervention or directly at the end that influence later changes.  Inputs are key project activities/interventions. Examples for how mapped outcomes can be placed along a results chain have been provided for two projects to demonstrate how this technique can be applied for any map, regardless of its complexity. Some maps reflect a relatively linear progression from inputs to impact, whereas others follow a more iterative process, gaining “early wins” to trigger additional inputs and/or behavior changes on the way to longer-term institutional change. Example 1. Mapped Outcomes in a Results Chain: Reforming Solid Waste Management in Bosnia Impact Development Goal: Improved solid waste coverage for citizens (12) Collection utility team (10) Municipality (9) Citizens in the pilot (13) Municipality remained operational unit, area expressed Institutional drafted new policy increased collection fee adding staff to expand by 10% and covered demand for better Changes to guide utility household coverage and waste collection operations deficit of dump utility fee collection service and paid fees (6) Collection (14) Collection (8) Collection (7) Collection utility re- utility obtained utility and dump utility used survey organized field workers so Changes in they could complete starter database provided input data to set new from another Behavior survey with wider into policy price for waste utility, with collection household coverage municipal approval (3) Reform team agreed to create a household database to manage the (1) Reform team collection process and met monthly Early (4) Reform team struggled to (5) Pilot team collaborated on 11-month outcomes or set up a household database formed and goal to increase solid waste (2) Reform team understood to increase waste collection decided to milestones collection coverage and fees and decided to set up a team the deeper problems conduct citizen blocking expanded waste to pilot a solution in one survey collection village and one urban area Project Advisory support provided activities or WBI technical support to help GTL Workshop WBI Rapid Results to implementation team inputs shape team’s proposal Coaching leader (2011-2012) 48 Example 2. Mapped Outcomes in a Results Chain: Improving Central and Decentralized Public Service Delivery for Citizens in Burundi 49 (Q) How is the program design advancing what needs to be achieved? The harvested outcome information can be used to determine the theory of change of the program or change process. This is the “big picture” of how the program’s outcomes come together to advance progress toward higher-level results. This documented sequence of changes can be compared against the intended program strategy and any needs of stakeholders to make adjustments in planned interventions. The map below shows how outcome information can be organized to describe a program’s change process, including information on the:  Development objective  Specific development problems addressed by the outcomes  Key change agents or social actors influencing outcomes  Other partners involved (such as development partner, World Bank teams, other)  Major outcomes from the harvest—these can be grouped into 3-5 outcome areas that demonstrate progress toward existing or new objectives  Outcomes from the harvest that are linked to the major outcomes—there could be 2-4 key intermediate changes or progress markers to highlight in each outcome area  Key activities that helped to achieve the outcomes Example from Utility Reform Case: Overview of Change Process Depending on the complexity of the theory of change, teams might find it useful to think through the overall change process by focusing on one problem or challenge at a time and reflecting on how the program helped which agent(s) change to address that particular challenge (see next table). Outcome areas for addressing each of the problems can then be summarized to diagram the program’s overall change process. 50 Example of Focusing on One Challenge to Look at Change Path Goal Improve the benefits of public goods and services for citizens Challenge Citizens have habit of not paying utility fees Key change Citizens in pilot area; implementation team in collection utility; municipality agent(s) Institutional Citizens’ participation in utility price-setting increased; public support increased for changes paying higher fees Intermediate New understanding of problem led to decision to pilot solution; survey used to changes communicate with citizens and increase demand for services Supporting Leadership workshop; follow-up Rapid Results Coaching to facilitate experiential interventions learning 51 (Q) Which interventions have been most beneficial? The process of examining the significance of each outcome and the way(s) in which a program appeared to contribute to the change can allow for a more systematic look at how important or beneficial specific program components were for advancing progress. If a team has mapped outcomes along a results chain, for example, an additional useful step is to review the main program activities and their roles in supporting or leading to advances toward a development objective. For the case in Burundi, focused on improving the delivery of central and decentralized public services for citizens, a key project component appears to have been the advisory support provided to the Steering Committee for conceptualizing the Leadership for Results program in Burundi and analyzing priorities. As shown below, this single component led to or supported a notable array of outcomes along the results chain. This simple exercise helps to highlight the critical value of this advisory support, without which the Steering Committee could have been less effective in driving the needed change. 52 (Q) What are essential “how to” lessons to adapt or inform scale-up? Changes within complex programs might occur through multiple paths. However, some outcomes are more “essential,” meaning that a development objective is unlikely to be achieved without these changes. The results chain of a project can be analyzed to help teams reflect on which milestones and outcomes are essential to scale-up the change process or adapt a similar process in another program. Milestones and outcomes can be identified as “essential” if they serve as critical junctions in the overall change process, leading to deeper, more transformational changes. Teams can identify these by considering the role of each milestone and outcome in facilitating progress along the results chain. For example, in the case related to reforming solid waste management, essential outcomes for scaling up change are circled in red. These include the collaboration of the municipality and two utilities via a reform team to understand deeper problems, the direct advisory support provided by the project team to the team leader in country, the decision to pilot a solution that responded to the deeper problem of demand from citizens, and the implementation of a citizen survey to set the new price for waste collection. Example of Essential Outcomes in a Results Chain: Reforming Solid Waste Management in Bosnia Impact Development Goal: Improved solid waste coverage for citizens (12) Collection utility team (10) Municipality (9) Citizens in the pilot (13) Municipality remained operational unit, area expressed Institutional drafted new policy increased collection fee adding staff to expand by 10% and covered demand for better Changes to guide utility household coverage and waste collection operations deficit of dump utility fee collection service and paid fees (7) Collection utility re- (6) Collection (14) Collection (8) Collection utility obtained utility and dump utility used survey organized field workers so Changes in they could complete starter database provided input data to set new from another Behavior survey with wider into policy price for waste utility, with collection household coverage municipal approval (3) Reform team agreed to create a household database to manage the (1) Reform team collection process and met monthly Early (4) Reform team struggled to (5) Pilot team collaborated on 11-month outcomes or goal to increase solid waste set up a household database formed and (2) Reform team understood to increase waste collection decided to milestones collection coverage and fees the deeper problems and decided to set up a team conduct citizen blocking expanded waste to pilot a solution in one survey collection village and one urban area Project Advisory support provided activities or WBI technical support to help GTL Workshop WBI Rapid Results to implementation team inputs shape team’s proposal Coaching leader (2011-2012) 53 (Q) What are lessons to overcome critical development challenges? Outcome maps provide valuable information about what changes were needed to overcome bottlenecks or challenges to development goals. Teams can use various approaches to extract lessons from their mapping experience to inform future replication or improvements. One technique is to use the map as a visual tool, particularly if the outcomes have been interpreted along the levels of a results chain (see the question, “How are the outcomes advancing impact?”) Outcomes that led directly to the next level along the results chain reflect important lessons. In the example on utility reform, the use of the citizen survey was critical for addressing challenges of social norms, accountability and legitimacy that blocked advancement of the goal. The survey was used to communicate with households, increasing both the demand for services and the willingness to pay the tariff, and it informed the creation of a legitimate policy. Pinpointing Lessons in a Results Chain: Reforming Solid Waste Management in Bosnia Impact Development Goal: Improved solid waste coverage for citizens (12) Collection utility team (10) Municipality (9) Citizens in the pilot (13) Municipality remained operational unit, area expressed Institutional drafted new policy increased collection fee adding staff to expand by 10% and covered demand for better Changes to guide utility household coverage and waste collection operations deficit of dump utility fee collection service and paid fees (7) Collection utility re- (6) Collection (14) Collection (8) Collection utility obtained utility and dump utility used survey organized field workers so Changes in they could complete starter database provided input data to set new from another Behavior survey with wider into policy price for waste utility, with collection household coverage municipal approval (3) Reform team agreed to create a household database to manage the (1) Reform team collection process and met monthly Early (4) Reform team struggled to (5) Pilot team collaborated on 11-month outcomes or goal to increase solid waste set up a household database formed and (2) Reform team understood to increase waste collection decided to milestones collection coverage and fees the deeper problems and decided to set up a team conduct citizen blocking expanded waste to pilot a solution in one survey collection village and one urban area Project Advisory support provided activities or WBI technical support to help GTL Workshop WBI Rapid Results to implementation team inputs shape team’s proposal Coaching leader (2011-2012) 54 (Q) How can a results framework be developed to assess the achievement of outcomes? Teams can develop a results framework to assess progress for key outcomes. The sample below is modeled after the framework structure used for projects in World Bank operations. Indicators can be identified and included for measurement as follows:  Development goal—to assess progress toward the higher-level objective over the longer term.  Institutional changes—at least one key institutional change should be measured for each outcome story. In cases where multiple related institutional changes are part of the same story, it might be sufficient to just reflect one in the results framework.  Major intermediate outcomes—these are major behavioral changes that are useful to track. These outcomes often reflect important gateways or bridges to other higher level outcomes and therefore reflect critical junctions in the outcome story. Example of a Results Framework Goal Indicator(s) Data Source(s) Use of Information Improved solid waste # of people in urban areas provided Household survey Evaluates progress towards coverage for citizens with access to regular solid waste in targeted solid the achievement of the collection* waste higher-level development management area objective % of households in target area served by formal waste management system Institutional Changes Indicator(s) Data Source(s) Use of Information Municipality drafted new Municipal approval (1) and Policy document Assesses increased policy to guide utility implementation (2) of new policy for legitimacy and local buy-in operations utilities that reflects input from Municipal records for policy guiding utility collection utility and dump. operations Institutional Changes Indicator(s) Data Source(s) Use of Information Collection utility team Formal establishment of unit within Utility records Confirms the improved remained operational collection utility to scale up waste management of solid unit, adding staff to collection services to more waste services for expand household households [Y/N] improved operational coverage and fee effectiveness collection Municipality increased % of waste generators in targeted Municipal records Measures the commitment collection fee by 10% area paying new fee Utility records of the municipality and and covered deficit of citizens to a financially dump utility Cost recovery ratio of dump and viable waste collection collection utilities system Citizens in the pilot area % of surveyed citizens demanding Household survey Assesses local citizen expressed demand for service in targeted waste participation in and better waste collection management area support for scaling up a service sustainable solid waste collection system *Core sector indicator 55 Intermediate Outcomes Indicator(s) Data Source(s) Use of Information Collection utility used Completion of household database Utility records Confirms innovation in survey data to set new (number and location of households) price setting price for waste collection Establishment of new process to align collection prices with survey data Collection utility # of field workers authorized to give Utility records Confirms new approach to reorganized field priority attention to the solid waste leveraging resources for workers so they could collection reform process broader citizen coverage complete survey with wider household coverage Pilot implementation Memorandum of Understanding that Municipal records Tracks formation of new team formed and accompanied the establishment of the structure for key decided to conduct implementation team stakeholders to participate citizen survey in decision process for reforming solid waste collection Depending on the time horizon and purpose of the results framework, a monitoring plan can also be developed to establish baseline and target values for assessing progress. 56 (Q) What progress has been achieved? Teams can assign indicators with data sources and/or methods to assess progress and gaps. They can develop a monitoring table to inform an existing results framework (see the question, “How can a results framework be developed to assess achievement of outcomes”) or through the following steps:  State the development goal  List the outcome statement from harvested information  Identify the indicator(s) and data source(s) to track the described change  Describe the baseline situation, current value and target value for each indicator Through this process, teams will have a monitoring plan that informs the current status of implementation and also gaps in what changes are still needed to advance the development objective. Examining the progress toward targets helps teams identify lessons on where adjustments may be needed and where interventions have had a critical role in advancing progress. Monitoring Plans for Mapped Outcomes: Examples from Three Cases Objective Outcome Indicators Baseline Situation Current Value Target Value Statement (Year) (Year) (Year) Improve the Citizens in the % of surveyed No survey in 61% of surveyed 85% of surveyed value of pilot area citizens in pilot area place to engage citizens expressed citizens express municipal expressed demanding service citizens demand for waste demand for services for demand for collection waste collection citizens better waste [solid waste collection service % of households in Current fee 46% of households 75% of collection] pilot area paying system does not paying new fee households new fee cover utility costs paying new fee Enhance the Regional multi- # of countries with Only informal 7 countries 8 countries transparency stakeholder formal membership network and coalition formed in regional coalition participatory online community process for of practice to # of practitioners Online 500 active 750 active setting better protect the active in online Community of members members priorities to right to health community of Practice does not (“active”= posting realize the practice yet exist and/or reading/ right to health downloading for all citizens resources) Improve the Organizations Days required to 365 60 45 delivery of improved deliver 25,000 central and performance textbooks in target decentralized through pilot province public services initiatives for citizens at # of pregnant 71 482 600 all levels in women tested for [country] HIV within 30 days in target province 12 or more 3 3 # Months for teachers to receive first paychecks Traditional monitoring and evaluation methods can also be used to complement the collection of outcome information and help ensure that isolated stories (anecdotal evidence) are combined into more rigorous bodies of information for systematic use. The monitoring stage presents a critical opportunity for analyzing the progress to date and assessing whether changes in planned activities or interventions are needed to better reach a development objective. 57 (Q) Which outcomes reflect setbacks or failures? Programs often contribute to development objectives through a non-linear stream of outcomes, and it can be a valuable exercise to identify which outcomes did not work well for a project and what actions or changes are needed to help advance progress. Teams can identify and make sense of project setbacks or failures by mapping outcomes along a results chain or by assessing progress toward established targets as described in the case examples below. EXAMPLE OF A SETBACK Excerpt from an Outcome Map: Reforming Public Financial Management in the Philippines  Finding setbacks in an outcome map. Using the results chain map as a visual tool, teams can explore how each outcome supported or led to outcomes at the next level. Outcomes that do not appear to have supported advances along the results chain should be examined, given that they sometimes provide important lessons about challenges in the development process. For example, in the Philippines, a team worked to establish an integrated financial management information system (FMIS) for a more accountable use of public money. However, it was challenging to motivate agencies to use the new online process for their budget submissions even when it was required (see red circle in example). It was necessary but not sufficient to make the online budget submissions a prerequisite for budget allocations: the first deadline passed with only a handful of agencies meeting the new requirements. Additional interventions were needed—including assigning budget specialists to assist line ministries and holding another leadership forum to build local ownership— before widespread compliance (institutionalized practice) was achieved. 58  Using indicators to track progress. Outcome maps do not tell the whole story about expected versus actual progress, so teams might also find it useful to monitor progress by assigning indicators and targets. The periodic review of results helps to inform implementation decisions. For example, the Pay and Performance Project in Sierra Leone is a results-based financing initiative in which disbursement-linked indicators (DLIs) are used to trigger the flow of project funding based on evidence of progress. An inter-agency performance management team made notable progress in instituting a performance appraisal system for civil servants, but the team was not on track to achieve performance targets due to a systemic problem across rapid results initiatives in which the allocated funding for program activities was delayed. Review sessions for teams to discuss results and challenges led to a collaborative learning process and a solution for the persistent funding delays. The Performance Management Team was then able to expand the number of pilot ministries and improve results to meet the target set by the DLI. A setback might not signal that interventions are on the wrong track but instead often indicates that additional support or guidance is needed. The use of adaptive management is critical for ensuring that desired outcomes are achieved. In both examples, the team mapped outcomes to review preliminary results against expected progress and agree on mid-stream adjustments. The use of team review sessions to support collaborative adaptive learning can be critical for advancing development impact. 59 Tools for Interpreting and Monitoring Outcomes The following checklists, tables, maps, and other support are assembled here to help teams interpret and monitor outcomes: A. Quality Checklist for an Outcome Statement B. Decision Tree for Identifying Institutionalized Results C. Sample Change Strategy Maps and Template for Organizing Outcome Areas D. Sample Results Framework Template E. Sample Monitoring Plan Template F. Sample Stakeholder Questionnaire for Tracking Progress Toward Targets A. Quality Checklist for an Outcome Statement An effective outcome statement:  Names the change agent (individual or group) as the subject of the sentence  Explains in detail what the change agent is doing differently  Specifies the timeframe for the change  Identifies where the change took place  Describes the change in qualitative and/or quantitative terms (How much? How many?)  Is based on clear evidence or is widely agreed upon by stakeholders 60 B. Decision Tree for Identifying Institutionalized Results Applying a framework such as the World Bank’s Capacity Development and Results Framework could be used to determine which changes are key institutional outcomes. Any institutional changes identified should be reviewed with stakeholders for broad understanding and agreement. Does the outcome statement reflect… 61 C. Sample Change Strategy Maps and Template for Organizing Outcome Areas The change strategy shows the “big picture”—how the outcomes come together to advance change toward the goal and development objective. This example shows how the outcomes can be grouped by common links into areas that show paths for change. 62 63 D. Sample Results Framework Template A results framework can be developed to assess progress for key outcomes. Rows can be added or removed as appropriate. Goal Indicator(s) Data Source(s) Use of Information Institutional Changes Indicator(s) Data Source(s) Use of Information Intermediate Outcomes Indicator(s) Data Source(s) Use of Information 64 E. Sample Monitoring Plan Template Objective Outcome Indicators Baseline Current Value Target Value Statement Situation (Year) (Year) (Year) Ex: Improve Citizens in the % of surveyed No survey in 61% of 85% of surveyed the value of pilot area citizens in pilot place to engage surveyed citizens express municipal expressed area demanding citizens citizens demand for services for demand for service expressed waste collection citizens better waste demand for [solid waste collection waste collection collection] service % of households in Current fee 46% of 75% of pilot area paying system does not households households new fee cover utility paying new fee paying new fee costs 65 F. Sample Stakeholder Questionnaire for Tracking Progress Toward Targets It is helpful for stakeholders to record information on significant milestones to track progress on outcomes as interventions proceed. Monitoring progress (or lack thereof) at regular intervals informs program adjustments and collaborative learning. A stakeholder questionnaire can capture data on changes resulting from interventions, as well as the cause and the effect of those changes. (This questionnaire is based on work by the Open Contracting team from a workshop by Kaia Ambrose.) A basic questionnaire for stakeholders at regional, country, community levels, etc. can be completed quarterly or monthly, depending on the pace of the project and when activities take place. The questionnaire can look at expected outcome areas of the project. These areas may change during implementation, so new areas can be added. The sample here may be modified based upon your (or stakeholder) needs. DATE: Name and Role: Organization and Location: Contact Information: Outcome Area: Ex. Leadership and transparency among CSOs Targets Rating – Scale Description Significance Contribution Evidence of 1 to 5 KEY TARGETS or EXTENT the WHAT WHY is this HOW did this If available, milestones to track target is being happened, relevant to the come about (how collect met (indicate WHEN did this objective or did the evidence in 1 for low, 3 happen, WHERE problem in this intervention support of the for medium, 5 did this take context? support it)? progress for high) place? (meeting minutes, press release) Problem: Ex. CSOs are interested in increasing communications and collaboration, but lack the mechanisms and commitment to do so Ex. Establishment of a new online CSO network Ex. Monthly knowledge exchanges among CSOs Ex. CSOs partnerships for engagement with high-level government officials 66 Guide for Identifying Lessons from Mapped Outcomes Analyzing and mapping the outcomes of a program can help teams learn from their implementation experience. Lessons from implementation provide important information about how different social actors influenced change across a timeframe and how some milestones or intermediate steps were instrumental for overcoming key challenges and supporting institutional changes. Codifying these lessons and possible indicators to track changes over time can inform learning on a particular program and others working in the sector and/or region to learn from successes and minimize failures. Lessons can be gained around various topics:  What can we learn about the sustainability of the changes to institutionalize results?  What can we learn about the effectiveness of the program design in advancing what needs to be achieved?  What can we learn about the mix of actors taking action to advance key changes? The following maps of outcomes are from the Leadership for Results (L4R) program in Burundi, which contributed to developing capacity and delivering results toward public sector reform. It offered reform teams a platform for shaping an in-country coalition, converging on a common vision, and working toward 100-day goals to improve local service delivery. The lessons documented are to demonstrate how outcome information could be used to draw lessons from a program. Identify Lessons from Results Chain A results chain in this case summarizes the different levels of results observed during implementation of the project. A useful technique for identifying lessons is tracing the progression from inputs and short- term milestones to intermediate outcomes and institutional changes. Lessons are derived from the “essential changes” that reflect steps that are deemed necessary for achieving progress toward a development objective. Such changes could reflect:  A critical junction—an outcome that stems from or leads to multiple other outcomes or milestones to support an integrated, complex change process.  A gateway or bridge—an outcome or milestone that leads directly to a higher level along the results chain. Figure 1 provides an example results chain in which the lightest levels are furthest from the project’s “sphere of influence” and closest to the impact. The darkest levels are more directly under project control.  Impact is the goal in the line of sight of the project.  Institutional changes are outcomes that reflect institutionalization or sustainability. Often these are more permanent changes in stakeholder engagement/responsiveness, formal policy/rules/guidance and organizational effectiveness. Each outcome at this level has an “outcome story” that outlines the project’s theory of change and informs sustainability.  Changes in behavior are outcomes that reflect new knowledge, skills, relationships, or implementation know-how after any intervention or with a different group than the direct participants.  Early outputs or milestones are changes observed during an intervention or directly at the end that influence later changes.  Inputs are key project activities/interventions. 67 The circled outcomes are examples of essential changes observed during implementation. For example, outcome 6 shows how the Steering Committee initiating a process to build knowledge among civil servants to use results-based methods for addressing constraints was a key early milestone. Figure 1. Example of a Results Chain: Improving Public Service Delivery in Burundi Look at Change Process Map of Project Diagramming the overall change process or theory of change in terms of the original program inputs, key change agents, and main change strategies to overcome challenges can help teams think through how the program design was most effective in advancing needed changes. In some cases, the review of the main outcome areas helps to highlight lessons that might not immediately surface when examining the results chain. For example, outcome area 2 in figure 2 shows the importance of the shift to an adaptive learning culture across government to address reform challenges and advance results. 68 Figure 2. Example of an Overview of the Change Process: Improving Public Service Delivery in Burundi EXAMPLE LESSONS Based on figures 1-3, what lessons can be taken from the public service delivery project in Burundi to inform the current program and or similar initiatives? Lesson 1. Obtaining high-level commitment provided an authorizing environment to demonstrate the benefits of the results-based L4R approach and transition from limited pilots by champions to the endorsement of its broad application to improve service delivery across government.  Government ministries and agencies at all levels faced inefficiencies that hindered service delivery, yet there was no system in place to support learning around achieving results. 69  The president created a Steering Committee to strengthen results and nominated the second vice president as chair. The committee provided leadership across the public administration to advance the new culture of delivery and problem solving required for the L4R approach to succeed.  The committee commissioned studies to inform the launching of initial pilot initiatives in the health and education sectors. When the Ministries of Education and Health publicized successful results of the pilots, the president and vice presidents expressed their buy-in for using a similar approach to unblocking problems in other areas of public sector reform.  The committee provided the leadership commitment to build knowledge in L4R methods to strengthen delivery across the public administration.  The committee monitored progress on pilot initiatives at key points to facilitate problem-solving to unblock slowed progress—particularly blocks that spread across different sectors or departments. The committee’s oversight role also helped identify follow-up projects to build on or scale-up.  The two vice presidents of Burundi signed a decree recommending the L4R approach be applied transversally throughout government to support implementation of the country’s Poverty Reduction Strategy Paper. The decree said each ministry should launch results-based initiatives every trimester that aligned with priorities in that sector and that progress should be recorded (quantitative and qualitative) at the end of each trimester. EXAMPLE INDICATORS FOR TRACKING THIS CHANGE PROCESS:  Establishment of a formal steering committee to manage Burundi’s L4R program (Yes/No)  % of results-based initiatives the steering committee supported on problem-solving  # of ministries launching results-based initiatives to pilot solutions for identified challenges (ministry records, reports to steering committee) Lesson 2. The cyclical implementation of rapid results initiatives provided opportunities for collaborative teams to adaptively learn how to develop sustainable solutions for overcoming implementation challenges and bottlenecks. This experience, over time, fostered a culture change across government, with a growing number of ministries implementing the results based method to achieve development objectives.  Government ministries and agencies at all levels lacked a results culture, with no system in place to support learning around achieving results. This lack of systematic learning supported widespread and persistent inefficiencies that hindered service delivery.  Ministers identified multi-agency stakeholder teams to implement pilot initiatives to advance priority results in 100 days.  The success of the initial pilots in the education and health sectors started a demonstration effect, with other ministries perceiving the potential benefits of the rapid results approach and piloting similar efforts in their sectors.  Results based initiatives were cyclically built on to provide lessons to adapt or scale-up across government. This momentum led steadily to a culture change, with an increasing number of ministries and agencies at all levels recognizing the value of implementing rapid results initiatives and using the results data to inform program planning. EXAMPLE INDICATORS FOR TRACKING THIS CHANGE PROCESS:  # of government ministries and agencies applying the rapid results approach to address implementation challenges and bottlenecks.  % of successful initiatives, as demonstrated by customized performance indicator(s) such as: a. Days required to deliver 25,000 textbooks in target province 70 b. # of pregnant women tested for HIV within 30 days in target province c. % of electricity grid customers satisfied with service  % of initiatives with follow-up actions to scale-up or build on the initiative Lesson 3. Using trained coaches that were embedded in ministries helped to institutionalize the results-based approach across sectors and levels of government. This design feature provided key support for the ongoing shift to a results culture in public service delivery.  Individual ministries had limited systems and processes to learn what works and apply lessons. Further, no mandate or resources existed that focused on developing leadership and learning for results.  Public servants with coaching capacities were embedded in national ministries to facilitate results-based initiatives but their numbers and reach were limited.  Support to the National School of Administration helped to train civil servants in the results- based method and to develop curriculum that could ensure the approach can be used across the public administration nationally and sub-nationally.  Coaches received training in local languages and support from the National School of Administration enabling the government to start to use the results- based methods sub- nationally. EXAMPLE INDICATOR FOR TRACKING THIS CHANGE PROCESS:  # of trained coaches embedded in ministries and agencies across government  % of ministries and agencies with trained coach embedded (by sector) Lesson 4. Engaging diverse stakeholders during the design of a strategy for scaling up the rapid results approach was critical for institutionalizing the L4R program. The participatory process led to a strategy that was broadly supported by government, civil society, and donors.  Government ministries and agencies at all levels faced severe problems with ineffectiveness and inefficiencies that hindered service delivery.  Limited or no collaboration existed among stakeholders to address service delivery problems.  The rapid results approach demonstrated success in selected pilots, but there was no strategy to scale it up for broader institutional change across agencies.  Engaging a broad range of stakeholders in strategy design ensured that diverse perspectives and needs were considered, increased awareness of the pilots’ results and broader potential of the approach, and ultimately fostered buy-in from stakeholder groups for scaling up the L4R program. EXAMPLE INDICATORS FOR TRACKING THIS CHANGE PROCESS INCLUDE:  # of stakeholder groups providing input for a new strategy to scale up the L4R program (attendance records of forums, administrative records of National School of Administration).  Development of a formal strategy for scaling up the L4R program (Yes/No, as evidenced by National School of Administration strategy documents).  % of ministries signing strategy; % of donors endorsing strategy; approval of budget to implement strategy (National School of Administration strategy documents, L4R program records, and administrative data). 71 Results Framework Informed by Lessons A results framework can also be developed to assess progress for key outcomes. The sample results framework below is modeled after the framework structure used for projects in World Bank operations. Indicators can be identified and included for measurement as follows:  Development goal—to assess progress toward higher-level objective over the longer term.  Institutional changes—at least one key institutional change should be measured for each outcome story. In cases where multiple related institutional changes are part of the same story, it might be sufficient to just reflect one in the results framework.  Major intermediate outcomes—major behavioral changes that are useful to track. These outcomes often reflect important gateways or bridges to other higher level outcomes and therefore reflect critical junctions in the outcome story. Figure 3. Example of Part of a Results Framework: Improving Public Service Delivery in Burundi Development Objective Indicator(s) Data Source(s) Use of Information Improved delivery of Development objective indicators for key target Ministry of Health Assesses the longer-term impact of central and sectors—for example: Administrative improving organizational decentralized public Health: Coverage of treatment to prevent mother to Records performance through the use of services for citizens at child HIV transmission Rapid Results Initiatives. Specific all levels in Burundi Transport: % of population less than 2km walk from a Ministry of Transport indicators can be added and motor able road and Public Works monitored over time as initiatives are Water Supply and Sanitation: % of population with implemented and scaled up. reliable access to treated water (disaggregated by rural Ministry of Water and urban areas) Institutional Changes Indicator(s) Data Source(s) Use of Information Organizations improved  MoE—Days required to deliver 25,000 textbooks in Agency administrative Demonstrates the success of the pilot performance through target province data systems, pilot initiative in unblocking a barrier for pilot initiatives  MoH—# of pregnant women tested for HIV within reports improving organizational 30 days in target province performance.  MoE and Civil Service—# of months for teachers to receive first paychecks Indicators are monitored at the pilot  Civil Service—# of fictitious staff within civil service level and a subset of initiatives are receiving salary payments (target is to decrease) Survey of state tracked at the program level.  State Electricity Company—% of electricity grid electricity company Additional indicators can be added as customers satisfied with service customers new initiatives are launched. ENA developed Development of formal strategy for scaling up the L4R ENA Strategy Confirms establishment of a strategy visioning paper and Program [Y/N] Documents; L4R for scaling up the program that budget to scale up Program records reflects the inputs of key program endorsed by % of ministries validating the strategy stakeholders, is owned by ministries ministries and donors % of donors endorsing strategy and endorsed by donors Approval of budget to implement strategy [Y/N] Intermediate Outcomes Indicator(s) Data Source(s) Use of Information A growing number of # of ministries launching results-based initiatives to pilot Ministry records; Assesses new results-based behaviors ministries implemented solutions for identified bottleneck reports to the L4R of ministries introduced by embedded initiatives using results- % of successful initiatives Steering Committee coaches and, in the aggregate, based methods % of initiatives with follow-up actions to scale-up or reflects growth of a common agenda build on the initiative for results-based learning in govt National network of # of trained coaches embedded in ministries Steering Committee Examines the decentralization of the coaches trained in # of ministries with at least one trained coach embedded records; Ministry capability for results-based method results-based methods # of teams sharing experiences in the use of the results records and formation of a new network for based methods for project management promoting/sustaining results culture Short-Term Outputs Indicator(s) Data Source(s) Use of Information 2nd VP established L4R Establishment of L4R Steering Committee, Chaired by Agreement with L4R Confirms the presence of both a Program and headed Vice President [Y/N] team; program strong authorizing environment and a Steering Committee # of meetings of Steering Committee records coordination mechanism for the # of Cabinet Retreats program’s implementation 72 Guide for Communicating Outcomes Real-time learning on results helps teams to address various communications needs before, during and after project implementation. External stakeholders and funders can gain a “big picture” understanding of how inputs are helping to advance a development objective. Outcome stories can spotlight promising results. Or, results reporting can emphasize lessons learned to inform practitioners working in similar contexts or to guide decisions for replicating or scaling up program activities. The example tables and maps in this toolkit are ways of communicating results; some proposed approaches for intentional results reporting are explored here. First, for a more comprehensive presentation of a program’s results, teams can develop a map with interpreted outcomes. As shown in the example below, such a map links outcomes and also can code major changes. The graphic depiction of sequenced outcomes provides an entry point for exploring individual outcome stories or for analyzing the overall change process. Example Map with Interpreted Outcomes: Utility Reform in Bosnia Another effective way to provide stakeholders a sense of what has been achieved is to separate out individual outcome stories. In this case, the team can focus on one key institutional change that has been achieved and show the progression of milestones and outcomes that contributed to that change. Summary text can be presented, as shown in the box below, with as much or as little detail as appropriate for the intended audience. 73 EXAMPLE OUTCOME STORY INCREASING THE COMMITMENT OF LEADERSHIP TO IDENTIFYING AND SCALING UP DEVELOPMENT INITIATIVES Outcome On July 14, 2009, a decree was promulgated and signed by the two then-Vice Presidents of Burundi recommending that the Rapid Results Approach (RRA) be applied transversally throughout the government to support implementation of the country’s Poverty Reduction Strategy Paper. Signaling high-level leadership and commitment, the decree prescribed that each ministry should launch initiatives every trimester aligned with the priorities in that sector and that progress (quantitative and qualitative) should be recorded at the end of each trimester. This resolve to apply the RRA reflected the implementation of one of the main recommendations of a Governmental retreat held earlier in 2009 and marked the culmination of a change process that started in 2006. Key milestones included the following:  In 2006, the President of Burundi, Pierre Nkurunziza made a commitment to strengthen the capacities of the country’s leaders to drive change at the institutional level to accelerate the implementation of national programs aimed at delivering results to benefit the country’s population.  A leadership program was then developed to help leaders tackle national development implementation challenges. A President-appointed Steering Committee was created to manage the process, and the Second Vice-President was nominated as head of the Steering Committee.  After commissioning background studies, the Steering Committee supported the launching of pilot projects in the health and education sectors. The Ministry of Education (MoE) delivered 25,000 textbooks throughout the province of Bubanza within 60 days instead of one year, and the Ministry of Health (MoH) more than quadrupled the number of pregnant women being tested for HIV in the province from 71 to 482 within 30 days.  When the MoH and MoE publicized the successful results of these pilots, the President and Vice Presidents expressed their strong buy-in for a similar approach to unblocking problems in other areas of public sector reform.  The Second Vice-President, as head of the Steering Committee, requested that staff within the sectors for which he was responsible be trained in a results-based management methodology that could be used as a tool to jumpstart new initiatives.  In September 2007, several of those how had undergone a five-day training created action plans to launch efforts to overcome challenges and bottlenecks in their own ministries.  The Steering Committee monitored progress and prioritized areas for new initiatives with government officials. Significance This decree reflected high-level commitment to the RRA process and served as an instrument to ensure the institutionalization of the capacities developed, with the new practices becoming self-sustaining and applied even more broadly through government. Despite the fact that the two Vice Presidents who signed this decree are no longer in their positions and ministers have changed in many ministries since 2009, the decree is still being applied to support implementation as, even now, there are still rapid results initiatives underway. Program Contribution The World Bank provided advisory support to Burundi’s Leadership for Results Steering Committee, recommended the preparation of background studies to inform the selection of the first pilots, and introduced the Rapid Results Approach as a tool for jump starting and accelerating project implementation. The Bank also provided support for the RRA training and acted as the convener and resource for knowledge exchange throughout this process. In March 2009, the Bank provided technical assistance and guidance in facilitating the Second Governmental Retreat in March 2009, during which the President and two Vice Presidents expressed their commitment for working towards the institutionalization of the culture of results by using the RRA more broadly. 74 Another approach for communicating results is to provide a timeline snapshot of key changes achieved to catch the attention of busy stakeholders who might not be expected to read a more lengthy text. Example Results “Snapshot”—Gaining Leadership Commitment for Rapid Results Initiatives Timeline Outcome Program Contribution Two Vice Presidents sign public decree that mandates expansion July 2009 of L4R Program across government In March 2009, the program More than 400 additional rapid results initiatives launched, with provided technical assistance and demonstrated success guidance in facilitating the Examples Second Governmental Retreat  The time required for newly recruited teachers to receive their during which the President and first paychecks in six provinces is reduced from one year to three two Vice Presidents expressed 2008 to months; 717 new teachers received their paychecks without their commitment for working 2009 being subset to corrupt fees towards the institutionalization  The Civil Service reduces fictitious staff, suspending payment of of the culture of results by using 728 salaries for “ghost” individuals, reflecting a monthly cost the RRA more broadly. saving of $530,759  Client satisfaction with the state electricity company increased from 18% to 65% 2,000 coaches were trained and embedded in ministries across 2007 to government for implementing additional Rapid Results 2009 Initiatives. All coaches are linked to regional network of results The program provided support coaches in Africa. for the RRI training and acted as The pilots demonstrate success, reported by the Ministries of the convener and resource for Education and Health—the in the target province decreased knowledge exchange throughout May 2007 from one year to this process. Health—the number of pregnant women tested for HIV within 30 days in the target province increased from 71 to 482 2006 to Two initial pilots are implemented in priority sectors (health and The program provided advisory 2007 education) support to Burundi’s Leadership for Results Steering Committee, recommended the preparation of background studies to inform the A Steering Committee is formally established, to manage selection of the first pilots, and June 2006 Burundi’s Leadership for Results Program introduced the Rapid Results Approach as a tool for jump starting and accelerating project implementation. 75 Snapshot of External Training Knowledge Services Project Outcomes The following describes outcomes of selected World Bank External Training (TE) projects in different sectors/thematic areas in 2013 and 2014. The outcomes present intermediate and deeper institutional changes reported toward the project’s development objective(s). The outcomes are based on qualitative outcome information jointly harvested by the project team members and clients. The charts present progress snapshots to communicate results of the projects. Case: Strengthening Capacities of Local Governments in South East Europe Objective: Progress toward more livable and sustainable cities in South East Europe that provide a high quality of life for citizens Total number of outcomes harvested: 28 institutional and 14 intermediate changes Institutional outcomes by actors: 25 by municipalities/local councils, 8 by mayors, 5 by Network of Associations of Local Authorities of South East Europe, 4 by coalitions/CSOs Institutional Changes in Cities and Regions During 2011-13 Increased commitment, transparency and decision- making 44% 44% Policy/strategy changes to combat corruption Increased effectiveness of 12% municipalities and regional networks Case: Strengthening Implementation of Legislation on Access to Information (ATI) across Latin America Objective: Progress toward improvement in service delivery for citizens across Latin America Total number of outcomes harvested: 9 institutional and 12 intermediate changes Institutional outcomes by actors: 10 by ATI agencies in Bolivia, Colombia, Chile, Mexico, Peru, Brazil and Uruguay, 10 by formal Latin American network of ATI agencies, 1 by Open Government Partnership Institutional Changes During 2011-13 Increased commitment of Access to Information agencies in Bolivia, Colombia, 45% 46% Chile, Mexico, Peru, Brazil and Uruguay Increased guidance to implement Access to Information policy 9% Increased effectiveness of Access to Information agencies 76 Case: Scaling up Capacity Development of City Officials and Practitioners across China Through eLearning Objective: Progress toward improvement in public service delivery in China’s rapidly growing urban areas Total number of outcomes harvested: 7 institutional and 13 intermediate changes Institutional outcomes by actors: 20 by Chinese Academy of Governance (national institution mandated to train public officials) Institutional Changes During 2009-12 Increased government 14% commitment to scale-up public officials learning 14% Policy change to scale-up public officials learnin 72% Increased effectiveness in delivering learning Case: Improving Open Contracting Processes at the Country and Global Level Objective: Progress toward improvement in the benefits of public goods and services for all citizens Total number of outcomes harvested: 17 institutional and 16 intermediate changes globally and in the countries of Uganda, Mongolia, Nigeria and Ghana Institutional outcomes by actors: 11 by contract monitoring coalitions, 4 by government ministries/bureau, 4 by CSOs, 4 by Steering Group (became known officially as Open Contracting Partnership), 2 by national procurement authorities that joined Steering Group, 2 by the World Bank, 2 by Federal courts, 1 by academia, 1 by state power holding company, 1 by government aid agencies, 1 by private sector initiatives Institutional Changes During 2011-13 Increased commitment, transparency and decision- making 44% 44% Policy/strategy changes to combat corruption Increased effectiveness of 12% municipalities and regional networks 77 Case: Strengthening Parliamentary Oversight of National Budgets in Africa Objective: Progress toward improvement in the benefits of public spending for citizens in Africa Total number of outcomes harvested: 17 institutional and 12 intermediate changes regionally and in the countries of Botswana, Ethiopia, Kenya, Rwanda, Swaziland, South Africa, South Sudan, Tanzania, Uganda and Zambia Institutional outcomes by actors: 15 by parliaments, 11 by public accounts committees, 3 by public accounts committee network Institutional Changes During 2009-13 15% Increased transparency of budget and audit processes 45% Increased legitimacy of budget process and corruption 40% mechanisms Increased effectiveness of parliament and public accounts committees Case: Leadership for Results: Developing Capacity and Delivering Results toward Public Sector Reform in Burundi Objective: Progress toward improvement in delivery of central and decentralized public services for citizens at all levels in Burundi Total number of outcomes harvested: 26 institutional and 21 intermediate changes Institutional outcomes by actors: 10 by high-level government officials, 10 by civil service, 10 by ministries, 6 by state electricity company, 6 by National School of Administration, 3 by steering committee, 2 by volunteers, 2 by communal administrators Institutional Changes During 2006-13 Increased public sector leadership and priority setting to 35% strengthen reforms 54% Increased policy changes to combat inefficiencies and 11% corruption in services Increased effectiveness, adaptability and responsiveness in service delivery 78 Case: Improving Social Accountability in the Philippines Education Sector Objective: Progress toward improvement in the quality of services and education performance of public schools for citizens in the Philippines Total number of outcomes harvested: 13 institutional and 10 intermediate changes Institutional outcomes by actors: 8 by infomediaries, 5 by Department of Education, 5 by Affiliated Network for Social Accountability, 2 by governments (Kenya, Indonesia), 1 by World Bank, 1 by Open Government Partnership, 1 by academia Institutional Changes During 2008-12 Increased commitment and innovations 25% to advance social accountability and transparency in public schools Increased legitimacy of public school 13% 62% data increased responsiveness of government to public education issues 79 Resources Guides and tools for outcome-based learning are continuously being developed. For those with access to the World Bank Intranet, visit http://outcomemapping. Outcome Mapping Learning Community www.outcomemapping.ca The Better Evaluation website http://betterevaluation.org/blog/drawing_logic_models Theory of Change Online or “TOCO” http://www.theoryofchange.org/toco-software The Capacity Development Results Framework http://wbi.worldbank.org/wbi/document/capacity-development-and-results-framework Designing a Multi-Stakeholder Results Framework http://wbi.worldbank.org/wbi/document/designing-multi-stakeholder-results-framework Cases in Outcome Harvesting (full report of 10 first cases in pilot experiences) http://wbi.worldbank.org/wbi/document/cases-outcome-harvesting 80 Ulaanbaatar, Mongolia case study in outcomes evaluation: Mongolia Showing the value of mapping outcomes to learn from complex programs SUMMARY In fall 2013, the World Bank and Mongolia office of From 2010–2013, the World Bank Governance Part- SDC decided to use an outcome mapping approach nership Facility (GPF) and the Swiss Agency for to evaluate the effectiveness, sustainability, and Development and Cooperation (SDC) helped build relevance of these interventions. Outcome mapping the capacity of Mongolian civil society organizations is a participatory methodology useful for evaluating (CSOs) to promote good governance and an effective complex programs that involve capacity and coalition civil society engagement in procurement and service building, multiple actors, and tacit knowledge. It looks delivery monitoring. An assessment of results from the beyond outputs and delivery efficiency to institutional interventions was needed to satisfy accountability and behavioral changes that occur in and among social learning needs and to inform decisions on future pro- actors influenced by interventions. grams and funding. However, the short-term, complex A total of 190 outcomes were collected through nature of the interventions, numerous CSOs involved, document analysis, interviews and surveys with perti- and scarce documentation meant that knowledge nent social actors, with nine elaborated as outcome of results was largely limited to activities and impact stories. The outcome stories identified what changes would be difficult to measure. took place, by whom, when and where, why they were 81 significant, and how the interventions contributed. Democracy Education Center, and Open Society Then, independent persons substantiated the out- Foundation. come stories to provide further evidence and cred- (3) World Bank—Public Procurement (2012–ongo- ibility on whether the World Bank and SDC support ing) to support the development of an effective, self- advanced development objectives. governing network of CSOs committed to monitoring The evaluation provided benefits to the stakehold- public procurement and supporting government in ers in several ways: results were packaged into an creating a framework for CSO participation in public accessible, narrative format for various communica- procurement monitoring. Implementing partner: Part- tion purposes; lessons were identified on what worked nership for Public Procurement. and did not work to inform the design of future CSO To evaluate the interventions, the team needed to support, particularly concerning social actors and their answer three questions: roles, innovative solutions, and how to adapt or scale • What is the effectiveness of the interventions in up a program; and the participatory process promoted contributing to the intended objectives? stakeholder learning and ownership of results achieved • What is the sustainability of changes influenced to date. by the interventions after donor funding ceases? Thus, the evaluation generated robust, locally • What is the relevance of the interventions’ validated data that demonstrated the value of the outcomes to the SDC and Bank programs in interventions to stakeholders and donors and revealed Mongolia, the Bank’s GPF objectives, and the needs ways to improve implementation and management for of target CSOs? future efforts. Other teams that need to assess similarly However, the nature of the interventions presented complex programs might consider the merits of using difficulties for assessment. First, they were short in the outcome mapping approach as well. timeframe. For instance, the social accountability project consisted of one workshop and small grants for pilot projects with mentoring, and it ended in 2011. CONTEXT Second, little data existed, and tacit knowledge had to Over the last decade the Mongolian economy has be collected to provide evidence of results. Third, and grown at a rapid pace based on the strength of its most important, changes in behaviors and relation- extractive industries sector. Good governance and ships within and among the CSOs and government an effective civil society are key to ensure that the needed to be captured to show the richness of the wealth produced is used in a way that benefits all change process. These were the intermediate changes citizens. Government must develop policies and that could lead to longer-term results. systems that are responsive to citizens and open to public scrutiny. Wider civil society can contribute to improved accountability, transparency, and openness OUTCOME MAPPING by monitoring government tenders and public fund Outcome mapping is an innovative assessment expenditures. methodology to learn from complex development To this end, the World Bank and SDC supported processes that involve behavioral changes, multiple Mongolian civil society in three interventions: social actors, and profound development challenges. (1) SDC—Local NGO Capacity Building (2011–2013) Although more commonly used at the design stage of to improve internal governance, oversight, financial an intervention, here it was used retrospectively in the and operational management of Mongolia’s most evaluation. active local CSOs. Implementing partner: Mongolian Outcome harvesting is one of the tools from the Center for Development Studies. Outcome Mapping Learning Community.1 In this (2) World Bank—Social Accountability (2010–2011) approach, an outcome is defined as a change in to strengthen the skills of CSOs to work with and moni- the behavior, relationships, activities, or actions of tor public sector organizations and service delivery the people, groups, and organizations with whom a and willingness of public sector organizations to work program works directly. For interventions promoting with CSOs. Implementing partners: Affiliated Network learning, this approach implies that participant learn- for Social Accountability in East Asia and the Pacific, ing outcomes are demonstrated when they apply the concepts and tools from their learning in their work. 82 Through collecting—or harvesting—bites of that interventions influence and/or contribute to but detailed outcome information from colleagues, cannot lay sole claim to results was therefore valu- partners, and stakeholders, one can identify, monitor, able. Further, the approach fitted with the evaluation’s and learn from changes in social actors. The collected purpose—that it was as much about learning with the information describes what changed, for whom, when local implementers of the project and informing future and where, why it matters to the development objec- work as it was about accountability. tive and particular development challenge(s)—the significance of the change—and how the program contributed to the change. PROCESS The harvesting process is stakeholder-centered Specifically, this was an outcomes evaluation of the and captures qualitative, tacit knowledge. It includes World Bank GPF and SDC CSO/NGO capacity build- tools to substantiate and analyze this knowledge col- ing interventions that took place in Mongolia, August laboratively and communicate progress toward impact 2010–September 2013. Richard D Smith (team leader), to clients, management, and partners. The method is Jeremy Gross, and Amarbayasgalan Dorj conducted flexible to adapt to a program’s design and comple- the evaluation from September–November 2013. ment other monitoring and evaluation and knowledge Using the outcome harvesting tool, the evaluators management tools. identified outcomes with the social actors the interven- Outcome mapping approaches follow the principle tions had been seeking to influence directly. They col- that evaluations should be focused on use and users. lected outcomes by means of one-on-one and group Use-focused evaluations seek to engage users of the interviews, focus groups, and surveys (see annex for evaluation from the outset in defining the scope and sample questions). The evaluators looked for observ- questions, and agreeing on information and substanti- able evidence that participants had applied concepts ation sources. In addition, the evaluation process itself and tools introduced by the interventions. engages users in generating or verifying data. Such Each outcome was precisely described so it is clear engagement may help users have more confidence in and verifiable who changed in what way, when and evaluation findings. where, and how the intervention contributed (see figure 1 on the next page for examples of outcome descriptions). Outcome harvesting includes a specifi- DECISION TO USE OUTCOME MAPPING cation for optional contribution descriptions for each The World Bank GPF and SDC commissioned a joint outcome; these were vital for understanding how the evaluation using the outcome mapping approach. interventions had contributed to outcomes, directly or The team decided outcome mapping was well suited indirectly. because the objectives of the interventions being The outcome definition used2 set a high bar for evaluated involved areas—capacity development, net- assessing the effectiveness and sustainability of Bank work building, and accountability—that meshed with and SDC contributions to capacity and coalition-build- the methodology. ing changes. Informants were given the opportunity to The effectiveness of network building and network- verify outcome descriptions. ing is demonstrated by behavior changes that can The evaluators entered the outcomes in a database be found through collected outcome information. and coded them by type of change to provide for a For example, outcome information on the voluntary program-level reflection of the outcomes to date. The association of organizations or individuals and their resulting findings were then interpreted to address the changes represent new ways of working collectively, evaluation questions as follows: such as cooperating to plan work, sharing knowledge, • Effectiveness was assessed against the pre-defined strategizing, and securing resources. The realization of objectives and theory of change to identify whether social accountability is also demonstrated by behavior intended outcomes were met, and any unintended changes, in this case of government, business, citizens, outcomes outside the intended scope. and CSOs. • Sustainability was assessed by whether institutional The effectiveness of the Bank and SDC interven- changes had been realized. tions depended on the extent to which they were able • Relevance was assessed by whether the outcomes to influence others. The outcome mapping concept aligned to local needs and program priorities. 83 In addition, the evaluators highlighted key out- comes in outcome stories to communicate specific Figure 1. Sample Outcome Descriptions achievements or lessons in a narrative format (see SDC—Local NGO Capacity Building annex for a sample story). They produced nine out- Outcome 11: Between 2012 and 2013, after the come stories, each with sufficient detail to allow project training, six organizations developed a written independent sources to substantiate the outcome, charter defining, for example, the organization’s the contribution of the intervention, and the claimed purpose; beneficiaries; activities; role of the board; significance of the outcome.3 elections; how meetings are run. Contribution: The lead implementor of the SDC project appointed the team of trainers, contributed to CHALLENGES the development of the modules and organized and The evaluators faced several challenges during the facilitated the trainings. harvesting process, which included: Limited documentation of outcomes—Lack of World Bank—Social Accountability existing monitoring data caused a high dependence Outcome 86: In March 2013, the Democracy on collecting data during the evaluation. For two Education Center (DEMO) was able to expand its interventions, there was limited knowledge of Check My Service program when the Asia Foundation outcomes from serving staff of the Bank, SDC, and awarded it a grant for the Check My School and Check My Clinic projects. their contracting implementing partners due to high staff turnover and lack of a strong monitoring Contribution: DEMO’s leading role in implementing framework for outcomes. the Social Accountability Learning-in-Action (SAcLAP) program enhanced its reputation and expertise in the Lack of experience in methodology—Local con- area so it could develop a convincing proposal for the sultants were not well versed in outcome mapping, Asia Foundation. but this was looked at as an opportunity to build up their capacity in outcome mapping and evaluations for World Bank—Public Procurement future work. The lead evaluator coached co-evaluators Outcome 11: In April 2013, the Public-Private on the necessary concepts and provided guiding ques- Partnerships Board and review committee adopted tions for interviews. The co-evaluators were willing and its first five-year strategic plan with an initial focus on able to learn and also brought complementary experi- shaping new procurement law implementation. ence and skills to the team. Contribution: The World Bank Institute and the Perceptions of methodology—Initially there was Affiliated Network for Social Accountability in East some hesitation from Bank and SDC staff about using Asia and the Pacific (ANSA-EAP) designed and outcome mapping given that it can be a time-con- facilitated the strategic planning working with the suming process. The evaluators overcame this notion board, and WBI provided feedback on drafts of by using key concepts of outcome mapping rather the plan. than all of the steps. The Bank also voiced concern about having clear evaluation criteria. The evaluators addressed this by using outcome mapping to answer FINDINGS questions solely on effectiveness, sustainability, and The 190 outcomes collected demonstrate that each relevance. Efficiency could be assessed with a more intervention met or exceeded its pre-defined objec- appropriate evaluation tool. tives (see table 1). This represents impressive results Shortened timeframe—The short timeframe for short-term interventions, two of which were ongo- limited travel time to engage with actors influenced by ing, leading to the conclusion that the interventions the interventions. This restricted the ability to engage have been effective and relevant. with those most knowledgeable when describing and substantiating outcomes. However, the local consul- SDC—Local NGO Capacity Building tant was able to stay and conduct these interviews at 69 outcomes in total later and more suitable times. Against the central objective of building the capac- ity of CSOs, outcomes show improved organizational 84 performance of CSOs even over a limited time. The Judged strictly against the mining (or extractive indus- greatest effect came when, beyond improving internal try) value chain, however, the intervention has not been organizational abilities, CSOs were empowered to be fully effective since it has almost without exception only active in their community and involved in activities that strengthened CSO capacity at the end of the value supported such endeavors, including local govern- chain farthest from the extractive industry. ment monitoring, providing data for community needs, A further benefit of assessing the program two years or acting as service provider. after it concluded is that it has been possible to inter- pret the extent to which the results are sustainable. World Bank—Social Accountability After the intervention, 71 of the outcomes materialized, 93 outcomes from all 13 pilot project grantees and many of which demonstrate organizational ownership of some from those who only participated in one training concepts and tools introduced. workshop World Bank—Public Procurement Against the objective to strengthen monitoring capac- 28 outcomes total ity of CSOs on the mining value chain and related economy outcomes, the intervention was successful at Against the objectives of support to the Ministry of introducing or significantly enhancing social account- Finance and CSOs, the intervention was successful at ability knowledge and skills in several organizations. supporting the ministry and the development of a self- Table 1. At a Glance: 190 Outcomes Collected from the Three Interventions SDC—Local NGO Capacity Buiding 69 Outcomes in total, each relevant to or exceed SDC’s pre-defined objectives 51 Outcomes demonstrate the application of knowledge from the intervention 7 Outcomes suggest a sustained influence of the intervention 5 Outcomes cite engagement between CSOs and their community World Bank—Social Accountability 93 Outcomes in total, each relevant to or exceed the Bank’s pre-defined objectives 71 Outcomes suggest a sustained influence of the intervention 59 Outcomes demonstrate the application of social accountability knowledge gained through the intervention 29 Outcomes show dissemination of social accountability 24 Outcomes cite constructive engagement, demonstrating a deepening awareness of social accountability 14 Outcomes show successful fundraising for implementation of social accountability activities after the intervention 11 Outcomes show demand for support in using social accountability concepts and tools 11 Outcomes show networking of practitioners 9 Outcomes directly relevant to the mining value chain 7 Outcomes describe working with the private sector 7 Outcomes demonstrate advocacy of social accountability 5 Outcomes involve engaging the media World Bank—Public Procurement 28 Outcomes in total, each relevant to or exceed the Bank’s pre-defined objectives 25 Outcomes suggest the potential sustainability of the Partnership for Public Procurement 14 Outcomes are at the aimag (provincial) level 12 Outcomes are at the national level 11 Outcomes demonstrate strengthened capacity of CSOs in procurement monitoring 8 Outcomes show influence on road maintenance, specifications and planning 7 Outcomes show support for self-governing CSO networks 5 Outcomes show support for Ministry of Finance in its development of implementing rules and guidelines on CSO participation and oversight 2 Outcomes at the Ulaanbaatar level 85 governing CSO network. But it had not yet succeeded outcome is the impact that empowered newly trained in developing monitoring tools for the network, and CSOs can have in their community. Six outcomes prove the extent of capacity strengthening of CSOs has this occurred because of the intervention. A local been limited to the few participants in two pilots. The trainer, motivated by what she learned, took it upon intervention was still young and continued for months herself to find resources and train a further 22 organi- more. While some objectives remain unachieved, zations. This unintended outcome that normally would the outcomes indicate progress in influencing rules not have been discovered adds to an appreciation of and guidelines and formal establishment of the the effectiveness and sustainability of the intervention. partnership. Engage stakeholders Outcome mapping encourages reflection and dia- BENEFITS logue—it is a truly participatory method in which Several benefits for the evaluators, Bank, and SDC counterparts are engaged, thereby giving them owner- came about from using outcome mapping as an ship over the process and ensuing results. It offers evaluative methodology in this particular case. Teams an opportunity to get at the collaborative theory of might want to consider these benefits when deciding change. on a methodology for assessing similarly complex In this particular case, the SDC and World Bank programs. worked together as a team to design the evaluation questions and in the process built a stronger relation- Examine multiple actors ship. Once data collection was completed, the evalu- Traditional evaluations tend to give credit to a single ators sent key informants their individual findings and contributor, when in complex development programs asked them to confirm whether the information was multiple actors drive change. It is important to dis- correct. This step enabled informants to play a main cover how and which actors worked together or built role and express their views, and added credibility as on each other’s actions over time to create results so well. future programs can maximize their potential and use the “right” mix of actors. Inform next steps For example, in the findings from the Bank—Social Outcome mapping enabled the evaluators to identify Accountability pilot in community monitoring of a links between multiple outcomes to uncover latent/ family hospital’s services and conditions, outcomes emerging knowledge, innovative solutions, and how to showed how multiple social actors brought about scale up the program. All of this helps to detect pieces change: patients, doctors, and hospital management. for prospective program design or further phases. Based on the engagement of patients, the hospital For example, in the area of NGO capacity building, changed its policy so all patients can use toilets that the outcome data provides a rich source for identify- were previously “staff only”; the hospital appointed a ing organizations that could fulfill particular roles in guide nurse to help patients arriving for treatment; and a new phase of funding support. Based on this data, the chief doctor appointed two additional doctors. the recommendation is that the SDC and the Bank, with selected stakeholders with firsthand knowledge Learn from the complete picture of relevant institutions in Mongolia, may review the Harvesting outcomes allows the exploration of sig- outcome data and other sources and, according to the nificant outcomes—whether intended or unintended, focus of any new intervention, identify potential actors negative, or tacit/unrecorded—to get a complete pic- for particular roles. ture of what went right or wrong and how to learn from the change process to inform future thematic areas. Communicate results Taking a narrow approach that considers only those Outcome mapping allows evaluators to go deeper outcomes that had immediate or direct contributions into what, why, and how changes happened than using from the interventions could miss stories of change more traditional methods, which can often rely on hard connecting related outcomes. numbers and indicators. Harvesting detailed outcome For example, in the findings from the SDC—Local information led to the writing of nine outcome sto- NGO Capacity Building intervention, an unintended ries that describe changes in people—how the roles 86 of actors affect results—to capture the flavor of what occurred. SDC wanted to have evidence of results presented in a storytelling format, and the outcome stories ended up being shared the most. The outcome map- ping methodology proved useful for reconstructing the storyline of change for the three interventions. n ACKNOWLEDGMENTS The case study is a collaborative effort based on documentation from the evaluation and interviews and feedback with: • Evaluation team—Richard D. Smith (leader), Jeremy Gross and Amarbayasgalan Dorj • World Bank—Marcela Rozo and Kathrin Frauscher from the Open Contracting team, and Governance Partnership Facility team members • Swiss Agency for Development and Cooperation team members Jenny Gold and Sharon Fisher, World Bank, led the case study team. Dawn Roberts provided input. Published by The World Bank, May 2014 FOOTNOTES 1 See www.outcomemapping.ca 2 Thisoutcomes evaluation followed the definition of “outcome” used in the outcome mapping methodology: a change in the behavior, relationships, activities, or actions of the people, groups, and organizations with whom a program works directly. 3 It is up to the principal evaluation user to decide what substantiation is needed to make the results credible. In this case, outcome data was provided nearly exclusively by those the interventions were seeking to influence and not by the World Bank or SDC staff. Thus, the outcomes were assumed to be credible, which substantiation of the nine outcome stories confirmed. 87 ANNEX Example of an Outcome Story: World Bank—Public Procurement Intervention The evaluators highlighted key outcomes in outcome stories to communicate specific achievements or lessons in a narrative format. They produced nine outcome stories. OUTCOME STORY 6—The Ministry of Finance adopted Implementing Rules and Regulations for the monitoring of public procurement that reflected suggestions from Civil Society Organizations (CSOs). Outcome In late 2012, the Ministry of Finance adopted new Implementing Rules and Regulations (IRR) for CSO participation in bid evaluation committees that included several requests of the Partnership for Public Procurement (PPP), such as the use of specific reporting templates and creation of a web portal for CSOs to use when reporting on the performance of bid evaluation committees. Significance The IRR are significant for being a necessary implementation instrument for the 2011 amendment to the Public Procurement Law, which mandated the involvement of civil society in public procurement for the first time. This was the first time that CSOs working on procurement in Mongolia successfully collaborated to advocate for regu- latory changes as a partnership network. Contribution Financial and technical support for the creation and strategic development of the PPP has been provided by the World Bank since 2012 through the Governance Partnership Facility it administers. The PPP provided coordinated inputs during the policy dialogue with the Ministry of Finance over the summer and autumn of 2012. N. Otgonjargal, chair of the PPP, led and coordinated the contributions of PPP members to the drafting of the rules and regulations. The Bank supported the Governance Partnership Facility in developing a united message to engage constructively with the government. Bank support included coaching, network build- ing and technical advice on procurement monitoring. 88 ANNEX Interview Guide: World Bank—Social Accountability Intervention Informants Purpose of Interview • We are conducting an evaluation of the work the World Bank and SDC have supported since 2009 on civil society/NGO strengthening and CSO engagement in procurement. • Specifically, we are assessing the Social Accountability Learning-in-Action (SAcLAP) and PPP projects of the Bank and the NGO Effectiveness Project of SDC. • The main focus of the evaluation is to learn what did and didn’t work. We are not looking at how the money was used. Key Facts Name: Position now: Position during SAcLAP: Participated in SAcLAP training/pilot? Others from your organization who participated in SAcLAP ? How many workshops/events were there? What were they on? Who hosted, participated? SAcLAP 1. What did you/your organization gain from the SAcLAP project ideas/tools? 2. Were the tools/ideas appropriate and tailored for the Mongolian context? 3. Have there been any effects/benefits because of the work you did during the pilot project? Any changes in policies, practices, relationships, or activities of those you have been trying to influence? 4. If yes, who changed, what, when and where? How, exactly, did the SAcLAP ideas/tools contribute? Before SAcLAP 1. Before your involvement in SAcLAP, had you heard of social accountability? 2. If yes, had you done any social accountability work/used social accountability tools? 3. If yes, what was the project and who funded it? 4. If yes, from where/which organizations did you receive materials/tools/trainings to understand the subject? After SAcLAP 1. Have you used the ideas/tools from SAcLAP in any other work? 2. Have there been any effects/benefits because of the work you did during the pilot project? Any changes in policies, practices, relationships, or activities of those you have been trying to influence? 3. If yes, who changed, what, when and where? How, exactly, did the SAcLAP ideas/tools contribute? Who funded it? 89 ANNEX Survey Questions: SDC—Local NGO Capacity Building Intervention 1. A clear mission statements that reflects your organization’s purpose? Y/N 2. A written charter defining, for example, the organization’s purpose, beneficiaries, activities, role of the board, elections, how meetings are run? Y/N 3. An active board that meets regularly in accordance with the organization’s rules? Y/N 4. A mechanism to review and update your organization’s strategic plan and annual work plan regularly? Y/N 5. An annual workplan? Y/N 6. A human resource policy for staff development? Y/N 7. Clear roles and responsibilities for staff members and/or volunteers? Y/N 8. A staff member capable of submitting a financial report to the tax office? Y/N 9. Capacity to raise funds from members, donations or government funds? Y/N 10. The capacity to write a funding proposal? Y/N 11. Mechanisms for beneficiary, partner and stakeholder feedback? Y/N 12. Sustainable activities to achieve your mission and vision statement? Y/N 13. The ability to partner with other NGO’s to benefit from pooled resources? Y/N 14. What have been the specific benefits of each internal management change your organization has made following your participation in the SDC funded training on Capacity Building and Training for Local NGOs. 15. As a result of changes to your organization’s internal management, has your organization carried out new types of actions or activities? Please describe. 90 3 Methods such as outcome harvesting offer approaches to manage knowledge and learn from complex and difficult-to-monitor development processes. From 2012–2014, the World Bank undertook pilot experiences to identify how outcome harvesting could be integrated with its results management approach. The guides and tools customized in the pilot experiences were brought together in this field guide to be a collaborative vehicle for staff, clients, partners, and other stakeholders to harvest, substantiate, interpret, monitor, and learn from outcomes across the project cycle to improve effectiveness and results. It is hoped that these first outputs of the pilot experiences will continue to be adapted in different ways and improved upon through ongoing application and learning. 2