90171 Ulaanbaatar, Mongolia case study in outcomes evaluation: Mongolia Showing the value of mapping outcomes to learn from complex programs SUMMARY In fall 2013, the World Bank and Mongolia office of From 2010–2013, the World Bank Governance Part- SDC decided to use an outcome mapping approach nership Facility (GPF) and the Swiss Agency for to evaluate the effectiveness, sustainability, and Development and Cooperation (SDC) helped build relevance of these interventions. Outcome mapping the capacity of Mongolian civil society organizations is a participatory methodology useful for evaluating (CSOs) to promote good governance and an effective complex programs that involve capacity and coalition civil society engagement in procurement and service building, multiple actors, and tacit knowledge. It looks delivery monitoring. An assessment of results from the beyond outputs and delivery efficiency to institutional interventions was needed to satisfy accountability and behavioral changes that occur in and among social learning needs and to inform decisions on future pro- actors influenced by interventions. grams and funding. However, the short-term, complex A total of 190 outcomes were collected through nature of the interventions, numerous CSOs involved, document analysis, interviews and surveys with perti- and scarce documentation meant that knowledge nent social actors, with nine elaborated as outcome of results was largely limited to activities and impact stories. The outcome stories identified what changes would be difficult to measure. took place, by whom, when and where, why they were 1 significant, and how the interventions contributed. Democracy Education Center, and Open Society Then, independent persons substantiated the out- Foundation. come stories to provide further evidence and cred- (3) World Bank—Public Procurement (2012–ongo- ibility on whether the World Bank and SDC support ing) to support the development of an effective, self- advanced development objectives. governing network of CSOs committed to monitoring The evaluation provided benefits to the stakehold- public procurement and supporting government in ers in several ways: results were packaged into an creating a framework for CSO participation in public accessible, narrative format for various communica- procurement monitoring. Implementing partner: Part- tion purposes; lessons were identified on what worked nership for Public Procurement. and did not work to inform the design of future CSO To evaluate the interventions, the team needed to support, particularly concerning social actors and their answer three questions: roles, innovative solutions, and how to adapt or scale • What is the effectiveness of the interventions in up a program; and the participatory process promoted contributing to the intended objectives? stakeholder learning and ownership of results achieved • What is the sustainability of changes influenced to date. by the interventions after donor funding ceases? Thus, the evaluation generated robust, locally • What is the relevance of the interventions’ validated data that demonstrated the value of the outcomes to the SDC and Bank programs in interventions to stakeholders and donors and revealed Mongolia, the Bank’s GPF objectives, and the needs ways to improve implementation and management for of target CSOs? future efforts. Other teams that need to assess similarly However, the nature of the interventions presented complex programs might consider the merits of using difficulties for assessment. First, they were short in the outcome mapping approach as well. timeframe. For instance, the social accountability project consisted of one workshop and small grants for pilot projects with mentoring, and it ended in 2011. CONTEXT Second, little data existed, and tacit knowledge had to Over the last decade the Mongolian economy has be collected to provide evidence of results. Third, and grown at a rapid pace based on the strength of its most important, changes in behaviors and relation- extractive industries sector. Good governance and ships within and among the CSOs and government an effective civil society are key to ensure that the needed to be captured to show the richness of the wealth produced is used in a way that benefits all change process. These were the intermediate changes citizens. Government must develop policies and that could lead to longer-term results. systems that are responsive to citizens and open to public scrutiny. Wider civil society can contribute to improved accountability, transparency, and openness OUTCOME MAPPING by monitoring government tenders and public fund Outcome mapping is an innovative assessment expenditures. methodology to learn from complex development To this end, the World Bank and SDC supported processes that involve behavioral changes, multiple Mongolian civil society in three interventions: social actors, and profound development challenges. (1) SDC—Local NGO Capacity Building (2011–2013) Although more commonly used at the design stage of to improve internal governance, oversight, financial an intervention, here it was used retrospectively in the and operational management of Mongolia’s most evaluation. active local CSOs. Implementing partner: Mongolian Outcome harvesting is one of the tools from the Center for Development Studies. Outcome Mapping Learning Community.1 In this (2) World Bank—Social Accountability (2010–2011) approach, an outcome is defined as a change in to strengthen the skills of CSOs to work with and moni- the behavior, relationships, activities, or actions of tor public sector organizations and service delivery the people, groups, and organizations with whom a and willingness of public sector organizations to work program works directly. For interventions promoting with CSOs. Implementing partners: Affiliated Network learning, this approach implies that participant learn- for Social Accountability in East Asia and the Pacific, ing outcomes are demonstrated when they apply the concepts and tools from their learning in their work. 2 Through collecting—or harvesting—bites of that interventions influence and/or contribute to but detailed outcome information from colleagues, cannot lay sole claim to results was therefore valu- partners, and stakeholders, one can identify, monitor, able. Further, the approach fitted with the evaluation’s and learn from changes in social actors. The collected purpose—that it was as much about learning with the information describes what changed, for whom, when local implementers of the project and informing future and where, why it matters to the development objec- work as it was about accountability. tive and particular development challenge(s)—the significance of the change—and how the program contributed to the change. PROCESS The harvesting process is stakeholder-centered Specifically, this was an outcomes evaluation of the and captures qualitative, tacit knowledge. It includes World Bank GPF and SDC CSO/NGO capacity build- tools to substantiate and analyze this knowledge col- ing interventions that took place in Mongolia, August laboratively and communicate progress toward impact 2010–September 2013. Richard D Smith (team leader), to clients, management, and partners. The method is Jeremy Gross, and Amarbayasgalan Dorj conducted flexible to adapt to a program’s design and comple- the evaluation from September–November 2013. ment other monitoring and evaluation and knowledge Using the outcome harvesting tool, the evaluators management tools. identified outcomes with the social actors the interven- Outcome mapping approaches follow the principle tions had been seeking to influence directly. They col- that evaluations should be focused on use and users. lected outcomes by means of one-on-one and group Use-focused evaluations seek to engage users of the interviews, focus groups, and surveys (see annex for evaluation from the outset in defining the scope and sample questions). The evaluators looked for observ- questions, and agreeing on information and substanti- able evidence that participants had applied concepts ation sources. In addition, the evaluation process itself and tools introduced by the interventions. engages users in generating or verifying data. Such Each outcome was precisely described so it is clear engagement may help users have more confidence in and verifiable who changed in what way, when and evaluation findings. where, and how the intervention contributed (see figure 1 on the next page for examples of outcome descriptions). Outcome harvesting includes a specifi- DECISION TO USE OUTCOME MAPPING cation for optional contribution descriptions for each The World Bank GPF and SDC commissioned a joint outcome; these were vital for understanding how the evaluation using the outcome mapping approach. interventions had contributed to outcomes, directly or The team decided outcome mapping was well suited indirectly. because the objectives of the interventions being The outcome definition used2 set a high bar for evaluated involved areas—capacity development, net- assessing the effectiveness and sustainability of Bank work building, and accountability—that meshed with and SDC contributions to capacity and coalition-build- the methodology. ing changes. Informants were given the opportunity to The effectiveness of network building and network- verify outcome descriptions. ing is demonstrated by behavior changes that can The evaluators entered the outcomes in a database be found through collected outcome information. and coded them by type of change to provide for a For example, outcome information on the voluntary program-level reflection of the outcomes to date. The association of organizations or individuals and their resulting findings were then interpreted to address the changes represent new ways of working collectively, evaluation questions as follows: such as cooperating to plan work, sharing knowledge, • Effectiveness was assessed against the pre-defined strategizing, and securing resources. The realization of objectives and theory of change to identify whether social accountability is also demonstrated by behavior intended outcomes were met, and any unintended changes, in this case of government, business, citizens, outcomes outside the intended scope. and CSOs. • Sustainability was assessed by whether institutional The effectiveness of the Bank and SDC interven- changes had been realized. tions depended on the extent to which they were able • Relevance was assessed by whether the outcomes to influence others. The outcome mapping concept aligned to local needs and program priorities. 3 In addition, the evaluators highlighted key out- comes in outcome stories to communicate specific Figure 1. Sample Outcome Descriptions achievements or lessons in a narrative format (see SDC—Local NGO Capacity Building annex for a sample story). They produced nine out- Outcome 11: Between 2012 and 2013, after the come stories, each with sufficient detail to allow project training, six organizations developed a written independent sources to substantiate the outcome, charter defining, for example, the organization’s the contribution of the intervention, and the claimed purpose; beneficiaries; activities; role of the board; significance of the outcome.3 elections; how meetings are run. Contribution: The lead implementor of the SDC project appointed the team of trainers, contributed to CHALLENGES the development of the modules and organized and The evaluators faced several challenges during the facilitated the trainings. harvesting process, which included: Limited documentation of outcomes—Lack of World Bank—Social Accountability existing monitoring data caused a high dependence Outcome 86: In March 2013, the Democracy on collecting data during the evaluation. For two Education Center (DEMO) was able to expand its interventions, there was limited knowledge of Check My Service program when the Asia Foundation outcomes from serving staff of the Bank, SDC, and awarded it a grant for the Check My School and Check My Clinic projects. their contracting implementing partners due to high staff turnover and lack of a strong monitoring Contribution: DEMO’s leading role in implementing framework for outcomes. the Social Accountability Learning-in-Action (SAcLAP) program enhanced its reputation and expertise in the Lack of experience in methodology—Local con- area so it could develop a convincing proposal for the sultants were not well versed in outcome mapping, Asia Foundation. but this was looked at as an opportunity to build up their capacity in outcome mapping and evaluations for World Bank—Public Procurement future work. The lead evaluator coached co-evaluators Outcome 11: In April 2013, the Public-Private on the necessary concepts and provided guiding ques- Partnerships Board and review committee adopted tions for interviews. The co-evaluators were willing and its first five-year strategic plan with an initial focus on able to learn and also brought complementary experi- shaping new procurement law implementation. ence and skills to the team. Contribution: The World Bank Institute and the Perceptions of methodology—Initially there was Affiliated Network for Social Accountability in East some hesitation from Bank and SDC staff about using Asia and the Pacific (ANSA-EAP) designed and outcome mapping given that it can be a time-con- facilitated the strategic planning working with the suming process. The evaluators overcame this notion board, and WBI provided feedback on drafts of by using key concepts of outcome mapping rather the plan. than all of the steps. The Bank also voiced concern about having clear evaluation criteria. The evaluators addressed this by using outcome mapping to answer FINDINGS questions solely on effectiveness, sustainability, and The 190 outcomes collected demonstrate that each relevance. Efficiency could be assessed with a more intervention met or exceeded its pre-defined objec- appropriate evaluation tool. tives (see table 1). This represents impressive results Shortened timeframe—The short timeframe for short-term interventions, two of which were ongo- limited travel time to engage with actors influenced by ing, leading to the conclusion that the interventions the interventions. This restricted the ability to engage have been effective and relevant. with those most knowledgeable when describing and substantiating outcomes. However, the local consul- SDC—Local NGO Capacity Building tant was able to stay and conduct these interviews at 69 outcomes in total later and more suitable times. Against the central objective of building the capac- ity of CSOs, outcomes show improved organizational 4 performance of CSOs even over a limited time. The Judged strictly against the mining (or extractive indus- greatest effect came when, beyond improving internal try) value chain, however, the intervention has not been organizational abilities, CSOs were empowered to be fully effective since it has almost without exception only active in their community and involved in activities that strengthened CSO capacity at the end of the value supported such endeavors, including local govern- chain farthest from the extractive industry. ment monitoring, providing data for community needs, A further benefit of assessing the program two years or acting as service provider. after it concluded is that it has been possible to inter- pret the extent to which the results are sustainable. World Bank—Social Accountability After the intervention, 71 of the outcomes materialized, 93 outcomes from all 13 pilot project grantees and many of which demonstrate organizational ownership of some from those who only participated in one training concepts and tools introduced. workshop World Bank—Public Procurement Against the objective to strengthen monitoring capac- 28 outcomes total ity of CSOs on the mining value chain and related economy outcomes, the intervention was successful at Against the objectives of support to the Ministry of introducing or significantly enhancing social account- Finance and CSOs, the intervention was successful at ability knowledge and skills in several organizations. supporting the ministry and the development of a self- Table 1. At a Glance: 190 Outcomes Collected from the Three Interventions SDC—Local NGO Capacity Buiding 69 Outcomes in total, each relevant to or exceed SDC’s pre-defined objectives 51 Outcomes demonstrate the application of knowledge from the intervention 7 Outcomes suggest a sustained influence of the intervention 5 Outcomes cite engagement between CSOs and their community World Bank—Social Accountability 93 Outcomes in total, each relevant to or exceed the Bank’s pre-defined objectives 71 Outcomes suggest a sustained influence of the intervention 59 Outcomes demonstrate the application of social accountability knowledge gained through the intervention 29 Outcomes show dissemination of social accountability 24 Outcomes cite constructive engagement, demonstrating a deepening awareness of social accountability 14 Outcomes show successful fundraising for implementation of social accountability activities after the intervention 11 Outcomes show demand for support in using social accountability concepts and tools 11 Outcomes show networking of practitioners 9 Outcomes directly relevant to the mining value chain 7 Outcomes describe working with the private sector 7 Outcomes demonstrate advocacy of social accountability 5 Outcomes involve engaging the media World Bank—Public Procurement 28 Outcomes in total, each relevant to or exceed the Bank’s pre-defined objectives 25 Outcomes suggest the potential sustainability of the Partnership for Public Procurement 14 Outcomes are at the aimag (provincial) level 12 Outcomes are at the national level 11 Outcomes demonstrate strengthened capacity of CSOs in procurement monitoring 8 Outcomes show influence on road maintenance, specifications and planning 7 Outcomes show support for self-governing CSO networks 5 Outcomes show support for Ministry of Finance in its development of implementing rules and guidelines on CSO participation and oversight 2 Outcomes at the Ulaanbaatar level 5 governing CSO network. But it had not yet succeeded outcome is the impact that empowered newly trained in developing monitoring tools for the network, and CSOs can have in their community. Six outcomes prove the extent of capacity strengthening of CSOs has this occurred because of the intervention. A local been limited to the few participants in two pilots. The trainer, motivated by what she learned, took it upon intervention was still young and continued for months herself to find resources and train a further 22 organi- more. While some objectives remain unachieved, zations. This unintended outcome that normally would the outcomes indicate progress in influencing rules not have been discovered adds to an appreciation of and guidelines and formal establishment of the the effectiveness and sustainability of the intervention. partnership. Engage stakeholders Outcome mapping encourages reflection and dia- BENEFITS logue—it is a truly participatory method in which Several benefits for the evaluators, Bank, and SDC counterparts are engaged, thereby giving them owner- came about from using outcome mapping as an ship over the process and ensuing results. It offers evaluative methodology in this particular case. Teams an opportunity to get at the collaborative theory of might want to consider these benefits when deciding change. on a methodology for assessing similarly complex In this particular case, the SDC and World Bank programs. worked together as a team to design the evaluation questions and in the process built a stronger relation- Examine multiple actors ship. Once data collection was completed, the evalu- Traditional evaluations tend to give credit to a single ators sent key informants their individual findings and contributor, when in complex development programs asked them to confirm whether the information was multiple actors drive change. It is important to dis- correct. This step enabled informants to play a main cover how and which actors worked together or built role and express their views, and added credibility as on each other’s actions over time to create results so well. future programs can maximize their potential and use the “right” mix of actors. Inform next steps For example, in the findings from the Bank—Social Outcome mapping enabled the evaluators to identify Accountability pilot in community monitoring of a links between multiple outcomes to uncover latent/ family hospital’s services and conditions, outcomes emerging knowledge, innovative solutions, and how to showed how multiple social actors brought about scale up the program. All of this helps to detect pieces change: patients, doctors, and hospital management. for prospective program design or further phases. Based on the engagement of patients, the hospital For example, in the area of NGO capacity building, changed its policy so all patients can use toilets that the outcome data provides a rich source for identify- were previously “staff only”; the hospital appointed a ing organizations that could fulfill particular roles in guide nurse to help patients arriving for treatment; and a new phase of funding support. Based on this data, the chief doctor appointed two additional doctors. the recommendation is that the SDC and the Bank, with selected stakeholders with firsthand knowledge Learn from the complete picture of relevant institutions in Mongolia, may review the Harvesting outcomes allows the exploration of sig- outcome data and other sources and, according to the nificant outcomes—whether intended or unintended, focus of any new intervention, identify potential actors negative, or tacit/unrecorded—to get a complete pic- for particular roles. ture of what went right or wrong and how to learn from the change process to inform future thematic areas. Communicate results Taking a narrow approach that considers only those Outcome mapping allows evaluators to go deeper outcomes that had immediate or direct contributions into what, why, and how changes happened than using from the interventions could miss stories of change more traditional methods, which can often rely on hard connecting related outcomes. numbers and indicators. Harvesting detailed outcome For example, in the findings from the SDC—Local information led to the writing of nine outcome sto- NGO Capacity Building intervention, an unintended ries that describe changes in people—how the roles 6 of actors affect results—to capture the flavor of what occurred. SDC wanted to have evidence of results presented in a storytelling format, and the outcome stories ended up being shared the most. The outcome map- ping methodology proved useful for reconstructing the storyline of change for the three interventions. n ACKNOWLEDGMENTS The case study is a collaborative effort based on documentation from the evaluation and interviews and feedback with: • Evaluation team—Richard D. Smith (leader), Jeremy Gross and Amarbayasgalan Dorj • World Bank—Marcela Rozo and Kathrin Frauscher from the Open Contracting team, and Governance Partnership Facility team members • Swiss Agency for Development and Cooperation team members Jenny Gold and Sharon Fisher, World Bank, led the case study team. Dawn Roberts provided input. Published by The World Bank, May 2014 FOOTNOTES 1 See www.outcomemapping.ca 2 Thisoutcomes evaluation followed the definition of “outcome” used in the outcome mapping methodology: a change in the behavior, relationships, activities, or actions of the people, groups, and organizations with whom a program works directly. 3 It is up to the principal evaluation user to decide what substantiation is needed to make the results credible. In this case, outcome data was provided nearly exclusively by those the interventions were seeking to influence and not by the World Bank or SDC staff. Thus, the outcomes were assumed to be credible, which substantiation of the nine outcome stories confirmed. 7 ANNEX Example of an Outcome Story: World Bank—Public Procurement Intervention The evaluators highlighted key outcomes in outcome stories to communicate specific achievements or lessons in a narrative format. They produced nine outcome stories. OUTCOME STORY 6—The Ministry of Finance adopted Implementing Rules and Regulations for the monitoring of public procurement that reflected suggestions from Civil Society Organizations (CSOs). Outcome In late 2012, the Ministry of Finance adopted new Implementing Rules and Regulations (IRR) for CSO participation in bid evaluation committees that included several requests of the Partnership for Public Procurement (PPP), such as the use of specific reporting templates and creation of a web portal for CSOs to use when reporting on the performance of bid evaluation committees. Significance The IRR are significant for being a necessary implementation instrument for the 2011 amendment to the Public Procurement Law, which mandated the involvement of civil society in public procurement for the first time. This was the first time that CSOs working on procurement in Mongolia successfully collaborated to advocate for regu- latory changes as a partnership network. Contribution Financial and technical support for the creation and strategic development of the PPP has been provided by the World Bank since 2012 through the Governance Partnership Facility it administers. The PPP provided coordinated inputs during the policy dialogue with the Ministry of Finance over the summer and autumn of 2012. N. Otgonjargal, chair of the PPP, led and coordinated the contributions of PPP members to the drafting of the rules and regulations. The Bank supported the Governance Partnership Facility in developing a united message to engage constructively with the government. Bank support included coaching, network build- ing and technical advice on procurement monitoring. 8 ANNEX Interview Guide: World Bank—Social Accountability Intervention Informants Purpose of Interview • We are conducting an evaluation of the work the World Bank and SDC have supported since 2009 on civil society/NGO strengthening and CSO engagement in procurement. • Specifically, we are assessing the Social Accountability Learning-in-Action (SAcLAP) and PPP projects of the Bank and the NGO Effectiveness Project of SDC. • The main focus of the evaluation is to learn what did and didn’t work. We are not looking at how the money was used. Key Facts Name: Position now: Position during SAcLAP: Participated in SAcLAP training/pilot? Others from your organization who participated in SAcLAP ? How many workshops/events were there? What were they on? Who hosted, participated? SAcLAP 1. What did you/your organization gain from the SAcLAP project ideas/tools? 2. Were the tools/ideas appropriate and tailored for the Mongolian context? 3. Have there been any effects/benefits because of the work you did during the pilot project? Any changes in policies, practices, relationships, or activities of those you have been trying to influence? 4. If yes, who changed, what, when and where? How, exactly, did the SAcLAP ideas/tools contribute? Before SAcLAP 1. Before your involvement in SAcLAP, had you heard of social accountability? 2. If yes, had you done any social accountability work/used social accountability tools? 3. If yes, what was the project and who funded it? 4. If yes, from where/which organizations did you receive materials/tools/trainings to understand the subject? After SAcLAP 1. Have you used the ideas/tools from SAcLAP in any other work? 2. Have there been any effects/benefits because of the work you did during the pilot project? Any changes in policies, practices, relationships, or activities of those you have been trying to influence? 3. If yes, who changed, what, when and where? How, exactly, did the SAcLAP ideas/tools contribute? Who funded it? 9 ANNEX Survey Questions: SDC—Local NGO Capacity Building Intervention 1. A clear mission statements that reflects your organization’s purpose? Y/N 2. A written charter defining, for example, the organization’s purpose, beneficiaries, activities, role of the board, elections, how meetings are run? Y/N 3. An active board that meets regularly in accordance with the organization’s rules? Y/N 4. A mechanism to review and update your organization’s strategic plan and annual work plan regularly? Y/N 5. An annual workplan? Y/N 6. A human resource policy for staff development? Y/N 7. Clear roles and responsibilities for staff members and/or volunteers? Y/N 8. A staff member capable of submitting a financial report to the tax office? Y/N 9. Capacity to raise funds from members, donations or government funds? Y/N 10. The capacity to write a funding proposal? Y/N 11. Mechanisms for beneficiary, partner and stakeholder feedback? Y/N 12. Sustainable activities to achieve your mission and vision statement? Y/N 13. The ability to partner with other NGO’s to benefit from pooled resources? Y/N 14. What have been the specific benefits of each internal management change your organization has made following your participation in the SDC funded training on Capacity Building and Training for Local NGOs. 15. As a result of changes to your organization’s internal management, has your organization carried out new types of actions or activities? Please describe. 10