81259 IEG Working Paper 2013 / 1 Adding Value to Evaluations: Applying the Governmental Learning Spiral for Evaluation-Based Learning Bidjan Nashat, Sandra Speer, and Raoul Blindenbacher © 2013 Independent Evaluation Group, The World Bank Group 1818 H St., NW Washington, DC 20433 IEG: Improving World Bank Group Development Results Through Excellence in Independent Evaluation The Independent Evaluation Group is an independent unit within the World Bank Group; it reports directly to the Bank’s Board of Executive Directors. IEG assesses what works, and what does not; how a borrower plans to run and maintain a project; and the lasting contribution of the Bank to a country’s overall development. The goals of evaluation are to learn from experience, to provide an objective basis for assessing the results of the Bank’s work, and to provide accountability in the achievement of its objectives. It also improves Bank work by identifying and disseminating the lessons learned from experience and by framing recommendations drawn from evaluation findings. IEG Working Papers are an informal series to disseminate the findings of work in progress to encourage the exchange of ideas about development effectiveness through evaluation . The findings, interpretations, and conclusions expressed here are those of the author(s) and do not necessarily reflect the views of the Board of Executive Directors of the World Bank or the governments they represent, or IEG management. The World Bank cannot guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply on the part of the World Bank any judgment of the legal status of any territory or the endorsement or acceptance of such boundaries. Bidjan Nashat is Director of Programs and Advocacy at Save the Children Germany. He can be reached at bnashat@gmail.com. Sandra Speer works internationally as an independent senior evaluation expert. She can be reached at Sandra_speer@t-online.de. Raoul Blindenbacher is a Senior Fellow at the Graduate Institute of International and Development Studies in Geneva. He can be reached at rblindenbacher@yahoo.de. ISBN-10: 1-60244-234-4 ISBN-13: 978-1-60244-234-7 Contact: IEG Communication, Learning, and Strategy (IEGCS) e-mail: ieg@worldbank.org Telephone: 202-458-4497 Facsimile: 202-522-3125 http:/www.worldbank.org/ieg Contents 1. Introduction ........................................................................................................................... 1 2. Learning Theories for the Governmental Level .................................................................... 2 3. The Concept of the Governmental Learning Spiral .............................................................. 4 4. Four Case Studies ................................................................................................................. 7 5. Lessons ................................................................................................................................ 13 6. Outlook ............................................................................................................................... 16 Tables Table 1: Questions for Applying the Governmental Learning Spiral ....................................... 7 Table 2: Key Priority Actions Selected by Country Participants of Workshop on Gender Equality in Southern Africa .................................................................................................... 15 Figures Figure 1: The Governmental Learning Spiral ........................................................................... 6 Abbreviations AfDB African Development Bank IEG Independent Evaluation Group IFAD International Fund for Agricultural Development IFC International Finance Corporation NORAD Norwegian Agency for Development Cooperation SDC Swiss Agency for Development and Cooperation iv Acknowledgments The authors would like to thank the Swiss Agency for Development and Cooperation and the Norwegian Agency for Development Cooperation for their support in realizing evaluation- based learning processes, and Hans-Martin Boehmer, Gita Gopal, Konstantin Atanesyan, Bahar Salimova, Nalini Kumar, Svetlana Markova, and Ali Khadr, and many other Independent Evaluation Group colleagues for their support, guidance, and their helpful comments and suggestions. Heather Dittbrenner and Joanna Mikulski provided editorial support. Finally, the authors would like to thank the approximately 400 learning actors from 37 mostly African countries whose contributions made all the efforts worthwhile. Raoul Blindenbacher and Bidjan Nashat are authors of The Black Box of Governmental Learning: The Learning Spiral—A Concept to Organize Learning in Governments, referred to in this paper. This book was published while they both were working with the Independent Evaluation Group. v Summary Since 2008, the Independent Evaluation Group (IEG) has carried out a number of evaluation- based learning processes that have brought together over 400 participants from World Bank operations, other international organizations, client governments, civil society and academics, and the private sector. This paper discusses different learning theories for the governmental level and summarizes and analyzes the IEG experience in applying the Governmental Learning Spiral as a concept for organizing evaluation-based learning. The Governmental Learning Spiral has three main stages. Before the Learning Process, the governance challenge is analyzed to identify the most relevant perspectives surrounding the problem and to frame the existing evaluative knowledge and experience around it and trust is established between the learning actors. During the Learning Process the learning actors review and adapt the new knowledge according to their personal, organizational and political needs and translate the evaluative knowledge into their contextual environment. After the Learning Process, the follow-up to the learning activity and its results takes place and the newly reframed knowledge around a governance challenge is made accessible to everybody involved in the learning activity as well as to a wider audience for further feedback. At this point, a new spin of the Governmental Learning Spiral begins. This paper summarizes four evaluation-based learning processes that have applied the concept. One focused on combining several evaluation studies to a thematic learning approach on public sector reform. It took place in East Africa in 2008. Another one combined a two-step process to bring together different Bank country teams and stakeholders and evaluate World Bank engagement at the state level in Washington, DC, and West Africa in 2009. A third learning process brought together World Bank clients with other multilateral lending institutions and stakeholders from the private and public sector around agricultural productivity in Southern Africa in 2010 and a fourth learning process built networks around gender equality using video-conferencing and an in-person workshop in Southern Africa in 2012. The analysis concludes with several lessons from this five-year experience. First, triangulation of content and stakeholders as an organizing principle requires a lot more conceptual preparation than traditional workshops because the issue or governance challenge has to drive the selection of the participants and the design of the learning process. Second, facilitating evaluation-based governmental learning requires close cooperation with partners throughout the process. Its successes have rested on the support of donors and the cooperation of IEG’s counterparts in the World Bank Group in its headquarters in Washington, DC, and all over the world in the country offices. And third, evaluation-based learning is as much about content as it is about communicating a culture of accountability and learning. Communicating independent evaluation as a norm in the organization can increase its legitimacy in the eyes of its stakeholders and shift the focus from a narrow view to a forward-looking learning organization. Evaluation-based learning has the potential to trigger single-, double-, and triple-loop learning. vii 1. Introduction 1.1 Governmental learning has a multidisciplinary research tradition and a plethora of literature exists on organizational as well as policy learning (Bennett and Howlett 1992; Etheridge 1981; Levy 1994). Many contributions are more conceptual and descriptive but theory based approaches and empirical case studies are mostly lacking. Of course, donor agencies are also aware that transplanting one reform, policy or program from one country to another is not always successful. Different concepts for structured learning from evaluation results on the governmental level exist (Speer 2011a, 2011b). It is common to all that they depend on a careful selection of participants and that the political, cultural, and institutional environment is key to the ultimate success of many governmental learning activities. Policy learning can be fostered by various types of organized activities, which range from peer review frameworks often focused on accountability to international learning processes based on concepts like the Governmental Learning Spiral (Blindenbacher and Nashat 2010). The first are based more on a rationalistic tradition, the latter on insights from individual, organizational and social learning theories. This paper will focus on these learning processes, including examples from the World Bank. 1.2 Various layers of evaluation systems as well as streams of evaluative information exist within the World Bank and its Independent Evaluation Group (IEG), one of the largest evaluation units among international organizations.1 It provides a platform for individual, organizational, and governmental learning. IEG carries out evaluation for accountability purposes but also tries to structure learning from evaluation within the World Bank and together with clients. The purpose of the former is in principle backwards looking, whereas the latter is used to improve future practices and is therefore considered forward looking. ‘Streams of evaluative knowledge’ are available through regular synthesis reports focusing on sectors or cross-cutting topics providing “channeled evaluative knowledge” as well as sometimes “channeled mixed kind of information” (Stame 2006). At the partner country level, evaluation dissemination workshops including various stakeholder groups have been organized regularly over the years. The role of the World Bank in improving governance and promoting complex reforms in recipient countries is multifaceted. However, its engagement as a multilateral organization in fostering structured diffusion and learning from evaluation across recipient countries and stakeholder groups is newer. 1.3 This paper will discuss and analyze four examples of evaluation-based governmental learning organized in the framework of the World Bank. First, this contribution will reflect on different streams of learning theories for the governmental level, as they represent assumptions and motivations for organized learning in governments. The Governmental Learning Spiral, an eight-stage approach to learning from evaluation, will be presented, 1 IEG is charged with evaluating the activities of the World Bank, the work of the International Finance Corporation in private sector development, and the Multilateral Investment Guarantee Agency’s guarantee projects and services. The Director-General of IEG reports directly to the World Bank Group's Board of Directors. The goals of evaluation are to provide an objective assessment of the results of the Bank Group’s work and to identify and disseminate lessons learned from experience. See http://ieg.worldbankgroup.org/content/ieg/en/home/about.html. 1 including in the case studies. This article will conclude by reflecting on the concept of the Governmental Learning Spiral and its relation to different levels of learning. 2. Learning Theories for the Governmental Level 2.1 The primary uses of evaluation findings are threefold: judging merit or worth, improving programs, and generating knowledge (Patton 2002; Weiss 1998). Governmental learning is about all three of them. Policy decisions across countries are informed by single evaluations, but more importantly, from the synthesized, cumulative knowledge from many evaluations. The cumulative knowledge is especially relevant for policy makers, who are interested in both the experiences from earlier replications in different contexts and in clear recommendations. 2.2 “The fundamental problem with social learning […] is that national policy makers often have difficulty assessing the consequences of the various policies. Policy makers are ‘cognitive misers’ … as much as anyone else. As bounded rational actors, they rely upon a set of cognitive heuristics to make sense of these sometimes complicated policy choices” (Elkins and Simmons 2005). Research has shown that policy actors prefer learning from policy models similar to their own context, culture, and geographic region. Imitation is one of the simplest cognitive heuristics. Meseguer (2005) sees emulation rather than learning playing a role for policy adoption, the first dominating the latter. However, institutions only work well when there is a firm understanding and commitment to them. The implementation of policies and programs allows for experimentation and to discover policies or programs close to what might be perceived as ideal. Experimentation involves risk taking. Imitation and borrowing from other countries avoids those risks, but at the same time, these imitated policies could prove inappropriate to national circumstances. 2.3 Governmental learning is about the content of policies, institution building, laws and programs, but may also comprise aspects of how to make and implement them or determine the suitable timing (Duina and Nedergaard 2010). It uses different types of evaluative knowledge, including “knowledge about organizational prerequisites, conditions, and procedures like ‘planning and control’ devices and management information systems” and “substantive or explanatory knowledge about mechanisms within society that make policies work (or not) and that are assumed to be of relevance to realize effective and efficient (public) policies” (Leeuw 2006). 2.4 The knowledge about “what works” is the core of evaluation reports, but how to adapt and implement is often less documented. Exchanges with officials from other jurisdictions or practitioners can help assess the applied relationship between interventions and outcomes, particularly in regard to the knowledge for fundamental policy decisions may. Governmental learning is therefore about policy and program choices but can also focus on the implementation process, combining tacit and explicit knowledge. Thus, in the context of learning from the experience of other countries, governmental learning is not limited to questions of effectiveness and implementation, but may also include political consequences 2 of reform projects, such as electoral consequences and other aspects of the governance process. 2.5 Knowledge from evaluation can be transmitted as evidence or as advice, but can also be further developed through interactive debates and communications. Efforts to transmit knowledge as evidence or advice are rooted in the scientific and evidence-based tradition of rationalistic learning, by which explicit and transferable knowledge is disseminated. These knowledge transfer efforts may include national and international policy discourses within formal or informal meetings. The evidence-based movement contributes mainly to predefined policy goals and problem definitions and focuses on scientific studies, with a preference for impact evaluations. The search for “what works” is based on a rationalistic ideal, which can also have its limits. The use of scientific evidence for policy making is often weak. One remedy might be the existence of knowledge brokers and personal contacts between researcher and politicians/public servants (Nutley et al. 2007). 2.6 As demonstrated by Sabatier and Jenkins-Smith (1999), policy changes may occur during a so-called “window of opportunity.” But policy changes might also be due to ‘advocacy coalition frameworks’, where policy change is coordinated among a range of individuals with the same core policy beliefs (Sabatier and Weibe 2007). This can lead to prioritizing ideology over scientific of practitioner’s knowledge. Learning is in this case intertwined with political negotiations and decision making. The political climate surrounding the relevant issue, as well as the way a problem is defined, influence the policy solution applied to the problem. Governmental learning might help policy makers and practitioners prepare for such a situation but is less likely to lead to immediate changes. Evaluation-based learning from evaluation might require changes to existing mental models and the rethink of assumptions. Rational learning will be influenced by the consistency of the evaluations results and by the strength of prior beliefs. Evaluation results showing that a policy is effective may not be enough to convince politicians, who have strong, contradictory ideological beliefs. 2.7 Learning from abroad can be fostered through various methods. Mechanisms such as the European Union Open Method of Coordination or the Organisation for Economic Co- operation and Development Peer Review System rely heavily on documents and previous self-evaluations, whereas the concept of the Governmental Learning Spiral uses such documentation as a starting point for a structured process on the interpersonal level (Speer 2011b). There are four distinct theories of governmental learning: (1) learning based on constructivism, which emphasizes the role of expert networks and interest groups in generating information; (2) coercion as often exercised by international and multilateral organizations; (3) competition as a trigger and pressure leading to natural selection; and, (4) the facilitation by international organizations and the use of technology is a way of fostering learning. This kind of governmental learning process is rather unique. Aside from the Governmental Learning Spiral described later in this paper, the European Training Foundation is one of the rare other examples active in this endeavor of social learning on the macro-level, using various methods for discussing deeper meanings of policies and coconstructing them between peers (Nikolovska and Vos 2008; Sultana 2008). 3 3. The Concept of the Governmental Learning Spiral 3.1 The Governmental Learning Spiral is rooted in ideas from constructivism and those gathered through facilitation. It takes into account that policy formulation is usually based on multiple perspectives and a wider range of evidence. The concept does not apply any hierarchy to evaluation designs. It also uses evaluative information in the form of several types of synthesis reports and underlying evaluation approaches. The idea of a knowledge or learning broker that designs and facilitates the learning process is a central part of the concept. The concept has been further developed over the last decade without being explicitly based on evaluative knowledge. Applying the concept with evaluations offers a great opportunity to facilitate an evidence-based learning process that combines both the rationalistic tradition with a more constructivist approach based on tacit knowledge and social learning. 3.2 The Governmental Learning Spiral was conceptualized and organized as interactive learning processes (for example, as workshops, conferences, e-learning events). It has been practiced by IEG since 2008. The concept has been derived from theory and practice alike. Its basic fundamentals were developed by following a heuristic procedure. The development of the concept was the result of a multiyear process, during which experiences in governmental learning processes and evaluation workshops were systematically reviewed and subsequently further developed by related analytical and theoretical approaches (Blindenbacher and Nashat 2010). To date, a broad range of contemporary theories of political history, policy analysis, sociology, and pedagogy are compiled into the concept template (Blindenbacher and Nashat 2010). 3.3 Among the many incorporated theories, the pedagogical approaches have the most important impact on shaping the different spiral stages, mainly because of their significant role in leveraging the learning actors’ willingness and ability to play their part in adopting evaluation-based knowledge in the course of the learning process. In fact, the following four major pedagogical learning orientations were considered (Tennant 1997). They are the humanistic approach, the cognitive approach, the social approach, and the behavioral approach. 3.4 According to the humanistic approach, the learning actor should experience a safe and comfortable learning environment that provides him or her with the opportunity to experience new knowledge, which may awaken his or her curiosity and interest in entering into the learning process. According to the cognitive approach, the learner’s motivation should be enhanced by setting up a deliberate didactic process in which he or she becomes rationally aware of the difference between the known and the unknown knowledge, as well as the importance of overcoming this knowledge gap. Following the social approach, learning should take place in a social context where the learning actor is exposed to different peer perspectives. This way he or she gets an opportunity to choose from a variety of different explicit or tacit knowledge and to think about how it may best fit into his or her individual social reality. And finally, according to the behavioral approach, if learning actors are exposed to positive incentives and rewards, their motivation to engage in a 4 learning process increases and supports the intended learning outcomes – a change of thinking and an intended change of behavior (Blindenbacher and Nashat 2010). 3.5 Complemented with the remaining theoretical considerations and practical reflections, each of these pedagogical aspects was incorporated into three main sequences: before, during, and after a learning process, which were split into a eight-stage template (see Figure 1): the conceptualization, triangulation, accommodation, internalization, externalization, reconceptualization, transformation, and configuration stages. As verifiable in the description of the sequences and stages below, the humanist approach shaped the conceptualization and transformation stages to a high degree. The cognitive approach shaped the internalization stage. The social approach governed the externalization, the reconceptualization and the transformation stages. The behavioral approach is a tool that should be considered in each of the stages, with the exception of the triangulation and accommodation stages. 3.6 Before the Learning Process: The conceptualization, triangulation, and accommodation stages are the stages that have to take place before the start of a learning process. Conceptualization and triangulation require the learning broker to step back and analyze the problem before the planning process begins. The key is to identify the most relevant perspectives surrounding the problem and to frame the existing evaluative knowledge and experience around it in a straightforward way with the kind of stakeholders that increase the likelihood for finding and implementing possible solutions (content and stakeholder triangulation). Through the accommodation stage, a broker tries to establish a sense of trust between the learning actors by communicating the selected evaluative knowledge together ground rules and the goals early on in the process. 3.7 During the Learning Process: The internalization, externalization, reconceptualization, and transformation stages represent the core of the didactical procedures, where the learning actors review and adapt the new knowledge according to their personal, organizational and political needs. The learning actors reflect and eventually modify their thinking and behavior in a theory-guided inter- and intrapersonal process. It can be described as a translation process from evaluative knowledge around a specific challenge to contextualized knowledge for each stakeholder. 3.8 After the Learning Process: The configuration stage is organized within a follow- up of the learning activity, in which a wider audience might be included. The newly reframed knowledge around a governance challenge is made available and accessible to everybody involved in the learning activity as well as to a wider audience for further feedback. This new knowledge can serve as the basis for the next spin of the Governmental Learning Spiral, as well as a feedback loop in the context of a new learning system. 5 Figure 1: The Governmental Learning Spiral Source: Blindenbacher and Nashat (2010). 3.9 Governmental learning is defined by voluntary participation and noncompetitive environment and approaches for individual as well as group reflection. Here it is assumed that behavior change cannot be triggered by simple exposure to information. Instead, the stakeholders have to be enabled to adopt and translate evaluative findings into their specific context. The concept emphasizes learning around a specific governance challenge using evaluative findings in order to strengthen the peers’ capacity to transfer evaluative knowledge and to implement policies. It rests on the assumption that we need to organize our learning processes around the governance challenges we are facing, not along existing organizational and political power structures. 3.10 The nature of the learning process is applicable to different types of settings. The range can be as broad as from working groups, international conferences, multiyear international roundtables, and study tours to virtual platforms including e-learning (Blindenbacher and Nashat 2010). These different formats vary in the intensity of linking individual learning to group learning and linking broader learning with action in the single countries. These forms for interaction and the scope of their content are usually determined by several single stakeholders, and respectively, stakeholder groups. Of course, the different forms of application will not be able to produce similar effects. The following table (Table 1) develops questions that should guide the planning process for applying the Governmental Learning Spiral. 6 Table 1: Questions for Applying the Governmental Learning Spiral  Has the governance challenge been defined in a systematic and holistic way?  Is the related evaluative knowledge available and accessible?  What are the relevant knowledge perspectives and levels of analysis on the governance challenge and who are the primary stakeholders to represent these?  What measures can be taken to ensure that there is a safe learning space and a trustful atmosphere between the learning actors?  Does the learning process design encourage self-reflection in response to the presentation of the governance challenge and evaluative knowledge?  Does the learning process design encourage group reflection triangulating different content and stakeholder perspectives?  Does the event design allow to reconceptualize the original evaluative knowledge?  Does the event design allow to translate the new reconceptualized knowledge and experience into the individual context of the learning actors to solve the given governance challenge?  How can the new knowledge best be transformed into concrete actions and follow up activities?  How can the new knowledge best be configured for the next loop of the Governmental Learning Spiral and how can the learning actors be encouraged to build a network to continue the exchange on the governance challenge over time? 4. Four Case Studies 4.1 From December 2008 to October 2010, IEG carried out four learning processes around evaluative knowledge and IEG reports using the Governmental Learning Spiral as an organizing concept. The Swiss Agency for Development and Cooperation (SDC) and the Norwegian Agency for Development Cooperation (Norad) provided support. The reviews usually draw on a wide range of project evaluations within a sector or a cross-cutting topic across many countries. These IEG reports synthesize previous evaluations, but also partly evaluations themselves after the project completion, and focus mostly on questions of effectiveness. They thus also include multiple perspectives from project beneficiaries, non- beneficiaries, implementing agencies, representatives from relevant ministries. 4.2 These IEG reports used the basis for the four cases were already intended for a target group including government officials and other stakeholders, who want to learn from World Bank project and program experiences. The translation process from evaluative findings to communication for a wider audience was thereby already under way. This evaluative knowledge drives the process and structures the learning process. The following paragraphs highlight the experience with and some lessons from planning, carrying out, and following up on the learning processes. 4.3 Case One: From several evaluation studies to a thematic learning approach on public sector reform. The workshop “Lessons of a Decade of Public Sector Reform,” held in Addis Ababa, Ethiopia, on December 9–10, 2008, was jointly organized by IEG, the 7 Africa Region Public Sector Reform and Capacity Building Unit of the World Bank, the Public Sector Governance Unit of the Bank, and the World Bank Institute. 4.4 The aim of the workshop was to bring IEG’s recent evaluative findings together with all the available knowledge and experience on how to improve public sector effectiveness and efficiency, a priority item on the reform agenda of most African countries. Therefore, the learning broker designed a thematic approach around four IEG evaluations that had dealt with public sector reform and developed a concept note together with the evaluator that had been leading some of these studies. 4.5 This idea of triangulating the available content around a specific challenge was followed by a stakeholder triangulation: 60 individuals from 8 African countries were invited to represent both French- and English-speaking countries on all levels of government, as well as several development partners and regional organization representatives. At the workshop, the participants—through a facilitated reflection and exchange process in breakout groups— translated the evaluative findings into relevant lessons for their own context (Internalization and Externalization Stage) and reported them back to the plenary (Reconceptualization Stage). In the transformation stage, the participants developed an action plan called with lessons and recommendations for all participants. 4.6 For the purpose of configuring the newly reframed knowledge and its contextual application, the workshop results, including the different action plans, were summarized into Lessons of a Decade of Public Sector Reform: Voices of African Client Stakeholders (Configuration Stage). Furthermore, the findings were publicly disseminated at a press conference and through numerous interviews, with the participation of prominent workshop participants. 4.7 The workshop received positive survey feedback from the participants. This feedback, along with a 12-month follow-up with written reflections from a high-level Ethiopian government representative, indicated a sustained effect of the exercise. Several immediate follow-ups took place between representatives from different countries, such as a video-conference organized by the World Bank between Rwanda and Madagascar in December 2008 on a capacity-building program. European Union representatives agreed to consider reinvigorating and supporting a former Civil Service Training Institute in Sierra Leone, an undertaking that started to materialize in August 2009 and was renewed in 2011. Although it is difficult to make any direct attributions, this learning process bringing together evaluative knowledge and the necessary stakeholders for implementation around public sector reform was an interesting example of feeding back IEG lessons into development practice. Nevertheless, to sustain the engagement with the emerging network, more structured follow-up could have taken place on the basis of IEG’s evaluative findings 4.8 Case Two: A two-step process bringing together different Bank country teams and stakeholders on evaluating World Bank engagement at the state level. As a lesson of the experiences of with the public sector reform workshop, IEG developed a more sequenced approach to learning processes. In the fall of 2009, IEG designed and carried out a learning process based on the evaluation World Bank Engagement at the State Level: The Cases of Brazil, India, Nigeria, and Russia and its in-depth country case studies. On discussing this 8 report, the World Bank’s Board had suggested that the organization should consider this report as a basis for further thinking toward a more comprehensive framework to guide the Bank’s engagement at the state level and to consider ways to strengthen systematic knowledge sharing and learning from ongoing work at the state level. 4.9 IEG seized this mandate to enhance the study’s utilization and pass the ownership for the findings to key Bank stakeholders by sharing lessons and facilitating a debate on related topics. The first internal workshop “Working at the State Level in Large Federations – Sharing Knowledge and Experience among Country Teams” was held in 2009 in Washington and brought together key Bank staff working on subnational projects in Brazil, India, Nigeria, and Russia, as well as in other federal countries with similar assistance programs via video conferencing (Conceptualization and Triangulation Stage). At the half-day workshop, the learning broker facilitated a comparative knowledge sharing exchange around differences and similarities between the case studies (Internalization and Externalization Stage). The format also distinguished between the strategic aspects of how to select which state-level governments to work with according to windows for reform and the poverty level, and the more pragmatic aspects such as how the support actual public-private partnerships on a municipal level. 4.10 The participants were also encouraged to give suggestions for the follow-up workshops in the country offices. The follow-up workshop built on this first exchange by focusing on the case of state-level engagement in Nigeria, which was organized in close cooperation with the World Bank’s Nigeria country team. It provided a forum for state and national level policy makers and other stakeholders, including Bank staff, civil society, and academia and donor partners to exchange views on best practices in effective development assistance at the sub-national level (Stakeholder Triangulation Stage). 4.11 At the event, IEG presented the comparative evaluation findings from Brazil, India, Russia, and Nigeria. In response, a Nigerian scholar who had written the country background paper commented from the national perspective and the learning broker facilitated the feedback from the different levels of government, and civil society (Internalization Stage). The participants were invited to reflect on and then translate these findings onto their level, sector, and organization. A modified understanding of how World Bank state-level engagement should be adapted to Nigeria emerged from this debate (Externalization and Reconceptualization Stage). 4.12 The workshop highlighted the need to plan for timely input of evaluation reports for new country strategy building on comparative country case experiences. Another lesson communicated back to the headquarters was the high level of demand by participants for building effective monitoring and evaluation systems and conducting independent evaluation at the state level in Nigeria. Civil society participants stressed that they had been previously unaware of independent evaluation at the World Bank, a point that would strengthen the organization’s legitimacy in their eyes. The then World Bank country director for Nigeria noted the usefulness of learning from very good examples in Brazil, India, and Russia and the benefit of knowledge sharing in the Nigerian context for improving the understanding of the World Bank work to stakeholders in beneficiary countries. The positive feedback of the country director is interesting because the evaluation report itself had been received with 9 skepticism by the country office staff. However, during the learning process, the opinion regarding the report and its recommendations was better understood. 4.13 Case Three: From studies to streams of knowledge and building networks around gender equality. The third learning process developed the idea of streams and networks further and also built in a social media component. In early 2010, IEG developed a learning process around the progress made in achieving gender equality. The findings of IEG’s evaluation of World Bank support for gender and development drove the process. A workshop, “Gender Equality in Southern Africa: Achievements and Challenges,” was jointly organized by IEG and the World Bank’s Africa Region and supported by the SDC. The workshop gathered around 50 participants from 9 African countries including ministers, high-level civil servants, donors, and representatives from nonprofit organizations, academia, and World Bank country offices in the region. 4.14 The aim was to create a venue to launch a sustained dialogue on the basis of evaluative evidence to better understand the weak progress in economic empowerment and country-related constraints to implementing initiatives for gender equality. After the presentation of IEG’s main evaluation findings (Internalization Stage), the participants were invited to reflect and discuss the relevance of the evaluation findings for their country and organization contexts. They shared their concerns and issues related to gender gaps in the region and obstacles for bridging the gap. Most participants expressed views that resonated with the evaluation findings—that issues of women’s economic empowerment and strategic areas such as the reconciliation of differences between the customary laws and existing national legal frameworks are key steps to bridging gender gap in the region. The conversation focused on better understanding the evaluation findings on what has worked and what has not in economic empowerment of women as well as in learning from other countries and Bank experiences (Externalization Stage). 4.15 On the second day of the workshop, and based on the shared international, regional, national, subnational, and societal perspectives on the challenge of gender equality, the workshop participants developed an action plan that spelled out seven general observations, such as the need to go beyond human development and focus on economic empowerment of women and enhance the access to microcredit for women. They split into three groups to develop concrete results framework for addressing and measuring the progress on the priority issues identified by each group (Reconceptualization Stage). IEG facilitated this process by providing the example of its own evaluation framework to assess Bank support for gender and development. Based on these observations the learning actors developed prioritized and measurable intermediate and long-term outcomes for each of the eight countries (Transformation Stage). 4.16 After the event, the workshop summary was shared with all participants for comments and then published on IEG’s gender evaluation website as well as disseminated to broader audiences. In response to the engagement and the concrete action plans of the workshop, SDC decided to fund a follow-up activity that would build on the lessons from the workshop, create a gender learning network, and broaden the dialogue into a worldwide platform. Furthermore, in October 2010, four follow-up video conferences were held with 170 participants from 12 countries. Together with the in-country facilitators, IEG drafted 10 and circulated a 20-page summary document of the four sessions and posted it on the gender evaluation network. 4.17 In a second step, IEG also created an online knowledge network, Gender in Evaluation, to deepen the dialogue through a subsequent electronic discussion and to share resources and knowledge across the countries. In the two months after its launch, 82 members joined this network which grew to 300 members within a year (Configuration Stage). 4.18 Overall, the following outcomes were achieved as a result of the initiative: The World Bank’s Gender and Sustainable Development network seized on the online knowledge network to gather expert stakeholder input for a new report on the link between gender, environment, and poverty in Ethiopia and Ghana. An online follow-up survey in 2011 showed that participants still rated the extent to which this exchange had been relevant to their work and gave them new information as very high. Some respondents called for more grassroots participation in follow-up video conferences pointing to the fact that nongovernmental organizations can only represent but really speak for the most affected populations. In conclusion, the gender evaluation learning process followed all stages and sequences of the Governmental Learning Spiral and built on the idea of moving from studies and single events to streams of knowledge and creating new learning spin-offs, such as a voluntary online network in a sustainable way. 4.19 Case Four: Bringing together World Bank clients with other multilateral lending institutions and stakeholders from the private and public sector around agricultural productivity. The fourth learning process that IEG initiated focused on bringing together three multilateral lending institutions around the challenge of agricultural productivity. In the conceptualization and triangulation stages of this process, IEG seized the opportunity to combine the findings of its 2010 evaluation of the World Bank’s and the International Finance Corporation’s (IFC) agriculture and agribusiness portfolios with another joint evaluation by the independent evaluation units of the African Development Bank (AfDB) and the International Fund for Agricultural Development (IFAD). With SDC’s support and the collaboration of the World Bank country office the learning event brought together around 65 stakeholders from the World Bank, including its Executive Director for Africa, IFC, Southern African Development Community, IFAD, AfDB, Food and Agriculture Organization of the United Nations, government officials, nongovernmental organizations, academic representatives, and private sector clients from seven different Southern African countries in October 2010 in Gaborone, Botswana. 4.20 The workshop aimed to share experiences on progress made in evaluating initiatives for agricultural growth and productivity from both the public sector and private sector perspective. As an organizing principle, the steps of the agricultural value chains were used to structure breakout groups led by Southern African experts in five key areas: rain-fed agriculture, regional research and development, markets, livestock sector development, policy reform and leadership (Conceptualization and Triangulation Stages). This focus on the whole process of agricultural production from water management, research, and extension (an area of mostly public sector intervention) to roads and infrastructure, market access, and 11 processing (areas where private sector engagement increases) helped further to structure the debate between public and private sector support and its bottlenecks. 4.21 The presentations by IEG, IFAD, and the AfDB on their evaluative findings compared both the global and the regional similarities and differences to provide a context for discussion. They were complemented by a keynote speech by one of the most prominent agricultural experts in the region, who also commented on the evaluations from a regional perspective. Both the evaluative and the academic perspectives enabled the participants to reflect on how the global and regional trends relate to their national and sectoral perspectives on agricultural productivity individually and in their breakout groups (Internalization, Externalization and Reconceptualization Stages). 4.22 The breakout groups focused on the biggest challenges in the topic area for the near future, the most important areas in which donor support could increase agricultural productivity, and the most important new insights participants are taking away from the evaluation findings and the sessions (Transformation Stage). Based on the participant feedback on these three areas, IEG drafted and circulated workshop findings (Configuration Stage). 4.23 Although the learning event received good overall feedback, some participants suggested that they would have liked a greater focus on capacity building in monitoring and evaluation rather than presenting evaluative findings. This was also a theme from the other learning processes and events. The participants further suggested staying in touch and fostering a network of participants to keep the exchange of evaluative findings alive and to disseminate the workshop findings to relevant organizations’ decision makers electronically and, if possible, through follow-up workshops and events. The need for incorporating the findings and recommendations into donor strategies and processes and monitoring the implementation of the workshop findings was mentioned as well. One of the most important lessons was the need for more dialogue between donors, and the public and private sector in the region to overcome gaps in communication and policy. Given that the World Bank and IFC have a decades-long experience with facilitating public-private dialogue, there is ample opportunity to build on existing networks and practices in this area. 4.24 Organizing and carrying out these four learning processes using the concept of the Governmental Learning Spiral has been a time- and resource-intensive undertaking. Large- scale workshops that require close and long-lasting partnerships and collaboration between many different actors across continents always are. Organizing and facilitating or “brokering” these processes added transaction costs, especially when evaluations had not been received well. Only the strong commitment from everybody involved, but particularly from colleagues at the World Bank Group, allowed the creation and continuation of the online discussions and social media activities. They made for example processes like the one that went from studies to sustainable streams possible. 4.25 There is valid criticism in the development community about the phenomenon of the “workshop hopping” and a “per diem culture” by certain development actors. This makes it all the more important to carefully select participants who are willing to commit to actively participate a long-term learning process. The more participatory we organized the evaluation 12 processes with our IEG colleagues, the more rewarding were the learning processes for everybody involved. 5. Lessons 5.1 Over the past five years, IEG has applied the Governmental Learning Spiral in a number of cases. For the following areas lessons can be summarized: 5.2 Cooperation as a learning process: Facilitating evaluation-based governmental learning requires close cooperation with partners throughout the process. Its successes have rested on the support of donors such as SDC and Norad and the cooperation of IEG’s counterparts in the World Bank Group in its headquarters in Washington, DC, and all over the world in the country offices. IEG itself would not have been able to tap into the vast network of policy makers, civil society stakeholders, and private sector clients that participated in these learning processes for two reasons. First, few people enjoy being evaluated and having their successes and failures discussed in front of their clients and stakeholders. Second, it requires evaluators to translate their findings and sometimes technical language into the language of policy makers, civil society members, and private sector clients. This is a learning process in itself for all parties involved but one that has led to more understanding about development effectiveness in many cases. 5.3 Learning about accountability: One of the lessons from applying the Governmental Learning Spiral over the last several years has been the fact that evaluation-based learning is as much about content as it is about communicating a culture of accountability. Time and again, both World Bank and external participants have given the feedback that they had either been skeptical of IEG’s independence or not aware of it in the first place. However, in most cases and as evidenced by the positive evaluations, they valued the forward-looking exchanges. Communicating independent evaluation as a norm in the organization can increase its legitimacy in the eyes of its stakeholders and shift the focus from a narrow view to a forward-looking learning organization. The Governmental Learning Spiral’s approach has been helpful here because it conceptualizes evaluation as a starting point and only one perspective of a learning process. One of the lessons from applying the concept in the field is that the sometimes heated exchanges between IEG and World Bank management appear less relevant when actual stakeholders are at the table with a vested interest in learning from past success and failure in order to improve future policies. 5.4 Triangulation of content and stakeholders as an organizing principle: The analysis of the Governmental Learning Spiral points to the need to approach learning in governments from an organizational and an individual perspective as well as to take a governance perspective that reaches beyond single government agencies. The concept tries to move beyond learning in organizations to learning around issues and challenges with all stakeholders involved. Preparing learning processes requires a lot more conceptual preparation than traditional workshops do because the issue or governance challenge has to drive the selection of the participants and the design of the learning process The fields of application are so far rather diverse, but many range in the field of governance including 13 aspects of political systems, public administration, and social governance, one had a sectoral approach, one a cross-cutting topic and another was more specifically on the World Bank engagement below the federal level. 5.5 In the last decades, the focus of World Bank programs—as development aid in general—has shifted toward funding governance projects, which often complement other sector programs. What all these subjects have in common is that they are at the heart of the debate on aid effectiveness and can be characterized by a high level of complexity. Furthermore, they involve a variety of societal stakeholders, which requires the need for this kind of intensive learning approach and for accomplishing the structured interaction between explicit knowledge with tacit knowledge. The same holds true for the cooperation of the public with private organizations, which was an issue in the events on public-private partnerships and on agriculture and agribusiness. The topic of gender equality contributes to wider development objects, which is on the one side targeted by specific programs and projects but also cross-cutting through all projects. 5.6 Results and impact of learning from evaluation: The Governmental Learning Spiral aims for three areas of results, the creation and implementation of action plans based on evaluation-based learning, the creation of networks around evaluative lessons, and the feedback and updating of existing knowledge. These goals are difficult to measure and rarely take place in linear processes. 5.7 When trying to facilitate learning from evaluation as an independent evaluation group, IEG will always have to look for ways to hand over learning results and developed networks to the World Bank Group for further operational action and dialogue. This requires planning and building networks for evaluative partnerships and ownership early on in the process. Carrying out the two last stages of the Governmental Learning Spiral—the Transformation and Configuration Stages—illustrate this point. Learning from evaluation can be understood here as single-, double-, and triple-loop learning. The evaluation results might change current practices, policies, and norms within the context they have been undertaken. From the World Bank perspective, lessons from the workshops could be identified on all three levels. Participants made comments on the single-loop level, such as concerning World Bank procurement procedures and guidelines, but also on more integrating local contexts. Many of the concrete messages were on the double-loop learning level focusing on how programs and projects could be aligned. However, many observations from the workshops had dual implications for learning, for both the country governments and the World Bank, which is not surprising in the development aid sector. For example, on the one side participants stressed the need of the governments’ commitment to mainstream gender equality, on the other side they said, “The World Bank should work with governments to strengthen existing legal frameworks and regulatory environments to enhance women’s empowerment and gender equality.” Triple-loop learning might also be triggered within the World Bank, especially concerning the relationship to the beneficiary countries and further learning from that. 5.8 From the beneficiary perspective, also much double-loop learning concerning (re)focusing their activities could be observed. This was especially evident for the agriculture and agribusiness event, where for different parts of the value chain challenges have been 14 identified, as well as for the series of events on gender equality, where for each of the participating countries key priority actions were selected (see Table 2). The participants defined both long-term outcomes as well as intermediates outputs on gender equality for the respective countries. But also triple loop learning could be triggered. For “leadership and policy formulation,” a limited capacity to identify and prioritize issues and to come up with clear and consistent policies, a lack of ownership of policies and their internalization, inconsistency in policies as well as the special challenge of unwritten policies have been highlighted (see examples of IEG workshop findings). Table 2: Key Priority Actions Selected by Country Participants of Workshop on Gender Equality in Southern Africa Country Long-Term Outcome Intermediate Outputs Botswana Greater equality in political Establish quota for women to achieve greater representation (baseline: 4 of 52 political representation ministers are women) Eliminate discrepancies between traditional and Improved gender equal legal regime civil systems of law and ensure implementation of rights under formal legal systems Lesotho Increased economic empowerment of Raise awareness of women to exercise their rights married women provided by the Married Women’s Act Raise awareness of banks to support women enforce their rights under the Married Women’s Act Malawi Improved gender equal legal regime Marriage law and inheritance (already prepared) Improved gender balance in decision law approved making positions Quota system established for women in decision making positions Mauritius Currently, there is complete alignment of No suggestion domestic laws with conventions, that is, discrimination on grounds of sex is prohibited, equal rights to property, as well as marital regimes are equal Namibia Improved gender mainstreaming in all Transparent and gender aware allocation of sectors development budget through a gender budget Improved gender aware monitoring and initiative evaluation Improved sex-disaggregated statistics and establish a gender barometer to assess costs and resources for enforcement South Africa Improved gender equality legal regime Reduced inconsistency between customary Law and Decreased violence against women statutes Reduced gender bias in laws related to marriage and abduction, inheritance and succession Tanzania Increased access to credit for women Train commercial banks to provide finance to Improved gender-equal legal regime women Reduced inconsistencies between customary and formal laws, particularly related to inheritance Zambia Improved gender-equal legal regime Reduced inconsistencies between customary and formal laws, particularly related to inheritance Collaboration with nongovernmental organizations to address cultural constraints Raise women’s understanding of their rights Source: IEG (2012). 15 5.9 Most learning took place at the level of donor strategies and program designs, which is not surprising as the topics were around high-level sectoral or cross-cutting issue. But the workshops identified also demand for further evaluation and the need for evaluation capacity building. For example, for public sector reforms, “regular monitoring and evaluation must be undertaken along with dissemination of results” (IEG 2012) and the Bank support in it was sought. There was less emphasis on generating specific evaluation questions, although, for example, in the agriculture and agribusiness case, participants asked for greater attention to long-term impacts concerning livestock and particularly by giving attention to exit strategies and sustainability (IEG 2012). The learning events also showed knowledge gaps and might trigger more South-South learning and cooperation, for example, learning from Brazil for rain-fed agriculture. Evaluation gaps had also been identified, for example, more impact evaluations of microfinance for women or certain evaluation designs recommended, for example, participatory risk assessments in Kenya. 6. Outlook 6.1 The Governmental Learning Spiral is about “nudging” potential behavior change and delivering evaluative information, which cannot be taken from evaluation reports written for other purposes. In the four cases described and analyzed here, conceptualization, stakeholder triangulation, and accommodation were always a necessary condition for success. The Governmental Learning Spiral’s focus is not only on knowledge transfer, but also on an active interaction between stakeholders from various countries, which encourages multilevel learning and personal understanding. The learning brokers do not advocate for or against specific evaluation results or judge participants’ decisions. Instead, they try to facilitate a forward-looking conversation on the basis of evidence. 6.2 Integrating knowledge management into evaluation systems and using evaluation findings within organizations is challenging. But learning from evaluation on the country level and across countries makes a sustainable learning process even more complex. In the future, international organizations will need to explore how to better share evaluative knowledge with stakeholders outside of their organization and across countries. The Governmental Learning Spiral will help in designing and making learning processes more effective. 6.3 The World Bank has been a provider of knowledge for assistance, but has also provided a public good by delivering very different knowledge products as well as research. Evaluations have always been part of the knowledge diffusion strategy. Events for knowledge exchange, including the recent South-South learning events, have also been offered by the World Bank Institute. Guidelines have been strengthened to include elements of the Governmental Learning Spiral. 6.4 However, participatory and networked learning around IEG evaluation reports is newer and certainly a promising way for both the beneficiary countries and the World Bank. Until now, the Governmental Learning Spiral processes have been organized after the evaluation has been finished. Whether the learning from evaluation will also influence 16 evaluation designs and more participation within evaluation remains an open question. However, participatory events after evaluation cannot make up for a lack of participation during the evaluation because the intended users may differ. The case studies at hand indicate that IEG has drawn lessons from the implementation of learning processes and has begun to involve stakeholders in the process of selecting, conducting, and disseminating evaluation results with the intent of triggering learning processes across countries. 6.5 Further research should shed light on the question of how to make evaluation-based learning processes more stakeholder-driven since the concept and its application can ensure stakeholder ownership only to a certain extent. More effort will be needed to move away from single events to thematic and networked learning processes that are driven by the demand for content and skills of the stakeholders in countries. 6.6 The four examples discussed here focused on feedback to the World Bank and its stakeholders on the strategic orientations of policies and programs. They deal less with learning from the process, which can also be the focus for cross-country learning (Sultana 2008). The workshops provided feedback to the World Bank. Although this feedback may not have been new information in all cases, it plays an important role in informing headquarters about globally relevant policy and program orientations, in which many country offices are involved. The concept of the Governmental Learning Spiral is open to future variations in its implementation and could shift focuses, such as to learning from political processes. It also can be applied on the level of learning from project evaluations across countries. 6.7 Within the evaluation literature, we always face the dichotomy of evaluation for learning versus evaluation for accountability reasons. The Governmental Learning Spiral, as applied here to the context of evaluation, can help overcome this dichotomy by strengthening learning from evaluation independently from the evaluation goals and designs. We therefore strongly invite practitioners as well as members of the scientific community to join our effort to further develop and improve governmental learning. 17 References Bennett, C.J., and Howlett, M. 1992. “The Lessons of Learning: Reconciling Theories of Policy Learning and Policy Change.” Policy Sciences 25: 275-294. Blindenbacher, R., and B. Nashat. 2010. The Black Box of Governmental Learning, The Learning Spiral – A Concept to Organize Learning in Governments. Washington, DC: World Bank. Dobbin, F., B. Simmons, and G. Garrett. 2007. “The Global Diffusion of Public Policies: Social Construction, Coercion, Competition, or Learning? ” Annual Review of Sociology 33: 449-472. Duina, F., and P. Nedergaard. 2010. “Learning in International Governmental Organizations: The Case of Social Protection.” Global Social Policy 10: 193-217. Elkins, Z., and B. Simmons. 2005. “On Waves, Clusters, and Diffusion: A Conceptual Framework.” Annals of the American Academy of Political and Social Science 598: 33-51. Etheridge, L. 1981. “Government Learning.” In The Handbook of Political Behavior, vol. 2, ed. Samuel L. Long. New York: Plenum. Gilardi, F. 2010. “Who Learns from What in Policy Diffusion Processes?” American Journal of Political Science 54(3): 650–666. IEG (Independent Evaluation Group). 2009. “Lessons of a Decade of Public Sector Reform: Voices of African Client Stakeholders.” IEG Workshop Findings. Washington, DC: World Bank. IEG. 2010. “Evaluating Support for Growth and Productivity in Agriculture and Agribusiness in Southern Africa.” IEG Workshop Findings. Washington, DC: World Bank. IEG. 2012. “Gender Equality in Southern Africa: Achievements and Challenges.” IEG Workshop Findings. Washington, DC: World Bank. Lindstädt, René, and Nathan M. Jensen. 2012. “Leaning Right and Learning from the Left: Diffusion of Corporate Tax Policy across Borders.” Comparative Political Studies 45(3): 283-311. Kingdon, J.W. 1995. Agendas, Alternatives, and Public Policies. New York: Longman. Leeuw, F.L. 2006. “Managing Evaluations in the Netherlands and Types of Knowledge.” In From Studies to Streams, eds. Rist, R.C., and N. Stame. New Brunswick NJ: Transaction Publishers. Levy, J.S. 1994. “Learning and Foreign Policy: Sweeping a Conceptual Minefield ”. International Organization 48( 2): 279-312. Meseguer, C. 2005. “Policy Learning, Policy Diffusion, and the Making of a New Order.” The Annals of the American Academy of Political and Social Science 598: 67-82. Nikolovska, M. & Vos, A. 2008. “ETF Peer Learning: From Policy Learning to Policy Change in Partner Countries.” In ETF Yearbook 2008: Policy Learning in Practice, Turin: ETF. Nutley, S., I. Walter, and H. Davies. 2007. Using Evidence: How Research Can Inform Public Services. Bristol: Policy Press. Patton, Q. M. 2002. Qualitative Research and Evaluation Methods. Thousand Oaks, CA: Sage. 18 Sabatier, P.A., and H.C. Jenkins-Smith. 1993. Policy change and learning: An advocacy coalition approach. Boulder, CO: Westview Press. Sabatier, P.A., and C.M. Weibe. 2007. “The Advocacy Coalition Framework: Innovations and Clarifications.” In Theories of the Policy Process, ed. P.A. Sabatier. Boulder, CO: Westview Press. Speer, S. 2011a. “Peer Evaluation – The Powerful Peer?” In Evaluation: Seeking Truth or Power? eds. Eliadis, P., and J.-E. Furubo St. Jacob. New Brunswick, NJ: Transaction Publishers. Speer, S. 2011b. “Organised Governmental Learning: Vocational Education and Training Practices between Peer Review and Peer Learning.” Research in Comparative and International Education 6(3): 330-340. Stame, N. 2006. “Introduction: Streams of Evaluative Knowledge.” In: From Studies to Streams: Managing Evaluative Systems, eds. Rist, R.C., and N. Stame. New Brunswick, NJ: Transaction Publishers. Sultana, R.G. 2008. “The Promises and Pitfalls of Peer Learning.” In ETF Yearbook 2008: Policy Learning in Practice. Torino: ETF. Tennant, M. 1997. Psychology and Adult Learning. London: Routledge. Weiss, C. H. 1998. Evaluation: Methods for Studying Programs and Policies. Upper Saddle, NJ: Prentice Hall. 19 Appendix A. Overview on Four IEG Applications of the Governmental Learning Spiral Topic and Focus and Level Relevant Stakeholders Methods Actions, Outcome and Follow-Up Location of Analysis Evaluations2 Included Lessons of a Analyzing the IEG: Addis Ababa: 60 Evaluation Level-One Evaluation (rated from 1-5): Average Decade of Public instrument and Decentralization in policy makers and lesson-driven relevance rating: 4.56 (4.8 by government participants), Sector Reform in institutional level: Client Countries: An civil servants from content and average usefulness rating 4.45. (4.8 by government Africa (Addis State-citizen Evaluation of World Anglophone stakeholder participants). Ababa, Ethiopia, relationship Bank Support, (Ethiopia, Nigeria, triangulation, Action plan: Participants drafted action plan with December 2008) (service delivery), 1997–2007 (2007) Sierra Leone, Sudan, break-out groups deliverables called “Lessons of a Decade of Public Sector Policy reforms on Public Sector Tanzania), and along shared Reform: Voices of African Client Stakeholders” (IEG different vertical Reform: What Francophone African experience, re- 2009) and horizontal Works and Why? countries conceptualizing Follow up: Written reflections in 2009 from Ethiopian levels of the state (2008); How (Democratic of evaluation high-level government representative.3 Follow-up between (local Effective Are World Republic of Congo, lessons, action representatives from different countries with, as a first step, governance), Bank Fiduciary Madagascar, planning a videoconference organized by the Bank between Rwanda efficiency of Diagnostics? Rwanda), and Madagascar in December 2008 on Bank capacity- financial (2008); Capacity nongovernmental building program. European Union representatives agreed instruments btw. Building in Africa organizations, World to consider reinvigorating a former Civil Service Training donors and (2005) Bank staff, donor Institute in Sierra Leone, which took place in 2009.4 recipients, partners developing policy options World Bank Lessons-driven IEG: World Bank Washington: 25 Evaluation-based Level-One Evaluation (rated from 1-5): Relevance: 4.2; Engagement at evaluation Engagement at the Bank staff in lessons as timely Quality of discussion, 4.3, Overall usefulness: 4.4. the State Level in approach State Level: The Washington and 3 follow up to Follow up: Dissemination of findings from both Brazil, India, (comparative Cases of Brazil, country offices (via 2009 country Washington, DC and Abuja event in World Bank Group Nigeria, and lessons from India, Nigeria, and video) Abuja: 30-35 strategy intranet story. 2 All IEG evaluations are available for download at http://ieg.worldbankgroup.org/. 3 Reflections from the Director of Planning and Programming Directorate, Ministry of Capacity Building of the Government of Ethiopia, Ato Ahmed Mohammed Ali, Addis Ababa, Ethiopia, January 12, 2010, in: Blindenbacher, R. and B. Nashat (2010). 4 In August 2009, the European Commission, United Kingdom Department for International Development (DfID), United Nations Development Programme (UNDP) and the World Bank under the leadership of the Public Sector Reform Unit (PSRU) jointly designed a new Public Sector Reform Programme for Sierra Leone. The support was renewed in 2011, see http://eeas.europa.eu/delegations/sierra_leone/press_corner/all_news/news/2011/20110623_1_en.htm. 21 Russia working below the Russia (2009) participants in Abuja, discussions, (Washington, federal level), including Bank staff, stakeholder November 2009, Federal-local representatives of triangulation of Abuja, Nigeria, relationship, Nigerian state all state levels, December 2009) Strategic level governments, federal comparative (support for government, NGOs, lessons from reformist or academia, and donor international poorest states), partners. cases (and connecting two learning events) Gender and Analyzing state- IEG: Gender and Pretoria: 50 Evaluation Level-One Evaluation (percentages on expectations): Development in citizen and Development: An participants from 9 lesson-driven Relevance: 88 percent “exceeded,” 12 percent “fully met,” Africa and Asia citizen-to-citizen Evaluation of World Southern African content and Quality of discussion: 59 percent “exceeded,” 24 percent (Pretoria, South relationship Bank Support, countries consisting stakeholder “fully met,” Overall usefulness: 63 percent “exceeded,” 38 Africa, May (gender), 2002–08 (2010) of high-level triangulation, percent “fully met” 2010, Video effectiveness of government officials, break-out Action Plan: Participatory action plan with key priority conferences in legal frameworks, donors, groups, re- actions selected by country participants (IEG 2012) October 2010) empirical nongovernmental conceptualizing Outcome: Donors funded follow up network and video economic organizations, of evaluation conferences with local facilitators focusing on priorities inequality, academia, and World lessons, results- identified such as developing results framework for developing Bank staff based action designing and implementing activities to support gender frameworks for Video conferences: planning, video equality. More than 300 participants joined the online results 170 government and conference network. World Bank used online network to gather input measurement civil society facilitation on gender report. participants from 12 Follow up evaluation 2011 (averages of level two and countries worldwide three evaluations): increased knowledge of the subject: Gender online 4.08, applied knowledge obtained in your work: research network: 300 3.38, policy design: 3.31. members participated in online facilitated discussions for more than a year. 22 Evaluating Value chain IEG: Growth and Gaborone: 65 Evaluation Level-One Evaluation (scale from 1-5): Relevance: 4.2, Support for approach, Productivity in participants from 7 lesson-driven New information: 3.0, Quality of discussion: 4.0, Overall Growth and relationship Agriculture and Southern African content and usefulness: 4.0 Productivity in between state and Agribusiness (2010), countries consisting stakeholder Action Plan: Participatory action plan with key priority Agriculture and private sector, AfDB and IFAD: of high-level triangulation actions selected by country participants6 Agribusiness in national and local “Towards government officials, across public and Follow up evaluation 2011 (averages of level two and Southern Africa government Purposeful NGOs, academia, private sector, three evaluations): increased knowledge of the subject: (Gaborone, capacity, Partnerships In World Bank and IFC break-out 3.92, applied knowledge obtained in your work: research Botswana, differences African staff, African groups, re- 3.13, operations: 3.13. October 2010) between center Agriculture—A Development Bank conceptualizing and periphery, Joint Evaluation of staff, and of evaluation regional the Agriculture and international donors lessons, cooperation Rural Development participatory Policies and action plan Operations in Africa of the AfDB and IFAD”5 5 AfDB: African Development Bank, IFAD: International Fund for Agricultural Development, http://www.ifad.org/evaluation/jointevaluation/docs/africa/africa.htm. 6 http://siteresources.worldbank.org/EXTGPAA/Resources/WorkshopFindings_Botswana.pdf. 23