89421 Learning and Results in World Bank Operations: How the Bank Learns E VA L U AT I O N 1 JULY 2014 Learning and Results in World Bank Operations: How the Bank Learns Evaluation I July 2014 © 2014 International Bank for Reconstruction and Development / The World Bank 1818 H Street NW Washington DC 20433 Telephone: 202-473-1000 Internet: www.worldbank.org This work is a product of the staff of The World Bank with external contributions. The findings, interpretations, and conclusions expressed in this work do not necessarily reflect the views of The World Bank, its Board of Executive Directors, or the governments they represent. The World Bank does not guarantee the accuracy of the data included in this work. The boundaries, colors, denominations, and other information shown on any map in this work do not imply any judgment on the part of The World Bank concerning the legal status of any territory or the endorsement or acceptance of such boundaries. Rights and Permissions The material in this work is subject to copyright. Because The World Bank encourages dissemination of its knowledge, this work may be reproduced, in whole or in part, for noncommercial purposes as long as full attribution to this work is given. Any queries on rights and licenses, including subsidiary rights, should be addressed to World Bank Publications, The World Bank Group, 1818 H Street NW, Washington, DC 20433, USA; fax: 202-522-2625; e- mail: pubrights@worldbank.org. ii Contents ABBREVIATIONS .................................................................................................................................. v ACKNOWLEDGMENTS ........................................................................................................................ vi OVERVIEW........................................................................................................................................... vii MANAGEMENT RESPONSE .............................................................................................................. xii CHAIRPERSON’S SUMMARY: COMMITTEE ON DEVELOPMENT EFFECTIVENESS .................... xv 1. APPROACH AND CONTEXT .......................................................................................................... 1 An Evaluation of Learning in Lending, 5 Context, 5 Lessons from Past Realignments: Technical versus Country Focus, 6 Scope and Methods of Evaluation I, 9 Evaluation Limitations, 12 Organization of Report, 13 2. KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION ........................................ 17 What the Literature Says, 17 What the Bank’s Evidence Shows, 17 Looking Ahead, 33 3. THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS ............... 37 What the Literature Says, 37 What the Bank’s Evidence Shows, 39 Vital Importance of Team Composition and Integrity, 48 Looking Ahead, 52 4. INCENTIVES, LEADERSHIP, AND CULTURE ............................................................................. 58 What the Literature Says, 58 What the Bank’s Evidence Shows, 60 Looking Ahead, 81 5. IMPLICATIONS .............................................................................................................................. 84 Implications for the World Bank, 84 Implications for the Independent Evaluation Group, 90 Implications for the Next Evaluation, 91 BIBLIOGRAPHY................................................................................................................................... 94 BOXES Box 1.1. How This Evaluation Program Defines Knowledge and Learning............................................................. 2 Box 2.1. Using the Bank’s Information Technology to Search for Bank Documents Can Be Frustrating.............. 24 Box 2.2. Lessons Learned and the Problem of External Validity .......................................................................... 32 Box 2.3. Spark Is Improving Knowledge Sharing across the World Bank Group .................................................. 34 Box 3.1. The Tendency for Organizations to Undervalue Interpersonal Learning ................................................ 38 iii CONTENTS Box 3.2. A View on the Demise of Thematic Groups ............................................................................................ 44 Box 3.3. Task Team Leader Rotation Can Be Painful........................................................................................... 52 Box 3.4. Mentoring World Bank Youth .................................................................................................................. 54 Box 4.1. Task Team Leaders Speak Out about the Lack of Institutional Incentives ............................................. 71 Box 4.2. Learning from Failure: Panama Land Administration Project .................................................................. 73 Box 4.3. Can the World Bank Learn from the U.S. Army? .................................................................................... 76 TABLES Table 1.1. Topics and Sources of Evidence by Chapter ....................................................................................... 12 Table 5.1. Detailed Implications for the Bank that Flow from Each of the Main Findings ...................................... 88 FIGURES Figure 1.1. From Knowledge to Improved Development Outcomes: A Stylized Learning Model ............................ 3 Figure 1.2. Commitments of the International Bank for Reconstruction and Development and the International Development Association, FY86–13 .................................................................................................. 4 Figure 2.1. Lending Pressure Crowds Out Learning? ........................................................................................... 19 Figure 2.2 Recommendations about How Best to Promote Learning in Lending .................................................. 20 Figure 2.3. I Can Easily Access the Relevant Knowledge I Need to Serve My Clients ......................................... 21 Figure 2.4. Knowledge Expenditures, FY12 .......................................................................................................... 21 Figure 2.5. The World Bank’s Restricted Capture of External Ideas ..................................................................... 25 Figure 2.6. Extent to Which Development Economics Reports Are a Source of Learning for Project Preparation ............................................................................................................................................... 27 Figure 2.7. Extent to Which Impact Evaluations Are a Source of Learning for Project Preparation as Reported by Staff Working in Three Sectors ......................................................................................................... 28 Figure 2.8. To What Extent Does the Bank Have Useful Technical, Operational, and Country-Specific Knowledge? .......................................................................................................................................................... 32 Figure 3.1. To What Extent Has Learning Occurred through Doing and by Talking to Others? ............................ 41 Figure 3.2. To What Extent Is Mentoring an Important and Adequate Source of Learning? ................................. 42 Figure 3.3. Connectivity in the Energy and Mining Family Varies across the Bank .............................................. 48 Figure 3.4. Perceptions of Project Teams ............................................................................................................. 50 Figure 4.1. Task Team Leaders in Country Offices Perceive More Opportunities for Learning than Those at Headquarters ......................................................................................................................................... 62 Figure 4.2. Confidence in the Bank’s Commitment to Learning in Lending........................................................... 63 Figure 4.3. Total Spending on Staff Learning, FY01–13 ....................................................................................... 64 Figure 4.4. Learning Days Delivered by Type, FY13 ............................................................................................ 65 Figure 4.5. To What Extent Did Learning Occur as a Result of Training Courses or Learning Events? ............... 65 Figure 4.6. To What Extent Did Learning Occur through Quality Assurance Procedures? ................................... 67 Figure 4.7. Staff Overall Performance Evaluation Ratings for Learning and Knowledge Sharing ......................... 72 APPENDIXES Appendix A. Does Learning Vary by Lending Instrument and Project Performance? Appendix B. Tables Appendix C. Methodology and Main Findings from Staff Survey Appendix D. Projects Reviewed by the Independent Evaluation Group Appendix E. Examples of Good Practice Appendix F. The Place of Knowledge, Learning, and Innovation in the Proposed Corporate Monitoring Framework Appendixes are available online at https://ieg.worldbankgroup.org/evaluations/learning-and-results. iv Abbreviations AAA analytic and advisory activities ADB Asian Development Bank AAR After Action Review BAR Before Action Review BBL brown bag lunch DEC Development Economics DECPI Development Economics Poverty and Inequality DFID U.K. Department for International Development DLC Development Learning Center DPO development policy operations ESW economic and sector work FPD Finance and Private Sector Development GDP gross domestic product IBRD International Bank for Reconstruction and Development ICR Implementation Completion and Results report IDA International Development Association IEG Independent Evaluation Group IEGPS Independent Evaluation Group Public Sector ILI Intensive Learning ICR report IT information technology LIL Learning and Innovation Loan MIGA Multilateral Investment Guarantee Agency OPCS Operations Policy and Country Services OPE Overall Performance Evaluation OPFOR Opposing Force (U.S. Army) ORAF Operational Risk Assessment Framework PAD project appraisal document PAR public administration reform PER public expenditure review PFM public financial management PPAR Project Performance Assessment Report QER Quality Enhancement Review SDN Sustainable Development Network SIL Specific Investment Loan TTL task team leader USAID U.S. Agency for International Development VPU Vice Presidential Unit All dollar amounts are in U.S. dollars unless otherwise indicated. v Acknowledgments Soniya Carvalho and John Heath jointly led this study by the Independent Evaluation Group (IEG) under the supervision of Marie Gaarder (Manager, IEGPS) and the general direction of Emmanuel Jimenez (Director, IEGPS) and Caroline Heider (Director General, IEG). The evaluation drew on background research and contributions at various times from Iradj Alikhani, Anna Amato, Ana-Maria Arriagada, Roman Balin, Cathy Cardona, April Connelly, Sabine Dinges, Navin Girishankar, Catherine Gwin, Xue Li, Eduardo Fernandez Maldonado, Stefano Migliorisi, Chris Nelson, Manuel Penalver, Bahar Salimova, Rino Schiavo-Campo, Matt Winters, and Hanlei Yun. Richard Kraus provided administrative support. Barbara Rice and Cheryl Toksoz provided editorial support. The peer reviewers were Marilyn Darling (Fourth Quadrant Partners), Jeffrey Gutman (Brookings, formerly of the World Bank), and Michael Woolcock (Lead Social Development Specialist, DECPI). IEG extends its sincerest thanks to all of the World Bank managers and staff who participated in interviews and focus groups and responded to the questionnaire survey. The contribution of interviewees at other development agencies is also gratefully acknowledged. vi Overview As the world’s leading development explore the extent to which learning finance agency, the World Bank has, in takes place within and between Bank principle, an unrivaled opportunity to projects in a sample of countries and promote learning and knowledge sectors, taking into account the views of sharing about development clients and development partners. effectiveness. Bank lending has fallen in relation to developing country gross In summary, this first report shows that domestic product. To remain relevant, although, in general terms, the staff the Bank must improve the quality of its perceive the Bank to be committed to services; learning and knowledge offer learning and knowledge sharing—as an important competitive edge. The attested by surveys—with respect to the challenge is to become better at learning particular case of learning in lending, from lending and feeding learning back the culture and systems of the Bank, the into lending, responding more quickly incentives it offers employees, and the to lessons from experience with both signals from managers are not as successful and failed efforts, and being effective as they could be. The Bank’s more alert to the creation and use of organizational structure has been cutting-edge knowledge. revamped several times, notably in 1987 and 1996, when the shift to the matrix The Independent Evaluation Group organization took place. These changes (IEG) is conducting a program of have not led to a significant change in evaluations that seek to answer the learning in lending because they following question: touched neither the culture nor the How well has the World Bank incentives. It remains to be seen if the generated, accessed, and used learning latest structural change—the and knowledge in its lending introduction of Global Practices—will operations, and what is the scope for avoid past pitfalls and be more effective improving how it does so? in creating the right culture and incentives for learning and knowledge To date, two evaluations have been sharing. planned. The first, represented by this report, is limited to the perspective from Findings are drawn from a wide range within the World Bank and does not of sources: the academic and include that of the International Finance management literature on Corporation or the Multilateral organizational learning; Bank strategies Guarantee Agency. The next report will and studies; Bank employee surveys; an also cover only the World Bank, but will IEG survey of Bank staff; interviews and vii OVERVIEW focus groups, including consultations ensure that the staff makes time for both with other development agencies; IEG knowledge exploitation and knowledge reviews of project documents as well as exploration. project evaluations and studies; and aggregate data on the learning section of The range of knowledge sources used for preparing and implementing the annual Overall Performance projects is more limited than it might be. Evaluation of Bank staff. Little use is made of sources external to the Bank and the range of Bank Knowledge Exploitation and documents consulted is fairly narrow— Knowledge Exploration economic and sector work is extensively used; Bank research, impact evaluations, Studies show that learning in projects and IEG reports less so. Access to the has two aspects: the creation of new knowledge needed for lending is knowledge (exploration) and the use of hampered by the shortcomings of the existing knowledge from various Bank’s information technology, and sources (exploitation). In the long-term, systems for capturing, storing, and both elements are crucial for the success collating knowledge. However, remote of an organization, but there is a tension access has improved and, through between them. A focus on immediate Spark, it is now easier to locate the outputs produced to exacting deadlines desired expertise, both inside and favors exploitation; a focus on long-term outside the Bank. outcomes will privilege exploration. Finally, there is some suggestion that In the Bank, lending pressure—the country-specific knowledge may be survival of what Willi A. Wapenhans shallower relative to operational and referred to in 1992 as the “approval technical knowledge, which could be a culture”—is seen by staff as crowding challenge for the sector and thematic out learning even today. When asked to remit of the new Global Practices select the three actions which they because of the importance of country thought most likely to encourage context for development outcomes. learning in the Bank’s lending operations, the highest percentage of staff chose allotting sufficient time for The Interpersonal Dimension: learning in the Work Program Connectivity and Teams Agreement (66 percent), followed by Scientific research shows that learning allotting sufficient budget (57 percent) that involves copying, modeling, and and giving greater recognition of other interactive behaviors is more learning in promotion criteria (38 effective than solitary learning. Indeed, percent). Creation of an enabling the Bank staff perceives that environment through adequate time interpersonal exchanges are the most and budget, when combined with important source of learning and incentives such as promotion, will viii OVERVIEW knowledge sharing. These exchanges which may favor the introduction of are mediated by the networks to which new ideas, but the contribution of team people are connected and the teams in members tends to be under which they operate. Interpersonal acknowledged relative to task team learning includes mentoring, a practice leaders. that is highly valued by Bank staff; there is less mentoring than there used to be Given that so much operational and although attempts are now being made technical knowledge is confined to to revive it, notable examples being the people’s heads, the handover between Y2Y (Youth to Youth) community team leaders of projects is a source of mentoring program and initiatives learning discontinuity. The Bank has taken by the Africa Region. recently sought to address this through learning events that explore ways to Connections to Bank workplace debrief departing staff. networks and broader social networks can be a powerful stimulus to learning. In the past, structural constraints on Incentives, Leadership, and Culture cross support and budget constraints on There is a consistently positive communities of practice tended to generalized perception by staff of the hamper connectivity, problems that the Bank’s commitment to learning and Global Practices are intended to solve. knowledge sharing. The Bank has made Recent Bank experiments with a sustained investment in training and organizational network analysis— learning events, and this is aligned to sponsored by the Finance and Private the staff’s perceived needs. Sector Development Network and the Energy and Mining Family—have Despite these positive general trends, helped identify the people who provide most knowledge, the extent to which aspects of the system and the culture these experts are overloaded, and the specific to learning in lending may length of time it takes for new recruits to discourage the innovation and the Bank to connect to networks. These adaptiveness called for by effective experiments suggest that there is scope lending. Quality assurance procedures, for using organizational network including peer review and Quality analysis as a management tool to Enhancement Reviews (QERs) have monitor knowledge flow and to made an uneven contribution to anticipate the effect of staff rotation. learning. QERs, for example, are used to varying degrees and for varying With respect to team dynamics, research purposes. They tend to focus on shows that heterogeneous teams may be compliance with safeguards and better equipped to innovate. Bank fiduciary protocols, leaving little space employee surveys and the IEG survey of for consideration of the technical Bank staff found that project teams are perceived to be sufficiently diverse, ix OVERVIEW matters bearing on development and Results (ILIs) reports. IEG’s review outcome. of these products found that neither LILs nor ILIs produced more evidence The evaluation confirmed what research of learning than, respectively, other has indicated. In an organization like investment instruments or the core the Bank, financial incentives such as implementation completion reports. pay raises and performance bonuses are Neither of these instruments embodied a less important stimulus than the better monitoring and evaluation to any satisfaction derived from recognition by significant degree. managers and peers and the opportunity to lead important tasks. The Management literature has explored the annual Overall Performance Evaluation use of balanced scorecards as a means to of Bank staff includes a rating of promote alignment between the mission learning and knowledge sharing which of an organization, the culture and appears not to discriminate by level of values of the workplace, and the learning. The distribution of ratings command structure. The Bank has varies little between different parts of recently renewed its pledge to align the Bank. leadership, culture, and values, as embodied in the Corporate Scorecard Although restructuring of projects is and the proposed monitoring generally perceived to be less framework. Knowledge, Learning, and problematic than it was, staff report that Innovation is singled out as a pillar of they are not always encouraged to the new architecture. acknowledge problems with projects. Some attempt has been made to address this by organizing “learning from Implications failure” events. Bank staff perceives the lack of Recently introduced smart learning institutional incentives as one of the tools, for example, checklists, hold biggest obstacles to learning and promise and are valued by staff but it is knowledge sharing in the Bank. While too early to say if they will be sustained. reorganizations have been relatively By themselves, they will not be common at the Bank, serious reforms of sufficient to consolidate a culture of the Bank’s internal incentives have learning in lending. lagged. The ongoing change process provides an opportunity to finally There are lessons to be drawn from the redress this long-standing gap. But time Bank’s experience with customized is of the essence as reorganizations tend learning instruments—Learning and to deplete the very two assets Innovation Loans (LILs) and Intensive management needs to push through a Learning Implementation Completion transformative shift in internal x OVERVIEW incentives—political capital and staff Third, learning and knowledge sharing good will. is only likely to flourish if there is senior management commitment, leadership, Fast and forceful action by senior Bank signaling, and role modeling. management in giving clear, concrete, and consistent signals on the importance IEG has a shared responsibility for of learning and knowledge sharing— promoting learning. It has made the including through the questions it commitment to assess how its continually asks and the behaviors it evaluation procedures balance models—can bring rich pay-offs. Staff accountability and learning as well as to expectations are high: nearly 60 percent revamp its suite of products to make of respondents to IEG’s survey of Bank more allowance for learning staff indicate that they strongly agree or evaluations. agree with the statement that the Bank is committed to promoting learning and The next report in this program of IEG knowledge sharing in its lending evaluations will examine the extent to operations. The Bank has a golden which learning takes place within and opportunity right now. between Bank projects in a purposive sample of countries and sectors, taking There are three major implications for into account the views of clients and the World Bank. First, there is a case for development partners. It will attempt— the Bank to pay more attention to how through the use of case studies—to knowledge flow and learning are assess the early influence of Bank mediated through interpersonal process and incentive reforms on exchanges, understanding how team learning and knowledge sharing, while dynamics and connection to social mapping the social network in which networks shape the potential for individual projects are embedded, and learning and knowledge sharing. to explore the learning trajectories of individuals and teams. Particular Second, there is a need for smarter attention will be paid to the context in approaches to rewarding learning and which lessons are generated and the discouraging the hoarding of extent to which the specificity of a given knowledge, including redesign of context limits the scope for transmitting individual results agreements and learning across time and space. performance evaluation criteria. xi Management Response The World Bank management thanks the Independent Evaluation Group (IEG) for the report entitled Learning and Results in World Bank Operations. Management commends IEG for producing an insightful report on a topic which is at the heart of the ongoing World Bank Group reform effort. The IEG report comes at a time when management has put knowledge, learning, innovations, and results at the center of its reform agenda and has already undertaken important steps in order to improve learning from operations. With the introduction of “Global Practices” (communities of experts on specific technical areas spanning across the whole World Bank Group) and “Cross-cutting Solutions Areas,” more effective transfer of knowledge and expertise will be facilitated, and the emphasis on learning and innovation will be stimulated. Global Practices/Cross-cutting Solutions Areas will create new and more opportunities for formal and informal learning exchanges within and outside the Bank. The aspiration of these reforms is for the World Bank to become a Solutions Bank, not just a repository of knowledge about development. The entire leadership of the Bank Group, including the heads of Global Practices, will be responsible for identifying, together with their staff, the best approaches to lift people out of poverty and promote shared prosperity and scaling up successful programs—what we have called a “science of delivery.” Through greater integration of the Bank, the International Finance Corporation, and the Multilateral Investment Guaranty Agency, it expected that staff will work more cohesively across the World Bank Group, to bring their collective experience together to better serve our clients. Thus, while management overall agrees with the reports’ main findings, it believes the reform of the World Bank Group and the implementation of its new strategy are addressing the implications flowing from those findings. Management generally agrees with the report’s conclusions regarding the importance of (1) the interpersonal dimension of learning; (2) rewarding learning and knowledge sharing; and (3) role modeling and leadership from senior management. These areas, especially the last two, are among the key drivers for the entire change effort at the Bank Group. Many of the report’s sources used to arrive at the conclusions have indeed also served to inform management’s initial decision to embark on the change effort as well as subsequent decisions, e.g., on the centrality of knowledge and learning in the terms of reference of the Global Practices and the Cross-cutting Solutions Areas. Management agrees with the need to strengthen the interpersonal dimension of learning. In addition to strengthening and streamlining the on-the-job learning xii CHAPTER 1 APPROACH AND CONTEXT component of the Operational Core Curriculum, management also believes that the emphasis on learning and talent management in the Global Practices/Cross-cutting Solutions Areas provides an excellent opportunity to take this conclusion to heart. Management is putting in place a new approach to Knowledge Management, with the creation, under the auspices of the Global Practice VPs, of solutions-oriented Global Practice portals. These portals are intended to improve the ability to search for knowledge and expertise, the curation and flow of knowledge, and, through the integration with the Spark collaboration platform, the collaborative work and support for communities of practice. Further, World Bank Group staff, clients, and partners will be invited to learn and collaborate in an Open Learning Campus through learning programs that package the state of knowledge on key development challenges in an easily accessible way. It is expected that through the development of these knowledge management and learning tools, person-to-person interaction and collaboration will be enhanced. Management agrees with IEG on the critical importance of rewarding learning and knowledge. Among other things, the World Bank Group is identifying, and will promote, a set of behaviors and incentives to develop the notions of knowledge citizenship and knowledge leadership, and to ensure that knowledge is everyone’s business. To that end, management has created a Working Group on Incentives and Culture that will provide specific recommendations on formal and informal incentives to promote the new knowledge behaviors across the World Bank Group. Management strongly believes that leadership, signaling, and role modeling are indeed fundamental for improving the World Bank Group’s learning and knowledge ecosystem. Management wants to take this opportunity to reiterate its full commitment to this agenda. The creation of the Learning, Leadership, and Innovation Vice-Presidency is a clear example of the fundamental role of the culture of learning going forward. Also, the various on-boarding and staff readiness programs, which have recently started to take shape, will put a strong emphasis on leadership and role-modeling knowledge and learning behaviors. Management recognizes the methodological challenges posed by this evaluation. The report mostly relies on the staff surveys and interviews. While other different research methods are also used (consultations, literature review, project reviews), a number of conclusions are drawn from single studies as well as from inconclusive evidence and/or unreliable data. The low response rate to the survey (only 18 percent), the fact that an important percentage of respondents skipped several questions, and the possible response bias, add to the concern over the sufficiency of data for coming up with findings and drawing conclusions. Additionally, the xiii MANAGEMENT RESPONSE limited number of task team leaders (TTLs) who participated in consultations (only 38 across the Bank) invites considerable caution in drawing firm conclusions. Management commends the comprehensive approach to the report, including the clear recognition of IEG’s role in the World Bank Group learning culture. IEG has an important role to play in the Bank Group’s knowledge and learning ecosystem and in learning from lending specifically. Through its own instruments, such as the Implementation Completion Report reviews, IEG is also a source of incentives and a driver of behaviors. IEG’s openness to look at how its own products are used and to investigate the implications thereof is very much appreciated. Management suggests removing the word “results” from the title. The title creates an expectation that the report provides a detailed analysis of the links between learning and results, which is not the case. While IEG, through its “stylized learning model” has done a commendable effort to help understand points of time in the project cycle where opportunities for learning exist, it does not delve into how learning in the lending cycle leads to better development results. Moreover, the conclusions of the report (called “implications”) are about how to strengthen learning, not about how to improve results through learning. Even if Phase II of the evaluation were to explore the links between learning and results, this otherwise valuable Phase I report is presented as a self-standing study and its contents do not live up to the expectations created by the title. Management values the collaboration and coordination that took place around this Phase I evaluation. The use of a “just-in-time” approach for this particular evaluation provides good learning value, given the timing of the ongoing reform. Looking forward to Phase II of the evaluation, management is concerned about the timing and relevance of the recommendations, given the ongoing change process, as well as about the trade-offs between the proposed breadth of the analysis, and the depth and rigor that can be achieved in the limited amount of time envisaged. Rather than being overly ambitious (e.g., the formulation of 11 research questions, in addition to the “attempt” to use country case studies to “map the social network into which the individual projects are embedded and explore the learning trajectories of individuals and teams”), management would recommend that IEG be more selective and specific in what it wants to achieve. Management is particularly interested in seeing clear evidence—if possible—of how “learning from lending” influences the achievement of development outcomes, and which types of learning (formats, methods) are more effective at improving development outcomes. Such evidence would be particularly useful for management for putting in place incentives or processes to foster the type of learning that is most likely to improve the achievement of development outcomes. xiv Chairperson’s Summary: Committee on Development Effectiveness On May 12, 2014, the Committee on Development Effectiveness (the committee) met to consider Learning and Results in World Bank Operations: How the Bank Learns, Evaluation I and Draft Management Response. Summary The committee welcomed the evaluation’s findings and commended IEG and Management for their collaboration. Members were encouraged that IEG’s findings were consistent with the analysis carried out by Management at the outset of the change process, and they reiterated the relevance of interpersonal learning, leadership, and incentives. Members appreciated that actions Management is taking in the context of the overall Global Practice (GP) process confirm and address the evaluation findings on improving learning in Bank operations. They welcomed concrete steps Management is taking in this regard, inter alia, addressing the issue of staff incentives through cascading incentives from Management to staff; ensuring knowledge and knowledge sharing are embedded in staff’s Overall Performance Evaluations; and establishing structured knowledge platforms on key topics. It was also noted that the MOUs of the GPs will hold GPs accountable on knowledge and learning. Members noted that knowledge needs to be transparent, readily available, and shared on time to foster learning, and that time needs to be allocated to staff for such learning. They recognized the need for an IT learning system that allows easy storage, search, and travel of information across the organization. Members further underscored the essential need for a culture shift in how Management and staff learn from lending operations. They noted that throughout the broader change process in the institution, in particular during discussions on the New Approach for Country Engagement, the Board has reiterated that without a cultural change in behavior the structural changes will not have the intended impact. They acknowledged that cultural shifts are not simple and they take time— and this serves to highlight the need for incentives, strong leadership, and a constant push led from the top. Members looked forward to Phase II of the evaluation, and asked IEG to sharpen the focus of the evaluation and draw conclusions on how the World Bank generates, accesses, and uses learning and knowledge in its lending operations to contribute to better results. The committee welcomed IEG’s ongoing efforts to identify how it could better contribute to learning in the institution. Juan José Bravo CHAIRPERSON. xv 1. Approach and Context Highlights  Bank lending has fallen in relation to developing country gross domestic product; to remain relevant and reposition itself as a Solutions Bank, the Bank will need to become better at learning from lending.  This evaluation examines the extent to which learning takes place in lending operations, the factors that facilitate or hinder learning, and how learning and knowledge sharing can be improved at the Bank.  The next evaluation will follow-up on hypotheses emerging from this evaluation and explore the extent to which learning takes place within and between Bank projects in a sample of countries and sectors, taking into account the views of clients and development partners. There is a problem with learning and knowledge sharing in the Bank, a problem that has long been recognized, notably since the 1992 Wapenhans report. “Something is not working because the problems we are encountering in today’s projects are the same problems encountered in projects many years ago.… [We] keep making the same mistakes because we do not learn from earlier experience” (Wapenhans 1992, B12–13). Today, recognition of this same problem seems to underlie the Bank’s flagship initiative—the science of delivery—which combines the art and science of delivery. The art lies in the innovation and adaptability of the actors and different delivery models, and has as its key aspect the “continuous interplay of designing interventions using evidence; implementing them in an iterative way; and, learning deliberately throughout the process,” and the science lies in replicating and scaling those models (Pradhan 2013). No systematic attempts have been made to evaluate how knowledge and learning is acquired, captured, and transferred in the course of lending, or how the extent and quality of learning help shape operational outcomes. World Bank Group President Jim Yong Kim’s commitment to improving how operations are delivered embraces a particular interest in learning by doing, including learning from failure. Knowledge, Learning, and Innovation is one of eight objectives that will be monitored in the Bank’s new strategy (Appendix F). The Independent Evaluation Group (IEG) is conducting a program of learning and results evaluations to promote a better understanding of how the World Bank acquires, captures, and transfers knowledge and learning in its lending operations (henceforth referred to as learning in lending), and what scope there is for 1 CHAPTER 1 APPROACH AND CONTEXT improving how it does so. Box 1.1 describes how the terms knowledge and learning will be used throughout the evaluation program. Box 1.1. How This Evaluation Program Defines Knowledge and Learning In this program of evaluations, knowledge refers to a content or stock: the data, information (both theoretical and practical), and skills acquired through research, study, and experience. Bank knowledge services involve the production, use, and sharing or transfer of that content both inside and outside of the Bank. Learning is a processor flow, which involves obtaining or acquiring knowledge and capabilities. As related to Bank lending operations, learning occurs importantly by:(i) bringing knowledge into the design of operations (learning into lending); (ii) gaining and using knowledge in the modification and implementation of ongoing projects (learning while lending; and (iii) transmitting or feeding back lessons from projects to other projects or follow-on projects(learning from lending). Learning in lending, a term used throughout this program, includes (i), (ii), and (iii). To the extent that learning in lending is effective, it results in changes in operational behaviors, policies, or processes that inform current or subsequent operations and helps to build the Bank’s base of operational knowledge. Thus, there is a feedback loop from knowledge to learning and back to enhanced knowledge with the purpose of improving development outcomes. Knowledge and learning in the World Bank can typically be divided into operational, what is needed to design and implement projects; sector or thematic, often geared to a community of practice; and country specific, referring to institutional capacity and political economy. Knowledge and learning can take the following forms: • Documented knowledge is written down or entered into a database. It is knowledge that is captured, stored, and collated, and in principle will remain permanently available, although the systems and technology for accessing it will influence how much it is used. • Tacit knowledge is contained in the heads of individuals. It may entail technical expertise or practical experience. • Learning through training refers to courses and learning events that would typically be part of a staff member’s work plan and explicitly budgeted. • Tacit learning refers to tacit knowledge that is transmitted to others through verbal exchanges and nonverbalized copying and mirroring of behaviors—the full gamut of interpersonal exchanges, which are powerfully mediated by connections to teams and social networks in and beyond the workplace. Overall, the knowledge and learning nexus in any institution is fostered or impeded by an institution’s organizational arrangements, processes, and incentives. The staff of a learning organization excels at creating, acquiring, and transferring knowledge. This presupposes: (i) an organization that embodies the structure, culture, and incentives needed to support learning (e.g., an organization where staff 2 CHAPTER 1 APPROACH AND CONTEXT are free to experiment); (ii) sound learning processes and practices (e.g., incentives to make midcourse corrections to active operations and opportunities for sharing knowledge between departments); and (iii) leadership behavior that reinforces learning (Garvin et al. 2008). In the case of the World Bank, the ultimate goal is for learning to lead to better development outcomes (World Bank 2013a,b). Based on the literature, Figure 1.1 presents a learning model, illustrating how the process of acquiring knowledge, sharing knowledge, and engaging with it could lead to learning, which when acted on should lead to better operations and improved development outcomes. This in turn would validate the old knowledge or signify new knowledge (created through the process of adapting and contextualizing old knowledge or through disruptive innovation). This learning model is inevitably stylized; learning in lending is likely to be complex and nonlinear. None of the steps is automatic. Knowledge can be hoarded rather than shared; it can be selectively perceived or retained (whereby people interpret and remember facts to suit their existing biases); and opportunities to apply lessons learned to improve impact on the ground may not be seized. For each of the steps in the virtuous learning cycle to materialize, the incentives, culture, structure, and processes need to be aligned. Figure 1.1. From Knowledge to Improved Development Outcomes: A Stylized Learning Model 3 CHAPTER 1 APPROACH AND CONTEXT The Bank’s declining role as a lender invites a reassessment of the lending–learning balance. As the world’s leading development finance agency, the World Bank has, in principle, an unrivaled opportunity to promote learning and knowledge sharing about development effectiveness. But the Bank’s core business of lending has created a set of institutional incentives that have downplayed the learning that lending requires. As the 2013 corporate strategy points out, Bank lending is dwindling in relation to developing country gross domestic product (GDP) (World Bank 2013a). The scale of operations for the International Bank for Reconstruction and Development (IBRD) and International Development Association (IDA) is now below 1 percent of the combined GDP of borrowing countries and falling (Figure 1.2). “Maintaining a minimum scale of operations is important if the World Bank Group is to influence the policy agenda and support clients in delivering effective development solutions”(World Bank 2013a, 14). IEG’s annual evaluation of the results and performance of the World Bank Group for 2013 reports a continued decline of overall portfolio performance in the Bank, driven by lower outcome ratings, mainly of investment projects (IEG 2014a). Development solutions may not be forthcoming if insufficient attention is paid to learning in lending. If effective development solutions are not seen to be delivered, then demand may also dwindle, making the reverse of the strategy statement equally true. If not being seen as providing effective solutions, a minimum scale of operations may be hard to maintain. Figure 1.2. Commitments of the International Bank for Reconstruction and Development and the International Development Association, FY86–13 Source: Reprinted from World Bank (2013a). 4 CHAPTER 1 APPROACH AND CONTEXT An Evaluation of Learning in Lending IEG is conducting a program of evaluations on learning, and this report covers the first evaluation in the series. The objective of the program is to delineate attributes of effective learning in World Bank lending. These attributes refer to learning into lending (inputs into project design); learning while lending (feedback and modifications of design and implementation while the project is underway); and learning from lending (lessons from the project that were transmitted to other projects) (Box 1.1 and Figure 1.1). The evaluation program will assess how the Bank can become better at generating, accessing, and using learning and knowledge in its lending operations. It acknowledges the importance of the feedback from knowledge to learning and from learning back to enhanced knowledge. As a whole, the evaluation program seeks to answer the following overarching question: How well has the World Bank generated, accessed, and used learning and knowledge in its lending operations and what is the scope for improving how it does so? There is an important caveat. While the next evaluation in the program will address the interaction between learning and results, the within-Bank focus of this first study does not allow conclusions to be drawn about development outcomes. Context Since 2000, the World Bank Group has been acknowledged as one of the Most Admired Knowledge Enterprises in the world. 1 In addition, corporate surveys of staff have found respondents to be consistently upbeat about the scope for learning and knowledge sharing in the World Bank. A 2012 Organizational Health Index survey found that 61 percent of respondents replied “always” or “often” to the statement, “the World Bank holds events to share knowledge and ideas across the organization.” 2 In response to the statement, “ideas and knowledge are freely shared within the Bank,” 57 percent of respondents agreed. In addition, almost two- thirds of respondents (62 percent) agreed, “the World Bank generates enough high- quality ideas to achieve its strategic objectives.” This2012 survey sought to benchmark the Bank against other, comparable organizations around the world. It found that, with respect to learning and innovation, the Bank was comparable to the average for public sector organizations but below the benchmark for private sector financial institutions (consistent with the most recent survey of Most Admired 5 CHAPTER 1 APPROACH AND CONTEXT Knowledge Enterprises, which included no non-private sector companies in the top 20). The Bank staff surveys did not refer specifically to learning and knowledge sharing as applied to lending. Nevertheless, the positive perception by the Bank’s staff down the years was echoed in the findings of the staff survey that IEG conducted for this evaluation in December 2013 through January 2014. Sixty percent of respondents to IEG’s survey replied “strongly agree” or “agree” to the statement, “I have confidence that the World Bank is committed to promoting learning and knowledge sharing in its lending operations.” Yet, as this report will show, when questions are asked about learning in specific aspects of the lending process the picture that emerges is more nuanced. IEG suggests that the framing of questions about knowledge sharing and learning is critical. Given the Bank’s status as the leading repository of knowledge about development and the undeniable pride that staff manifest about working for the institution—another consistent theme from corporate surveys—general questions about knowledge and learning probably reflect the halo effect of the Bank’s reputation. This effect is possibly accentuated by a certain fuzziness about what learning actually entails for the Bank. Lessons from Past Realignments: Technical versus Country Focus The Bank’s past efforts at organizational restructuring and strategic realignment have not eliminated what this report will identify as persistent shortcomings in learning in lending. Most recently, the World Bank has established 14 Global Practices and five Cross-cutting Solution Areas. It is too early to say how these global practices and cross-cutting areas will affect learning and knowledge at the technical and country levels. In the second phase of the evaluation program, IEG will provide an initial assessment of the implications of the new organizational arrangements on learning as well as suggestions on how to achieve a stronger learning culture. The Bank at its inception was organized around units dealing with each of the major thematic areas that the Bank supported through projects. The Bank staff members recruited for these thematic areas were usually experts who had built or run such things as water systems, education departments, and power companies. Many came from former colonial services. They were the technical experts who had a strong say on the design and project components, and on whether a Bank project went forward or not. 6 CHAPTER 1 APPROACH AND CONTEXT The 1972 reorganization ushered in by President Robert McNamara was the first attempt at bringing some “country focus” to the Bank. McNamara had plans for a dramatic growth in Bank lending and believed that a global Bank would not be able to deliver the dramatic growth in lending volumes he had in mind. He reorganized the Bank into five regions, each headed by a vice president, with a “projects” and a “programs” department. The projects departments were essentially technical departments, and the program departments comprised of country-focused staff. Technically, both were on equal footing, but gradually it became clear that the programs department had a major call on big decisions. This did not mean that technical focus was abandoned. But the new configuration was based on the recognition that the design of any project involved compromises and trade-offs that were largely driven by country circumstances. The 1972 reorganization endured for the following 15 years, accompanied by a massive expansion of lending under McNamara auspices. Bank lending grew from $1billion in 1968 to $13 billion in 1982. The 1972 reorganization was followed by the 1987 reorganization, when each of the Regions was divided into several self-contained Country Departments, including both country and sector divisions. The country department director was responsible for the design of each country program and for the delivery of its lending and analytical components. The third major reorganization came in 1997 when, under the auspices of President James Wolfensohn, the Bank moved to a matrix organization, with sector and thematic units on one side of the matrix and country units on the other. Each staff member in operations now became accountable to two bosses—a sector manager and a country director. The move to the matrix was accompanied by the following measures: increased decentralization of staff to strengthen the exchange of knowledge and learning with clients; a new policy to foster greater openness and access to Bank knowledge internally and externally; adjustments in budget processes and dual reporting of sector staff to incentivize enhanced capture, use, and feedback of operationally relevant knowledge and learning; and intensified formal staff and client learning programs. Also, from 1996 forward, the Bank began to promote itself as the Knowledge Bank, based on the principle that the provision of knowledge for development was no less central to the Bank’s mission than lending. 3 Steps were taken to upgrade internal information technology (the intranet), expand formal and informal knowledge sharing activities, and establish new knowledge partnerships. The post-1997 structure of the Bank included many complex organizational arrangements. There was a high transaction cost entailed by the many interactions between a few large sector and thematic departments and the many small country units, the number of the latter having mushroomed. The 22 country departments of 7 CHAPTER 1 APPROACH AND CONTEXT the 1987 structure had now become 60 plus country departments. The sector and thematic departments, which were designed to restore the focus on technical excellence, were overwhelmed by the need to manage the burgeoning number of transactions with the country departments. Today the Bank is more client-focused than it was mainly because of decentralization, but progress on knowledge and learning in support of improved development results remains elusive. 4 Bank and IEG reports from 2005 to 2013 are in broad agreement that the Bank’s matrix system has not proved conducive to achieving the objective of a global Knowledge Bank (IEG 2013). “[T]he goal of building thriving global technical networks was not achieved. Over time, the dual matrix has created silos within and between the Regions and Network anchors. Each region has striven to provide responsive products and services to country clients, while anchor departments have enhanced the Bank’s role on global issues and global public goods. But this fragmentation has come at the expense of nurturing well-integrated technical practices. As a result, transmission mechanisms, such as networks, weakened, and global knowledge exchanges suffered. Knowledge flows were hindered by low levels of cross support and limited staff mobility” (World Bank 2010). Work within the Bank around 2007–2008 led to the development in 2010 of the Bank’s first-ever knowledge strategy and helped identify internal obstacles to learning and knowledge sharing (World Bank 2010): • Senior leadership is lacking. Leadership has not been a priority in recent years and as result, a coherent strategic direction to guide staff knowledge and learning priorities and programs is missing. • Knowledge and learning are not integrated into core business processes. Cutting-edge knowledge is an “add-on”— nice to have but not embedded into core lending and nonlending processes. • Learning programs are not always aligned with best practices. Programs have particular shortcomings in building core competencies. • Disincentives to learning and knowledge sharing persist. Recruitment and promotion, operational processes, budget allocations, do not facilitate systematic learning and knowledge sharing as part of the normal work of teams. • Governance for staff knowledge and learning is weak. It is difficult to adopt common standards and processes for staff knowledge and learning, and to ensure compliance. • The information technology architecture does not enable effective staff knowledge sharing and learning. Bank units have created their own systems, 8 CHAPTER 1 APPROACH AND CONTEXT which are not necessarily compatible or easily accessible throughout the institution. What lessons from these past reorganizations can be drawn by the new Global Practices? As noted above, this will be explored further in the second phase of the evaluation program. At this point, however, two initial comments are worth making. First, careful thought needs to be given to the relationship between the practices and the country units, ensuring that the capacity of practice managers to focus on technical excellence is not crowded out by the sheer volume of transactions with country units. Second, it may be necessary to regroup the countries into fewer units. The many small country units that exist are highly dependent on lending prospects in just one or two countries. To justify their existence, they face an incentive to push lending irrespective of whether resources would be better invested in countries beyond their jurisdiction. Scope and Methods of Evaluation I This first evaluation is limited to IBRD and IDA investment operations and development policy operations. It does not address the operations of the International Finance Corporation and the Multilateral Investment Guarantee Agency. Also, the study does not make a standalone assessment of the Bank institutions that support operational learning (i.e., Development Economics, IEG, Information and Technology Services, Network anchors, and World Bank Change Knowledge and Learning,). Literature Review. The purpose of the review was to generate hypotheses about how organizations like the Bank learn, hypotheses that could be tested in the interviews and the staff survey that IEG conducted. Attributes considered were the organizational structure and culture, the system of learning practices and processes, and the role of leaders in reinforcing a culture of learning (Garvin et al. 2008). The review was divided into three parts. First, IEG assessed the academic literature on project-based learning. This was complemented by a review of the management literature on good practice in organizational learning. Finally, IEG searched for Bank documents on corporate policy and strategy that bear on learning, dating back to the late 1980s. This included a review of the findings from past IEG studies, notably the 2012 evaluation of the Bank’s experience with the matrix organization. Consultation. Throughout the evaluation, IEG exchanged ideas with the managers and senior staff who have been leading the change, knowledge, and learning reforms at the Bank. Guidance from the three peer reviewers of this evaluation was 9 CHAPTER 1 APPROACH AND CONTEXT also taken into account. Sixty-two interviews were held with directors, managers, and advisers in the World Bank. In addition, IEG organized six focus groups and conducted one-on-one interviews with 38 Bank staff active as task team leaders (TTLs). 5 In addition, IEG invited the leading staff of the sector boards to recommend projects that were, in their view, exemplars of innovation and effective learning. Finally, knowledge leaders in other leading development agencies, including the U.S. Agency for International Development, the U.K. Department for International Development, and Asian Development Bank, were interviewed. 6 Staff Surveys. IEG conducted a questionnaire survey to understand staff perceptions of how the World Bank promotes learning and knowledge sharing in its lending operations. The survey was fielded from December 2013 to January 2014 and was sent by email to all Bank staff at grade F and above. In all, 6,800 staff members received the questionnaire and 1,239 responded, a response rate of 18 percent. The survey asked the staff about: (i) factors favoring or hindering learning and knowledge sharing at the various stages of project cycle; (ii) the effect on learning of Bank products, processes, and the incentives that staff work with; and (iii) ways to improve learning in lending. The results of this survey were triangulated against comparable findings from the various surveys of staff conducted by the World Bank, including the 2012 Organizational Health Index Survey and the 2013 Employee Engagement Survey. Project Review. This part of the evaluation considered to what extent the products and processes associated with project preparation, implementation, and completion (and associated project evaluations) provide a sound indication of how knowledge and learning are generated, used, and transferred. IEG sought to balance the twin objectives of ensuring that the operations reviewed were as recent as possible, and they contained enough evidence on which to base a judgment about how much had been learned. Therefore, selection was limited to 134 completed projects, broken down by lending instrument, whose Implementation Completion and Results (ICRs) reports had been validated by IEG. For each of the five lending types examined, the approach was to count back in descending chronological order from a date in mid- October 2013 until the quota for that particular instrument was reached. The quotas, which were intended to be roughly proportional to the share of each instrument in the IBRD and IDA portfolio, were: Specific Investment Loans or SILs (n = 60), adaptable program loans (n = 10), Learning and Innovation Loans or LILs (n = 10), technical assistance loans (n = 12), and development policy operations, both standalone and those forming part of a series (n = 42). Although the Bank has not approved any LILs since FY08, they were included in the cohort because of their explicit learning objective. For the purposes of this study, it was important to 10 CHAPTER 1 APPROACH AND CONTEXT understand why this instrument failed to prosper. The projects are listed in Appendix D. IEG examined to what extent these projects were associated with knowledge sharing and learning, primarily based on a review of project documents but supplemented by interviews with TTLs. IEG began by identifying moments in the lending cycle that might be conducive to learning. These learning opportunities included use of research and impact evaluations to inform project design; peer review; signals from managers in Implementation Supervision Reports; feedback from monitoring; and participation in workshops with clients. IEG tested a set of hypotheses about the association between, on the one hand, the learning opportunities arising in the project cycle and, on the other hand, the type of lending instrument and the performance trajectory of investment projects. The approach to this investigation, the results (inconclusive), and the lessons that will be taken forward to the next evaluation are presented in Appendix A. The project review had three other dimensions. First, IEG assessed FY13 project appraisal and program documents to identify the sources and types of knowledge used for their preparation. Second, a sample of Intensive Learning ICRs (ILIs) was compared to a sample of core ICRs to assess whether the extra spending involved in ILIs resulted in richer learning. From the pool of 35 ILIs, IEG randomly selected 10 and compared them with 10 randomly selected core ICRs from the same period (FY05–13). The sample was weighted to ensure an appropriate spread across the sector boards where most of the ILIs were conducted and these were then matched to the same sectors for core ICRs to balance the comparison. Third, the type and extent of learning in selected Project Performance Assessment Reports (PPARs) were examined with a view to determining what value these reports add to the ICR from a lesson-learning perspective. From the 92 reports published in FY09–12, a purposive sample was drawn. The first step in the sampling process was to select all single-project PPARs referring to projects approved from FY04 onward plus any cluster PPARs that included at least one project approved from FY04 onward. This increased the likelihood that the most up-to-date learning would be captured. From the 16 reports left, nine were chosen to include the widest possible range of sectors and lending instruments, and to contain examples of both cluster and single-project PPARs. OPE Data. IEG discussed with Human Resources (HR) what data could be released from the annual performance assessment—the Overall Performance Evaluation 11 CHAPTER 1 APPROACH AND CONTEXT (OPE)—of staff members without breaching confidentiality norms. HR released aggregate information of the ratings on learning and knowledge sharing for each of five years, broken down by gender, grade level, and sector board mapping. It was not possible to obtain data linking staff with above average learning ratings to projects on which they had worked. Nor was it possible to correlate the learning and knowledge sharing rating with other OPE ratings. Table 1.1 shows how the topics covered by this report are distributed between chapters and the evidence that underpins the analysis of each topic. Table 1.1. Topics and Sources of Evidence by Chapter Sources of Evidence World World IEG Academic Bank Bank survey IEG IEG IEG project and corporate employee OPE of interviews project evaluations management documents surveys, data Bank and focus document and studies literature and 1997– staff, groups review No. Topics studies 2013 2014 Time √ √ √ √ Sources √ √ √ √ Knowledge √ √ √ √ management 2 Document √ √ √ √ √ use External √ √ √ validity Mentoring √ √ √ √ Connectivity √ √ √ √ √ 3 Teams √ √ √ √ √ Staff rotation √ √ √ √ Commitment √ √ √ √ Bank √ √ √ √ learning Quality √ √ √ √ 4 Incentives √ √ √ √ √ √ Adaptiveness √ √ √ √ √ Smart tools √ √ √ √ Customized √ √ instruments Evaluation Limitations This first evaluation has five limitations. First, there is no predefined benchmark of learning in Bank lending—thus no readily available evaluation template was 12 CHAPTER 1 APPROACH AND CONTEXT available. Also, no expert consensus exists about what constitutes learning relevant to lending. IEG tackled this challenge by taking an interactive approach, involving regular exchanges with staff leading the most recent set of reforms in addition to the multiple sources of views and information outlined in the methods section. Second, the 18 percent response rate to the questionnaire survey of Bank staff may be considered low, although the absolute number of staff responding (1,239) was substantial. There might also have been a response bias. IEG did as much as possible to boost the number of questionnaires completed, including exhortations to staff from top management and a vigorous campaign of follow-up by phone (randomly sampled). It also used the standard method of triangulation: comparing survey results with the findings of previous studies and the output of open-ended, in-depth interviews and focus groups with TTLs, quality assurance advisers, and representatives of senior management. Particularly important for triangulation purposes were the comparable findings from the 2012 Organizational Health Index Survey and the 2013 Employee Engagement Survey, both of which were responded to by a majority of Bank employees. Third, in the project review, there was an inevitable trade-off between breadth and depth. With a cohort of 134 projects (Appendix D), it was not possible to interview all of the TTLs, let alone the managers and advisers associated with each project. Privileging breadth over depth meant that most of the assessment was based on a review of project documents, which only partially reflect the learning that occurred. Fourth, the selection of projects captured intraproject learning but shed little light on the transfer of learning between projects or across countries (unless captured in the project documents)—perhaps the biggest challenge facing the new Global Practices. In the time available for this first evaluation, it was not possible to trace the learning threads that run from one operation to another. Also, no attempt was made to assess the broader programs in which individual projects were inserted. No reference is made to country partnership strategies or related country program evaluations. Fifth, this first evaluation omits consideration of the views of country clients and development partners concerning how effective the Bank is at learning from its lending operations. Also, the association between learning and development outcomes will be captured in the next evaluation. Organization of Report The report logic is consistent with the interpretation of knowledge and learning that is set out in Box 1.1 above. A distinction is drawn between retrievable, documented, 13 CHAPTER 1 APPROACH AND CONTEXT and programmed knowledge and learning, on the one hand, and the tacit knowledge and learning that are typically transmitted though conversations and nonverbalized copying or reacting to the behavior of others (“interpersonal exchange”). The report begins by assessing the access to and use of documented knowledge in the Bank. It then considers the various facets of interpersonal exchange, considering how these are mediated by connections to organizational networks and teams. Finally, it considers how leadership, incentives, and culture influence the pattern of learning and knowledge sharing in the Bank. Chapter 2 explores two essential aspects of learning—knowledge exploitation and knowledge exploration and the factors influencing them. It also addresses the range of documentary sources and databases that the Bank staff uses when preparing and implementing projects, considering how technology and systems affect access to these sources of knowledge and the wide variations in the type of documents consulted. It further considers whether the current approach to learning hampers the development of country-specific knowledge, a limitation of which the new Global Practices will need to be aware. Building on the observation that most learning and knowledge sharing occurs through interpersonal exchanges, chapter 3 begins by examining the contribution of mentoring. It then considers how network connectivity influences learning, exploring the problems encountered with cross support and previous experience with communities of practice. Recent Bank experiments with organizational network analysis have highlighted the extent of connectivity constraints and suggest that this type of analysis may be used as a management tool to monitor knowledge flow. The chapter then considers how the composition of teams and their functioning may influence learning. Finally, it explores how interpersonal learning and knowledge exchanges may be affected by staff rotation, with particular reference to the learning discontinuities attendant on the handover between task team leaders. Chapter 4 addresses the extent to which incentives, leadership, and culture are aligned to promote learning in lending. It begins by showing that there is a consistently positive generalized perception by staff of the Bank’s commitment to learning and knowledge sharing, a perception that is undoubtedly informed by the organization’s sustained investment in training and learning events. However, this positive perception seems at variance with the specific problems bearing on the culture of learning in lending. The chapter considers the uneven contribution that quality assurance procedures make to learning. It shows that the annual assessments of individual performance do not significantly reward learning and knowledge sharing. The chapter then considers the scope for adaptiveness in lending, as reflected in processes such as project restructuring and the extent to which 14 CHAPTER 1 APPROACH AND CONTEXT admitting to operational shortfalls and learning from failure is promoted. It ends by considering how smart tools and customized learning instruments may enhance learning, concluding that these will have limited effect if the Bank’s top leaders do not take the steps needed to embed a culture of learning in lending. Chapter 5 considers the implications of the report’s findings, for the Bank’s change process, for IEG, and for the design of the second evaluation in IEG’s Learning and Results series. References Garvin, David A., Amy C. Edmondson, and Gino Francesca. 2008. “Is Yours a Learning Organization?” Harvard Business Review 86 (3): 109–116. IEG (Independent Evaluation Group). 2013. The Matrix System at Work: An Evaluation of the World Bank’s Organizational Effectiveness. Washington, DC: World Bank. ______. 2014. Results and Performance of the World Bank Group 2013. Washington, DC: World Bank. Oppenheimer, D., and L. Prusak. 2011. “Knowledge Management at the World Bank.” Case Number 1936. Case Program of Harvard Kennedy School, Cambridge, Mass. Pradhan, Sanjay. 2013. “A Solutions Partnership to End Poverty.” Speech delivered at the World Knowledge Forum, October 16, 2013, Seoul, Korea. Wapenhans, W. 1992. Effective Implementation: Key to Development Impact. Report of the Portfolio Management Task Force. Washington, DC: World Bank. World Bank.2010. Transforming the Bank’s Knowledge Agenda: A Framework for Action. Washington, DC: World Bank. ______. 2013a. World Bank Group Strategy. Washington, DC: World Bank. ______. 2013b. FY13 Annual Report: Staff Learning—The State of Staff Learning at the World Bank. Washington, DC: World Bank. 1For each year from 2000 to 2004, the World Bank Group was featured in the top 20 Most Admired Knowledge Enterprises, as identified in an annual global survey organized by Teleos, an independent research company specializing in knowledge management. After 2004 the Bank Group featured less prominently and made the top 20 in 2011. In 2013, the Bank Group was one of 52 finalists but did not proceed to the next round. Samsung was the overall winner, and no nonprivate sector companies made the top 20. 2The Organizational Health Index survey was conducted in October 2012. There were 6,450 respondents, which is a response rate of 55 percent. 3For a thorough examination of the limitations of the Knowledge Bank, see Oppenheimer and Prusak (2011). 4The assessment was made both through self-evaluation (Quality Assurance Group, Organizational Effectiveness Task Force, and World Bank [2010]) and by IEG. 15 CHAPTER 1 APPROACH AND CONTEXT 5 To address the difficulty of tracking down task team leaders (TTLs) in a timely manner, IEG conducted a series of impromptu phone interviews, working down a list of TTLs with known attributes, posing a limited number of questions to whichever TTL picked up the phone. These impromptu interviews were targeted toward experienced TTLs. Using Information and Technology Solutions data, IEG compiled a list of TTLs who had delivered five or more operations to the Board, including at least one in FY12–13. 6 The results of these consultations on good practice are shown in Appendix E. 16 2. Knowledge Exploitation and Knowledge Exploration Highlights  The Bank staff perceives that there is a lack of time for learning.  The range of knowledge sources consulted is limited, in particular the use made of non-Bank sources.  Of the various documentary sources used, country and region-focused analytic work is used the most for project preparation—less attention is given to research, impact evaluations, and IEG evaluations.  The written record provides a poor map of Bank learning.  The Bank’s technology and systems for capturing, storing, searching, and collating knowledge do not allow staff to make the most of documentary sources of learning.  Recent technology developments, such as Spark, SkillFinder, and CommunityFinder, are promising ways to build learning exchanges within and beyond the Bank.  Knowledge that is country specific in nature appears to be shallower than technical and operational knowledge; this may be a concern given that lessons generated in one country may not easily be generalized to other countries. What the Literature Says The academic and management literature indicates that learning in projects has two aspects: the creation of new knowledge (exploration) and the use of existing knowledge from various sources (exploitation). In the long-term, both elements are crucial for the success of an organization (Eriksson 2013). Still, there is potentially a tension between exploration and exploitation (Andriopoulos and Lewis 2010; Katila and Ahuja 2002; Lavie and Rosenkopf 2006; O’Reilly and Tushman 2011; Uotila et al. 2009). “A short-term focus on efficiency, based on exploitation of existing knowledge and technologies” may conflict with “a long-term focus on innovation and strategic development, based on exploration of new knowledge and technologies” (Eriksson 2013, 333). Organizations need to succeed in both exploration and exploitation and keep an appropriate balance between them. March (1991) argues that firms focusing too much on exploration may suffer the costs of experimentation without gaining many 17 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION of its benefits due to many undeveloped new ideas. Exploiters on the other hand may obtain short-term efficiency gains based on current competences, leading to success and thereby more exploitation. Due to the direct benefits of exploiting current competencies, firms may get stuck in a competence trap. Because of exploration's greater uncertainty, most organizations focus more on exploitation than on exploration (Uotila et al. 2009). This may result in short-term success but long-term stagnation and failure (O´Reilly and Tushman 2008; Eriksson 2013, 334). The tension between short-term and long-term goals resonates throughout the literature. A study on project-based learning in different organizations found that project-oriented organizations tend to privilege actions that produce the quickest acceptable outcomes rather than actions that produce optimal outcomes in the long term. This approach not only limits the learning within projects but also restricts the transfer of knowledge across projects by not allowing time for it (Swan et al. 2010). Time for reflection is one of the building blocks of the learning organization. Being too busy or overstressed by deadlines and scheduling pressures affects people’s ability to think analytically and creatively, which calls for thoughtful review of work processes (Garvin et al. 2008). What the Bank’s Evidence Shows Faced with pressure to meet short-term goals (lending and disbursement targets), Bank employees may feel that they do not have the time to search widely for knowledge or to experiment with new ideas and approaches. In an organization like the Bank, learning must go hand in hand with lending: it is not an either-or choice. At the same time, there may be a case for adjusting the balance to allow more time for learning. The approval culture is alive and well. According to the 2013 Employee Engagement Survey, 26 percent of all respondents disagreed or strongly disagreed that the World Bank Group prioritizes development results over the number and volume of transactions. But 40 percent of task team leaders (TTLs) showed this level of disagreement. This squares with the findings from the survey of Bank staff conducted for this evaluation by the Independent Evaluation Group (IEG). Seventy percent of the respondents strongly agreed or agreed that lending pressure crowds out learning; although managers are significantly less persuaded that this is the case than the staff who report to them (Figure 2.1). 18 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION Figure 2.1. Lending Pressure Crowds Out Learning? Non-TTLs TTLs Staff at grade GG* Staff at grades GI-GK* 0 10 20 30 40 50 60 70 80 90 Percentage responding "strongly agree" or "agree" Source: IEG survey of Bank staff conducted for this evaluation. *p = 0.00. Respondents to IEG’s survey of Bank staff indicated that learning would be more likely to receive the attention it needs if time and budget are earmarked for this purpose. When asked to select from a list of options the three actions that they thought most likely to encourage learning in the Bank’s lending operations, the highest percentage of staff chose allotting sufficient time for learning in the work program agreement (66 percent), followed by allotting sufficient budgets (57 percent) and giving greater recognition to learning in the staff promotion criteria (38 percent) (Figure 2.2). Creating an enabling environment through time and budgets when combined with incentives such as promotion will ensure that the staff makes time for both knowledge exploitation and knowledge exploration. While time could be a factor, employees’ use of documents may also be hampered by the Bank’s system for collating knowledge and by the shortfalls in the Bank’s information technology (IT). In addition, to the extent that they consult documents, employees will focus on Bank documents rather than external documents. Finally, the range of Bank documents referred to may itself be limited. The evidence for all three observations will now be examined. 19 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION Figure 2.2. Recommendations about How Best to Promote Learning in Lending CAS and Corporate Scorecard monitor Salary review increase reflects Top management signals priority Staff promotion criteria acknowledge Budget allotted for preparation and supervision Time allotted in Work Program Agreement 0 10 20 30 40 50 60 70 Percentage of respondents recommending these incentives Source: IEG survey of Bank staff conducted for this evaluation. Note: CAS = country assistance strategy. KNOWLEDGE IS EASY TO ACCESS In surveys dating back to 1997, 60 percent or more of the Bank staff reported that it is easy to access the knowledge they need. 1 The 2012 Organizational Health Index survey found that 61 percent of respondents replied “always” or “often” to the statement that the World Bank holds events to share knowledge and ideas across the organization. In response to the statement “ideas and knowledge are freely shared within the Bank,” 57 percent of respondents agreed. Also, almost two-thirds of respondents (62 percent) agreed, “the World Bank generates enough high quality ideas to achieve its strategic objectives.” The survey sought to benchmark the Bank against other, comparable organizations around the world. It found that, with respect to learning and innovation, the Bank was comparable to the average for public sector organizations but below the benchmark for private sector financial institutions. 2 Compared to the earlier surveys, a similar level of favorable response about knowledge access is conveyed by the 2013 Employee Engagement Survey (Figure 2.3). But there are some unanswered questions. Why do employees in country offices report significantly greater ease of access relative to those at headquarters? Why do TTLs report significantly less ease of access than non-TTLs? How much does access refers to knowledge gleaned from talking to people as opposed to reading documents? The findings from IEG interviews and focus groups suggest that while in general people are able to find the knowledge they need, they experience frustration in navigating the systems that the Bank uses for capturing, storing, and collating information and knowledge. Time involved in searching for knowledge reduces time for reflection and exploration. 20 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION Figure 2.3. I Can Easily Access the Relevant Knowledge I Need to Serve My Clients CO based* 74 HQ based* 68 TTL** 64 Non-TTL** 74 TTL, CO 64 TTL, HQ 64 Percentage of responses: to a "very high" or "substantial" extent Source: 2013 Employee Engagement Survey; results for IBRD and IDA. Note: Country office (CO) based (n = 3,782); headquarters (HQ) based (n = 5,727); non-TTL (n = 6,150); TTL (n = 2,924); TTL, CO (n = 655); TTL, HQ (n = 1,216). For two of the three comparison groups (headquarters versus country office and TTL versus non-TTL), the differences were statistically highly significant (p = 0.00). For the third group (TTL based at headquarters versus TTL based in country offices), there was no difference. Spending on all knowledge products rose from $300 million in FY02 to $690 million in FY12. After technical assistance, analytic work (i.e., economic and sector work [ESW]) is the largest knowledge product produced by the Bank (Figure 2.4). Figure 2.4. Knowledge Expenditures by Product Line, FY12 Other knowledge products 112 World Development Report 6 Global monitoring 19 Research 40 External client training 63 Impact evaluation 11 Technical assistance 272 Economic and sector work 167 0 50 100 150 200 250 300 US$, millions Source: World Bank Business Warehouse, February 22, 2013. FRUSTRATIONS WITH HOW THE BANK MANAGES KNOWLEDGE Of the various obstacles to learning in lending, 36 percent of respondents to the IEG survey of Bank staff singled out the fragmented system of knowledge management, 21 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION making it the third highest ranked obstacle. The staff’s frustration with the Bank’s IT is a relatively small part of the larger problem of poorly collated knowledge. Only 11 percent indicated that the Bank’s lack of state-of-the-art information technology was an obstacle to learning and knowledge sharing in lending operations. The main problem lies with the Bank’s system for capturing and collating knowledge and learning, which is only partly a hardware or software problem. IEG’s project review demonstrated the limited extent to which the learning embedded in Bank project files can be readily coded and captured. The written record provides a poor map of Bank learning. This is not a trivial observation. Given staff turnover and the associated risk of learning discontinuity, unless outgoing staff are systematically debriefed, it is likely that the learning they have acquired will be lost and hard to reconstruct in retrospect based on a review of project documents. Even if the written record were a sufficient guide to learning, in the course of this investigation IEG found that the filing of these records is not systematic. In particular, peer review comments and the minutes of decision meetings are often hard to track in the Operations Portal and in some cases are simply missing. IEG’s evaluation of the Bank’s matrix organization reported: “Most staff, particularly those in the Regions and country offices, are unable to draw efficiently on knowledge generated inside and outside the Bank. Knowledge products are not stored in an easily searchable and retrievable form and are rarely used by staff outside the units where they are produced. As a result, and notwithstanding the analytical quality of the Bank’s AAA [analytic and advisory activities], much of the Bank’s knowledge has limited shelf life and use value. The knowledge produced by the Development Economics Vice Presidency (DEC), the Bank’s research department, is widely disseminated to a global audience. But only 7 percent of operational staff report making direct use of DEC’s knowledge in operational work” (IEG 2012a, xxi). The difficulty of gleaning evidence of learning from project documents emerged from IEG’s review of the 20 most recently evaluated development policy operations. All of the program documents in this cohort referred to a comprehensive list of sources. However, the link between the documentary sources and project design is rarely made explicit, an observation that has also been made in the various retrospective reports that have been commissioned by Operations Policy and Country Services. The minutes from the Regional Operations Committee’s meetings are scarcely more enlightening, even though the project team is explicitly instructed to show how the proposed reforms built on the use of background documents. Mere citation of documents does not amount to assimilation of their findings. The IEG review found little reference to and use of documents produced outside the Bank. 22 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION The sources most regularly cited are government strategy papers and International Monetary Fund reports. The program documents that make the fullest use of knowledge products are those prepared for multisector and human development operations. Interviewees told IEG that the Bank’s information and knowledge are poorly collated. Project document storage is not systematic; the records in electronic archives are patchy. Also, there is no system for summarizing or synthesizing information, and there are no standardized headings (equivalent to a Dewey library classification system). This makes it hard for the TTL to assess rapidly what information is available and to make full use of what is there. In country offices, it is particularly difficult to obtain the necessary knowledge quickly and to distill it in a way that meets client needs. Some sector specialists launch ad hoc initiatives to store and classify knowledge. But these archives are not regularly updated, and the initiatives tend to fade when the initiator moves on or when the budget dries up. Stored knowledge rapidly becomes obsolete from an operational perspective; this may reduce the incentive to invest in systems for capturing and archiving knowledge. If the overall system of knowledge management is the main problem, technology is nevertheless a contributing factor (Box 2.1). Satisfaction appears to have dwindled over time. In the 1999 World Bank staff survey, 63 percent of respondents replied favorably to the statement, “The Intranet is a user friendly tool to find the information I need to do my job more effectively.” But in replying to IEG’s survey of Bank staff, only 31 percent of respondents replied that it was very easy or easy. Strikingly, the country-based staff is relatively positive about the intranet system, and the statistical difference between these groups and those based in Washington, DC was highly significant: 45 percent of the respondents in country offices replied that the intranet was a very easy or easy way to find relevant knowledge for lending operations, compared to 26 percent of respondents located at headquarters. The reasons for this discrepancy are not obvious given good global access to the worldwide web (the benchmark), meaning that expectations for the Bank’s intranet may be assumed to be ratcheted up for staff in country offices as much as for those in Washington, DC. Participants in IEG focus groups and interviews expected more of the Bank’s IT, noting that it compared unfavorably with that of other leading knowledge management enterprises, notably Google and Bing. One person commented, “I cannot find on the intranet my own working papers written for the Bank, but they pop up right away on other websites.” Unlike Google, Bing, or similar systems that were built from scratch and for specific goals, Bank databases have grown piecemeal 23 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION over the years by a process of accretion. The original hard files—stored in a Pennsylvania mine—were scanned and digitized and new materials, differently coded, were progressively added. Interviewees noted that, to complicate matters, IT users and IT providers find it difficult to talk to each other in a mutually comprehensible way. Box 2.1. Using the Bank’s Information Technology to Search for Bank Documents Can Be Frustrating Finding documents on the Bank’s external website is often difficult, even when the exact title of the document is known. For example, a search for “Senegal Public Expenditure Review” pulls up 921 items, the first three of which have nothing to do with the document in question presumably because the search is drawing in everything triggered by the word “public.” In addition, the external website is not intelligent about interpreting typographic or spelling errors in the search terms, unlike Google or Bing. The Open Knowledge Repository, launched in April 2012, is a better external source. However, this too has its limitations. A search for Senegal's public expenditure reviews (PERs) begins by listing the most recent one but then skips to PERs for the Democratic Republic of Congo, Maldives, Poland, and Russia rather than to earlier reports for Senegal. The International Monetary Fund and many other international organizations are ahead of the Bank in this respect. Searching on the Bank’s intranet is not much easier. Operations Portal and Image Bank will work if the searcher enters the precise title or code, but these systems are incapable of intuiting based on incomplete or partially correct information. If searchers don’t exactly know what they want, the systems are not very helpful. Recent improvements have been ad hoc rather than part of a comprehensive data management reform. Two new tools were introduced in January 2013: a new PER search tool (search.worldbank.org/per) and a new query (bireporting.worldbank.org > Shared Services/Reports > Image Bank), which lists all new, completed, and dropped activities. However, without clear and adequately funded responsibility for monitoring the process and maintaining these tools, the initiative is likely to evaporate. LIMITED USE OF OTHER DOCUMENTS Bank staff appears to make less use of documents produced outside the Bank when preparing and implementing projects. The IEG survey of Bank staff found that, during preparation, one-third of respondents cited non-Bank products as an important source of learning; for implementation, the proportion was one-quarter. This picture is reinforced by a separate investigation that IEG conducted for this study. A review of all 97 project appraisal documents and program documents produced in the second and third quarter of FY13 revealed that only 36 percent of these documents drew on non-Bank research or other external sources of knowledge. In addition, participants in IEG interviews and focus groups were 24 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION unanimous in pointing out that, when preparing projects, TTLs use Bank documents more than documents produced outside the Bank. More broadly, in their response to another recent survey, staff members indicated that the Bank makes limited use of external knowledge (Figure 2.5). The same message was echoed in a Harvard case study: “The Bank remains strongly inward- oriented and insular in its knowledge activities.…Bank operations are exactly the opposite of the open-source movement in software; until very recently, the Bank predominantly relied on its own knowledge rather than opening the institution up for broad-based collaboration with other knowledge centers” (Oppenheimer and Prusak 2011, 5). Figure 2.5. The World Bank’s Restricted Capture of External Ideas “The World Bank imports ‘best practices’ from other organizations and industries.” “The World Bank uses external contacts to maximize the flow of ideas into the World Bank.” “The World Bank creates active networks with thought leaders to bring new ideas into the World Bank.” 0 5 10 15 20 25 30 35 40 45 50 Percentage of responses: "always" or "often" Source: 2012 Organizational Health Index survey. USE OF DIFFERENT TYPES OF BANK DOCUMENTS WIDELY VARIES Bank analytic work is an important source of learning. Respondents to the IEG survey cited country or region-focused analytic work as a source of significant learning more often than corporate analytic work, and analytic work was a more important source for preparation than for implementation. In the importance assigned to country-focused analytic work, no significant difference was found between respondents who described themselves as TTLs and those who said they were not; or between TTLs of investment projects and development policy operations. However, to a significant extent, country-based staff valued country- focused analytic work more highly relative to other knowledge sources than headquarters-based staff: 65 percent of the former said that in the past two years this 25 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION work had been a source of learning for project preparation, compared to 57 percent of the latter. The importance of Bank analytic work as a source of knowledge for project preparation has come to light before. A 2008 IEG study found that development policy operations were potentially more likely to have been informed by Bank analytic work (ESW) than investment loans. A sample of 119 loans approved during FY03–05 that was representative of both investment and development policy operations found that 91 percent of the development policy operations were preceded by ESW that could have informed the loans, whereas 61 percent of investment loans were preceded by such ESW. The operative word is “potentially”—in this part of the study, IEG did not attempt to assess actual use of ESW findings for project identification and design. However, the same study also asked Bank staff to rate on a scale of 1 to 6 (with 1 denoting “no extent” and 6 denoting “great extent”) to what extent they used ESW to inform lending. This question was examined in two ways: first, through a review of 12 countries, according to which 74 percent of project TTLs gave a rating of 4 or higher; second, in a separate survey of project TTLs, 87 percent rated the use of ESW at 4 or above (IEG 2008, 21–22). This message was reinforced in IEG interviews and focus groups. AAA was acknowledged as an important part of the knowledge harnessed for project preparation. In the words of one, “If you don’t have good AAA, you don’t have good projects.” Bank research is a less important source of learning. In the IEG survey of Bank staff, less than 15 percent of respondents described documents from DEC as having been very large or substantial sources of learning—either for preparation or for implementation. To the extent that DEC research informs the thinking of the experts who staff consult or the analytical work that staff use, DEC’s influence may be underestimated here. Respondents were particularly likely to say that DEC reports were either not applicable to learning or that they did not know about the extent to which DEC reports were a source of learning. Thirty-one percent of respondents replied either “not applicable” or “don’t know” for the project preparation stage, and 38 percent gave one of those answers for the project implementation stage. There were statistically significant differences between sectors and lending instruments (Figure 2.6). In a separate enquiry, IEG found that of the evaluated projects in the study cohort (Appendix D), 52 percent cited research findings in the appraisal or program 26 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION document and 31 percent in the Implementation Completion and Results (ICR) report. Figure 2.6. Extent to Which Development Economic Reports Are a Source of Learning for Project Preparation Human development sectors Policy sectors Hard sectors TTLS, development policy operations TTLs, investment projects 0 5 10 15 20 25 30 35 Percentage of responses: "very large" or "substantial" extent Source: IEG survey of Bank staff conducted for this evaluation. Note: As defined by IEG for this evaluation, human development refers to education, health, social protection, and social development; policy refers to economic policy, poverty reduction, and public sector governance; hard sectors refers to agriculture, energy, transport, urban development, and water. DEC has conducted research on the use that operational staff makes of its work (Ravallion 2011; DEC 2012). A majority of the 555 staff members (grade GG or above) who responded to its survey valued Bank research. Differences between regions were smaller than those between sectors. The proportion of respondents that were highly familiar with DEC research ranged from 67 percent in the Middle East and North Africa to 45 percent in East Asia and Pacific. Staff working on poverty, human development, and economic policy were more familiar with Bank research (over 60 percent responding “highly familiar”) than staff in the more traditional sectors of Bank lending—agriculture and rural development (50 percent), energy and mining (32 percent), transport (45 percent), and urban development (32 percent). Familiarity with research correlated positively with the value placed on it from an operational perspective. The DEC survey revealed that vice-presidential units with higher shares of economists and doctorates in any field tended to value and use Bank research more. The sectors that made the least use of Bank research also tended to rely less on research produced outside the Bank. Participants in IEG interviews and focus groups shed more light on this topic. Although DEC makes a significant contribution to the analytic work conducted by the Bank Regions—DEC staff are expected to commit 30 percent of their time to 27 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION cross support—interviewees said that little of this work feeds directly into project design, and DEC is not strategic in reaching out to TTLs. It does not actively promote its services and instead waits to be called on by the TTL. Sometimes that call never comes from the TTL. Use of impact evaluations as a source of learning in lending is substantial but with room for further uptake, while systematic reviews are surprisingly missing in action. The learning and utility that operational staff derives from impact evaluations is substantial. IEG’s evaluation of Bank Group impact evaluations reported a substantial number of evaluators and TTLs who perceived them to have contributed to the global knowledge of what works and to be useful in dialogue with clients and donors, though perhaps their contribution was more modest than might be expected given the high profile of this work in recent years (IEG 2012a). According to the IEG survey of Bank staff, at the preparation phase, 28 percent of respondents rated impact evaluations (whether or not they were Bank-sponsored) as a very large or substantial source of learning; for project implementation, the corresponding proportion was 22 percent, in line with the fact that impact evaluations are less well suited and designed to look at implementation issues. When the data are broken down by the self-identified sector board mapping of respondents and the analysis is confined to three stylized sectors as defined by IEG, statistically significant differences were found between the policy sector and the other two stylized sectors (Figure 2.7). Figure 2.7. Extent to Which Impact Evaluations Are a Source of Learning for Project Preparation as Reported by Staff Working in Three Sectors Policy sectors Hard sectors Human development sectors 0 10 20 30 40 50 60 70 Percentage of responses: to a "very large" or "substantial" extent Source: IEG survey of Bank staff conducted for this evaluation. Note: Human development refers to education, health, social protection, and social development; policy refers to economic policy, poverty reduction, and public sector governance; hard sectors refer to agriculture, energy, transport, urban development, and water. In its review of 134 recently evaluated operations, this evaluation found that only 15 percent cited impact evaluations in the appraisal or program document and 17 percent in the ICR. Given the limited external validity of individual impact evaluations to other settings and countries, one would mainly expect the ICRs to refer to impact evaluations of the project in question. Recent data from Development 28 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION Impact Evaluation, commonly known as DIME, show that about 25 percent of projects have impact evaluations attached to them; hence a 17 percent citation in historic projects is probably accurate. What is more surprising is that systematic reviews—that is, reviews that sum up the best available research on a specific question using a systematic and transparent approach to synthesize evidence mainly derived from high-quality impact evaluations—are surprisingly absent from the evidentiary basis in the World Bank’s project documents. One issue relates to how many projects build in impact evaluations for learning purposes. Given the lack of verified knowledge of effects in many sectors, there is clearly a need to do more. Another issue is the use that is made of existing impact evaluations. An IEG study found that the results of only 47 percent of completed World Bank impact evaluations were used in ICRs to demonstrate project effectiveness. It drew on World Bank team leader and evaluator surveys to report that 37 percent of the impact evaluations linked to a lending project were used as an input to the ICR or midterm review. The report concludes that the feedback loop between impact evaluations and operations is not yet well developed and suggests that this may be associated with factors such as their relevance, timeliness, dissemination, and engagement with local counterparts as well as with monitoring and evaluation culture and political environment (IEG 2012a, xxiii). While this evaluation report refers to impact evaluations finalized prior to 2012, and measures have reportedly been put in place to improve the feedback loop between impact evaluations and operations since then, this first phase of the evaluation program does not look at those measures. Nevertheless, the TTLs interviewed by IEG were relatively lukewarm in their assessment of the contribution that impact evaluations make to preparation and implementation. Some suggested that the findings of these evaluations were difficult to operationalize. Timeliness of the impact evaluations may also be a factor. They also pointed out that this type of evaluation is not appropriate for all sectors and has mainly been applied to human development. Others said impact evaluations were too costly and time consuming to be a regular part of lending operations. To the extent that case studies selected for the second phase of the learning-in-lending evaluation have associated impact evaluations, IEG will be able to explore this further in that context. Most staff members do not use IEG products to inform learning in lending. Respondents to the IEG survey of Bank staff also indicate that they make less use of IEG evaluations than they do of impact evaluations. At the preparation phase, 22 percent of respondents rated IEG evaluations as a very large or substantial source of learning; for project implementation, the corresponding proportion was 17 percent. But there was a statistically significant difference between headquarters-based and 29 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION country-based staff, with 25 percent of the former and 34 percent of the latter indicating that IEG evaluations were a source of learning for project preparation. A separate source—IEG’s most recent client survey—corroborates this evidence. Of the 755 Bank staff who responded to the client survey (IEG 2013), only 13 percent indicated that they frequently read IEG reports compared to 25 percent of the 456 external clients who participated in the survey. There is a huge gap between country-based and headquarter-based staff: 23 percent of the former, but only 6 percent of the latter indicated that they frequently read IEG reports. The product most frequently cited was the ICR Review. In its review of 134 recently evaluated operations, this evaluation found that only 13 percent referred to IEG in the appraisal or program document and 14 percent in the ICR. Comparing investment and policy-based lending, the latter cited IEG more frequently. Of the 14 appraisal or program documents referring to IEG, 11 were for development policy operations. To what extent do IEG’s standard products inform lending? The Project Performance Assessment Report (PPAR) is IEG’s oldest product line, dating back to the creation of an operations evaluation unit at the Bank in 1972. When IEG conducted a search in July 2013, Image Bank listed about 2,200 PPARs, the first issued in October 1972 (OED 1972). Yet this is a product that is rarely read by World Bank staff—or by members of the Board to whom IEG reports. IEG’s 2012 client surveys asked 434 Bank staff which reports they had read in the past 12 months. There were only 12 mentions of the PPAR (IEG 2013). It may be that PPARs are not widely used as a source of learning because the staff finds that lessons drawn are not sufficiently detailed or generalizable to be of operational significance. From the very beginning of its existence, IEG sought to maximize the learning potential of PPARs by examining clusters of similar projects. 3 But there is no evidence that Bank staff use cluster PPARs more than single-project PPARs. In a review of PPARs conducted for this evaluation, IEG found that the cluster PPARs did not generally adduce richer lessons than the reports devoted to single operations. There are some exceptions. For example, a cluster PPAR on finance sector development, which drew on experience in four countries, was particularly thorough in assessing the lessons drawn from the Bank analytic work that presaged operations in these countries (IEG 2012b). The format of this report is particularly attractive, combining a seven-page chapter on Conclusions and Lessons Learned that is well enough evidenced to stand on its own; readers requiring more information on individual country cases can refer to Appendixes, each of which is 30 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION equally sound and stand-alone. This is one of the few cases where, from a learning perspective, the PPAR added real value to the ICR. LIMITED UTILITY OF LESSONS LEARNED IN ICRS Participants in IEG interviews gave differing opinions about the usefulness of ICRs but generally agreed that these are more oriented to accountability than lesson learning. Moreover, people look at are the ratings not the lessons. Some said that reading ICRs before designing projects helped to ensure that mistakes were not repeated. But ICRs for the second or third project in a series rarely convey any sense of cumulative learning. Others said that the lessons cited in ICRs were too general to be operationally useful, and that it was hard to translate lessons from one country context to another. It was acknowledged that there was a tendency to “copy and paste” ICR lessons into appraisal documents without any attempt to adjust project designs to reflect this learning. There is an important question about the external validity of lessons learned: how generalizable are they to contexts other than the one’s in which they were generated (Box 2.2). In its review of ICRs, IEG has the means to amplify lessons already mentioned or to suggest new lessons. This opportunity is seized to a limited extent. Research conducted for this evaluation led IEG staff to conclude that the learning element in ICR Reviews has been faulted for being drawn largely from ICRs, as being superficial, and as having weak evidence that is poorly substantiated. Enhanced timeliness and operational relevance of the lessons presented in ICR Reviews could enhance their impact. There is an important question about the external validity of lessons learned: can they be generalized to contexts other than the one’s in which they were generated? It may be that the Bank does not pay sufficient attention to the country specificity of the lessons that are extracted from its operations and the knowledge that is accumulated. Respondents to ICR’s staff survey were asked to what extent useful technical, operational, and country-specific knowledge existed in the Bank. The last of these three was the laggard. Comparing TTLs at headquarters with those in country offices there was no statistically significant difference between them with respect to the extent of the Bank’s useful knowledge on the country context (Figure 2.8). This may appear surprising. Locating staff in country offices is supposed to enhance knowledge of local constraints and opportunities. It is possible that the finding is driven by the fact that country office staff (and other staff for that matter) may have interpreted the question as relating to knowledge they can access, rather than knowledge they themselves possess. 31 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION Box 2.2. Lessons Learned and the Problem of External Validity Over a decade ago, Eliot Berg, a prominent consultant to the Bank, noted the difficulty of generalizing lessons learned in one context to other contexts (Berg 2000, 30). “A former department director in the World Bank tells me that every year the Executive Board would ask: What is the Bank doing to see that the staff learns from the many ‘lessons’ emanating from the reports of the Operations Evaluation Department? He would distribute relevant reports and have a meeting to discuss them. The staff would say: ‘None of this applies to me. The situations I confront are unique.’” Furthermore, “the greatest weakness in Bank operations [is the] inability to customize programs to country-specific needs” (Berg 2000, 38). Similar observations about the weak external validity of best practices have been recently made by Woolcock (2013) who also commented: “The primary rationale for an organization-wide focus on ‘learning’ is that ‘lessons’ can in fact travel across countries (and perhaps even across sectors). But can they? Or rather, under what conditions is it reasonable to presume that ‘lessons’ from project X in country Y translate to country Z? No one seems to have a really good answer to that question. At present, the default assumption is that a sufficiently ‘rigorous’ empirical finding provides warrant for claims regarding the likelihood that the same project implemented elsewhere (or at a larger scale of operation) will attain correspondingly similar findings, but in recent years this assumption has been increasingly (and properly) called into question. The implication should be clear: if the Bank is to become a bona fide ‘learning organization,’ it must of analytical necessity be able to articulate a credible basis on which the various ‘lessons’ emanating from its programs can and cannot be deployed elsewhere.” Figure 2.8. To What Extent Does the Bank Have Useful Technical, Operational, and Country-Specific Knowledge? Technical Operational* TTL, CO TTL, HQ Country specific 0 10 20 30 40 50 60 70 Percentage of reponses: to a "very large" or "substantial" extent Source: IEG survey of Bank staff conducted for this evaluation. *p = 0.02. 32 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION The logic of the new Global Practices is that sector or thematic knowledge is globally fungible. Various staff interviewed by IEG suggested that to the extent that sector and thematic specialists are now expected to cover the world, there is a risk that the steady accumulation of in-depth knowledge about particular countries will be neglected. One TTL said that if the Global Practices make it mandatory for him to devote 20 percent of his time to working as a non-TTL on teams in the other countries, “his” operations (the one for which he is TTL) would suffer. Some queried what value could be added by “parachuting in”—joining a team in another country for just two weeks. Also, given that the Global Practices will still be sector or theme specific, it is not clear to staff how they will facilitate knowledge transfer in multisector operations. A recent IEG report on the Bank's response to avian influenza highlighted how cooperation at the strategic level between staff working on animal health and staff working on human health broke down at the project level during supervision because of institutional incentives within the Bank (IEG 2014). Looking Ahead It is a source of concern that many staff perceive that they do not have enough time for learning. Time pressure may compromise the two essential aspects of learning— knowledge exploitation and knowledge exploration. Earmarking time for learning in the staff’s work program agreements may be one solution, although staff will also need to be given proper incentives to learn in order to ensure that they make time for knowledge exploitation and knowledge exploration. Some recent innovations address the underuse of knowledge. With respect to increasing the use of findings from impact evaluations, the Africa Region has introduced its “Smackdown Series,” which pits teams from impact evaluation and operations against each other for debates on a priority topic, such as microfinance, youth training, and agricultural innovation. With respect to underuse of Bank research by staff in operations, DEC launched its Visiting Experts Program in FY13, which allows operational staff to take time from their regular assignments to share their field experience with research economists (Secretariat to the Learning Board 2013, 30). With respect to the Bank’s management of knowledge, despite the shortcomings that have been examined here, some steps have been taken in the right direction. Remote access to the Bank’s systems is now much easier than before. IEG interviewees expressed particular enthusiasm for Spark and other recent initiatives that help TTLs connect with each other and with external experts (Box 2.3). However, more thought is called for concerning ways to improve the accessibility and usability of the enormous corpus of World Bank documents. 33 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION Box 2.3. Spark Is Improving Knowledge Sharing across the World Bank Group What is Spark? Spark is a virtual collaboration platform that cuts across Bank Group institutions and enables staff to exchange ideas, gather feedback, co-create documents, and easily share knowledge and experiences. In an effort to move toward a “One World Bank Group,” the launching of Spark marked the convergence of Scoop and iCollaborate, the respective online platforms of the Bank and the International Finance Corporation. It also unveiled a more robust, faster, and easier way to navigate a virtual forum for staff to find communities of practice, collaborate with practitioners across the Bank, and create an open space for senior management and staff to transparently discuss decisions associated with the Bank’s internal change process. The platform was launched on August 26, 2013. Over a 30-day period, Spark attracted 12,632 active users (those who viewed at least one document), including 2,824 participating users (who replied to posts) and 872 contributing users. Why is it useful? The tools and capabilities offered on Spark are helpful as they can improve the World Bank Group’s knowledge management and sharing potential and facilitate new avenues for staff to identify and access expertise, feedback, and experiences across the World Bank Group. Given the ease with which staff can create conversation threads on Bank issue areas, it has become a means through which staff members share knowledge and best practices on an array of operational issues. Other initiatives? CommunityFinder provides a directory of communities of practice, which assists staff in identifying online and offline communities and accessibly organizes key pockets of knowledge embedded at the Bank. SkillFinder enables staff to search the World Bank Group’s people pages for qualified skills, expertise, and specializations among staff featured in the Bank’s enhanced directory. TalentMarketplace features the latest on-the-job opportunities across the organization including cross support for programs and projects, corporate initiatives, innovative pilots, and opportunities related to fragile and conflict- affected states. References Andriopoulos, C., and M.W. Lewis. 2010. “Managing Innovation Paradoxes: Ambidexterity Lessons from Leading Product Design Companies.” Long Range Planning 43 (1): 104–122. Berg, Elliot. 2000. “Why Aren’t Aid Organizations Better Learners?” In Learning in Development Cooperation, edited by Jerker Carlsson and Lennart Wohlgemuth. Stockholm: Almqvist & Wiksell International. DEC (Development Economics Vice Presidency). 2012. Research at Work: Assessing the Influence of World Bank Research. World Bank: DEC. Eriksson, P.E. 2013. “Exploration and Exploitation in Project-Based Organizations: Development and Diffusion of Knowledge at Different Organizational Levels in Construction Companies.” International Journal of Project Management 31 (3): 333–341. 34 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION Garvin, David A., Amy C. Edmondson, and Gino Francesca. 2008. “Is Yours a Learning Organization?” Harvard Business Review 86 (3): 109–116. IEG (Independent Evaluation Group). 2008. Using Knowledge to Improve Development Effectiveness: An Evaluation of World Bank Economic and Sector Work and Technical Assistance, 2000–2006. Washington, DC: World Bank. ______. 2012a. World Bank Group Impact Evaluations: Relevance and Effectiveness. Washington, DC: World Bank. ______. 2012b. “An IEG Comparative Review Based on Project Performance Assessments of: Egypt Financial Sector Development Policy Loan (Loan No. IBRD-73910); Egypt Second Financial Sector Development Policy Loan (Loan No. IBRD-75280); Guatemala Financial Sector Adjustment Loan (Loan No. IBRD-71300); Morocco Financial Sector Development Policy Loan (Loan No. IBRD-73500); Pakistan Banking Sector Restructuring and Privatization (Loan No. IDA-35710); Banking Sector Development Policy Program (Loan No. IDA-40310 FSLT- 72700).” Project Performance Assessment Report 70030, World Bank, Washington, DC. ______. 2013. IEG’s 2012 Client Surveys: Topline Report of Key Findings. GlobeScan project 2557. London: GlobeScan. ______. 2014. Responding to Global Public Bads: Learning from Evaluation of the World Bank Experience with Avian Influenza 2006–13. Washington, DC: World Bank. Katila, R., and G. Ahuja. 2002. “Something Old, Something New: A Longitudinal Study of Search Behavior and New Product Introduction.” Academy of Management Journal 45 (6): 1183–1194. Lavie, D., and L. Rosenkopf. 2006. “Balancing Exploration and Exploitation in Alliance Formation.” Academy of Management Journal 49 (4): 797–818. March, J. 1991. “Exploration and Exploitation in Organizational Learning.” Organization Science (2):71–87. OED (Operations Evaluation Division). 1972. “Audit of Early Power and Telecommunications Lending to the Instituto Costariccense de Electricidad (Costa Rica).”October 20, 1972, World Bank, Washington, DC. O'Reilly, C. A., and M.L. Tushman. 2011. “Organizational Ambidexterity in Action: How Managers Explore and Exploit.” California Management Review 53 (4): 5. Oppenheimer, D., and L. Prusak. 2011. “Knowledge Management at the World Bank.” Case Number 1936. Case Program of Harvard Kennedy School, Cambridge, Mass. Ravallion, Martin. 2011. “Knowledgeable Bankers? The Demand for Research in World Bank Operations.” Policy Research Paper 5892, World Bank, Washington, DC. Secretariat to the Learning Board. 2013. Staff Learning: The State of Staff Learning at the World Bank— FY13 Annual Report. Washington, DC: World Bank. Swan, J., H. Scarbrough, and S. Newell. 2010. “Why Don't (or Do) Organizations Learn from Projects?” Management Learning 41 (3): 325–344. Uotila, J., M. Maula, T. Keil, and S. A. Zahra. 2009. “Exploration, Exploitation, and Financial Performance: Analysis of S&P 500 Corporations.” Strategic Management Journal 30 (2): 221– 231. Woolcock, Michael. 2013. “Using Case Studies to Explore the External Validity of ‘Complex’ Development Interventions.” Evaluation 19 (3): 229–248. 35 CHAPTER 2 KNOWLEDGE EXPLOITATION AND KNOWLEDGE EXPLORATION 1The World Bank surveyed staff perceptions of how well the organization was performing in 1997, 1999, 2002, 2005, 2007 and 2013. In addition, there was an Organizational Health Index survey in 2012, which was intended to provide a benchmark for the latest round of Bank reforms. 2The Organizational Health Index survey was conducted in October 2012. There were 6,450 respondents, which is a response rate of 55 percent. 3 “To the extent possible, lessons are sought as to how the lending activity might have been improved. The 'series' characteristic of the audit arises from the fact that the Bank has made a series of loans to the borrower and that the individual loans cannot be reviewed in isolation from one another” (OED 1972). 36 3. The Interpersonal Dimension of Learning: Connectivity and Teams Highlights  Interpersonal exchanges are the most important source of learning and knowledge sharing in the Bank: these exchanges are mediated by the networks to which people are connected and the teams in which they operate.  Interpersonal learning includes mentoring, a practice that is highly valued by Bank staff; there is now less mentoring than there used to be although attempts are now being made to revive it.  Network connectivity can be a powerful stimulus to learning; structural constraints on cross support and budget constraints on communities of practice suggest there is scope for improving connectivity in the Bank.  Recent Bank experiments with organizational network analysis have further highlighted the extent of connectivity constraints; they also suggest that this type of analysis may be used as a management tool to monitor knowledge flow.  Project teams are perceived to be sufficiently diverse, which may favor the introduction of new ideas, but the contribution of team members tends to be under acknowledged relative to task team leaders.  The handover between team leaders of projects is a source of learning discontinuity that the Bank has sought to address through learning events. What the Literature Says Abundant literature exists that indicates how connections to social networks and team dynamics influence learning. This may be described as the “interpersonal dimension” of learning, to distinguish it from the individualized learning that comes from perusing documents and databases as previously discussed. To a large extent interpersonal learning involves copying the behavior of others in the work group— whether it is a project team or a larger network. Research has found that learning from others is much more efficient than learning solely from one’s own experience (Rendell et al. 2010; Lazer and Friedman 2007; Glinton et al. 2010; Anghel et al. 2004; Yamamoto et al. 2013; Sueur et al. 2012; Farrell 2011). 1 Organizations often underestimate the importance of interpersonal exchanges as a source of learning, relative to learning through isolated study (Box 3.1). 37 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS Box 3.1. The Tendency for Organizations to Undervalue Interpersonal Learning “In India, one of the biggest problems I encountered was that researchers at each university were isolated from one another and therefore their research was stagnant and unproductive. People working in the same field, and sometimes even at the same university, had literally never met each other because the university administrators and the funding agencies thought it was sufficient to have the researchers read each other’s papers and that they didn’t need to travel to meetings or conferences. It was only when they began to meet and spend informal time together that new ideas began to bubble up and new ways of approaching problems began to spread” (Pentland 2014, viii–ix). The more connections that are established between people, the greater the scope for learning from others. Social network experiments show a learning curve that moves from low returns, when individuals largely work in isolation from each other, to high returns, when individuals interact more and there is an exchange of diverse ideas, the best being copied (Pan et al. 2012; Pentland 2014). 2 The key to productivity is ensuring that groups are diverse enough in composition, and open enough to knowledge from outside the group, to ensure a continuous competition between new and old ideas—resulting in the same process of pruning and strengthening that characterizes the development of the individual brain. 3 New ideas are more likely to arise in heterogeneous teams with different backgrounds and perspectives (Andriopoulos and Lewis 2010; Beckman 2006; Eriksson 2013; Lavie and Rosenkopf 2006; Lin et al. 2007). Teams that are good at exploring tend to contain highly diverse talents, including contrarian thinkers. These observations are borne out by research conducted at Bell Laboratories in the mid-1980s. Bell routinely recruited the best talent in the world but found that, between equally gifted individuals, there were large disparities in research productivity. They wanted to discover what made the difference between an average performer and a star. There were three findings from the research. First, the star performers were good at identifying, early on, the people with complementary talents that they would need at a later point. Second, the stars engaged continually and closely with the people in their networks, with the result that these people responded more quickly and helpfully when called on. Third, star performers’ networks were more diverse, containing individuals with widely ranging roles, the variation between which allowed the team to analyze any given problem from several viewpoints (Kelly 1999). In organizations like the Bank, much of the learning that takes place is generated within project teams. Teams need nurturing. Effective teams are not only diverse in composition they also exhibit high levels of trust between members, with the 38 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS contributions of each team member being acknowledged. The culture of the organization may influence the extent of knowledge sharing. In a project-based firm that creates a friendly, noncompetitive atmosphere at work based on participation, teamwork, and informality, teams are more likely to share knowledge with each other. Firms with a culture that emphasizes competition, achievement, and winning are likely to experience less knowledge sharing and may even inadvertently encourage teams to hoard rather than share information with other teams (Wiewiora et al. 2013, 1,170; De Long and Fahey 2000; Janz and Prasarnphanich 2003; Pilbeam 2013; Bartsch et al. 2013; Han and Hovav 2013). The interpersonal dimension of learning reflects the way in which the human brain evolved. Kahneman (2010) distinguishes two ways of thinking: one mode is fast, automatic, and largely unconscious (characteristic of the brain that primates needed to survive predators on the savanna); and the other mode is slow, rule-based, and largely conscious (typical of the brain needed to function in a bureaucratic society). Fast thinking relies on habits and intuitions. It involves making associations between personal experiences and those gained by observing others. Slow thinking uses reasoning, combining beliefs to reach new conclusions. People use both types of thinking—both are needed, and neither is superior. It may be that in project-based organizations where time for learning is limited and project deadlines are always pending, the staff’s learning style may be more inclined to fast thinking, which may reduce the demand for training and learning events geared to developing slow- thinking skills, particularly on the part of the oldest and most experienced staff. There may be a case for paying more attention to ways of enhancing fast brain skills through tweaking the networks and the social interactions that influence learning— particularly learning by task team leaders (TTLs). What the Bank’s Evidence Shows This section presents evidence on the importance of interpersonal learning in the Bank and examines three aspects. First, members of task teams learn, largely, by studying and selectively mirroring the behavior of colleagues. Mentoring is a powerful stimulant to this type of learning, and there is a strong, partially unmet, demand for it in the Bank. Second, the connection to networks strongly influences knowledge flow. The Bank’s experience with cross support and communities of practice is illustrative of how this connectivity may be thwarted. Recent experiments with using organizational network analysis as a managerial tool for enhancing knowledge flow are a promising new departure, although the results are still preliminary. Third, the integrity and effectiveness of the project team may be 39 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS compromised when members’ input is not acknowledged, when there is an abrupt change of task team leader, and when the team composition is insufficiently diverse. EMPLOYEES MAINLY LEARN BY DOING AND TALKING TO OTHERS According to one study, people are five times more likely to likely to turn to a coworker rather than a book, a manual, or a database (Davenport and Prusak 1998). TTLs seem to be more likely to behave this way than non-TTLs. A number of TTLs and senior managers told the Independent Evaluation Group (IEG) that the learning process in projects is implicit and leaves little trace in project documents, an observation borne out by an examination of documents from the project cohort selected for this study (Appendix D). A director summed it up nicely. Part of this implicit, or tacit, learning involves an intuitive sense of what works, and TTLs’ effectiveness depends on their ability to rapidly filter out information that is not directly relevant to the operational task at hand. This process may be verbalized, but it is likely to be only partially written down—it may be totally undocumented. This learning through doing would appear to draw on the fast thinking part of the brain as opposed to the reading and inward reflection of slow thinking (Kahneman 2010). When the IEG survey asked Bank staff how they learned in the course of lending, the largest group of responses corresponded to learning by doing (87 percent) followed by person-to-person conversations (83 percent), with a statistically significant difference between TTLs and non-TTLs. Participants in IEG focus groups and interviews stressed that documents of any sort are a less significant source of knowledge than what is picked up from talking to people—mainly fellow staff members but also clients. This is partly because of time pressure. It is quicker to obtain vital information by asking an expert than it is to scan the entirety of a document, and the expert’s knowledge may be more up to date. MENTORING IS AN IMPORTANT ASPECT OF INTERPERSONAL LEARNING The Bank’s staff is persuaded that mentoring is one of the best ways to promote learning and knowledge sharing. In IEG’s survey of Bank staff, 56 percent of respondents reported that learning and knowledge sharing had occurred through mentoring and coaching by experienced staff to a very large or substantial extent. Although managers rated mentoring most highly as a learning source, the differences between staff groups in this respect were not statistically significant (Figure 3.1). Impressions were much less positive when respondents were asked whether new staff received sufficient mentoring. The implication is that respondents—in all staff groups—perceive that they learned more through mentoring than recent recruits. The shortage of mentoring opportunities today compared to the past is perceived as critical. Only 14 percent of respondents agreed 40 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS or strongly agreed that new staff have sufficient operational and technical experience to operate effectively as TTLs. Figure 3.1. To What Extent Has Learning Occurred through Doing and by Talking to Others? Non-TTLs Person-to-person conversations** TTLs Learning by doing* 78 80 82 84 86 88 90 92 94 96 Percentage of responses: to a "very large" or "substantial" extent Source: IEG survey of Bank staff conducted for this evaluation. This message was reinforced in the interviews and focus groups conducted by IEG, with strong support being voiced for mentoring (Figure 3.2). Interviewees observed that mentoring has taken a variety of forms at the Bank. Some participants noted that when they joined the Bank a formal mentoring program was in place. Others noted that they worked for years as part of a collaborative team, where they were given a chance to take on some responsibility but not expected to do everything on their own. Only after they had substantial experience were they allowed to take over supervision and after that given a project to design. Others described co-TTL arrangements or operating as a “shadow TTL.” And some were informal arrangements. “My mentors were all informal.… All of them went to the field with me.” Whether mentoring is formal or informal, what counts is the opportunity for novices to work side by side with seasoned TTLs (particularly on missions). One participant said that the people skills that mentoring fosters are perhaps more important than the technical skills. “You need knowledge of how to bring people together, build trust with the client.… You get this through mentoring.” To work well, mentors have to be vetted, not randomly paired off with new staff. Participants stressed that whatever form it takes, mentoring is only effective if it has strong management support and a dedicated budget. 41 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS Figure 3.2. To What Extent Is Mentoring an Important and Adequate Source of Learning? Non-TTL TTL GG Adequate for new staff Importance GI-GK 0 20 40 60 80 100 Percentage who said that learning through mentoring was (a) important and (b) adequate for new staff Source: IEG survey of Bank staff conducted for this evaluation. Interviewees told IEG that problems arise in the absence of mentoring. If a TTL is launched without adequate preparation, he or she may seek to compensate for their lack of skills by hiring consultants; but unless they are former Bank TTLs, consultants will only bring technical knowledge to bear, not the operational know- how that is needed to deliver projects that perform well. Novices are sometimes recruited as TTLs before they are ready. Some interviewees observed that this tends to happen more in country offices where staff insist on becoming TTLs, but have neither the international experience to draw on nor the network of contacts at headquarters. Most important, they lack the opportunities to receive mentoring. When they fail, they are dropped as TTLs, but they had been setup for failure. The Operational Core Curriculum course, which was formally rolled out in FY14, now requires staff to undergo training before they can become a TTL, but the effectiveness of this measure is yet to be evaluated. Mentoring is generally perceived to have fallen by the wayside, partly because the budget for supervision missions is too tight to accommodate both the TTL and a fledgling. Interviewees told IEG that managers need to invest in mentoring. However, managers can’t mentor when they have more than 25 people reporting to them—the span of control (e.g., in the Sustainable Development Network [SDN]) is too great. Also, the pool of people suitable to serve as mentors is now very shallow. Many of the more seasoned staff have left during the Bank’s previous episodes of restructuring. 42 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS GREATER CONNECTIVITY TO PROMOTE KNOWLEDGE FLOW In the past, the Bank sought to promote knowledge flow across the organization by encouraging exchange of expertise across regions (cross support) and by building communities of practice. The new Global Practices can learn from studying the strengths and weaknesses of these previous initiatives. Cross support was constrained by structural boundaries. Cross support involves the ad- hoc purchasing of staff time between different Bank units. It is intended to promote the temporary sharing of knowledge and skills between individuals located in different parts of the Bank for a specific task, often lending preparation or implementation. Staff value cross support as a source of knowledge sharing and learning but the opportunities for it are more limited than they would like. In IEG’s survey of Bank staff, 52 percent of respondents reported that learning and knowledge sharing had occurred through cross support to a very large or substantial extent. But there was a statistically significant difference between the proportion of TTLs who responded this way (49 percent) and non-TTLs (59 percent). It may be that non-TTLs rated cross support more highly because they do not use it themselves and therefore are unaware of the significant constraints. IEG’s evaluation of the Bank’s matrix organization found “the flow of knowledge through cross support has been rather limited, is declining in volume, and is constrained by structural boundaries” (IEG 2013a, 62). First, inter-vice-presidential- unit cross support had fallen; as a percentage of staff time; it declined from 7.6 percent in FY02 to 5.2 percent in FY10. Second, the percentage of cross support from Network anchors to the Regions halved over the same period, dropping from 23 percent to 12 percent of Network anchor staff time. Third, the incentive of staff to prioritize operational services in their own Region reduced the amount of cross support from one Region to another, creating strong regional silos. Fourth, cross support from the Network anchors to the Regions and from the Regions to the Network anchors occurred within network silos. There was virtually no cross support from SDN sector units to sector units of other networks and not much cross support among sector units across other network boundaries. Communities of practice were squeezed in the past by resource constraints. Communities of practice (or thematic groups) facilitate learning and knowledge sharing by: (i) providing a safe space for people to share tacit knowledge; (ii) helping people gain access to new ideas and methods, inside and outside the organization;(iii) reducing the time needed to disseminate knowledge; (iv) discouraging the reinvention of the wheel and the repetition of mistakes; and (v) generating innovation (Wenger et al. 2009). 43 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS In the Bank, communities of practice had their moment in the late 1990s when they were funded under the Strategic Compact, but staff report that they are not now among the most important sources of learning. In IEG’s survey of Bank staff, 26 percent of respondents indicated that learning and knowledge sharing had occurred through communities of practice to a very large or substantial extent, and there is no statistically significant difference between respondent groups (TTLs and non-TTLs, and staff at different grades). However, a higher proportion of staff (39 percent) said that brown-bag lunches (BBLs), conferences, and workshops supported learning to a very large or substantial extent. Since these events are often sponsored by communities of practice, the role of these groups may be more important than the response to the question about communities of practice suggests. In the IEG interviews and focus groups, there was a strong sense that the original thematic groups varied widely in terms of level of activity and effectiveness. It was observed that these groups have become less active over time. They were thought to have been a vital source of learning in the late 1990s, but then their budgets were cut and many went into decline (Box 3.2). Some interviewees noted that there are insufficient incentives for TTLs to participate in thematic groups. One respondent said, “The personal cost for me to attend a seminar or go to a BBL is very high. I simply don’t have time to do it.” Box 3.2. A View on the Demise of Thematic Groups During an interview, a task team leader (TTL) told IEG that the Water Resources Thematic Group had greatly contributed to knowledge transfer when it had the resources to do so. “In 2010, the thematic group had a budget of $60,000 and developed a huge program. We brought in people from outside the Bank and did a study tour of the Chesapeake Bay. We had a program for young staff that paid the cost for them to join a mission in another region with an experienced TTL. We also organized to bring experienced retirees who had been out of the Bank for a while to come back and talk about their experience. That year was a lot of fun. The next year the thematic group got zero for its budget.” Older evidence bears out the findings that lack of dedicated budget and time has hampered the effectiveness of the communities of practice. A 2009 Bank survey received responses from 57 of the 120 listed communities of practice. It found that the most important functions supported by these groups were brokering knowledge (84 percent of responses), gathering appropriate practices and lessons (61 percent), and supporting lending-oriented tasks (28 percent). The most frequently cited service was the BBL. Of the communities of practice that reported, the primary source of funding in FY09 was Bank budget, but 47 percent had received no budget in the previous year. The lack of a dedicated budget, the difficulty of finding the time to participate, and the lack of connection between participation and staff 44 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS performance assessment or promotion were cited as the main obstacles to the promotion of communities of practice. To the extent possible, the second evaluation will examine the communities of practice that staff perceive to be working well; for example, those dealing with social safety nets and land administration. Organizational network analysis may be a tool for improving knowledge flow. Access to the knowledge needed to prepare and implement projects, and the scope for learning in the project process, are influenced by the organizational networks in which staff are embedded. The number and type of connections between the members of a given network—a Global Practice, for example—exercise a powerful influence on the flow of ideas. The Bank has recently conducted two experiments with organizational network analysis the findings from which have important implications for the process of learning and knowledge sharing. Case 1: The Finance and Private Sector Development Network The Finance and Private Sector Development (FPD) network was charged with piloting the Global Practices that are now being introduced Bank-wide. FPD conducted an organizational network analysis before and after the pilot was launched to assess the impact of the new arrangements on staff connectivity and knowledge flow. The first network analysis was conducted in March 2011 and the second in October 2012—15 months after launch of the FPD Global Practice (FPD 2013). More than 80 percent of FPD staff participated in both surveys. The surveys measured (i) the total number of connections among FPD staff and (ii) the number of steps needed for one person—the knowledge seeker—to reach another person, the knowledge provider. To reach the knowledge provider, the seeker typically had to deal with several other people on the way, by face-to-face meetings, phone calls, or e-mail exchanges. Each of these steps was counted. Before the pilot was launched the number of steps between seeker and provider averaged 3.6. Eighteen months later the following changes were observed: the total number of connections among FPD staff had increased by 44 percent, and the average number of steps decreased from 3.6 to 3.2. Between the two surveys, the number of isolated staff members who had no connections within their own part of FPD fell sharply. Links between the Network anchor and the regions increased. Overall, members of the FPD network became better connected. The network analysis identified the top three activities that enhanced connectivity as cross support and joint work, Global Practice-oriented events, and learning events. However, integrating new employees remains a slow process. The first survey found that it took more than three years for newcomers to become well integrated in the 45 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS network, as measured by the time it took them to reach the average level of ties within the network. The second survey found that newcomers still take more than three years to become well integrated. The second round of analysis was also used to identify areas of expertise (subpractices) where there is an imbalance between the demand for and the supply of knowledge. The largest imbalance was for the Competitive Industries: Economic Zones subpractice, where the number of self-identified experts was 13 and the number of seekers was 25. To increase network integration, data on staff expertise was shared through an expertise locator on the FPD website, helping seekers to contact experts. Echoing the findings of IEG’s matrix evaluation, a review of the FPD pilot by Deloitte LLP found weak alignment between the country and the sector sides of the matrix. The consulting firm recommended that Bank leaders assume responsibility for correcting this imbalance, which had left the country dimension of the matrix stronger than the sector side. Case 2: The Energy and Mining Family Following the lead of FPD, the Bank’s Energy and Mining Family (EMF) subsequently conducted its own organizational network analysis, with 70 percent of employees participating in a survey intended to map where people obtained the knowledge they needed to do their job. Focusing on links between knowledge providers and knowledge seekers, EMF averaged 10.1 connections per knowledge provider, higher than FPD before its pilot (9.4) but lower than FPD post-pilot (13.5). The range of such provider–seeker connections in EMF was huge: from 0 to 63. The survey highlighted the extent of skill gaps and pointed to the potential overloading of the most connected knowledge providers. The imbalance between the supply of and demand for technical expertise was measured for each of 10 EMF communities of practice, with the largest gap found for the energy efficiency and renewable energy practices. People often turned to the most convenient source of advice, or colleagues they had consulted in the past, rather than the best-qualified expert. As a result, important expertise went untapped and the most connected experts were overburdened. Most of the experts doing the connecting were at grades GH and above and had over 20 years of Bank experience. Those at GH and above accounted for 22 percent of staff but were responsible for 40 percent of all connections. Planning for the retirement of more senior staff and taking steps to debrief them before they leave are obvious implications if the inevitable loss of knowledge and acquired learning is to 46 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS be contained. Replacement of expertise is far from straightforward. As was the case in FPD, this survey found that it takes a long time for new recruits to become fully integrated in the family—ranging from 6 to 10 years for entrants to EMF. Collaboration between units in the family was limited by the silo effect described by several sources, including IEG’s 2012 matrix evaluation, with the demand for greater connectivity most strongly expressed by respondents in the Africa and the Middle East and North Africa Regions. The survey found substantial variation between Bank units in the average number of times that experts connected with other people in the family who were seeking their advice: comparing regions, Latin America and the Caribbean was the most connected (Figure 3.3). The unit identified by the survey as most in need of interaction with other work units in the family was the World Bank Institute. Also, the network analysis found that the staff in country offices was isolated from knowledge and learning networks relative to headquarters staff. People tended to connect with others who were recruited into EMF by the same route, with little outreach by headquarters staff to locally hired staff in country offices. Two caveats apply to both cases of organizational network analysis. First, the surveys did not distinguish between the different types of knowledge that people were typically seeking and providing (technical, process, interpersonal, and country). This type of information would have helped to identify misuse of expert’s time. A person may be both a technical and an operational expert (competitive advantage in both), but if there are relatively few technical experts, it would be better to use his or her time on technical questions and force people to go to others for the operational expertise (comparative advantage). Second, integration of newly recruited staff may lead to attrition of the distinctive knowledge they brought from outside the Bank. In the three to five years that it takes for new recruits to become “institutionalized” in the Bank, they may lose the cutting-edge knowledge they brought with them. Thus, to the extent that slow onboarding temporarily insulates this distinctive external knowledge, it may be a plus not a minus—contrary to what is suggested in the write-up of these two cases of network analysis. 47 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS Figure 3.3. Connectivity in the Energy and Mining Family Varies across the Bank Average number of connections per knowledge provider to knowledge seekers Source: Reprinted from Gray and Cross (2013). Note: AFR = Africa Region; DEC = Development Economics; EAP = East Asia and Pacific Region; ECA = Europe and Central Asia Region; FEU = Finance, Economics, and Urban Development; IFC = International Finance Corporation; MIGA = Multilateral Investment Guarantee Agency; MNA = Middle East and North Africa Region; SAR = South Asia Region; SEG = Sustainable Energy Department; WBI = World Bank Institute. Vital Importance of Team Composition and Integrity The relative contribution of team leaders and members needs to be appreciated. Interviewees told IEG that, in the words of one director, the best TTLs are heroic enablers, capable of “bringing it all together.” They do not need to be expert in any of the particular execution or technical disciplines that the project design calls for. But they do need to know where to find the right experts and how to create and (more important) sustain the nurturing environment that is needed for team members to work together fruitfully. The TTL was not always the prime mover. Before 1987, a technical team would prepare projects while specialized loan officers would be in charge of negotiations. The 1987 reorganization changed this, putting the TTL at the center of project delivery. Contrary to earlier practice, the TTL would negotiate the loan, which would increase his authority and his inclination to identify with a quality product and make best use of resources. In 1992, the working group on information technology that contributed to the Wapenhans report noted that the quality of 48 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS project supervision is closely linked to the experience and dedication of the TTL. The group concluded that the increasing use of relatively inexperienced staff as TTLs was one of the major contributors to supervision shortcomings. World Bank research has shown a strong association between project outcome ratings and the identity of the TTL. Some TTLs work on projects that are consistently rated more highly than projects led by other TTLs. Moreover, “task team leader fixed effects are of comparable importance to country fixed effects in accounting for the variation in project outcomes” (Denizer et al. 2011, 2). However, this research was not able to sort out the effect of individual TTL attributes, such as education and experience, from the effect of team attributes. Maybe the TTLs of “satisfactory” projects were not individually more gifted than their less successful peers; they simply had better teams to work with. Teams allow for the pooling of ideas, and research has shown that pooling results in an average “wisdom of the crowd” judgment that is better than the judgment of individual team members, however gifted the individuals may be (Surowiecki 2004). When the Bank was reorganized in 1987, there was a keen awareness of the need for project teams to back up the TTL, based on previous experience in the projects divisions, which were characterized by a strong team spirit. It was emphasized that task teams needed to have the right skills mix and experience, and that there should be a TTL backup—a permanent staff member who could take over if the TTL moved on. Participants in IEG interviews and focus groups observed that the most effective TTLs are those capable of mobilizing a diverse team, with members whose skills complement, rather than substitute or duplicate, the skills of the TTL. Questionnaire respondents and interviewees told IEG that budget cuts and the persistence of a “compliance culture” undermine team diversity. When budgets are tight there is less scope for contracting technical experts in general or the most talented in particular. First priority is always given to the specialists in safeguards, procurement, and financial management. One interviewee told IEG,“I have plenty of examples of technical specialists that I haven’t been able to bring on mission to deal with an issue that needs attention because there is no money in my supervision budget after hiring fiduciary, procurement, safeguard specialists, etc.” Others confirmed that on supervision missions “compliance specialists” tend to crowd out sector and thematic specialists. But two survey data sources do not bear out the impression that diversity is lacking. In the 2013 Employee Engagement Survey, 72 percent of Bank employees strongly agreed or agreed that “my work group has a climate in which diverse perspectives are valued”—compared to 73 percent in the 2009 survey. In 2013, 69 percent of TTLs 49 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS responded strongly agree or agree. Although this question did not refer specifically to the diversity of perspectives in project teams, the experience of these teams probably colored the response to the question, particularly for TTLs. IEG posed the question in more precise terms. Its 2014 staff survey found that almost two-thirds of respondents (62 percent) regard project teams to be diverse, encompassing individuals with different perspectives. This perception was more frequently held by country-based TTLs than TTLs located at headquarters (Figure 3.4). Figure 3.4. Perceptions of Project Teams "Project teams comprise individuals with diverse perspectives."* "There has been no real team work…teams rarely meet." TTL, CO TTL, HQ "The Bank rewards the whole project team, not just the TTL." 0 10 20 30 40 50 60 70 80 90 Percentage of responses: "strongly agree" or "agree" Source: IEG survey of Bank staff 2014. Note: CO = country office; HQ = headquarters; TTL = task team leader. *p = 0.00. With respect to the distribution of rewards within the team, about 40 percent of the respondents strongly agreed or agreed with the statement that good performance by the whole team is recognized, not just the performance of the TTL. There was no significant difference in this respect between TTLs at headquarters and those in country offices. In terms of team cohesiveness, the message from the IEG survey was relatively positive. Less than one-third of respondents strongly agreed or agreed that there was a lack of teamwork. However, in interviews and focus groups, several people pointed out that team integrity had eroded over time. They noted that past projects were staffed with larger teams, and the full team went on mission for 10 days to two weeks. As budgets tightened missions became shorter, and it was rare to have the full team together. Increasingly, only the TTL is there for the full length of the mission. Interviewees observed that the rest of the team tends to fly in for a day or two at different times, each working in separate cells. This was attributed in part to 50 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS budget constraints, but it also has to do with availability. Everyone is doing more with less, and this adds demands to everyone’s time and makes it difficult to schedule people at the same time. STAFF ROTATION MAY UNDERCUT TEAM INTEGRITY AND KNOWLEDGE FLOW There is a potential knowledge trade-off in moving staff. On the one hand it can imply bringing new knowledge and a fresh perspective to a different project or part of the organization. On the other, it tends to remove acquired knowledge from site where it was generated. Knowledge flow and the transmission of learning may be disrupted by the lack of a proper handover between incoming and outgoing TTLs of a particular project. Because it is unusual for a single person to follow the same operation from start to finish, a holistic perception of the factors responsible for success or failure is generally lacking. This problem was evident some years ago: “Different people appraise, design, implement, supervise, and evaluate. Everybody presides over someone else’s project, which will be evaluated by a third person—an arrangement not conducive to effective learning” (Berg 2000, 38). Rapid staff turnover may lead to learning being forgotten—and even to the resuscitation of bad ideas. The handover challenge has been addressed by other organizations. In 2003, Kaiser Permanente introduced the Nurse Knowledge Exchange, which was intended to improve how nurses exchanged patient information between shifts at hospitals. New software was developed to help nurses compile and share complete shift information in a standard format (McCreary 2010). The U.S. Agency for International Development arranges for the systematic debriefing of staff moving to new assignments or leaving the organization. Interviewees told IEG that there is no systematic attempt in the Bank to ensure that incoming TTLs are properly briefed by the people they take over from, and it is not necessarily the case that the new and the former TTL will do at least one mission together. There are instances though where the new TTL may benefit from going on his first mission without the outgoing TTL, especially if the client relationship had been a difficult one under the outgoing TTL. However, even here, the new TTL will benefit from a full briefing from the outgoing TTL so he or she can learn from the previous experience and avoid any mistakes made. One interviewee observed, “managing this transition is the biggest [learning] challenge.” One of the responses to the IEG survey of Bank staff had some bearing on TTL rotation. Only 15 percent of respondents reported that learning occurred to a very large or substantial extent because of handover notes or exit interviews with staff. The absence of systematic briefings of incoming TTLs is corroborated by the outcome of a learning-from- 51 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS failure event, involving 23 TTLs and a sector manager, which the Bank organized in April 2013. This event graphically illustrated the problems arising from TTL handover (Box 3.3). In the course of the event, participants came up with issues such as the time gap between the outgoing and incoming TTL and the outgoing and incoming manager, and the sector manager or past TTL’s unwillingness to acknowledge issues or failures as factors likely to result in a difficult transition between TTLs. Box 3.3. Task Team Leader Rotation Can Be Painful A participant at a learning-from-failure event organized by the Bank in April gave the following account of a particularly challenging task team leader (TTL) handover: “The project was a complex corporate risk project. The TTL was new to the Bank and did not speak the client’s language. The relationship with the client was at a very sensitive moment—actions the government had taken were jeopardizing the Bank’s continued involvement. A verbal agreement had been reached between Bank managers and client counterparts. At the same time, almost the entire management changed (the VP, sector director, country director, and sector manager). The former TTL had already been assigned and was in country on a new project, leaving a gap of supervision for several months. The new TTL, who was from a different sector, took the project to the Board without having a single conversation with the former TTL. As the former TTL described it, ‘It’s your baby and it’s been several years of your life and you really want to make sure the right thing gets done.’ This is a really extreme case. Everything that can go wrong went wrong in this one project. But I think it’s important to learn the lessons from this” (Darling 2013). In the IEG survey of Bank staff, about 30 percent of all staff reported that the 3-5-7 rule for staff rotation supported or greatly supported learning in lending, while 17 percent held the view that it hindered learning and another 21 percent felt it had no impact. High-ranking staff (grade GI and above) thought that staff rotation was contributing to learning and knowledge sharing to a greater extent than grade GG staff, and the difference was statistically significant. Whereas only 28 percent of the latter said that rotation contributed to learning to a very large or substantial extent, 64 percent of managers reported this. Looking Ahead The literature indicates that a considerable part of learning occurs through interpersonal exchanges, involving mirroring and copying of behavior. Mentoring is one way to promote this exchange. There are some promising indications that the mentoring deficit is now being addressed. In the Africa Region, there has been a recent move to appoint co-TTLs. Bank-wide, a youth mentoring program is booming (Box 3.4). 52 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS The new Global Practices will remove some of the structural barriers to knowledge flow that previously impeded cross support, and they will have the dedicated, predictable budget that past communities of practice often lacked, although the size of this budget (relative to the Regions) remains to be defined. There are some indications that organizational network analysis along the lines promoted by FPD and EMF could serve as a management tool. By identifying the primary knowledge providers in a network and the areas where providers are overloaded with demands, managers may plan for recruitment and retirement more effectively by anticipating the threat to the effectiveness of the network posed by the removal of key players. However, the surveys required to support this analysis are expensive (around $120,000 by one FPD estimate) and time consuming. The two networks that carried out the work have created useful benchmarks but it remains to be seen if the budget will be found to allow for regular updating. Also, more work is needed to establish what implications greater connectivity has for the quality of knowledge sharing and learning that takes place. IEG will address this in the next evaluation. With respect to the effectiveness of project teams, it may be that different types of projects call for different levels of team diversity. Blueprint projects (typical of the energy and infrastructure sectors where approaches are tried and tested) may be more likely to benefit from homogenous teams acting cooperatively, whereas projects requiring experimentation and innovation would perhaps benefit from heterogeneous teams where team members bring different perspectives. This idea merits investigation. Matching team characteristics to project characteristics may help to promote learning and knowledge sharing. The Bank’s reward system will also affect the team dynamic. If the TTL rather than the entire team gets all or most of the recognition, the cohesiveness of the team and collaborative spirit will suffer. It remains to be seen whether the Bank’s budget cutbacks can be implemented without compromising team diversity or the proper reward of team efforts. It will help if the effects of employee turnover can be better anticipated. One implication is that it is not enough to prepare training programs individual by individual, because individuals rotate. The learning plan should perhaps embrace the collectivity, the social network in which individual TTLs are embedded. A future learning agenda at the Bank must go beyond a focus on preparing programs for individuals and pursue initiatives that build collective capabilities to implement. 53 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS Box 3.4. Mentoring World Bank Youth What is the Youth Mentorship Program? The goal of the World Bank Group's Youth-to-Youth (Y2Y) Mentorship Program is to facilitate interaction between young staff and more experienced senior staff, and provide a mechanism for professional support, informal learning, and guidance. The number of young staff enrolled in the program rose from 123 in FY13 to 184 members in FY14. The mentors include staff with varying levels of seniority that are mapped across regions, sectors, and institutions of the Bank. There are currently 87 mentors. How does the program work? The mentors are approached by the Y2Y Mentorship Program and those who agree to meet with one or more mentees in person or virtually on a periodic basis are selected. They are then assigned a mentee(s) according to sector and regional experience as well as career interests. Upon joining the program, mentors and mentees are provided with a “Y2Y Mentoring Handbook” and encouraged to define a professional development strategy at the beginning of the program. Following an initial meeting, pairs agree to set-up bimonthly virtual or in-person meetings and report feedback and progress to the Y2Y Mentorship team over the course of the fiscal year. Upon completion of the formal requirements of the program, both mentors and mentees are asked to submit their feedback via a survey. To encourage senior staff to share their experiences, the Y2Y Mentorship team organizes professional development and skill-building workshops, featuring mentors and senior development experts from across the Bank. How do the mentees rate it? IEG randomly contacted 9 youth (junior professional associate, short-term consultant, extended-term consultant) at the Bank – of them, four had heard of Y2Y’s Mentorship Program and had a largely positive experience. Two youth had heard of it but felt that the program (or the Mentors) didn’t meet expectations. Two youth contacted had heard of it and would be interested in learning more about the program. One youth had not heard of the Mentorship Program but felt that Mentorship should be a priority at the Bank. Of those who had had a largely positive experience, multiple youth said that the Mentorship Program represented one of the most helpful initiatives launched by Y2Y. Although some of the mentees mentioned gaps in areas of expertise between themselves and their mentors, each of the mentors was perceived to be fully committed to the program and responsive to mentee needs. In many cases, mentors agreed to outline the main objectives of the partnership in advance and to provide mentees with advice on the working culture at the Bank. Multiple mentees mentioned that meeting with their mentors soon after joining the Bank contributed to informal learning and their acclimatization to the Bank. What are the challenges ahead? Reaching country offices has been a problem. Although the program is offered to staff across the institution, a substantial portion of participants are located at headquarters. Some youth have not heard about the program, indicating the need for more systematic outreach and communication. Some think that offering the program on a rolling basis would help because they had joined the Bank after the program deadline and the program had then not been available to them. 54 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS References Andriopoulos, C., and M.W. Lewis. 2010. “Managing Innovation Paradoxes: Ambidexterity Lessons from Leading Product Design Companies.” Long Range Planning 43 (1): 104–122. Anghel, M., Z. Toroczkai, K. Bassler, and G. Korniss. 2004. “Competition in Social Networks: Emergence of a Scale-Free Leadership Structure and Collective Efficiency.” Physical Review Letters 92 (5): 058701. Bartsch, V., M. Ebers, and I. Maurer. 2013. “Learning in Project-Based Organizations: The Role of Project Teams’ Social Capital for Overcoming Barriers to Learning.” International Journal of Project Management 31 (2): 239–251. Beckman, C.M. 2006. “The influence of Founding Team Company Affiliations on Firm Behavior.” Academy of Management Journal 49(4):741–758. Berg, Elliot. 2000. “Why Aren’t Aid Organizations Better Learners?” In Learning in Development Cooperation, edited by Jerker Carlsson and Lennart Wohlgemuth. Stockholm: Almqvist & Wiksell International. Darling, Marilyn J. 2013. Learning from Failure: TTL Handovers Meeting Report. Washington, DC: World Bank and Fourth Quadrant Partners. Davenport, T.H., and L. Prusak.1998. Working Knowledge. Boston: Harvard Business School Press. De Long, D.W., and L. Fahey. 2000. “Diagnosing Cultural Barriers to Knowledge Management.” Academy of Management Executive 14 (4): 113–127. Denizer, C., D. Kaufmann, and A. Kraay. 2011. “Good Countries or Good Projects? Macro and Micro Correlates of World Bank Project Performance.” Policy Research Working Paper WPS5646, World Bank, Washington DC. Eriksson, P.E. 2013. “Exploration and Exploitation in Project-Based Organizations: Development and Diffusion of Knowledge at Different Organizational Levels in Construction Companies.” International Journal of Project Management 31 (3): 333–341. Farrell, S. 2011. “Social Influence Benefits the Wisdom of Individuals in the Crowd.” Proceedings of the National Academy of Sciences 108 (36): E625. FPD (Finance and Private Sector Development). 2013. “Organizational Network Analysis,” Product and Practice Overview, January 2013, World Bank, Washington, DC. Glinton, R., P. Scerri, and K. Sycara. 2010. “Exploiting Scale Invariant Dynamics for Efficient Information Propagation in Large Teams.” Proceedings of the Ninth International Conference on Autonomous Agents and Multiagent Systems 1: 21–30. Gray, Peter, and Rob Cross. 2013. “Energy and Mining Family Organizational Network Analysis.” Slide presentation, June 25, 2013, World Bank, Washington, DC. Han, J., and A. Hovav. 2013. “To Bridge or to Bond? Diverse Social Connections in an IS Project Team.” International Journal of Project Management 31 (3): 378–390. IEG (Independent Evaluation Group). 2013. The Matrix System at Work: An Evaluation of the World Bank’s Organizational Effectiveness. Washington, DC: World Bank. “In the beginning was the word.” 2014. The Economist, February 22.http://www.economist.com/news/science-and-technology/21596923-how-babbling- babies-can-boost-their-brains-beginning-was-word. 55 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS Janz, B.D., and P. Prasarnphanich. 2003. “Understanding the Antecedents of Effective Knowledge Management: The Importance of a Knowledge-Centered Culture.” Decision Sciences 34 (2): 351–384. Kahneman, D. 2010. Thinking, Fast and Slow. New York: Farrar, Straus, Giroux. Kelly, R. 1999. “How to be a Star Engineer.” IEEE Spectrum 36 (10): 51–58. Lavie, D., and L. Rosenkopf. 2006. “Balancing Exploration and Exploitation in Alliance Formation.” Academy of Management Journal 49 (4): 797–818. Lazer, D., and A. Friedman. 2007. “The Network Structure of Exploration and Exploitation.” Administrative Science Quarterly 52 (4): 667–94. Lin, Z., H. B. Yang, and I. Demirkan.2007. “The Performance Consequences of Ambidexterity in Strategic Alliance Formations: Empirical Investigation and Computational Theorizing.” Management Science 53(10):1645–1658. McCreary, Lew. 2010. “Kaiser Permanente’s Innovation on the Front Lines.” Harvard Business Review 88 (9): 94–97, 126. Pan, Wei, Yaniv Altshuler, and Alex (Sandy) Pentland. 2012. “Decoding Social Influence and the Wisdom of the Crowd in Financial Trading Network.” In 2012 International Conference on Privacy, Security, Risk, and Trust and 2012 International Conference on Social Computing, 203–209. Institute of Electrical and Electronics Engineers, 2012. Pentland, A. 2014. Social Physics: How Good Ideas Spread—The Lessons of a New Science. New York: Penguin Press. Pilbeam, C. 2013. “Coordinating Temporary Organizations in International Development Through Social and Temporal Embeddedness.” International Journal of Project Management 31 (2): 190– 199. Rendell, L., Boyd, D. Cownden, M. Enquist, K. Eriksson, M. W. Feldman, L. Fogarty, S. Ghirlanda, T. Lillicrap, and K.N. Laland. 2010. “Why Copy Others? Insights from the Social Learning Strategies Tournament.” Science 328 (5975): 208–13. Sueur, C., A. King, M. Pele, and O. Petit. 2012. “Fast and Accurate Decisions as a Result of Scale-Free Network Properties in Two Primate Species.” Proceedings of the Complex System Society (January). Surowiecki, J. 2004. The Wisdom of Crowds. New York: Random House. Wenger, E., R. McDermott, and W.M. Snyder. 2009. Cultivating Communities of Practice A Guide to Managing Knowledge. Boston: Harvard Business School Press. Wiewiora, A., B. Trigunarsyah, G. Murphy, and V. Coffey. 2013. “Organizational Culture and Willingness to Share Knowledge: A Competing Values Perspective in Australian Context.” International Journal of Project Management 31: 1163–1174. Yamamoto, S., T. Humle, and M. Tanaka. 2013. “Basis for Cumulative Cultural Evolution in Chimpanzees: Social Learning of a More Efficient Tool-Use Technique.” PLoS ONE 8 (10): e55768; doi: 10.1371/journal.pone.0055768. 1Mathematical models of learning in complex environments suggest that the best strategy for learning is to spend 90 percent of time on finding and copying others who appear to be 56 CHAPTER 3 THE INTERPERSONAL DIMENSION OF LEARNING: CONNECTIVITY AND TEAMS doing well. The remaining 10 percent should be spent on individual experimentation and thinking things through (Rendell et al. 2010). 2A high level of interactions in a network is a necessary but not a sufficient condition for realizing learning potential; diversity is essential. Social network experiments have shown that a herd-like mentality, when individuals slavishly follow the single most frequently voiced idea, is counterproductive. This was demonstrated in a financial market experiment involving data from 10 million trades, which showed that returns on investment followed a parabola, peaking at the midpoint where traders were neither too isolated nor too herd- oriented (Pan et al. 2012; Pentland 2014). 3 The learning effect of increased connectivity in organizations mirrors the growth of connections in the developing brain (“In the beginning was the word” 2014).Babies are born with about 100 billion neurons, and connections between these form at an exponentially rising rate in the first years of life. It is the pattern of these connections that determines how well the brain works and what it learns. By the time a child is three there will be about 1,000 trillion connections in his brain, and that child’s experiences continuously determine which are strengthened and which pruned. 57 4. Incentives, Leadership, and Culture Highlights  There is a consistently positive generalized perception by staff of the Bank’s commitment to learning and knowledge sharing.  The Bank has made a sustained investment in training and learning events, and this is aligned to the staff’s perceived needs.  Despite these positive general trends, aspects of the system and the culture specific to learning in lending may discourage the innovation and adaptiveness called for by effective lending.  Quality assurance procedures have made an uneven contribution to learning.  Annual assessments of individual staff performance do not appear to reward learning and knowledge sharing.  Although restructuring of projects is generally perceived to be less problematic than it was, the staff report that they are not always encouraged to acknowledge problems with projects; some attempt has been made to address this by organizing learning-from-failure events.  Some recently introduced “smart learning tools” hold promise and are valued by staff, but it is too early to say if they will be sustained; by themselves, they will not be sufficient to consolidate a culture of learning in lending.  Customized learning instruments—the Learning and Innovation Loan and the Intensive Learning Implementation Completion and Results report—have not led to more learning.  The commitment to align leadership, culture, and values is explicit in recent corporate presentations, and Knowledge, Learning, and Innovation is singled out as a pillar of the new architecture; evidence from the literature on organizational change suggests that this initiative will only work if the Bank’s top leadership takes the lead by committing to, signaling, and modeling the culture and values appropriate to learning in lending. What the Literature Says With respect to incentives, studies of government agencies in the United States and Australia have shown that people who self-select to work in the public sector have a commitment to the values of public service, which appear to motivate them beyond pay (Milne 2007; Senge et al. 2007). Other surveys of knowledge workers in both the private and public sectors indicate that pay is not the primary factor when it comes to employee retention; more important are opportunities to learn new skills and 58 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE positive feedback from managers and peers, as reflected in the Hay survey of more than 300 companies, cited by Davenport (2009). In public sector organizations, incentives are typically geared to status differentiation. Bank staff grading is an example of this because it is difficult to quantify the difference any individual makes. It is unclear what results should be measured (effort, outputs, project outcomes, or progress made against initial benchmarks); how results should be measured; and how to separate team and individual performance, which is difficult when most people work in teams. Inappropriately designed performance evaluations may discourage staff from devoting time to learning (Kerr 1995; Milne 2007). Clarifying what types of knowledge sharing and learning are actually being rewarded is the first step to strengthening organizational learning. The behavior of managers breathes life into the culture and the incentives of the organization, helping to define the scope and the outcome of any reforms, including the attempt to nurture a learning culture. Influencing the culture is seen as one of the critical jobs of leaders in any organization. Experience both with successful and failed efforts at culture change underscore that leading by example is the only way by which leaders can effect culture change. They must articulate and model the behaviors and values that define the evolving culture of the organization, and then spread them constantly through personal contact and communication. “Leaders are considered the primary influencers to transmit, embed, and reinforce organizational culture through what they pay attention to, measure, and control; how they react and manage crisis; who they hire, promote, push aside, exit; how they allocate resources; and how they role model, teach, and coach” (Schein 1997, 15). It may be inferred that in an organization like the Bank learning in lending will not prosper unless managers send the right signals. In recent years, the management literature has made a case for aligning the goals, values, and activities of organizations, using a Balanced Scorecard that enables leaders to keep their eye on the big picture and to monitor progress from the top to the bottom of the organization. This involves a downward cascade with the work contracts of people at the top of the organization aligned with those lower down. According to Kaplan and Norton (2006), the Balanced Scorecard articulates the corporate vision and strategy by bringing together four perspectives: efficiency, client orientation, internal processes, and learning and growth. The Bank introduced this principle in 2011 with its Corporate Scorecard, the latest draft of which was reviewed by the Board in April 2014. Learning and the flow of knowledge have their rightful place in the Scorecard. The Scorecard can be a useful management tool, but it will not lead to better learning and knowledge sharing if it is simply grafted onto an organization that 59 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE lacks the attributes conducive to learning or the incentives that stimulate it. Based on two decades of research, Garvin et al. (2008) concluded that the following cultural attributes are essential for learning: (i) psychological safety, meaning that people are comfortable expressing their views and do not fear disagreement with peers or authority figures or making mistakes; (ii) appreciation of differences, meaning that employees are aware of the value of opposing ideas; (iii) openness to innovation, meaning a willingness to craft new approaches and to take risks; and (iv) time for reflection, meaning that the necessary focus on meeting immediate targets does not undercut what should be a long-term commitment to patient analysis and creative thinking. Ayas and Zeniuk (2001) argued that a sense of psychological safety was critical for sustaining an open learning culture, a culture without the fear of failure that produces defensive behavior. Data from more than 500 organizations across the world and interviews with thousands of managers show that 70 percent of organizational change efforts fail, and the main factors contributing to failure include employee resistance to change and management behavior that does not support the professed change (Keller and Price 2011). Employees resist change when the signals they receive from their managers are unclear (for example, when everything is labeled a priority) and when bosses’ behavior fails to model the reforms that they are trying to promote. What the Bank’s Evidence Shows This section begins by showing how, according to both World Bank employee surveys and the survey of Bank staff by the Independent Evaluation Group (IEG), the Bank is perceived to support learning and knowledge sharing. However, there are significant differences of view between task team leaders (TTLs) in country offices and those at headquarters, a divergence of opinion that warrants investigation. Second, spending on training and learning events appears to bear out the Bank’s commitment to building a learning culture. This affirmative evidence is then counterpoised against some important caveats, which suggest that the Bank’s top leaders and managers can do more to promote the culture and incentives needed to sustain learning. Quality assurance of lending does not strongly reinforce learning. The system for assessing staff performance—the annual Overall Performance Evaluation—assigns a low weight to learning and knowledge sharing in the overall calculus. Despite the Bank’s avowed commitment to promoting adaptiveness in lending (the science of delivery) and employees’ willingness to adapt, the IEG survey and interviews found that staff experience pressures to avoid risks and admissions of failure, and that in the view of many, there is limited learning from mistakes—factors that would tend to limit adaptiveness. While IEG is 60 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE mandated by the Board to perform an accountability function, the staff interviewed by IEG reports that the accountability focus has discouraged the development of an open and nondefensive dialogue between evaluators and operational staff about what works and doesn’t work in lending. Finally, although it is a critical aspect of monitoring the alignment of the Bank’s stated mission with the actual work that is done, the Corporate Scorecard remains a work in progress with respect to tracking improvements in learning and knowledge sharing. BANK IS PERCEIVED TO BE COMMITTED TO LEARNING AND KNOWLEDGE SHARING The 2013 Employee Engagement Survey found that for the World Bank, 50 percent of respondents replied favorably to a question about opportunities to learn new skills and develop talents; 60 percent said that the Bank performed well on learning opportunities relative to other organizations. Yet with respect to TTLs—the most relevant group for assessing learning in lending—there were significant differences between those located at headquarters and those based in country offices (Figure 4.1). On these, and several other measures, TTLs in country offices were more upbeat than their counterparts in Washington, DC. Why this should be so is unclear. IEG will probe the reasons for these differences in the next evaluation. There is an important caveat to this optimistic reading of where the Bank stands on learning and knowledge. The Bank staff surveys did not refer specifically to learning and knowledge sharing as applied to lending. When questions are asked about learning in lending the picture that emerges is less rosy. IEG suggests that the framing of questions about knowledge sharing and learning is critical. Given its status as the leading repository of knowledge about development and the undeniable pride that staff manifest about working for the institution—another consistent theme from corporate surveys—general questions about knowledge and learning probably reflect the halo effect of the Bank’s reputation, which is possibly accentuated by a certain fuzziness about what learning actually entails for the Bank. Despite this caveat, responses to IEG’s survey—which, unlike the Bank surveys, was geared to learning in lending—were distributed similarly to those in the 2013 Employee Engagement Survey (even though the IEG survey had a lower response rate). Fifty-nine percent of all respondents strongly agree or agreed with the statement that the Bank is committed to promoting learning and knowledge sharing in its lending operations. More than 70 percent of respondents strongly agreed or agreed with the statement that the employees in their unit “constantly consider better ways of doing things.” Sixty-seven percent reported that the staff in their units was eager to share information about what works and what does not, and 64 percent reported that colleagues in their units engaged in productive debates and discussions. Sixty-five percent of respondents disagreed with the notion “unless an 61 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE idea has been around for a long time, no one in my unit wants to hear it,” and 53 percent strongly agreed or agreed that institutional incentives support openness to new ideas at the Bank. Figure 4.1. Task Team Leaders in Country Offices Perceive More Opportunities for Learning than Those at Headquarters a. Perceived opportunities TTL, CO* "How much does your job offer you a chance to learn new skills and develop your talents?" TTL, HQ* 46 48 50 52 54 56 58 Percentage who responded "very great" or "substantial extent" b. Perceived opportunities relative to other organizations TTL, CO** "Compared to other organizations you may know about, how would you rate the WBG on providing learning TTL, HQ** opportunities so that you can handle your present job properly?" 50 52 54 56 58 60 Percentage who responded "very good" or "good" Source: Employee Engagement Survey 2013; data only from the International Bank for Reconstruction and Development. Note: CO = country office; HQ = headquarters; TTL = task team leader. *p = 0.02. **p = 0.05. With respect to perceptions of the Bank’s commitment to learning in lending, the pattern of differentiation by TTL location that emerges from the IEG survey echoes what was found in the Employee Engagement survey (Figure 4.2a). Also, taking 62 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE TTLs and non-TTLs together, there were significant differences between regions, with the East Asia and Pacific Region and the Europe and Central Asia Region expressing the highest level of confidence in the Bank’s commitment to learning in lending (Figure 4.2b). Figure 4.2. Confidence in the Bank’s Commitment to Learning in Lending a. By staff location TTL-CO* "Considering your experience over the past two years, to what extent do you agree or disagree with the statement 'I have confidence that the World TTL-HQ* Bank is committed to promoting learning and knowledge sharing in its lending operations.'" 0 20 40 60 80 100 Percentage of responses: "strongly agree" or "agree" b. By Region MNA SAR "Considering your experience over the past two years, to LCR what extent do you agree or AFR disagree with the statement 'I have confidence that the World EAP Bank is committed to ECA promoting learning and knowledge sharing in its 0 20 40 60 80 100 lending operations.'" Percentage of responses: "strongly agree" or "agree" Source: IEG survey of Bank staff conducted for this evaluation. Note: AFR = Africa; EAP = East Asia and Pacific; ECA = Europe and Central Asia; LCR = Latin America and the Caribbean; MNA = Middle East and North Africa; SAR = South Asia. The difference between the matched groups was statistically highly significant; p = 0.01 for regions; and p = 0.00 by staff location. LEARNING IS STEADILY FUNDED AND ALIGNED WITH PERCEIVED NEEDS Except for a blip in 2011, Bank spending has remained steady since 2004 (Figure 4.3). In 2013, South Asia and East Europe and Central Asia were the Regions that spent most on training as a proportion of the payroll (around 7 percent), with Africa and East Asia and Pacific at the other extreme (about 4 percent). The Learning Board is 63 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE charged with oversight responsibility for implementation of the Staff Learning Strategy, which has four pillars: (i) Corporate Core Curriculum; (ii) Professional and Technical Learning; (iii) Unit and Individual Learning; and (iv) On-the-Job Learning (such as mentoring and coaching). According to the Bank’s FY13 Staff Learning Annual Report, there is a demonstrated connection between robust on-the-job learning and the delivery of strong business results; an on-the-job learning assignment is now part of the Corporate Core Curriculum (World Bank 2013). Figure 4.3. Total Spending on Staff Learning, FY01–13 100 80 US$ millions 60 40 20 0 FY01 FY02 FY03 FY04 FY05 FY06 FY07 FY08 FY09 FY10 FY11 FY12 FY13 Source: Staff Learning Annual Reports for FY09, FY12, and FY13. Note: Includes all direct costs for learning delivered (design and delivery of staff learning) and learning received (participant staff time, related travel, and external fees). The learning that is delivered appears to be aligned with the skills that staff perceive to be most important for their work. Operational learning was the largest category by days delivered (Figure 4.4). This reflects the premium that staff place on this type of hands-on, project-related training, which includes the Operational Core Curriculum. The 2009 staff survey included a question (not repeated in the 2013 survey) on the skills that would most enhance job performance: 44 percent of respondents replied that operational skills bearing on Bank instruments and processes were the most important; 30 percent rated project management skills as the top priority (the response categories were not mutually exclusive). 64 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE Figure 4.4. Learning Days Delivered by Type, FY13 Managerial learning Unit or individual learning Orientation Professional and technical: ACS Professional and technical: other Professional and technical: sector weeks Interpersonal and communications learning Operational learning 0 5 10 15 20 25 Percentage Source: World Bank (2012). Note: ACS = administrative and client support. See endnote for definitions of the different types of learning. 1 Averaging across all responses to the IEG’s survey of Bank staff, 45 percent of respondents replied that learning occurred to a very large or substantial extent through training courses, including the Operational Core Curriculum,). With respect to learning events, including Sector Weeks, the corresponding percentage was 43 percent. Once again, location counts. The results for TTLs show that, compared to their counterparts at headquarters, a higher proportion of those based in country offices report that learning occurred through training courses and learning events (Figure 4.5). Figure 4.5. To What Extent Did Learning Occur as a Result of Training Courses or Learning Events? Training courses TTL, CO* TTL, HQ* Learning events 0 10 20 30 40 50 60 70 Percentage of responses: "very large" or "substantial" extent Source: IEG survey of Bank staff conducted for this evaluation. *p = 0.00. 65 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE Participants in IEG focus groups and interviews said that there is now less opportunity than before to learn through Sector Weeks. This is partly because, as the span of the Networks enlarged, Sector Weeks became more diffuse. For example, the Water Sector previously had had its own week, which led to a lot of idea sharing between TTLs. But once it was absorbed by the Sustainable Development Network, the pressure to “force cross-sectoral exchange” reduced the space for Water Sector TTLs to interact and learn from each other. Also, budget cuts have reduced the opportunity for staff from country offices to attend Sector Weeks and similar learning events. This trend may be exacerbated by the actions taken following the FY14 Expenditure Review, which aims to save $400 million. The measures that took effect on February 1, 2014 included a 15 percent reduction in the volume of travel to Washington, DC. 2 QUALITY ASSURANCE PROCEDURES FOR LENDING DO NOT ALWAYS FACILITATE LEARNING This section and those that follow consider instances of apparent misalignment— cases where Bank processes appear not to be fully aligned with the change agenda’s avowed commitment to installing knowledge, learning, and innovation as a discrete objective, embodied in the new Corporate Scorecard. In each of these cases, better alignment will not be realized unless the Bank’s top leaders—IEG included—take the first step, setting an example for staff to follow. What is needed is a change in the culture of the Bank and the set of incentives that it offers its employees. To start with, there is scope for making quality assurance of lending services more learning oriented. It is perhaps to be expected that the custodians of the quality assurance process (high-level staff) will rate the contribution that it makes to learning more highly than those whose work is subject to this control. The IEG survey found that, to a statistically significant extent, a higher proportion of staff in grades GI through GK reported that the Quality Enhancement Review (QER) was an important source of learning (Figure 4.6). Eighty percent of GI through GK staff said that to a very large or substantial extent, learning occurred through the QER; 37 percent of GG staff (and 36 percent of TTLs) responded this way. On the other hand, in the case of peer review—another quality assurance mechanism—there was no significant difference between the two groups. It could be that factors other than learning colored the perceptions of the quality enhancement review. Maybe the more positive response from managers and directors was a reflection of the high value they placed on these mechanisms as a way to contain risks (peer reviews are less risk centered). Staff members at lower grades were perhaps more inclined to view the quality enhancement review as a hurdle to overcome; this negative perception may have edged out any consideration of the learning that resulted. 66 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE Figure 4.6. To What Extent Did Learning Occur through Quality Assurance Procedures? Peer review GI-GK GG Quality Enhancement Review* 0 10 20 30 40 50 60 70 80 90 Percentage of responses: "very large" or "substantial" extent Source: IEG survey of Bank staff conducted for this evaluation. *p = 0.00. In the IEG interviews and focus groups, the choice of peer reviewers was regarded as significant but many interviewees said that the process is not rigorous and quality has declined over time. This is partly because peer review is not adequately funded. Sometimes managers send the wrong signals—they express a preference for hiring “soft” reviewers who won’t hold up project preparation. It is the TTL who usually proposes the peer reviewer to the manager and there is a tendency for TTLs to “go to the person you know” rather than to the person with the best skills. Overall, the staff interviewed by IEG reported that the quality assurance process is patchy, with differences across lending instruments. Some observed that the quality assurance of development policy operations seems to be more standardized across the Bank—more systematic than investment lending. The QER is not mandatory, and wide variations exist between Regions in the incidence and the timing of these reviews. For both peer review and QER, decisions reached are not uniformly recorded, and there is no sure way to quickly locate the minutes in the Bank’s electronic filing system. Some interviewees said that QERs are used mainly to expedite loan processing rather than to invite the discussion of opposing views about how to design the project; matters of technical substance get short shrift. It all depends on the manager. One director said that the aim was to demonstrate how the lessons from the Bank’s worldwide experience had been adapted to local conditions. But he also stressed that this questioning should be a continuous process, extending well beyond the QER. Other TTLs said that, from a learning perspective, more could 67 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE be obtained from informal clinics where TTLs could “safely” expose their project plans to critique. If the quality assurance process imparts lessons from operational experience, it should serve to prevent unworkable projects from being approved. However, IEG’s interviewees said that this was not always the case. Overly complex projects (“Christmas trees”) were often still approved even though experience demonstrates, in the words of one, “complexity is the enemy of success.” Projects that are too complex are more likely to lead to delays, cancellations, and restructuring. Opinions were divided about the origins of the problem. Some interviewees maintained that “the pressure to over complicate” comes mainly from within the Bank. But others said that the client was primarily responsible for the push to include more and more components in projects, making them difficult to manage. The intensity of quality assurance varies by lending phase. According to the IEG survey of Bank staff, 41 percent of respondents said that learning occurred to a very large or substantial extent during project preparation review meetings; for project supervision, the corresponding percentage was 32 percent. Participants in IEG interviews and focus groups said that managers tend to be less involved in the quality assurance of implementation and completion than they are in preparation. According to interviewees, there is limited managerial oversight of implementation status and completion reports. But a distinction clearly has to be drawn between what is written up and what remains an oral comment. During implementation, informal feedback from managers is often more important than what is written up every six months or so. Nevertheless, IEG’s review of more than 100 recently evaluated projects found that the majority of implementation status reports do contain comments from sector managers, country directors, or both—although these tend to focus on implementation progress rather than discussing matters with a bearing on development outcomes. Several TTLs noted that decision meetings have been taken over by the Operational Risk Assessment Framework (ORAF). “The ORAF is a nightmare and does not provide much practical value.” Respondents felt that people spend decision meetings obsessing over every potential scenario that could go wrong at the expense of attention to other more substantive issues. They also noted that the Bank has moved toward compliance by checklists and box ticking, which they felt was not conducive to learning and knowledge sharing. Some highlighted a culture of complying with the status quo and moving things along. “No one wants to slow things down, so it is frowned upon to ask critical questions in decision meetings.” 68 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE IEG’s evaluation of the Bank’s matrix organization bears out these observations. It found that the selection of peer reviewers was not well managed, with deadline pressures reducing the time available for TTLs to respond substantively to peer review comments. The evaluation found that QERs were a better way to boost lending quality, but they were optional and often not timely. Also, it revealed that the Bank’s quality control systems focused more on fiduciary and safeguard risks than on other aspects of quality (IEG 2013, 84). INCENTIVES COULD BE BETTER ALIGNED TO PROMOTE LEARNING In corporate reports and messages to staff, the Bank has acknowledged the lack of alignment in incentives for learning and knowledge sharing. “Most staff feel, despite the growing importance of knowledge work, that the Bank’s main internal incentives are still related to lending” (World Bank 2011). Some staff have argued that additional financing also masks the Bank’s push to lend. This continuing push to lend increases the opportunity cost of dedicating time and effort to learning and knowledge sharing, and distorts the metrics of performance in favor of inputs and short-term outputs rather than on results and development solutions. A recent statement from the top of the Bank confirms the incentive problem. “On any given day, the World Bank Group is engaged in thousands of operational interactions in well over 100 countries. But sharing this operational knowledge is hampered by weak incentives” (Pradhan 2013). The Bank has often given conflicting signals to staff on the importance of learning and knowledge sharing. A 2007 Issues Note observed: “While the Bank’s declared values espouse such a culture [of learning and knowledge sharing], the incentive systems (operational processes, budget, Human Resources, Information Technology, etc.) often promote the opposite” (World Bank 2007). Strategic alignment can help ensure consistency in signals across the organization, but the Bank has failed in the past on strategic alignment: • First, most efforts at alignment have been isolated, partially implemented or not sustained over time. • Second, knowledge sharing and learning have not been incorporated into how managers do their jobs. • Finally, knowledge sharing and learning remain outside staff jobs and the project cycle. They are a separate thing to do, “nice to have, but not necessary,” and assigned to “the K and L people” (World Bank 2007). It is one thing if the Bank provides insufficient incentives to staff for learning and knowledge sharing. It is quite another if it provides perverse incentives or disincentives. TTL focus groups conducted for this evaluation seem to point to the 69 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE existence of perverse incentives and disincentives for staff. There is a disincentive to share information with other Bank teams who compete for funds, specifically in the context of work program agreements. There is an incentive to hoard knowledge. The immediate measures for cost savings announced on January 23, 2014, included some which may adversely affect knowledge sharing and learning, such as the reduction of training and operational mission for GA–GD staff and the reduction of volume of travel to Washington D.C. by country office staff. While other measures may yet be put in place to compensate for these decreased learning opportunities, the signals sent should not be underestimated. More than one-third of respondents indicated the need for greater attention and signals by senior management on the priority of learning and knowledge sharing. In addition, many open-ended suggestions focused on management’s engagement and support to learning. Staff asked for stronger senior management leadership and support to knowledge and learning demonstrated through actions not just words, including leading by example and dedicating their own time to such activities. They said, “Senior management should influence the culture to move us away from a culture of information hoarding to information sharing.” IEG interviewees indicated that senior management should give knowledge the same status as lending in management attention, monitoring, and promotion. While 18 percent of respondents to IEG’s survey identified the lack of sufficient focus on results and outcomes by Bank management as one of the three biggest obstacles to learning and knowledge sharing, only 3 percent identified the Bank’s Board of Executive Directors as such. This could be either because the Board does not figure in the minds of Bank staff or because they feel it does indeed focus on results and outcomes. In any case, the bottom line is that for 97 percent of Bank staff, the Board is not among the three biggest problems for learning and knowledge sharing. Staff perceives the lack of institutional incentives as one of the biggest obstacles to learning and knowledge sharing. In the IEG survey of Bank staff, more than any other factor, the lack of institutional incentives was most frequently singled out as one of the three biggest obstacles to learning and knowledge sharing: 52 percent of respondents named incentives as an obstacle. Participants in IEG interviews and focus groups strongly reinforced this message (Box 4.1). 70 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE Box 4.1. Task Team Leaders Speak Out about the Lack of Institutional Incentives • “There is no incentive to go the extra mile during design. It is possible to prepare a project with less effort. It is up to me as a TTL whether I want to bring in best practices to the design or just deliver another project to the Board. I won’t get a better SRI or OPE for going the extra mile to make sure I have a project that is well designed and implementable. My reward is the personal satisfaction of designing a good project.” • “If you want to do something new, you have the freedom to do things, but you won’t get rewarded or go to the top because of it. I can design what I want. My boss trusts me, but you are not rewarded for ‘innovation.’” • “Too many people bring bad projects to the board and are promoted. It’s not about whether the project can be implemented and gets results.” • “No one is looking at how many problems you solved or how many problem projects you rescued.” • “The reward system should not be based on how many projects you bring to the board without looking at if things can be implemented. Rewards should be given elsewhere. They should consider things like whether you moved the dialogue forward. Reward knowledge.” • “There are disincentives to go the extra mile in design and implementation because of the slippery Human Resource framework. No one gives a hoot about implementation. You don’t get promoted based on performance.” Note: OPE = Overall Performance Evaluation; SRI = salary review increase; TTL = task team leader. The annual staff Overall Performance Evaluation (OPE) does not reinforce learning. Only about 7 percent of respondents to IEG’s survey of Bank staff said that the OPE rating on learning and knowledge sharing influences the overall performance evaluation to a very large or substantial extent. Over half the respondents said it influences it slightly or not at all. When asked to recommend one thing that Bank could do to further promote learning in lending, 14 percent of respondents to IEG’s survey advocated changing OPEs and results agreements to place more weight on learning and knowledge sharing. In IEG’s interviews, a recurring theme was the divergence between the OPE’s formal learning and knowledge sharing rating, which is typically “fully satisfactory” or better, and the low weight this rating receives in the overall assessment of performance. In focus groups also, participants observed that the rating carries little weight in the OPE. No units in the Bank stand out, either for exemplary or weak performance on the OPE assessment of learning. IEG analyzed aggregate data on the learning and knowledge sharing OPE rating for all World Bank staff with a grade of GF or higher, over a five-year period (FY09–13). Averaging over this period, most staff were rated fully successful (Figure 4.7). There was little variation in the distribution of ratings from one year to next. Also, for any given year, there was no significant difference between Regions or sector boards. On two dimensions, however, the variation 71 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE between groups was statistically significant. A man was 20 percent more likely than a woman to get an “outstanding rating.” Totaling the ratings of females over five years, 5 percent achieved an outstanding rating, compared to 7 percent for males. There were also significant differences between staff grades. Six percent of the staff of GG grade earned an outstanding rating, compared to 11 percent of those at GH grade and 22 percent at GI grade. Figure 4.7. Staff Overall Performance Evaluation Ratings for Learning and Knowledge Sharing Less than fully successful Fully successful Superior Outstanding/best practice 0 10 20 30 40 50 60 70 Percentage of staff distribution (mean for FY09-13) Source: World Bank Human Resource Analytics. CONTINUING OBSTACLES TO ADAPTIVENESS IN LENDING Hirschman (1967) and Rondinelli (1993) made the case early on for operations that are small-scale, exploratory, and risky—operations that do not always provide immediate economic returns or yield quick results. More recent studies have strongly argued for an adaptive approach to lending, emphasizing the importance of learning from failure (Andrews et al. 2012). Adaptiveness is at the heart of what the Bank Group’s president has championed as the science of delivery. 3 It has been presented as an art as well as a science. The art lies in the innovation and adaptability of the actors and different delivery models. Its key aspect is the “continuous interplay of designing interventions using evidence; implementing them in an iterative way; and, learning deliberately throughout the process,” and the science of delivery lies in “replicating and scaling those models” (Pradhan 2013). Adaptiveness presupposes a culture where employees learn from mistakes, are able to admit to failure without fear of reprisal, feel encouraged to take informed risks, and are able to innovate. This is a lot to ask of any organization. There are mixed messages about the extent to which this culture already exists in the Bank. In 2012, a learning- from-failure event sponsored as part of President Kim’s change agenda produced a rich trove of reflections on the challenges of adaptiveness (Box 4.2). 72 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE Box 4.2. Learning from Failure: Panama Land Administration Project Adapted from a presentation by Jorge A. Muñoz, task team leader, at the 2012 learning- from-failure event: I would say there were two critical, defining moments during project implementation: restructuring and review by the Inspection Panel (IP). By mid-2005, the project was essentially stuck. Disbursements were not flowing, and institutions were fighting over turf. As a result, project staff were not going to the field, conflicts were not being solved, and public frustration was building. The obvious solution was to restructure the project, but this proved to be very time-consuming and more bureaucratic than I imagined. Actually, what this project faced was not that unusual for complex policy reforms. Unfortunately, the Bank is not flexible enough to adapt to change. We spend too many resources designing airtight rigid projects and too little supervising them and adapting to inevitable changes. Restructuring should be the norm, not the exception. Addressing IP cases, Indigenous Peoples in Panama have some long-standing historical land claims that periodically result in violent events (unrelated to the project). The two IP cases resulted from poor consultation and the inability of the project to title some Indigenous Peoples lands. Although the component related to Indigenous Peoples lands represented only 5 to 10 percent of project resources, addressing these issues occupied 80 percent of my time and my team’s time in the last year and a half of the project. This issue also created considerable friction in relation to government which had different policy priorities. My team’s ability to address these issues was made possible by full support from all levels of Bank management. It required very intensive supervision and spending considerable amount of time in the field in direct contact with key stakeholders. Most Important Lessons • The Bank needs to be more flexible in adapting to change and managing risks. • At present, projects are designed according to rules and procedures suitable for discrete infrastructure type projects, not for complex policy reform programs. • We spend too much time designing Plan A, as if it will never change, but we do not plan for a Plan B when things change. • We react to risks; we do not manage risks effectively. • Flexibility is the key because complexity is the rule. What Would I Do Differently • Design a simpler project, not because the issues are simpler but rather to allow the Bank to have meaningful operational engagement, launch activities in the field, and expand operations as opportunities open up and circumstances change. • The solutions governments seek are often operational in nature—the how-to-do reforms. • This is learned largely by implementing projects, learning by doing, and supporting governments throughout the process. • Spend time in the field, spend time in the field, and spend time in the field. • Do not rely solely on project reports prepared by government agencies and discussed in the country’s capital. 73 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE The results of the 2013 Employee Engagement Survey are cause for reflection about whether the Bank’s culture embodies the trust needed for adaptiveness to flourish. One-third of all respondents strongly agreed or agreed that senior management creates a culture of openness and trust; for TTLs, the corresponding proportion was 23 percent. There is a much more favorable perception about incentives to innovate and to take risk. For all respondents, as well as for TTLs in particular, roughly two- thirds report feeling encouraged to find new and better ways of doing things (about 5 percent less than 2009). With respect to freedom to take informed risks, 57 percent of all IBRD employees and 55 percent of TTLs responded favorably (compared to 63 percent and 67 percent, respectively, in 2009). On both innovation and risk, there was no significant difference between TTLs at headquarters and those based in country offices. These results are paradoxical: if a culture of openness and trust is widely perceived to be lacking, why do the majority of employees report that they feel free to innovate and take risks? Senior management presumably sign off on projects that are risk taking and innovative. Yet they appear not to have created the culture of openness and trust that would be likely to encourage experimentation with new, possibly risky approaches. Responses to the IEG survey convey a powerful impression of the culture in which lending unfolds—and the scope for an adaptive approach to preparation and implementation. Only 5 percent of respondents felt to a very large or substantial extent that the Bank has encouraged informed risk taking in its lending operations. Seventeen percent of respondents replied that the Bank’s staff was able to learn from its mistakes to a very large or substantial extent. Managers were much more sanguine than staff at lower grades, with the difference between the groups being statistically highly significant: 41 percent of staff at grade GI and above replied that mistakes were learned from compared to 17 percent of GG staff. It could be that managers were promoted precisely because they had successfully learned from their mistakes, and because, unlike staff at a lower grade, they were more confident about their own ability to learn from mistakes, a conviction that was reinforced by their very promotion. Managers have a key role to play in creating a safe space for staff to candidly discuss operational problems and how to address them. Therefore, it is a matter of concern that only one-third of respondents to the IEG questionnaire survey opted for the response very large or substantial extent when asked if they felt able to discuss with their management what is not working in a lending operation. There was a statistically significant difference between Regions in the response to this question, ranging from Middle East and North Africa (where 69 percent replied very large or 74 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE substantial extent) to East Asia and Pacific (43 percent). There were no significant differences when respondents were compared by location or sector board mapping. The scope for adaptiveness is probably influenced by the ease with which projects may be restructured during implementation. According to the IEG survey of Bank staff, 51 percent of respondents agree or strongly agree that current Bank procedures for project restructuring have supported course corrections. In this respect, there was no significant difference between managers and staff or between TTLs and non- TTLs. On restructuring, the IEG focus groups and interviews found that managers and quality assurance advisers tended be more bullish, emphasizing that in recent years Operations Policy and Country Services (OPCS) has facilitated restructuring. TTL’s were less convinced but, in general, they said that there is greater willingness to make changes that did not require Board approval (typically involving the reallocation of loan proceeds between components) than there is to countenance a Board-endorsed revision of the project development objective. Various people said that such Level 1 restructuring is “stigmatized,” partly because there is a fear that it reflects badly on the competence of the TTL. But there are signs of adaptiveness in relation to lessons learned. Respondents to the IEG survey of Bank staff were asked, in an open-ended question, to give one example of how they changed the design or implementation of their lending operation in response to learning. The examples were so diverse that they proved hard to code into response categories. The category with the largest share of responses (10 percent of the 356 respondents to this question) was coded as “change occurred in the light of lessons learned from previous operations.” The next evaluation in IEG program will probe this further, trying to identify the learning chains and social networks involved in the transmission of lessons across time and space. SMART LEARNING TOOLS MAKE SENSE BUT ARE NOT ENOUGH TO TRANSFORM THE CULTURE A variety of new tools for facilitating learning has been showcased at the Bank in recent years. The 2013 workshop on TTL handover was examined above. The organizer of that workshop has also championed the use of after-action reviews, which have yielded good results for the U.S. Army. These reviews are resolutely low-tech, using field notebooks rather than formal briefs or databases (Box 4.3). What counts is the liveliness of the discussion about what happened, not the quality of the write-up. 75 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE Box 4.3. Can the World Bank Learn from the U.S. Army? The U.S. Army’s Opposing Force (OPFOR) is considered to be consistently successful, and its success is attributed to the way it uses after action reviews (AARs), a method for extracting lessons from one event or project and applying them to others (Darling et al. 2005). Most corporate AARs end up being faint echoes of OPFOR’s AARs. Whereas OPFOR converts its postmortems of past failure into aids for future success, most corporate AARs stop at being just postmortems. Backward and Forward Accountability Corporate AARs are often convened around failed projects (Darling et al. 2005). The patient is dead, and everyone weighs in on the mistakes that contributed to his demise. The word “accountability” comes up a lot; generally, it means “blame” which participants expend considerable energy trying to avoid. There is a sense of finality to these sessions. The team is putting a bad experience behind it. “Accountability” comes up a lot during OPFOR’s AARs as well, but in that context it is forward looking rather than backward looking. Units are accountable for learning their own lessons. And OPFOR’s leaders are accountable for taking lessons from one situation and applying them to others—for forging explicit links between past experiences and future performance. At the end of an AAR meeting, the senior commander stands and offers his own assessment of the day’s major lessons and how they relate to what was learned and validated during earlier actions. He also identifies the two or three lessons he expects will prove most relevant to the next battle or rotation. After Action Review and Before Action Review The AAR can be customized for corporate environments. To an AAR would be added a before action review (BAR). It requires teams to answer four questions before embarking on an important action: • What are our intended results and measures? • What challenges can we anticipate? • What have we or others learned from similar situations? • What will make us successful? The responses to those questions align the team’s objectives and set the stage for an effective AAR meeting following the action. In addition, breaking projects into smaller chunks, bookended by short BAR and AAR meetings conducted in task-focused groups, establishes feedback loops that can help a project team maximize performance and develop a learning culture over time. Four Fundamentals • Lessons must first and foremost benefit the team that extracts them. • The AAR process must start at the beginning of the activity (i.e., the BAR). • Lessons must link explicitly to future actions. • And leaders must hold everyone, especially themselves, accountable for learning. Another option is the checklist—a concise inventory of the essential steps in designing and implementing projects, requiring TTLs to systematically “tick each of the boxes.” The exponent of this approach, which he adapted for use by surgical 76 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE teams operating under pressure, argues that the volume and complexity of knowledge has now exceeded any single individual's ability to manage it consistently without error despite advances in technology, advanced training, and specialized functions and responsibilities (Gawande 2009). Gawande stresses the need to allow for the fallibility of human memory when it comes to mundane, routine matters that are easily overlooked under the strain of pressing events. In its survey of staff, IEG asked respondents to assess the usefulness of checklists and other smart tools. Three-quarters separately rated checklists, how-to guidance, one-stop shops, and just-in-time help desks as useful to a very large or substantial extent. When asked to what extent these aids already existed, 54 percent said that checklists were now being used; 48 percent said the same for how-to guidance. But only 20 percent reported that one-stop shops and just-in-time help desks were now in operation. The Bank is an organization where financial incentives are not the primary motivator—only one-quarter of respondents to the IEG survey included higher pay among the three actions most likely to encourage learning and knowledge sharing. Interviewees told IEG that recognition by peers and managers is the most important incentive. Recent events showcasing and rewarding innovation and learning from failure show promise. An example of this is the Bank’s Innovation Marketplace where solicits nominations for new ideas and approaches and awards significant prizes to the winners. When it comes to the use of awards for learning, the International Finance Corporation (IFC) appears to have promising practices. Smart Lessons, a voluntary program started in 2005, offers guidance for writing narratives to post online and editorial services to ensure that the articles and multimedia presentations posted on the SmartLessons site really work as stories (Morris and Oldroyd 2009). Since 2011, performance evaluations have considered SmartLessons, and this evidence helps inform decisions about promotions. The total award budget is $50,000 to $65,000 per year. However, there are mixed reports about the initiative’s impact. Although the program had 500 to 1,000 web hits per month, an internal IFC survey found that about 47 percent of Advisory Services staff and 60 percent of Advisory Services results measurement officers said they “never used them.” On the other hand, according to a Harvard Business Review article on Smart Lessons, survey results showed that more than 80 percent of IFC employees who read Smart Lessons found them relevant (Morris and Oldroyd 2009). One user reported, “Smart Lessons are an integral part of how I think through project design,” and a contributor said, “I have felt empowered to write a no-holds barred account of lessons I learned.” 77 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE The Knowbel award is a more recent IFC initiative, introduced in 2011 by the Global Knowledge Office. This program is administered by operations staff, and is designed to recognize those who promote knowledge sharing and learning. For example, the Facility for Investment Climate Advisory Services was awarded a prize for Excellence in Knowledge Sharing for work on peer-to-peer learning leading to new bilateral technical assistance. This involved over 36 events, including seminars and workshops, attracting more than 1,500 staff and external participants. While it makes sense to promote “smart” interventions like these, research evidence shows that these expedients are not sufficient in themselves to transform the culture of the organization. There is a “tendency to use formal project learning tools in a tokenistic way or to ignore them altogether” (Swan et al. 2010, 334). Smart tools will not flourish unless senior managers take them seriously enough to apportion sufficient budget, and unless the various awards offered are taken into account in staff performance assessments and promotion decisions. CUSTOMIZED LEARNING INSTRUMENTS NEED TO HAVE THE RIGHT INCENTIVES Like smart tools, learning instruments intended to privilege new approaches and experimentation will only work if they are embraced by senior managers (and clients), and if the incentives employees receive to use them are sufficient and in line with the organization’s overall commitment to learning. They won’t work as instrumental fixes for an organizational culture that does not reward learning. IEG assessed experience with two instruments specifically designed to promote learning: the Learning and Innovation Loan (LIL) and the Intensive Learning Implementation Completion and Results (ILI) report. The assessment was based on a comparison with, respectively, investment projects in general and standard or core Implementation Completion and Results (ICRs) reports. Both instruments have languished, for reasons that hold lessons for future directions. The LIL has been phased out, but the experience with this instrument is worth reflecting on in order that the Bank does not repeat similar mistakes in the future. IEG reviewed the 10 LILs most recently evaluated by IEG. The project documents for LILs were not notably different from those for Specific Investment Loans (SILs). They tended to give a thin description of the knowledge and learning on which these operations built. Reference to the analytical work that underpins project design was often sketchy. References to previous projects in Project Appraisal Documents (PADs) and ICRs often consisted of unsubstantiated assertions that the project built on Bank experience in the project country or other countries. Sometimes PADs identify generic operational lessons that inform design only for the subsequent ICRs to observe that these lessons were not acted on. But, in a minority 78 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE of cases, IEG found cases where the PAD gives a detailed account of the evidence base for the project, showing how the design takes account of what was previously learned. Five of the 10 LILs that were reviewed helped set up Development Learning Centers (DLCs) as part of the Global Development Learning Network. DLCs were intended to transfer knowledge among countries participating in the global network and to host training courses as an alternative to sending government officials abroad on costly study tours or importing expensive foreign consultants. The rationale for using this particular instrument is not always obvious. Indeed, the Midterm Review Aide Memoire from the Kenya DLC (P078209) asks why it was financed as a LIL, and if there was still a need for this sort of instrument. In addition, there is a conspicuous lack of detail about the actual learning mechanisms used by these projects. Also, there is little cross-referencing between the DLC projects. However, some cross-fertilization between these projects is evident. The ICR for the Côte d’Ivoire DLC (P066353) reports that DLC projects in Mali and Nigeria drew on the experiences of the Côte d’Ivoire project and provides some specific examples in the Lessons Learned section. The ICR also makes a specific operational comparison between this project and the one in Burkina Faso regarding the use of the project preparation fund. Moreover, although LILs were intended to pay particular attention to monitoring and evaluation (M&E) in order to facilitate learning, on average IEG ratings of M&E for LILs were not higher than ratings for other lending instruments. Three of the 10 lessons learned in the ICR for the Legal Reform Project in Mongolia (P074001) dealt with ways in which project M&E could have been improved. In that project, the indicators were little more than a checklist of project outputs, compiled with no apparent attention to what was to be learned or how that learning would be used. LILs often did not serve their avowed learning purpose. The ICR for the Cultural Heritage Project in Ethiopia (P057770) concluded: “The Bank mistakenly handled this LIL as a small investment project although the intent of a LIL is different due to its learning emphasis.” This problem also affected the Community School Support Project in Nepal (P082646) where the PAD promised an impact evaluation and yet relied largely on a count of schools that had been transferred to the community as its outcome indicator. The ICR Review and the Project Performance Assessment Report argued that the goal of the LIL was to create a knowledge base about community management and not to catalyze school transfer in advance of the development of such a knowledge base. In other words, the criticism is that the project was run like a SIL and not like a LIL. 79 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE Participants in IEG interviews and focus groups said that LILs failed to thrive because, although they were supposed to be cheaper to prepare and quicker to implement than SILs, they ended up costing just as much to administer, require the same approval-processes in client-countries, and fell out of favor because they were too small to have an impact, the ceiling being $5 million. Also, clients were not supportive of this instrument because the efforts to get them approved by government were no less onerous than for large projects. Most participants in IEG focus groups and interviews indicated that learning occurred irrespective of the choice of lending instrument. One interviewee responsible for quality assurance said, “We don’t need ‘learning-centered’ instruments; we should reduce the number of lending instruments while ensuring that all instruments accommodate learning.” One way to examine the value that ICRs add from a learning perspective is to compare the small subset of ICRs that are explicitly learning oriented with the standard (core) ICR. ILIs were introduced by OPCS to allow for a deeper analysis of the outcomes and lessons of given projects, based on stakeholder workshops and a beneficiary surveys for which additional funding was available. Only 35 ILIs were produced between FY05 and FY13. Have they been a richer source of learning than core ICRs? In the cohort examined by IEG, there was no obvious difference between the two groups in how the reports were prepared or in their quality or outcomes. Most of the ILIs included the mandatory stakeholder workshop (6 of 10) and beneficiary survey (7 of 10), but the findings from these events were orphaned in separate appendices and were not well integrated with the body of the report. Only two ILIs refer to the outcome of the stakeholder workshop or the beneficiary survey in the main text. The quality of the ICRs varied between the samples, and ILIs were not conspicuously superior to core ICRs. There was little difference between the two samples in the type and depth of lessons learned. ILIs and core ICRs both gave most space to generic operational matters, calling for more capacity building, increased institutional engagement, stronger partner relations and better technical inputs. The more useful lessons were those that explored in depth a particular feature unique to the project and then discussed its broader relevance. None of the ILIs in the sample based their lessons explicitly on the stakeholder workshops or beneficiary surveys for which they had received additional funds, and it is not clear what value this funding added to the learning that could be gleaned from the ICR. 80 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE Looking Ahead The literature points out that enabling environment factors—notably, time and budget—when combined with appropriate staff performance evaluation and promotion criteria; senior management signaling, leadership, and role modeling; corporate monitoring indicators; and salary increase criteria can create the necessary incentives for learning and knowledge sharing. Addressing these factors will inevitably involve trade-offs, which the Bank will need to make carefully. References Andrews, Matthew, Lant Pritchett, and Michael Woolcock. 2012. “Escaping Capability Traps through Problem-Driven Iterative Adaptation.” Harvard Kennedy School Faculty Research Working Paper, No. RWP12-036, Harvard University, Cambridge, Mass. Ayas, K., and N. Zeniuk. 2001. “Project-Based Learning: Building Communities of Reflective Practitioners.” Management Learning 32 (1): 61–76. Barber, A., A. Moffit, and P. Kihn. 2011. Deliverology 101: A Field Guide for Educational Leaders. Thousand Oaks, California: Corwin. Darling, M., C. Parry, and J. Moore. 2005. “Learning in the Thick of It.” Harvard Business Review 83 (7): 84–92. Davenport, T. 2009. “Managing Knowledge Workers.” In Thinking for a Living: How to Get Better Performance and Results from Knowledge Workers. Boston: Harvard Business School Press. Garvin, David A., Amy C. Edmondson, and Gino Francesca. 2008. “Is Yours a Learning Organization?” Harvard Business Review 86 (3): 109–116. Gawande, A. 2009. The Checklist Manifesto: How to Get Things Right. New York: Metropolitan. Hirschman, Albert O. 1967. Development Projects Observed. Washington, DC: Brookings Institution. IEG (Independent Evaluation Group). 2013. IEG’s 2012 Client Surveys: Topline Report of Key Findings. GlobeScan project 2557. London: GlobeScan. Kaplan, Robert S., and David P. Norton. 2006. Alignment: Using the Balanced Scorecard to Create Corporate Synergies. Cambridge, Mass.: Harvard Business Review Press. Keller, S., and C. Price. 2011. Beyond Performance: How Great Organizations Build Ultimate Competitive Advantage. New York: John Wiley and Sons. Kerr, S. 1995. “On the Folly of Rewarding A, while Hoping for B.” Academy of Management Executive 9 (1): 1–16. Milne, P. 2007. “Motivation, Incentives, and Organizational Culture.” Journal of Knowledge Management 11 (6): 28–38. Morris, S., and J. B. Oldroyd. 2009. “To Boost Knowledge Transfer, Tell Me a Story.” Harvard Business Review 87(5): 23. Pradhan, Sanjay. 2013. “A Solutions Partnership to End Poverty.” Speech delivered at the World Knowledge Forum, October 16, 2013, Seoul, Korea. 81 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE Rondinelli, Dennis A. 1993. Development Projects as Policy Experiments. Second edition. London: Routledge. Schein, E.H. 1997. Organizational Culture and Leadership. Second edition. New York: Jossey-Bass. Secretariat to the Learning Board. 2013. Staff Learning: The State of Staff Learning at the World Bank— FY13 Annual Report. Washington, DC: World Bank. Senge, P., B. Lichtenstein, K. Kaeufer, H. Bradbury, and J.S. Carroll. 2007. “Collaborating for Systemic Change.” MIT Sloan Management Review Winter. Swan, J., H. Scarbrough, and S. Newell. 2010. “Why Don't (or Do) Organizations Learn from Projects?” Management Learning 41 (3): 325–344. World Bank. 2007. Issues in Staff Knowledge and Learning: A Discussion Note. Washington, DC: World Bank. ______. 2011. The State of World Bank Knowledge Services: Knowledge for Development 2011. Washington, DC: World Bank. ______. 2012. FY12 Annual Report: Staff Learning—Linking Learning to Business Results. Washington, DC: World Bank. ______. 2013. FY13 Annual Report: Staff Learning—The State of Staff Learning at the World Bank. Washington, DC: World Bank. 1The following are definitions, drawn from Learning Secretariat (2013), for the different types of learning: Orientation activities are designed to induct staff into the Bank and impart the mission, values, strategy, and culture of the institution. The activities lay foundations for basic organizational and operational knowledge and allow staff to reach full productivity more quickly. Operational learning activities are designed to build basic project processing skills to enable staff to adequately manage Bank operations and projects. This includes the Operational Core Curriculum. Managerial learning activities are designed to update and strengthen the basic managerial skills of current managers and high potential staff. They include the corporate managerial programs organized by HRS as well as managerial training activities organized by VPUs/Departments/Units for their staff. Professional and technical learning activities are designed to enhance core sectoral capacity and develop and maintain cutting-edge technical skills related to a specific profession. This includes Sector Weeks. Interpersonal learning activities are designed to develop interpersonal (e.g., working in teams, managing relationships, etc.) and communications (e.g., language training, speaking, presentation, writing, etc.) skills in individuals. Unit and individual learning activities are those designed to develop general skills in staff that are fungible across professions and/or targeted toward specific learning needs in the unit. These include computer skills (e.g., Microsoft Office, Lotus Notes). 82 CHAPTER 4 INCENTIVES, LEADERSHIP, AND CULTURE 2“Expenditure Review—Immediate Measures,” an intranet communication from Bertrand Badre, managing director, January 23, 2014. 3The term was coined by Michael Barber who was hired by Tony Blair to set up his Delivery Unit. In his words, the science of delivery is “a systematic process for driving progress and delivering results in government and the public sector” (Barber et al. 2011). 83 5. Implications Highlights  There is a case for the World Bank to pay more attention to how knowledge flow and learning is mediated through interpersonal exchanges, understanding how team dynamics and connection to social networks shapes the potential for learning and knowledge sharing.  There is a need for smarter approaches to rewarding learning, including redesign of individual results agreements and performance evaluations.  Learning and knowledge sharing is only likely to flourish if there is senior management commitment, leadership, signaling, and role modeling.  The Independent Evaluation Group (IEG) has a shared responsibility for promoting learning, and it has made the commitment to assess how its evaluation procedures balance accountability and learning, and to revamp its suite of products to make more allowance for learning evaluations.  The next evaluation in this program of IEG evaluations will examine the extent to which learning takes place within and between Bank projects in a purposive sample of countries and sectors, taking into account the views of clients and development partners. Implications for the World Bank The Bank staff perceives the lack of institutional incentives as one of the biggest obstacles to learning and knowledge sharing in the Bank. While reorganizations have been relatively common at the Bank, serious reforms of the Bank’s internal incentives have lagged. The ongoing change process provides an opportunity to finally redress this long-standing gap. But time is of the essence as reorganizations tend to deplete the very two assets management needs to push through a transformative shift in internal incentives—political capital and staff good will. Fast and forceful action by senior Bank management in giving clear, concrete, and consistent signals on the importance of learning and knowledge sharing, including through the questions it continually asks and the behaviors it models, can bring rich pay-offs. Staff expectations are high. Nearly 60 percent of respondents to the Bank staff survey conducted by the Independent Evaluation Group (IEG) indicate that they agree or strongly agree with the statement that the Bank is committed to promoting learning and knowledge sharing in its lending operations. The Bank has 84 CHAPTER 5 IMPLICATIONS a golden opportunity right now. Three main implications emerge for the Bank from the analysis presented in this evaluation. STRENGTHENING THE INTERPERSONAL DIMENSION OF LEARNING AND CONNECTIVITY The new Bank aspires to be a Solutions Bank, not a Knowledge Bank. In the past, the Bank sought to strengthen its position as the world’s leading repository of knowledge about development. The Bank’s new thrust recognizes, first, that stored knowledge quickly gets stale, partly because updating websites and archives is time consuming and tedious; and second, that the Bank needs to become smarter at linking to knowledge outside the Bank, rather than focusing solely on improving the quality of its own products. The literature shows that it is the interpersonal dimension—from mentoring to informal conversations with experts—that drives learning more than individual work in isolation. Improving the format and surface polish of documents may produce fewer results than strengthening the opportunities for informal exchanges within the Bank and outside the Bank. Connectivity needs to be enhanced, drawing on the insights gleaned from the Bank’s recent experience with organizational network analysis. The need for documentation will vary from project to project. If a project involves tried and tested solutions that are not subject to immediate change, the project experience will likely be amenable to codification and distillation. If, however, the project involves solutions whose effectiveness in particular circumstances is not yet fully known or whose solutions vary significantly depending on the context, experimentation, iteration, flexibility, and adaptation will be key. In these cases, any written or electronic documentation of the project’s experience would best be in the form of options considered, pros and cons of each option, the option chosen and why, what trade-offs were made, and what the preconditions of success were or why the project failed, while also identifying a series of questions to ask that help customization to the local context. Furthermore, an electronic rolodex of experts organized in a search-friendly way to encourage person-to-person conversations and brainstorming would be helpful in these cases since there are no ready-made, off-the-shelf solutions. At the same time, connectivity requires ease of navigating the systems for capturing, storing, and collating knowledge. This is reflected in the change agenda of the World Bank Group, which includes as one of its objectives transforming the organization’s information technology. The lack of systematic documentation and difficulty retrieving key operational information has hampered learning and knowledge sharing in the Bank Group. Obtaining the right information comes down too often to luck. Better documentation that is easily searchable and retrievable would enhance operational effectiveness while also bringing efficiency gains. 85 CHAPTER 5 IMPLICATIONS REWARDING LEARNING AND KNOWLEDGE SHARING Revamping the organizational structure may be an important component of change. Yet, by itself, it has serious limitations as shown by evaluations of past Bank reorganizations, notably the reorganization that created the matrix system. Without tackling the underlying constraints relating to incentives, the benefits of reorganizations and other measures aimed at fostering learning and knowledge sharing will be limited. The Bank’s failure to adequately address the issue of incentives for learning and knowledge is an illustration of the discrepancy between private and social costs and benefits. The benefit to the individual of dedicating time, resources, and attention to knowledge is lower than the benefit to the institution. As a result, knowledge is under-produced and under-consumed. This discrepancy arises in part from the fact that development outcomes can be assessed only in the medium and long term, whereas performance evaluation of staff and managers has to be done annually. The challenge is to bridge the timing disconnect in a manner that makes conceptual and operational sense. Recognition programs to reward staff should be perceived to be fair and transparent. Measuring staff performance to base recognition programs on is not straightforward. In addition to the time lags between inputs and outcomes, the nonlinear links between most outputs and outcomes, and the difficulties in separating team and individual contribution, make the assessment particularly complex. A supportive environment (mainly adequate time and budgets) and nonmonetary recognition are more important than individual financial incentives in motivating staff behavior at the Bank. Nonmonetary recognition can include praise from the manager, leadership attention, and opportunities to lead institutionally important tasks. The Overall Performance Evaluation (OPE) rating on learning and knowledge sharing carries little weight in influencing the overall assessment. Work program agreements, individual results agreements, and individual performance assessments could reward learning more effectively, but this presupposes systematic alignment throughout the organization, as envisaged by the new architecture of the Bank embodied in the Corporate Scorecard. The Bank needs to find an appropriate system to effectively track commitment to learning, ensuring that this is reflected in staff performance. LEADERSHIP FROM SENIOR MANAGEMENT Literature on organizational change (Schein 1997; Senge 2010) suggests that learning and knowledge sharing only flourishes when there is senior management commitment, leadership, signaling, and role modeling. 86 CHAPTER 5 IMPLICATIONS With regard to commitment, the Bank is perceived by staff to be committed to learning and knowledge sharing. Nearly 60 percent of respondents to IEG’s survey indicate that they strongly agree or agree with the statement that the Bank is committed to promoting learning and knowledge sharing in its lending operations. The findings of the 2013 Employee Engagement also show that Bank staff rate highly the opportunities the Bank gives them for learning. The Bank’s leadership can capitalize on this favorable staff perception and contribute in the following ways: • Ensuring that the pressure to lend complements rather than competes with learning. About 70 percent of respondents to IEG’s survey of Bank staff feel that the pressure to lend has crowded out learning. The conversation in the Bank needs to shift from lending versus learning to learning from lending, striking the right balance between the two. This then necessitates an ongoing conversation regarding lending with learning versus lending without learning. • Determining the amount, nature (monetary compensation versus nonmonetary recognition), and basis for rewarding staff for learning and knowledge sharing (outcome versus Bank performance), and penalties for repeating mistakes and hoarding knowledge. Both carrots and sticks will be required. • Indicating to staff how much risk is permissible to take and how much failure the Bank is willing to tolerate (the Bank should not backtrack once increased risk taking begins to result in increased failure). This step requires an understanding of how good management can be linked to poor performance but appropriate risk. • Requiring evidence from the task teams that they have consulted and used available evidence relevant to the lending operation at hand, and that the selected peer reviewers have the necessary expertise and provide a critique rather than rubberstamp. With regard to signaling, budget and time allocations are key levers. If results, development outcomes, implementation, and learning are important, the Bank needs to fund supervision sufficiently. Similarly, if knowledge transfer is important, the Bank needs to signal this by allowing staff adequate time for sharing knowledge. A critical requirement for signaling is consistency. Unless signals converge and point in the same direction, they will be ineffective or even counterproductive. Stated and unstated signals, formal and informal ones, all need to say the same thing with the same force, and all parts of Bank management need to talk with one voice. So if learning matters on paper, it also needs to mater in practice. And if an OPE 87 CHAPTER 5 IMPLICATIONS rating on learning and knowledge sharing is introduced on paper, it needs to also be taken seriously in practice. With respect to role modeling, experience both with successful and failed efforts at culture change underscore that leading by example is the only way by which leaders can bring about culture change. Leaders must articulate and model the behaviors and values that define the evolving culture of the organization, and then spread them constantly through personal contact and communication. Therefore, if the Bank wants to encourage learning from failure and risk taking, staff has to feel it has a safe environment in which to experiment, raise questions and concerns, and propose new ideas. This will require senior management to: • model openness, intellectual curiosity, and humility; • acknowledge lack of definitive answers to the problems the organization is dealing with; • not react defensively or become risk averse when failure occurs, but rather take failure in its stride; and • ask questions demonstrating genuine and authentic interest in what people have to say. Table 5.1 provides additional detail about the implications for the Bank that emerge from the main findings of this evaluation. For ease of navigation, the implications are broken down by the chapters and the topics under which the findings were presented. Table 5.1. Detailed Implications for the Bank that Flow from Each of the Main Findings Chapter Topics Findings Implications Time The staff’s number one Work program agreements recommendation is to throughout the organization 2 earmark time for learning in could be better aligned, all lending. allocating time for learning. Sources of knowledge Interpersonal exchanges Connections to experts inside are the main source of and outside could be learning in lending; strengthened, with time knowledge is perceived as allowed for networking. easy to access but... Knowledge management ... The Bank’s system for Implement the Bank Group’s capturing, storing, and recently declared objective of collating knowledge is less designing and implementing smart than that of other information technology leading knowledge systems that deliver organizations. transformative change; monitor knowledge flow in the 88 CHAPTER 5 IMPLICATIONS Chapter Topics Findings Implications Corporate Scorecard. Document use Little use is made of non- Rather than revamping the Bank sources, and many documents and creating new types of Bank documents websites, it may be more are not consulted. effective to foster learning through interpersonal exchanges such as mentoring and quality assurance clinics. External validity Country-specific knowledge The Global Practices and IEG may have been crowded out could separately monitor by operational and technical operational, technical and knowledge; lessons learned country-specific lesson in one country may be hard learning, ensuring global to generalize to others. reach does not compromise country specificity. Mentoring Mentoring is critical, and the There would seem to be a staff wants more of it. case for building on the Youth 3 to Youth program, Africa Region, and other recent mentoring initiatives. Connectivity Cross support and The work by the Finance and communities of practice Private Sector Development languished respectively for Network and the Energy and structural and budgetary Mining Family could be reasons; organizational usefully consolidated and network analysis has further selectively replicated, using highlighted the constraints, organizational network including slow integration of analysis as a management new staff. tool to pinpoint knowledge gaps and bottlenecks and to plan for staff rotation. Teams Team diversity is perceived There may be a case for to be adequate and is vital research on how team for learning; but team composition influences project members are not always performance. sufficiently recognized. Staff rotation Task team leader (TTL) Co-TTLs and backstopping handover is a potential may be indicated. learning discontinuity. Commitment The staff generally Future staff surveys could ask perceives the Bank to be questions about learning in committed to learning, but lending specifically, assessing 4 this is not specific to adaptiveness line with the new learning in lending. emphasis on the science of delivery. Bank learning Bank spending on learning More could be made of has been stable and has training in political economy 89 CHAPTER 5 IMPLICATIONS Chapter Topics Findings Implications emphasized operational and and institutional analysis, technical learning, in line emphasizing country with employee perceptions specificity; this would help of their jobs. anticipate the external validity problem that Global Practices may encounter. Quality Quality Enhancement The One World Bank Group Reviews and peer reviews aspiration suggests that do not consistently take diverse quality assurance account of lessons learned. practices should be harmonized. Incentives Recognition by managers Staff performance evaluations and peers is a bigger and promotion decisions could motivator than pay. take more account of learning. Adaptiveness Restructuring is not Regular learning-from-failure perceived to be an obstacle, events could be promoted; but the culture does not U.S. Army-style after-action reward adaptiveness. reviews could be promoted. Smart tools There is too little evidence The impact of checklists, help yet to assess their impact desks, and awards could be on learning; by themselves, evaluated; if they are shown to they won’t transform the work, they could be expanded. culture. Customized learning instruments Learning and Innovation Instruments like these are still Loans and Intensive valid and could work if Learning Implementation incentives were better aligned. and Completion reports have not led to better learning. Implications for the Independent Evaluation Group The impact of IEG’s evaluation methodologies on Bank staff behavior, including risk aversion, needs to be examined. There is a perception among Bank staff that IEG’s internal culture is systematically biased toward downgrading projects. In this view, “IEG is a big reason why we cannot afford to be honest and is a major obstacle to learning.” Bank staff also reports that when projects are downgraded on outcome because of poor performance that is caused by factors beyond the Bank’s control, their incentive is to set the bar low and undertake risk-proof projects. Bank staff feels that being monitored by their management on the disconnect between IEG and self-rating on outcome leads to risk aversion. IEG has pointed out that by taking greater account of the Bank’s performance rating, which is based 90 CHAPTER 5 IMPLICATIONS entirely on factors within the Bank’s control, the possibility of risk aversion can be reduced. In any case, whether the perception that IEG is liable for the Bank’s risk aversion is justified or misplaced, perception is reality, and IEG needs to take serious note. Three actions are urgently warranted on IEG’s part: (i) reviewing its evaluation criteria and methodologies to examine their impact on Bank staff behavior, including risk aversion; (ii) communicating to Bank staff the underlying rationale for its evaluation criteria and methodologies; and (iii) ensuring that project iteration and project adaptiveness are adequately factored into the relevant ratings. Given that person-to-person conversations are such an important part of learning and knowledge sharing, IEG needs to strengthen its interpersonal communication with Bank staff. A clear benefit could include a stronger feedback loop of evaluation lessons into Bank lending. IEG has made a start by engaging with Regional staff through its brown-bag seminars and question-and-answer sessions on the quality of Implementation Completion and Results reports. IEG is taking steps to improve delivery on the learning part of its mandate, correcting some misalignments of the recent past. This study identified two such misalignments. First, the IEG’s 2011 self-evaluation report footnoted the learning part of the dual mandate (accountability and learning) that dates back to the inception of an independent evaluation department in 1973, but it did not assess IEG's contribution to learning (IEG 2011). Second, the terms of reference of the Director General, Evaluation (DGE), which are in the public domain and therefore part of IEG’s public image, are explicit about accountability but not about learning. These shortcomings have been addressed. The DGE's FY14 results agreement for staff spells out a commitment to promote learning as well as accountability. Moreover, IEG's long-term agenda and current work plan have sought to enhance systematically the learning function in IEG. To this end, in FY14, a budget of almost $1 million was allocated specifically for the production of a new brand of learning reports, entailing close engagement with, and broad dissemination to, Bank operational staff. IEG is committed to a medium-term assessment of the use made of these learning products and will monitor carefully the impact on the Bank staff’s perception of how well it discharges its responsibility for learning. Implications for the Next Evaluation The second evaluation will further investigate issues emerging from this first evaluation. It will explore the extent to which learning takes place within and 91 CHAPTER 5 IMPLICATIONS between Bank projects in a purposive sample of client countries and sectors. Part of this exercise will involve asking the Bank’s clients and development partners to assess how well the Bank is learning from lending. It will also involve an attempt— using country case studies—to map the social network in which individual projects are embedded and to explore the learning trajectories of individuals and teams. The country case studies will also provide an opportunity to test and refine the learning model, relating to how the process of acquiring knowledge and sharing and engaging with it leads to learning. Particular attention will be paid to the context in which lessons are generated and the extent to which the specificity of a given context limits the scope for transmitting learning across time and space. To the extent possible given time and budget constraints and assuming that implementation by the Bank of incentive reforms and identified good practices keeps apace and allows for early evaluation, Evaluation II will address the following questions: • To what extent do staff not have time for learning and knowledge sharing versus to what extent are they reluctant to make time for learning and knowledge sharing? • How has the scale and nature of a project’s monitoring and evaluation data influenced learning within and across projects, and to what extent has it contributed to project performance? • How effective are any ongoing Bank incentive reforms? How can the Bank structure its reward and recognition programs to encourage learning and knowledge sharing and discourage the hoarding of knowledge? • How do learning incentives vary between country- and headquarters-based staff? How has decentralization of Bank staff to country offices affected learning and knowledge sharing among Bank staff and with country clients? • What has been the Bank’s experience in encouraging smart risk taking and learning from failure, and how can it improve its record? • How can specific processes and instruments such as peer review, quality enhancement reviews, mentoring, and handover processes be strengthened to help improve learning from lending? • How can corporate work such as Development Economics reports, IEG evaluations, reviews and retrospectives from Operations Policy and Country Services, World Bank Institute reports, and Bank and non-Bank impact evaluations better inform the project cycle? • To what extent are the good practices identified by sector boards and Regional staff effective, and should they be promoted more widely? • To what extent is the Bank’s how-to guidance effective for designing and implementing Bank lending operations, just-in-time help desks for 92 CHAPTER 5 IMPLICATIONS knowledge, one-stop shops for operational knowledge, and checklists identifying critical actions or guidance in designing and implementing Bank lending operations, and how can they be improved? • How effective are different types of learning, i.e., technical, operational, and country-specific, and how can they be strengthened? • Based on a discussion with Bank staff on what they consider to be different formats for learning, which formats are effective under which circumstances? References IEG (Independent Evaluation Group). 2011. Self-Evaluation of the Independent Evaluation Group. Washington, DC: World Bank Group. Schein, E.H. 1997. Organizational Culture and Leadership. Second edition. New York: Jossey-Bass. Senge, P. 2010. The Fifth Discipline: The Art and Practice of the Learning Organization. Revised edition. New York: Crown Business/Random House. 93 Bibliography Ahuja, G., and C.M. Lampert.2000. “Entrepreneurship in the Large Corporation: A Longitudinal Study of How Established Firms Create Breakthrough Inventions.” Strategic Management Journal 22(6–7):521–543. Alavi, M., T.R. Kayworth, and D.E. Leidner.2005. “An Empirical Examination of the Influence of Organizational Culture on Knowledge Management Practices.” Journal of Management Information Systems 22(3):191–224. Argote, L., and P. Ingram. 2000. “Knowledge Transfer: A Basis for Competitive Advantage in Firms. “Organizational Behavior and Human Decision Processes 82(1):150–169. Arthur, M. B., R. DeFillippi, and C. Jones. 2001. “Project-Based Learning as the Interplay of Career and Company Non-Financial Capital.” Management Learning 32(1):99–117. Arumugam, V., J. Antony, and M. Kumar.2013. “Linking Learning and Knowledge Creation to Project Success in Six Sigma Projects: An Empirical Investigation.” International Journal of Production Economics 141(1):388–402. Bakker, R.M., B. Cambre, L. Korlaar, and J. Raab.2011. “Managing the Project Learning Paradox: A Set-Theoretic Approach toward Project Knowledge Transfer.” International Journal of Project Management 29(5):494–503. Burns, T., and G.M. Stalker.1961. The Management of Innovation. London: Tavistock. Butler, R. J., and D.C. Wilson. 1990. Managing Voluntary and Non-profit Organizations. London: Routledge. Cacciatori, E. 2008. “Memory Objects in Project Environments: Storing, Retrieving, and Adapting Learning in Project-Based Firms.” Research Policy 37 (9): 1591–1601. Cameron, K.S., and R.E. Quinn. 2005. “Diagnosing and Changing Organizational Culture: Based on the Competing Values Framework.” Revised edition. San Francisco: Jossey-Bass. Chakrabarty, B., and A. Shkilko.2013. “Information Transfers and Learning in Financial Markets: Evidence from Short Selling Around Insider Sales. “Journal of Banking and Finance 37(5):1560– 1572. Chinowsky, P.S., H. Brown, A. Szajnman, and A. Realph.2006. “Developing Knowledge Landscapes through Project-Based Learning. “Journal of Professional Issues in Engineering Education and Practice 132(2):118–124. Cohen, W.M., and D. A. Levinthal. 1990. “Absorptive-Capacity—A New Perspective on Learning and Innovation.” Administrative Science Quarterly 35(1):128–152. Cooper, K.G., J.M. Lyneis, and B.J. Bryant. 2002. “Learning to Learn, from Past to Future.” International Journal of Project Management 20 (3): 213–219. Cross, R., and L. Baird. 2000. “Technology Is Not Enough: Improving Performance by Building Organizational Memory. “Sloan Management Review 41(3):69–79. Cross, R., and L. Sproull.2004. “More than an Answer: Information Relationships for Actionable Knowledge.” Organization Science 15(4):446–462. DeFillippi, R.J. 2001. “Introduction: Project-Based Learning, Reflective Practices and Learning Outcomes.” Management Learning 32(1):5–10. 94 BIBLIOGRAPHY DeFillippi, R.J. and M. B. Arthur.1998. “Paradox in Project-Based Enterprise: The Case of Film Making.” California Management Review 40(2):125. Dyer, J., H. Gregorsen, and C. Christensen. 2009. “The Innovator’s DNA.” Harvard Business Review 87 (12): 60–67. Easterby-Smith, M., M. A. Lyles, and E.W. Tsang. 2008. “Inter-Organizational Knowledge Transfer: Current Themes and Future Prospects.” Journal of Management Studies 45(4): 677–690. Edmondson, A.C. 2002. “The Local and Variegated Nature of Learning in Organizations: A Group- Level Perspective.” Organization Science 13(2):128–146. _____. 2008. “The Competitive Imperative of Learning.” Harvard Business Review, July–August. Edmondson, C., and I.M. Nembhard.2009. “Product Development and Learning in Project Teams: The Challenges Are the Benefits.” Journal of Product Innovation Management 26(2): 123–138. Estis, A. 1998. “The Balanced Scorecard—Applying a Private Sector Technique to the Public Sector.” Presentation to the Conference of the Association for Public Policy Analysis and Management, New York City, October 31, 1998. Foss, N.J., K. Husted, and S. Michailova.2010. “Governing Knowledge Sharing in Organizations: Levels of Analysis, Governance Mechanisms, and Research Directions.” Journal of Management Studies 47(3):455–482. Gann, D.M., and A.J. Salter. 2000. “Innovation in Project-Based, Service-Enhanced Firms: The Construction of Complex Products and Systems.” Research Policy 29(7–8):955–972. Garrick, J., and S. Clegg.2001. “Stressed-Out Knowledge Workers in Performative Times: A Postmodern Take on Project-Based Learning.” Management Learning 32(1):119–134. Ghosh, A. 2004. “Learning in Strategic Alliances: A Vygotskian Perspective.” The Learning Organization 11 (45): 302–311. Gibson, C. B., and J. Birkinshaw. 2004. “The Antecedents, Consequences, and Mediating Role of Organizational Ambidexterity.” Academy of Management Journal 47(2):209–226. Grabher, G. 2004. “Learning in Projects, Remembering in Networks? Communality, Sociality, and Connectivity in Project Ecologies.” European Urban and Regional Studies 11(2):103–123. Gray, J.H., and I.L. Densten.2005. “Towards an integrative model of organizational culture and knowledge management.” International Journal of Organizational Behavior 9(2): 594–603. Hall, M., E. Kutsch, and D. Partington. 2012. “Removing the Cultural and Managerial Barriers in Project-to-Project Learning: A Case from the UK Public Sector.” Public Administration 90(3): 664–684. Hammami, H., N. Amara, and R. Landry.2013. “Organizational Climate and Its Influence on Brokers’ Knowledge Transfer Activities: A Structural Equation Modeling.” International Journal of Information Management 33(1):105–118. Harford, Tim. 2011. Adapt: Why Success Always Starts with Failure. New York: Farrar, Straus and Giroux. Hobday, M. 2000. “The Project-Based Organization: An Ideal Form for Managing Complex Products and Systems?” Research Policy 29(7–8):871–893. Holzmann, V. 2013. “A Meta-Analysis of Brokering Knowledge in Project Management.” International Journal of Project Management 31(1):2–13. 95 BIBLIOGRAPHY Hult, G. T.M., D.J. Ketchen Jr., and E.L. Nichols Jr. 2000. “Organizational Learning in Global Supply Management: A Model and Test of Internal Users and Corporate Buyers.” Decision Sciences 31(2):293–325. IEG (Independent Evaluation Group). 2010. Poverty Reduction Support Credits: An Evaluation of World Bank Support. Washington, DC: World Bank. ______. 2013a. Biennial Report on Operations Evaluation: Assessing the Monitoring and Evaluation Systems of IFC and MIGA. Washington, DC: World Bank. doi: 10.1596/978-0-8213-9918-7. Creative Commons Attribution CC BY 3.0. ______. 2013b. Knowledge-Based Country Programs: An Evaluation of the World Bank Group Experience. Washington, DC: World Bank Group. Jo, I.H. 2011. “Effects of Role Division, Interaction, and Shared Mental Model on Team Performance in Project-Based Learning Environment.” Asia Pacific Education Review 12(2):301–310. Jones, Harry, and Enrique Mendizabal. 2010. Strengthening Learning from Research and Evaluation: Going with the Grain. London: Overseas Development Institute. Keegan, A., and J.R. Turner.2001. “Quantity versus Quality in Project-Based Learning Practices.” Management Learning 32(1):77–98. Koners, U., and K. Goffin. 2007. “Learning from Postproject Reviews: A Cross-Case Analysis.” Journal of Product Innovation Management 24 (3): 242–258. Koskinen, K.U. 2004. “Knowledge Management to Improve Project Communication and Implementation.” Project Management Journal 35 (2): 13–19. Koskinen, K.U. 2012. “Problem Absorption as an Organizational Learning Mechanism in Project- Based Companies: Process Thinking Perspective.” International Journal of Project Management 30(3):308–316. Laurikkala, H., H. Vilkman, and K. Tanskanen. 2002. “Project Management Knowhow Transfer in a Networked Multi-Project Environment.” Paper presented at the 8th International Conference on Concurrent Enterprising, Rome, Italy, June 17–19, 2002. Lin, Y. C., and H.Y. Lee.2012. “Developing Project Communities of Practice-Based Knowledge Management System in Construction.” Automation in Construction 22:422–432. Lindkvist, L. 2004. “Governing Project-Based Firms: Promoting Market-Like Processes within Hierarchies.” Journal of Management and Governance 8(1):3–25. Lundin, R. A., and A. Söderholm.1995. “A Theory of the Temporary Organization.” Scandinavian Journal of Management 11(4):437–455. March, J.G., and J.P. Olsen. 1995. Democratic Governance. New York: Free Press. McKee, D. 1992. “An Organizational Learning Approach to Product Innovation.” Journal of Product Innovation Management 9(3):232–245. McKinsey Global Public Sector Practice.2010. Making It Work in Government. Perspectives on Transforming Performance in the Public Sector. Washington, DC: McKinsey &Company. Mezias, S., and M. A. Glynn. 1993. “The Three Faces of Corporate Renewal: Institution, Revolution, and Evolution.” Strategic Management Journal 14:77–101. Midler, C., and R. Beaume.2010. “Project-Based Learning Patterns for Dominant Design Renewal: The Case of Electric Vehicle. “International Journal of Project Management 28(2):142–150. 96 BIBLIOGRAPHY Nation, M.L. 2008. “Project-Based Learning for Sustainable Development.” Journal of Geography 107 (3): 102–111. Newell, S., M. Bresnen, L. Edelman, H. Scarbrough, and J. Swan. 2006. “Sharing Knowledge across Projects—Limits to ICT-Led Project Review practices.” Management Learning 37(2):167–185. Newell, S., C. Tansley, and J. Huang.2004. “Social Capital and Knowledge Integration in an ERP Project Team: The Importance of Bridging AND Bonding.” British Journal of Management 15:S43–S57. O’Dell, C., and C. Hubert. 2011. The New Edge in Knowledge: How Knowledge Management Is Changing the Way We Do Business. Hoboken: John Wiley and Sons. Orton, J.D., and K.E. Weick.1990. “Loosely Coupled Systems – A Reconceptualization.” Academy of Management Review 15 (2): 203–223. Papadopoulos, T., T. Stamati, and P. Nopparuch. 2013. “Exploring the Determinants of Knowledge Sharing via Employee Weblogs.“ International Journal of Information Management 33 (1): 133– 146. Pemsel, S., and R. Muller.2012. “The Governance of Knowledge in Project-Based Organizations.” International Journal of Project Management 30(8):865–876. Pemsel, S., and Wiewiora, A. 2013. “Project Management Office: A Knowledge Broker in Project- Based Organisations.”International Journal of Project Management 31(1):31–42. Phillips, David A. Reforming the World Bank: Twenty Years of Trial and Error. Cambridge: Cambridge University Press 2009. Pinto, J.K. 2013. “Lies, Damned Lies, and Project Plans: Recurring Human Errors that Can Ruin the Project Planning Process.” Business Horizons56 (5):643–653. PMI (Project Management Institute). 2004. A Guide to the Project Management Body of Knowledge. Third edition (PMBOK Series). Newtown Square, Pa.: PMI. Poell, R.F., L. Yorks, and V.J. Marsick.2009. “'Organizing Project-Based Learning in Work Contexts: A Cross-Cultural Cross Analysis of Data from Two Projects.” Adult Education Quarterly 60(1):77–93. Prencipe, A., and F. Tell. 2001. “Inter-Project Learning: Processes and Outcomes of Knowledge Codification in Project-Based Firms.” Research Policy 30(9):1373–1394. Pritchett, Lant, Salimah Samji, and Jeffrey Hammer. 2013. “It’s All About MeE: Using Structured Experiential Learning (“e”) to Crawl the Design Space.” Working Paper 322, Center for Global Development, Washington, DC. Prusak, L. 1997. Knowledge in Organizations. Oxford: Butterworth-Heinemann. Raelin, J. A. 2001. “Public Reflection as the Basis of Learning.” Management Learning 32(1):11–30. Ramalingam, Ben. 2013. Aid on the Edge of Chaos: Rethinking International Cooperation in a Complex World. Oxford: Oxford University Press. Riege, A. 2005. “Three-Dozen Knowledge-Sharing Barriers Managers Must Consider.” Journal of Knowledge Management 9(3):18–35. Robinso, B. 1994. “Voluntary Bodies as Learning Organizations.” The Learning Organization 1(3):10–15. Scarbrough, H., J. Swan, S. Laurent, M. Bresnen, L. Edelman, and S. Newell.2004. “Project-Based Learning and the Role of Learning Boundaries.” Organization Studies 25(9):1579–1600. Schein, E.H. 1990. “Organizational Culture.” American Psychologist 45(2):109–119. 97 BIBLIOGRAPHY Senge, P.M. 1991.”Transforming the Practice of Management.” Paper presented at the Systems Thinking in Action Conference, Boston, November 14, 1991. Sense, A. J. 2003. “A Model of the Politics of Project Leader Learning.” International Journal of Project Management 21 (2): 107–114. ______.2007a. “Learning within Project Practice: Cognitive Styles Exposed.” International Journal of Project Management 25 (1): 33–40. ______. 2007b. “Structuring the Project Environment for Learning.” International Journal of Project Management25:405–412. ______.2008. “The Conditioning of Project Participants’ Authority to Learn within Projects.” International Journal of Project Management 26:105–111. ______.2011. “The Project Workplace for Organizational Learning Development.” International Journal of Project Management 29(8):986–993. ______.2013. “A Project Sponsor's Impact on Practice-Based Learning within Projects.” International Journal of Project Management 31(2):264–271. Sense, A.J., and M. Antoni. 2003. “Exploring the Politics of Project Learning.” International Journal of Project Management 21:487–494. Sommer, S.C., and C.H. Loch. 2004. “Selectionism and Learning in Projects with Complexity and Unforeseeable Uncertainty.” Management Science 50 (10): 1334–1347. Sud, Inder. 2014. “Technical Focus or Country Focus?” The 1818 Society Bulletin 3 (19): 36–38, March. Van den Bosch, F. A.J., H.W. Volberda, and M. de Boer.1999. “Coevolution of Firm Absorptive Capacity and Knowledge Environment: Organizational Forms and Combinative Capabilities.” Organization Science 10 (5): 551–568. vanRooij, S.W. 2009. “Scaffolding Project-Based Learning with the Project Management Body of Knowledge (PMBOK).”Computers & Education 52 (1): 210–219. vanWijk, R., J.J.P. Jansen, and M. A. Lyles. 2008. “Inter- and Intra-Organizational Knowledge Transfer: A Meta-Analytic Review and Assessment of Its Antecedents and Consequences.” Journal of Management Studies 45 (4): 830–853. Williams, T. 2004. “Identifying the Hard Lessons from Projects—Easily.” International Journal of Project Management 22 (4): 273–9. Wong, P.S.P., S. O. Cheung, R.Y.L. Yiu, and M. Hardie. 2012. “'The Unlearning Dimension of Organizational Learning in Construction projects.” International Journal of Project Management 30 (1): 94–104. World Bank Institute. 2011. The Art of Knowledge Exchange: A Results-Focused Planning Guide for Development Practitioners. Washington, DC: World Bank Institute. Yakhlef, A. 2010. “The Three Facets of Knowledge: A Critique of the Practice-Based Learning Theory.” Research Policy 39 (1): 39–46. 98