43166 The Alignment of Staff Knowledge and Learning with the World Bank's Strategic Priorities: An Assessment Marlaine Lockheed Violaine Le Rouzic The Alignment of Staff Knowledge and Learning with the World Bank's Strategic Priorities: An Assessment Marlaine Lockheed Violaine Le Rouzic WBI Evaluation Studies EG08­133 World Bank Institute The World Bank Washington, D.C. November 2007 Acknowledgments The authors are very grateful to the 23 Regional managers and 17 knowledge and learning coordinators who generously answered our interview questions, to the 299 World Bank staff members who completed our survey, and to the Chief Learning Officer's office, notably Ana-Maria Arriagada, Jan Weetjens, Donald MacDonald, Brigitte Petit, Kristin Strohecker, Hye-jung Hwang, Katharina Ferl, and Morallina George, who provided feedback, data, and information used in the study. This report also benefited from the thoughtful review of Klaus Tilmes and Michael Randel from the Independent Evaluation Group Knowledge and Evaluation Capacity Development (IEGKE), and of Rema Balasundaram from the Quality Assurance Group (QAG). Thanks also to Dawn Roberts for her contribution to the coding of the learning activities recorded in the World Bank's Learning Management System (LMS), to Maurya West Meiers for early contributions to project conceptualization, to Anwesha Prabhu and Izlem Yenice for excellent research assistance, to Elisabeth De Meuter and La Shann Johnson for scheduling the interviews, to Nidhi Khattri and Jaime Quizon for reviewing our survey instrument, and to Humberto Diaz and Sadaf Tajammal for document formatting. The World Bank Institute Evaluation Group (WBIEG) evaluates learning by staff of the World Bank and activities of the World Bank Institute. The findings, interpretations and conclusions expressed in WBI Evaluation Studies are entirely those of the authors and do not necessarily represent the views of the World Bank Group, including WBI or the Bank's Knowledge and Learning Board, for which this study was completed. WBI Evaluation Studies are available at http://www.worldbank.org/wbi/evaluation Suggested citation: Lockheed, Marlaine, and Violaine Le Rouzic. 2007. The Alignment of Staff Knowledge and Learning with the World Bank's Strategic Priorities: An Assessment. Report No. EG08­133 Washington, DC: World Bank Institute. Vice President, World Bank Institute Rakesh Nangia, Acting Chief Learning Officer Ana-Maria Arriagada, Acting Manager, Institute Evaluation Group Nidhi Khattri, Acting Task Team Leader Violaine Le Rouzic ii Table of Contents ACRONYMS AND ABBREVIATIONS ...................................................................................v EXECUTIVE SUMMARY.................................................................................................. vii 1. INTRODUCTION............................................................................................................1 2. DETAILED ANALYSIS OF ALIGNMENT.........................................................................4 Data sources..............................................................................................................4 Regional Strategy and Performance Contracts identify staff skills as risk...............4 FY06 Regional K&L Plans lack specificity about learning programs .....................6 A glass half full: Learning received is only partially aligned with corporate priorities....................................................................................................................9 3. WHAT ACCOUNTS FOR THE GLASS HALF EMPTY?..................................................11 Governance of learning is multi-layered and transaction intensive........................11 Identifying good available learning opportunities can be difficult.........................12 The LMS is an inappropriate tool for accountability..............................................13 Some learning is not recorded ................................................................................13 What managers see as impediments to learning.....................................................14 4. RECOMMENDATIONS.................................................................................................15 Actionable recommendations not requiring changes in budget management ........15 Actionable items requiring agreement of CSRRM.................................................17 A radical reform specific to governance.................................................................18 REFERENCES ..................................................................................................................20 APPENDIXES Appendix A: Risk Assessment, FY06 and FY07 Regional Strategy and Performance Contracts.....................................................................21 Appendix B: List of Documents Consulted...........................................................22 Appendix C: Managers' Ascriptions of Decision-making Authority for Learning (pro-rated votes)...............................................................27 Appendix D1: Regional Learning Received for Raising Awareness (in hours) .....28 Appendix D2: Regional Learning Received for Building Individual Capabilities (in hours)......................................................................32 iii Appendix E: Logistic Regression Analysis of LMS Data for Six Regions, FY06 ................................................................................................36 Appendix F: Organizational Chart for Regional Learning Coordinators...............40 Appendix G: Survey of Staff from Sampled Regional Units.................................41 Appendix H: List of Recommendations for the LMS ............................................42 iv Acronyms and Abbreviations ACS Administrative and Client Support AFR Africa Region AFRVP Office of the Regional Vice President of the Africa Region AFTQK Africa Technical Families: Operational Quality and Knowledge AFTRL Africa Technical Families: Operational Quality and Knowledge Results and Learning AIS Activity Initiation Summary BBL Brown Bag Lunch CD-ROM Compact Disk - Read Only Memory CLO Chief Learning Officer CO Country Office CSRRM Resource and Performance Management Department EAP East Asia and Pacific Region EAPCO East Asia and Pacific Central Operational Services Unit EAPVP Office of the Regional Vice President of the East Asia and Pacific Region ECA Europe and Central Asia Region ECAVP Office of the Regional Vice President of the Europe and Central Asia Region ESSD Environmentally and Socially Sustainable Development Network EXT External Affairs, Communications and United Nations Affairs FSE Financial Sector Vice Presidency FY Fiscal Year GA-GD Grade A through Grade D GDLN Global Development Learning Network GE Grade E GF Grade F GG Grade G GH Grade H HDN Human Development Network HQ Headquarters HR Human Resources HRS Human Resources Vice Presidency IBRD International Bank for Reconstruction and Development ILP Individual Learning Plan ISG Information Solutions Group IT Information Technology JPA Junior Professional Associate K&L Knowledge and Learning v KLB Knowledge and Learning Board LCR Latin America and Caribbean Region LCRVP Office of the Regional Vice President of the Latin America and Caribbean Region LCSLU Latin America and Caribbean Region Operational Services Regional Learning Unit LCSOS Latin America and Caribbean Region Operational Services LIC Low-income country LMS Learning Management System MIC Middle-income country MNA Middle East and North Africa Region MNACS Middle East and North Africa Region Operational Core Services MNAOS Middle East and North Africa Region Operations Services MNAVP Office of the Regional Vice President of the Middle East and North Africa Region OECD Organisation for Economic Co-operation and Development OPCS Operations Policy and Country Services Vice Presidency P&T Professional and Technical PREM Poverty Reduction and Economic Management Network PSD Private Sector Development Network RLC Regional Learning Coordinator RMT Regional Management Team SAP Systems, Applications, and Products SAR South Asia Region SAROQ South Asia Region Operations Quality Unit SARSQ South Asia Region Operational Services and Quality SARVP Office of the Regional Vice President of the South Asia Region SFR Strategy Finance and Risk SLC Strategic Learning Center SPC Strategy and Performance Contract TF Trust Fund TLAP Trust Fund Learning and Accreditation Program TTL Task Team Leader VP Vice President VPU Vice Presidential Unit WBI World Bank Institute WBIEG World Bank Institute Evaluation Group WPA Work Program Agreement vi EXECUTIVE SUMMARY 1. By the beginning of fiscal year 2006 (FY06), a combination of three conditions--the sophistication of middle-income country counterparts, decentralization of the World Bank to over 100 country offices and increased reliance on newly hired, local (frequently junior) staff and consultants--had created a situation whereby the highest performance risks in all Regions related to staff capabilities. Regional Managers expressed concern about shortcomings in staff (a) professional and technical expertise, including sensitivity to the political economy of the country and ongoing global issues, (b) practical operational knowledge, including project implementation skills, and (c) communications skills and country knowledge needed for being effective on the ground. Yet nearly half of observable learning opportunities offered to and received by Regional staff in FY06 were not aligned with these priorities, having instead the objective of raising awareness rather than building key skills. 2. Compounding this and despite the recognized efforts of many Regional and Strategic Learning Center learning coordinators, internal formal staff learning remained a low priority with front line managers. Moreover, the Bank's budget process was not designed to deliver resources for learning where they were most needed, with all Regions receiving the same notional share of budget for learning in their annual budget envelope and most of this budget distributed to and managed by front-line units. Over three-quarters of Regional learning resources were, de facto, not subject to Knowledge and Learning Board (KLB) review or included in Knowledge and Learning (K&L) Plans. 3. This report examines the alignment of staff learning with World Bank business priorities, and makes recommendations on how to improve alignment. The data analyzed in this report come from (a) interviews with a random sample of Regional unit managers, stratified by headquarters and country office and by Region, (b) a two- question survey of the staff of those managers, (c) interviews with Learning Coordinators from all Regions, all Network Anchors and designated other representatives on the KLB, (d) Learning Management System (LMS) records of all learning sessions for FY06 and FY07 as of May 1, 2007, (e) selected human resources' data regarding learning participants, (f) Regional Strategy and Performance Contracts for FY06 and FY07, (g) the World Bank Group Staff Survey of 2005, (h) K&L Plans for FY06, (i) quarterly staffing reports for FY06, and (j) background studies and reports. 4. The study concludes that staff learning is only partially aligned with corporate and Regional priorities, and that major revisions in current practices will be required to improve alignment considerably. The most essential will be to: vii · subject staff learning to the standard World Bank business processes for Work Program Agreements; · consolidate learning into major programs for Professional and Technical and Operational learning; · focus on more tailored approaches targeted at different groups of staff; and, · utilize the learning budget more strategically through formula funding, conditional cash transfers, reduction in fungibility, and revision of learning fiscal year. 5. It will also require some "quick fixes" to the LMS, to provide better user interfaces for essential information and facilitate better targeting of learning opportunities. A radical reform in terms of the governance of learning is also suggested. viii 1. INTRODUCTION 1.1 By the beginning of fiscal year 2006 (FY06), a combination of three conditions--the sophistication of middle-income country (MIC) counterparts, decentralization of the World Bank to over 100 country offices and increased reliance on newly hired, local (frequently junior) staff and consultants (table 1)--had created a situation whereby the highest performance risks in all Regions related to staff capabilities (appendix A).1 Table 1: High Regional decentralization and "juniorization" of staff, June 2005 AFR EAP ECA LCR MNA SAR Regional offices (number) 39 13 27 19 9 8 All staff with < 3 years in Bank (%) 29 32.5 23.4 29.7 23.3 32.6 E-level staff with < 3 years in Bank (%) 38.4 40 23.6 38.8 9.6 55.1 Source: IBRD Quarterly Staffing Report, FY04 Quarter 4 1.2 Regional Managers expressed concern about shortcomings in staff professional and technical expertise, absence of sensitivity to country context and political economy, and lack of practical operational and implementation knowledge. Yet nearly half of observable learning opportunities offered to and received by Regional staff in FY06 were not aligned with these priorities, having instead the objective of raising awareness rather than building key skills (figure 1).2 1.3 Compounding this and despite the recognized efforts of many Regional and Strategic Learning Center (SLC) learning coordinators, formal internal staff learning remained a low priority with front line managers. Consensus among managers is that the best learning comes from interaction with peers and the mentoring of junior staff by senior staff, particularly in the field. Managers also report that senior staff are best served by external learning opportunities that maintain their professional and technical expertise. Unit budgets for learning, however, are typically insufficient to support much peer learning, senior staff on-the-job mentoring of junior staff, or external professional and technical learning. 1As reported in FY06 and FY07 Regional Strategy and Performance Contracts (SPCs) 2Observable learning opportunities are those recorded in the Learning Management System (LMS). 1 Figure 1: Nearly half of staff learning sessions and hours of learning received by Regional staff in FY06 were for raising awareness rather than building staff capabilities3 2,229 sessions 196,423 hours received Develop Develop Build team leaders leaders effective- Build team ness effective- ness Raise Raise Build aw are- aw are- capabilities Build ness ness capabilities Source: Authors' analysis of LMS data for FY06 1.4 The Bank's budget process is not designed to deliver resources for learning where they are most needed. All Regions receive the same notional share of budget for learning in their annual budget envelope, without regard to need. Charges against this budget can include Regional management of learning, creation and delivery of learning, and participation in learning (learning received). Resources for knowledge are managed separately. Approximately half of the observable costs for learning received are directed to raising awareness (figure 2). Figure 2: More learning resources for raising awareness than for building capabilities4 Cost of observed learning received, FY06 Develop Build team leaders effective- ness Raise aw areness Build capabilities Source: Authors' analysis of LMS data for FY06 3In FY06, 2,832 sessions (including internal, external and on the job-learning) were recorded in the LMS. Out of them, 2,229 sessions (79 percent) were attended by at least one Regional staff member. All World Bank employees received 425,574 hours of learning, of which 196,423 hours (46 percent) were received by Regional staff members with regular, open and fixed term contracts. 4Observable costs of learning received are those imputed to observable learning sessions on the basis of session duration and the average hourly wage rate of participants; they do not include development or delivery costs. For FY06, $3,882,151 was spent on raising awareness and $3,074,558 was spent on building staff capabilities. The report's data include Brown Bag Lunches (BBLs) recorded in the LMS. Excluding BBLs, the distribution of learning received (in hours and in cost) among raising awareness, building individual staff capabilities, building team effectiveness and developing leaders, remains equivalent. 2 1.5 The remainder of this report elaborates on these conclusions and offers two sets of actionable recommendations: one set that the Knowledge and Learning Board (KLB) could implement independently and a second set that would require the agreement of the Resource and Performance Management Department (CSRRM). A third recommendation, a radical reform of learning governance, is also described. 3 2. DETAILED ANALYSIS OFALIGNMENT 2.1 We look for alignment in three dimensions: between Regional priorities and Regional Knowledge and Learning Plans, between these plans and observed learning, and between observed learning and Regional priorities. The two main sources for information about Regional priorities are the Regional Strategy and Performance Contract (SPC) and our interviews with a random sample of managers selected from country offices and headquarters, who were asked to comment on the written document and provide their own observations. Data sources 2.2 Data analyzed in this report come from (a) interviews with a random sample of Regional unit managers, stratified by headquarters (HQ) and country office (CO) and by Region, (b) a two-question survey of the staff of those managers, (c) interviews with Learning Coordinators from all Regions, all Network Anchors and designated other representatives on the KLB, (d) Learning Management System (LMS) records of all learning sessions for FY06 and FY07 as of May 1, 2007 attended by staff working in Regional VPUs, (e) selected Human Resources' (HR) data regarding learning participants, (f) Regional Strategy and Performance Contracts (SPCs) for FY06 and FY07, (g) the World Bank Group Staff Survey of 2005, (h) Knowledge and Learning (K&L) Plans for FY06, (i) quarterly staffing reports for FY06, and (j) background studies and reports (see appendix B for list of documents consulted). Table 2: Population, samples and response rates for selected data sources Sampled or Response Population designated Respondents rate Regional unit managers 279 24 23 96% Staff of Regional unit managers 4,964 457 299 65% Learning Coordinators/KLB Members 23 18 17 94% LMS learning sessions attended by at least one Regional staff member5 2,229 2,229 n.a. n.a. Source: Authors Regional Strategy and Performance Contracts identify staff skills as risk 2.3 We examined Regional SPCs for FY06 (spring 2005) and FY07 (fall 2006), to obtain an indication of Regional priorities, and--more importantly--the risks for achieving these priorities as perceived by Regional management around FY06. In all 5The study focuses on learning of Regional staff for two reasons: they represent the core business of the Bank, and they comprise over 50 percent of staff. 4 six Regions, indicators of staff skills were rated as serious risks (defined as the impact of the risk multiplied by its likelihood on scales of 1-9) for achieving Regional objectives (see appendix A for details). For AFR, ECA, and SAR, no category was given a higher risk rating than "staff skills and mix" (72, 56 and 56, respectively). For all six Regions, indicators of "agility" (competitive positioning, responsiveness to global developments, readiness for crisis assistance) were given risk ratings over 45. For AFR, EAP, and SAR implementation skills (reliable procurement, disbursements, implementation management, Trust Fund compliance, managing of projects with complex safeguard issues) were rated 56 and in LCR implementation was rated over 45; only in ECA were implementation issues not rated as risks. To place these ratings in context, in LCR, MNA and SAR workplace security issues were accorded high risk ratings (56, 63 and 56 respectively), and in LCR and MNA these issues were paramount. 2.4 In general, the interviewed managers agreed with the strategic priorities and the performance risks as identified in the Regional SPCs. All interviewed HQ managers and half of interviewed CO managers mentioned concerns related to shortcomings in staff skills, in three main areas: (a) professional and technical (P&T) expertise, including sensitivity to the political economy of the country and ongoing global issues, (b) practical operational knowledge, including project implementation skills, and (c) communications skills and country knowledge needed for being effective on the ground (table 3). Table 3: Priorities for staff learning as identified by Regional managers AFR EAP ECA LCR MNA SAR P&T knowledge (Advanced) Operational skills (including procedures, policies) Communications skills Knowledge transfer (OECD to MICs, MICs to LICs) Political economy Country and cultural knowledge Operations implementation (including reforms) P&T knowledge (Introductory) Governance and anti-corruption Cross-regional experience Source: Authors' analysis Note: This table represents a summary of 23 confidential interviews and the checks indicate whether or not a particular topic was mentioned; no statistical tests of differences were made and no statistical significance is implied by the check marks. 2.5 With respect to professional and technical expertise, Regional managers observed that senior staff working in middle-income countries (MICs) are often no better informed on issues than their sophisticated counterparts, who may have attended the same graduate programs in the United States or Europe as attended by Bank staff. 5 In AFR and SAR, managers reported that junior staff needed basic sectoral learning. Staff in LCR and AFR were seen as lacking in cross-regional experience; managers viewed LCR staff as "insular" and staff in AFR as being "stuck" in the Region. In both Regions, managers observed that clients could be better served by learning from country experience outside the Region. In particular, knowledge and skill transfers from MICs whose experience could be adapted to the low-income countries (LICs) and from OECD countries whose experience could be adapted to MICs, would be valued in several Regions. 2.6 On the operational side, the principal observations were that countries value knowledge about "how to do" rather than "what to do," and that junior task team leaders (TTLs) lacked this practical knowledge, particularly about how to implement reforms. In addition, managers noted that CO staff were less well socialized into Bank practices and culture and often lacked the full range of Operational skills. 2.7 Communication skills were consistently identified by CO managers as an issue. They reported that some international staff lacked country knowledge and cultural sensitivity, including the "soft skills" needed in MNA. The ability to deal with the political economy was identified as a learning need in EAP, LCR, and SAR. Junior TTLs, in particular, were identified as "too green" to engage in dialogue with governments; they were viewed as technically proficient but lacking in knowledge of the political economy of the country. In some Regions, the international language skills (principally spoken and written English) of local staff were weak. FY06 Regional K&L Plans lack specificity about learning programs 2.8 No discussion of Regional K&L Plans can begin without a discussion of the budgets that support these plans. Budget allocations for learning are not strategic. All Regions receive the same notional budget share for learning (approximately 3.9 percent) and most of this budget is distributed to and managed by units. Over three-quarters of Regional learning resources are, de facto, not subject to KLB review. Staff and managers drive the bulk of learning in the Regions, and K&L Plans play a small role. 2.9 Regional K&L Plans seem to apply only to the share of the Regional K&L budget that is managed by the Regional Learning Coordinator (RLC), which for FY06 ranged from less than 10 percent in AFR to nearly 40 percent in LCR and MNA (figure 3). For ECA the share was 17 percent, and for EAP and SAR it was 25 percent. This amount included resources for the RLC (salary, travel, support, and in some cases evaluations) and to underwrite Regional learning activities including staff travel (in SAR); staff-initiated "skills development grants" (in ECA); cross-sectoral, task or country teams (LCR, ECA, MNA, SAR); and mentoring or coaching programs (in LCR, EAP, ECA, MNA). In several Regions staff or units competed for resources. The remainder of the learning budget, identified in the K&L Plans for "individual learning" (or "unallocated"), was typically distributed to units and managed by unit managers. In all, therefore, 22 percent (about $7 million) of the Regional learning budget was subject to KLB review and/or authorization. 6 Figure 3: Regions vary widely with respect to the share of the learning budget retained for Regional learning Regional learning as share of FY06 Knowledge and Learning Board distributed budget 50% 40% 30% 20% 10% 0% AFR ECA SAR EAP MNA LCR Source: FY06 Regional Knowledge and Learning Plans; FY06 Annual Report on Learning 2.10 Consistent with their control of 60-90 percent of the Regional learning budget, unit managers view their staff and themselves as primarily responsible for determining how knowledge and learning resources are used, and note that most decisions regarding resource use are in response to staff initiatives (appendix C). Indeed, managers--other than those with direct involvement in Regional Learning Committees--do not view the Regional Learning Coordinators as influential with regard to staff learning. Managers report interest in distributing resources "fairly" and often establish unit learning committees to decide on the use of the unit-level learning budget, which is small, averaging $40,000-$100,000 depending on the number of staff. Some managers note that staff travel associated with periodic sectoral events held in Washington (e.g., PREM Learning Forum, Financial Management ­ Fiduciary Forum) use all their variable cost budget for learning. Managers of small remote COs view the learning budget as insufficient, particularly to cover travel to Washington or Regional hubs for learning, whereas CO managers in hubs (such as Bangkok) view staff time as a major impediment to learning. HQ managers also view staff time, rather than budget, as a concern. 2.11 In determining how to use the FY06 budget, Regional Learning Coordinators undertook serious efforts to develop K&L Plans in coordination with Regional Management Teams (RMTs). This involved staff surveys, one-on-one meetings with staff, meetings with Country and Sector Directors and Managers, reviews of staffing by unit, reviews of past learning of staff and the active involvement of Regional Learning Committees. In Regions where the Regional Learning Coordinator was a member of the RMT, this process was facilitated by regular contact with members of the RMT. 7 2.12 Clarifying alignment between the resulting plans and Regional strategies and performance risks is difficult, because the descriptions of the strategic focus/priority areas for Regional FY06 K&L Plans are typically comprehensive, but rarely indicate detailed content for the learning program. Rather, lists of anticipated courses are provided, and in many cases, only the modality of learning, rather than its content, is described (see table 4 for examples). One exception to this observation is the reference to building team effectiveness, which occurs in all K&L Plans. Moreover, the K&L Plans are viewed as "living" documents, subject to change as the fiscal year progresses and responsive to unanticipated requests from managers. Table 4: Summary of FY06 K&L Plans' strategic focus/key priority areas Region Strategic focus/key priority areas AFR [no text] List of 50 learning activities. EAP "Continue to provide delivery of core programs to country office staff...complement face-to-face course delivery at HQ, country offices and the Bangkok hub with an increasing number of workshops, short modules and distance learning...continued focus on teams and team leaders...implement an operations course...place greater attention on on-the-job- learning...explore the use of e-learning...implement the advanced operations and leadership course for managers..." List of 50 learning activities. ECA "We are focusing on the skills mix in our strategic staffing effort, retooling staff skills, including by learning from other Regions and Part I best practice institutions, and building stronger teams." List of 72 learning activities. LCR "...provide maximum support to staff on the front-line, by equipping them with strong operational, client engagement and team skills, maintaining a high level of awareness of developments on the cutting edge, and providing strong managerial and administrative support." List of 75 learning activities. MNA "...continued Regional delivery of Field Orientation workshops and of Introduction to Bank Operations...five-day workshops tailored to the needs of field office and HQ staff...allocate a significant amount of resources for...supporting operational team efficiency..." List of 26 learning activities. SAR [same as EAP] List of 69 learning activities. Source: FY06 Regional Knowledge and Learning Plans 2.13 K&L Plans for FY06 did not consistently indicate how learning resources were to be allocated across priorities or activities, and most of the activities listed in the plans did not have costs associated with them. For example, the AFR plan called for $9.6 million for learning, including $780,000 reserved for Regional activities, of which the lion's share ($700,000) was allocated to the Regional operational learning course, Assuring the Quality of Bank-Assisted Operations in Africa. The rest ($8.9 million) was designated as "unallocated" and apparently programmed at the unit level. 2.14 Many weaknesses in the FY06 K&L Plans were to have been repaired in FY07, and these comments apply only to the FY06 plans. What is clear, however, is that the Regional learning programs only partially addressed the risks identified in the SPCs and further elaborated by the Regional managers. As an example, implementation of operations was notably absent from the list of SPC performance risks in ECA, and managers emphasized the need for new, cutting edge P&T skills and expertise. Yet about one-third of the Regional learning budget was allocated to courses devoted to 8 operations and project implementation, for which only half the expected participants showed up, suggesting a supply that was not aligned with demand. By comparison, a well-regarded learning program on a new P&T topic, "Regional development," was oversubscribed, with nearly 350 participants. A glass half full: Learning received is only partially aligned with corporate priorities 2.15 The literature identifies four major objectives of staff learning: raising staff awareness, building individual staff capabilities, building team effectiveness and developing leaders (Lockheed and Arango 2005). In the World Bank, typical activities to build awareness include the orientation program for new staff, sector forums and professional and technical (P&T) brown bag lunches. Activities to build staff capabilities include formal courses such as Fundamentals of Bank Operations, and other non-course approaches, such as on-the-job training or mentoring. Team effectiveness is built through programs that directly support teams, such as the Multi-sectoral Team Learning Program. Leaders are developed through formal courses and coaching. Regional SPCs and managers emphasize the importance of building staff capabilities for carrying out operational responsibilities and for engaging in professional and technical dialogue with clients. 2.16 Analyses of the observed learning received by all Regional staff in FY06 find that the learning sessions received by Regional staff were only partially aligned with the Regional risks and weaknesses in staff skills as identified by managers (see appendix D for observed learning hours for specific topics). Specifically, the following suggest misalignment: · Nearly half of learning received was for raising awareness (accounting for 46 percent of learning sessions and 44 percent of total observed learning hours). · About half of learning received had the objective of building staff capabilities (50 percent of sessions and 44 percent of observed learning hours). · Building team effectiveness was the objective of less than 10 percent of learning (3 percent of sessions and under 10 percent of learning hours). · Most activities designed to build staff capabilities were not differentiated according to level (introductory, intermediate, advanced) to facilitate appropriate targeting of staff. · Very little learning to build staff capabilities related to the corporate topic of governance and anti-corruption (less than half a percent of learning hours). · Very little learning focused on other Regional priorities; only a few recorded sessions dealt with issues of political economy, and none dealt with issues of "agility" in responding to changes in country conditions or requests from governments. 9 2.17 In other respects, however, learning was well aligned with corporate priorities. · Nearly half of learning for building staff capabilities focused on Bank operations skills (41 percent); the remainder focused on professional and technical skills (21 percent), communications skills (17 percent), IT skills (14 percent), resource management skills (4 percent), and other various skills (3 percent).6 · Among Bank operational skills, the most frequently learned involve procurement and portfolio management. · Among professional and technical skills, the most frequently learned involve monitoring and evaluation and economics. · Learning activities received by country office staff are more likely to focus on building staff skills and less likely to focus on raising awareness, and are twice as likely to focus on building operational skills, as compared with learning activities received by headquarters staff (see appendix E for details). · Ethics is a major topic of learning for raising awareness. Ethics awareness comprises 8 percent of learning for raising awareness; only rural development and procurement receive more hours of learning. · Governance/anti-corruption is a topic of learning for raising awareness, accounting for nearly 3 percent of learning hours received for raising awareness. 6Communication skills include country knowledge and cultural sensitivity, ability to deal with the political economy, engaging in dialogue with clients, conveying messages effectively and written and spoken language skills (including English). 10 3. WHAT ACCOUNTS FOR THE GLASS HALF EMPTY? 3.1 Misalignment may arise from the governance of learning; difficulty staff and managers face in obtaining information about the quality, relevance and availability of learning opportunities; the weakness of the LMS for use as an accountability tool; failures of staff to register some team and individual learning in the LMS; and other impediments as identified by managers and prior studies. Governance of learning is multi-layered and transaction intensive 3.2 Governance of staff learning is complex and inconsistent across Regions in terms of organizational form, strategies used to identify staff learning needs, and degree of cooperation with other Regions and SLCs. A visible difference among the Regions is the seniority of the "learning coordinator" and his or her organizational proximity to the Regional Management Team (RMT) or Vice President's (VP) office. ECA's Regional Learning Coordinator (RLC) reports directly to the VP, EAP's RLC reports through one administrative level, while the RLCs in AFR, LCR, MNA and SAR report through at least two levels of administration (see appendix F). Moreover, multiple learning committees operate within Regions, typically beginning at the unit level and culminating at the level of the VPU front office. In addition, Regional sector managers and directors serve on sector boards and councils, respectively, which authorize sector learning programs. 3.3 While RLCs strive to ensure alignment of learning with corporate priorities at least on the portion of the budget that they oversee, managers in all Regions attribute to themselves and their staff the greatest influence over the production and utilization of learning opportunities, with Sector Boards being a distant second for all but advisory services (figure 4 and appendix C). 3.4 Standard procedures for identifying staff learning needs and ensuring that learning responds to these needs are lacking. RLCs engage in a wide range of approaches for identifying learning needs. Strategies mentioned by RLCs include (a) conducting "needs assessments" via surveys and one-on-one meetings with staff, (b) reviewing corporate priorities such as "governance and anti-corruption" or "results" and focusing on these areas, (c) reviewing human resources' data to focus on new local staff with orientation and operations courses, (d) reviewing the staff survey results, (e) meeting with Regional directors with lists of new staff that have not received training, (f) consulting with the Operations Policy and Country Services vice presidency (OPCS) on operational learning, and (g) creating a database on all Regional staff and training received. These strategies are used in various combinations. 11 Figure 4: Managers view themselves and their staff as influencing most of 11 observed decisions about the creation, dissemination and use of knowledge and learning 12 10 s onisic 8 High de of 6 Medium erb Low muN 4 2 0 Staff Manager Sector Board RMT/VP RLC Source: Authors analysis; see appendix C. 3.5 Cooperation between Regions and other SLCs varies considerably across the Bank. RLCs attributed lack of cooperation to (a) the number of units that "own" certain topics (e.g., governance, results, evaluation), (b) the absence of a formal coordinating mechanism including some means for sharing "visibility" for learning, (c) personalities and "ownership" by certain Regional staff of certain topics, and (d) differences among Regions with respect to importance of various topics. Successful examples of cooperation were mentioned, particularly for cross-regional cost sharing on specific topics. Identifying good available learning opportunities can be difficult 3.6 Numerous learning opportunities are advertised daily to managers and staff alike. Faced with this amount of information, some managers expressed the need to know which learning opportunities--internal or external--would be valuable to their staff. To reduce the amount of messages and help staff plan their learning, some groups (e.g., Asia Learning, Information Solutions Network) advertise events monthly. Nevertheless, the large volume of messages offering learning opportunities impedes strategic choices. 3.7 Given the abundance of learning opportunity information, a good Learning Management System is needed to help managers, learning coordinators, and staff make strategic decisions about learning. However, searching activities in the LMS can be cumbersome and misleading. Getting a calendar of upcoming events sorted by dates is not possible. Additional issues with the search functions of the LMS are described in appendix H. 12 The LMS is an inappropriate tool for accountability 3.8 As a tool for accountability, the LMS is not viewed positively by either managers or learning coordinators. It is viewed as "user-unfriendly" and as a consequence it remains an underutilized tool. Comments were decidedly negative. Managers and learning coordinators cannot easily extract the training history of staff. They rely on the KLB secretariat to provide data that they want to query on their own for just-in-time use. Once provided, the data are often seen as too unreliable to be used. Appendix H details issues that make the LMS a poor accountability tool. 3.9 The LMS has earned such a poor reputation as an accountability tool that some managers and learning coordinators try to avoid using it. In response to the shortcomings of the LMS, some Regional Learning Coordinators have even established their own local database to provide them with the staff learning information they need, such as the training history of staff in their Regions, and to monitor the learning budget. Some learning is not recorded 3.10 Some observed misalignment may be due to failures in registering learning in the LMS and in constraints imposed by higher managers. One indication that learning is not recorded adequately is that it is difficult in the LMS to account for the entire Regional budget allocation (table 5). Table 5: Unexplained KLB Regional budget allocation, FY06 Item Cost Source (a) Budget allocation, all Regions $32,421,000 FY06 KLB Annual Report (b) Staff time cost of learning (imputed) $7,678,364 LMS staff participant hours multiplied by net market rate of participants (c) Regional consultants, temporaries $1,279,684 LMS participant hours of consultants, and JPAs participation cost7 temporaries and JPAs multiplied by their rate (d) Travel cost charged to all Regions for $5,604,011 SAP FY06 Travel Expense/Forecast Report training received (e) Staff travel time to and back from the $3,438,613 16 hours multiplied by net market rate of training8 participants (f) Learning development and delivery $7,400,000 FY06 KLB Annual Report (g) a ­ (b+c+d+e+f) $7,020,328 Unexplained Source: See table 7 Consultants, temporaries and Junior Professional Associates (JPAs) were excluded from our main analysis. However, in table 5 to provide a conservative estimate of the unexplained budget allocation we included them as follows. For short-term consultants and temporaries, LMS participant's hours were multiplied by the participant's actual hourly wage rate. For extended term consultants and temporaries, we applied the distribution by grade group of the regional staff to impute their participation costs. For JPAs, we used the net market reference point-hourly wage rate of Grade D staff. 8 When the country of delivery of a face-to-face learning session was different from the country of duty of the staff member, we assumed that the staff member spent two days (16 hours) traveling for this learning session. To compute the cost of this traveling time, we multiplied the net market reference point-hourly wage rate by 16. 13 3.11 In table 5, the imputed staff time cost of learning recorded in the LMS accounts for approximately 24 percent of FY06 allocation and an additional 4 percent could be explained by the time of temporaries, consultants and Junior Professional Associates (JPA) assuming that they were paid for attending the learning sessions. The cost of participants' travel and the estimated cost of participant staff travel time to and back from the training account for 17 percent and 11 percent respectively. Another 23 percent correspond to the Regional learning development and delivery costs. This analysis leaves a conservative estimate of 22 percent of the budget allocation to all Regions unexplained, possibly in part a consequence of failures in recording participation. 3.12 Participation under-recording is particularly relevant to peer learning. Managers observed that on-the-job learning from more knowledgeable and experienced peers is universally valued; they provided several examples. Staff agreed (appendix G); among staff surveyed, 46 percent reported that they learned on the job from a colleague and/or had participated in a group peer learning program; this was second only to reading a study, operational guide or knowledge website (49 percent). This learning is often not captured in the LMS; for example, only 9 percent of the same staff had their peer learning recorded in the LMS.9 In addition, some types of team learning organized at the country office unit level, including "retreats" or "study tours" to visit projects under implementation, were administratively curtailed, according to several managers. What managers see as impediments to learning 3.13 Misalignment may be due to constraints on Regional staff. Country managers identified three main impediments for learning: (a) distance (in time zones and travel time) between the CO and HQ or Regional hubs, (b) demands on staff time for operational work, and (c) in a few cases, staff lack of familiarity and comfort with distance learning (e-learning and video-conferencing). Budget was rarely mentioned as a defining impediment, although it was an issue in three cases. First, small, remote COs found that the unit budget for learning was too small to allow for much staff travel. Second, for the years in which major sectoral events (such as sector forums) occurred, travel budgets for sectoral staff were viewed as insufficient. Third, budgets were generally viewed as insufficient for obtaining senior level staff (or retiree) mentors for junior TTLs. 3.14 Another source of misalignment may come from staff motivation for learning. Staff ratings for learning sessions are generally positive, but relatively low with respect to relevance to their present job (Ling, Dasgupta, and Yenice 2006). It may be that career development rather than job performance is the main incentive for learning. 9The period observed in the survey and the corresponding LMS data cover FY06 and the first ten months of FY07. 14 4. RECOMMENDATIONS 4.1 Alignment of staff learning with corporate and Regional priorities can be considerably improved with a number of major revisions in current practices. The most essential will be to subject staff learning to the standard Bank business processes for Work Program Agreements, consolidate learning into major programs, and focus on more tailored approaches targeted at different groups of staff. It will also require some LMS fixes, to provide better user interfaces for essential information and facilitate better targeting of learning opportunities. A radical reform in terms of the governance of learning is also suggested. 4.2 The following "basket" of learning, tailored to staff job responsibilities, would improve learning alignment with Regional corporate priorities: · For ACS and local staff, continuous updating on Bank operational skills (portfolio management, procurement, etc.) and Bank policies, largely through e- learning · For junior task team leaders (TTLs), (a) coaching by senior staff on political economy and project implementation, within the country context, (b) a standard TTL learning program (expanding on Fundamental of Bank Operations) across Regions, and (c) introductory P&T learning through sectoral "flagship" courses or externally provided courses · For senior experts, particularly for those working with MIC clients, professional expertise enhancement through external learning · For international staff moving to COs, pre-departure learning on country and culture. 4.3 The challenge is how to achieve this "basket" in the absence of clear budget authority and incentives for implementation or compliance. The following recommendations include those that can be achieved without changes in budget management as well as those that would require coordination with Resource and Performance Management Department (CSRRM). Actionable recommendations not requiring changes in budget management 4.4 The following are recommendations that the KLB could decide to implement independently. 15 KLB actions to improve targeting and alignment of learning · Eliminate "open enrollment" for most face-to-face learning opportunities other than BBLs. · Use LMS and HR data to identify staff movement (from HQ to CO, CO to HQ, from one position to another) and provide staff, managers and RLCs with a list of learning aligned with the requirements of the new position. · Target junior task team leaders for TTL learning and sectoral P&T courses (including "canned" courses and those provided outside the Bank). · Schedule longer formal courses for the first two quarters of the fiscal year. KLB actions to improve business processes for learning · Improve transparency, by (a) sending a Learning Dashboard to Regional VPs to document learning received and resources spent, and (b) sending summary unit data on learning received and resources spent to managers and staff of Regional units and to the Regional Learning Coordinator. · Follow the World Bank's standard business processes for review and approval of learning programs, i.e., concept note, review meeting, Work Program Agreement (WPA). KLB actions to improve P&T skills alignment with business priorities · Differentiate courses by level, i.e., basic, intermediate, and advanced. (OPCS does this already in their "roadmap.") · Identify new learning opportunities for advanced P&T (through partnerships with universities, think tanks, etc.); RLCs could identify decentralized opportunities and Network Anchors could identify international opportunities. · Commission a study group to examine how to respond to learning needs on "political economy" and country context. KLB actions to improve operational skills alignment with business priorities · OPCS and the Learning Support Group should expand the number of short, online or CD-ROM based courses for specific operations skills (procurement, supervision, disbursement, etc) with regular (at least annual) updates to keep current with operational requirements. (Financial Management does some of this already.) · Establish a "Senior Operations Specialist" fund to support on-mission mentoring of junior TTLs by senior TTLs (possibly those 60 or older). 16 · Establish peer learning and mentoring awards as incentives. · Load non-course/event learning tools into the LMS (e.g., self-assessments, checklists, etc). KLB actions to make the LMS more user-friendly (Appendix H details all LMS recommendations.) · Create user-friendly search tools and reports to provide unit managers with information on staff learning history and plans. · Create user-friendly interfaces for staff to easily find relevant learning opportunities and sort them by dates. KLB actions to use LMS more strategically · Send managers (with copy to the RLC) a quarterly email list of their staff with (a) their past learning received, (b) suggestions for their next learning, linked to upcoming events, (c) links to the unit's manager reports to enable autonomous just-in-time monitoring, and (d) links to tutorials for managers and staff to find and enter information in the LMS. KLB actions to improve LMS data · Better define the target audience by (a) replacing the field "grade groups" and "HR manager flag" with "function," (b) adding "intact team" versus "individual" learning, and (c) specifying CO/HQ target differently. · Better define the activity content by (a) mandating the completion of the "learning objectives" field in a structured results-oriented format, and (b) differentiating all activities by difficulty levels (i.e., introductory, intermediate, and advanced). · Capture in the LMS sessions that would otherwise be unrecorded, by enabling learners to add sessions that they attended but that were not recorded in the LMS, thereby creating a flag prompting the retrospective entry of sessions in the LMS. · Add a field to the LMS to identify peer learning or mentoring providers (name of staff). · After fixing the LMS, communicate all improvements to users to rebuild their trust in using the system's data. Actionable items requiring agreement of CSRRM 4.5 The following recommendations should help improve the strategic use of learning budget. If the variable cost learning budget were managed by the Chief 17 Learning Officer's office (rather than included in each Region's budget envelope from CSRRM), alignment with corporate priorities could be improved. Some alternatives to consider are: · Formula funding of variable cost budget: Allocate unit funding for learning based on a formula that takes into account such factors as (a) travel cost to HQ or Regional hub, (b) time zone distance and overlap with HQ, (c) internet connectivity speed at office, (d) access to GDLN/videoconferencing service, (e) share of new and/or junior staff, (f) number of peers in CO to provide peer learning, (g) country governance index, etc. · Conditional cash transfers: Distribute budgets to units on the basis of quarterly performance measures based on observed staff learning participation in activities for which they match the target audience, up to the limit of the budget envelope. · Reduce fungibility of learning budgets by: (a) limiting use of learning budgets to pay for staff participating in learning (allow limited use of Regional training development/delivery budget to adapt--not reinvent--programs, (b) limiting use of SLC budgets to design and delivery of learning (except for their units' budgets for staff learning), and (c) monitoring WPAs for compliance. · Change the fiscal year for learning: A learning fiscal year aligned with the calendar year would reduce many of the time conflicts for staff and could impose greater discipline on learning providers to develop most learning in the first two quarters of the Bank's fiscal year. A first step is to develop an 18- month WPA covering fiscal year 2008 and calendar year 2008. A radical reform specific to governance 4.6 A change in learning governance might be in order. This could involve the creation, in each Region, of a small "Learning Unit" (and Learning Manager) that would be responsible for staff learning and that would manage the entire Regional budget for learning (that is, the entire 3.9 percent received by Regions for this purpose). Learning deliverables would be organized into programs and segmented by targeted staff. Learning Units would follow standard budget procedures for monitoring the progress of these programs (e.g., identification, concept note review, draft review of learning content, delivery to country or sector client, evaluation, and implementation report). The scale of learning resources at the Regional level (an average of about $5 million per Region) would justify the creation of such small specialized units for learning. Activity Initiation Summaries (AISs) would be created in SAP for each learning program (not activity) with specific deliveries identified. Since the Learning Unit would manage the budget, the Regional managers would not have the incentive to use the learning budget for non-learning purposes or as rewards to staff. 4.7 Learning Units would serve as brokers, but would neither create nor deliver learning opportunities. Instead, Learning Units would enter into contracts with Country 18 or Sector units to deliver a set of products that would be aligned with the country or sector strategies, and into contracts with Network Anchors, other SLCs or outside vendors to develop and deliver the learning sessions called for in the Country or Sector unit contracts. Since the Learning Units would not design or deliver learning, they would not have an incentive to use the learning budget to "reinvent the wheel," which should reduce the number of nearly identical learning offerings. 19 REFERENCES Lockheed Marlaine, and Diana Arango. 2005. Features of Staff Learning that Contribute to Effectiveness in the World Bank. Report No. EG05-107. Washington, DC: World Bank Institute. Ling, Cristina, Basab Dasgupta, and Izlem Yenice. 2006. Annual Review of Staff Learning: Lessons from Participants' Assessments of Quality in FY05. Report No. EG06-119. Washington, DC: World Bank Institute. 20 APPENDIX A: RISK ASSESSMENT, FY06 AND FY07 REGIONAL STRATEGY AND PERFORMANCE CONTRACTS Highest of any category FY06 or FY07 Staff skills and mix Agility (competitive positioning, responsiveness to global developments, readiness for crisis assistance) AFR EAP Workplace safety and security, safety of facilities ECA LCR Implementation (reliable procurement, MNA disbursements, implementation management, TF compliance, managing projects with complex SAR safeguard issues) Staff motivation Results (results oriented culture, timely evaluation, timely and accurate reporting) 0 10 20 30 40 50 60 70 80 90 100 Risk ratings Note: Risk ratings are obtained by multiplying "potential impact" with "likelihood of adverse outcomes." Both risk elements are rated on an ascending 10-point scale. Thus, the highest possible risk rating is 100. In each category the chart indicates the highest risk rating between FY06 and FY07. 21 APPENDIX B: LIST OF DOCUMENTS CONSULTED Building Staff Capacity for Development Results: Update on the Staff Learning and Knowledge Program FY06. Office of the Chief Learning Officer/ Knowledge and Learning Board, October 2006. Davenport, Thomas, Ralph Poole, Laurence Prusak and Bruce Strong. June 16, 2006. Knowledge and Learning Governance: Cross-institutional Comparison with Recommended Options. FY06 Learning Plan, Africa Region, (last modified on 5/18/06). Downloaded 12/27/06. FY06 Learning Plan, East Asia and Pacific Region, (last modified on 4/20//06). Downloaded 12/27/06. FY06 Learning Plan, Europe and Central Asia Region, (last modified on 6/8/06). Downloaded 12/27/06. FY06 Learning Plan, Latin America and Caribbean Region, (last modified on 9/1/05). Downloaded 5/10/07. FY06 Learning Plan, Middle East and North Africa Region, (last modified on 8/16/05). Downloaded 12/27/06. FY06 Learning Plan, South Asia Region, (last modified on 2/6/06). Downloaded 12/27/06. FY06 Learning Plan, Administrative and Client Support, (last modified on 9/13/05). Downloaded 12/27/06. FY06 Learning Plan, Client Engagement and Team Learning, (last modified on 6/21/06). Downloaded 12/27/06. FY06 Learning Plan, Environmentally and Socially Sustainable Development, (last modified on 8/24/05). Downloaded 12/27/06. FY06 Learning Plan, Financial Sector, (last modified on 8/17/05). Downloaded 12/27/06. FY06 Learning Plan, Human Development, (last modified on 8/17/05). Downloaded 12/27/06. FY06 Learning Plan, Infrastructure, (last modified on 8/16/05). Downloaded 12/27/06. FY06 Learning Plan, Information Solutions Group, (last modified on 8/31/05). Downloaded 12/27/06. 22 FY06 Learning Plan, Learning and Organizational Effectiveness, (last modified on 1/12/06). Downloaded 12/27/06. FY06 Learning Plan, Operations Policy and Country Services, (last modified on 6/15/06). Downloaded 12/27/06. FY06 Learning Plan, Poverty Reduction and Economic Management, (last modified on 6/13/06). Downloaded 12/27/06. FY06 Learning Plan, Private Sector Development, (last modified on 8/16/05). Downloaded 12/27/06. FY06 Learning Plan, External Affairs, Communications and UN Affairs, (last modified on 2/14/06). Downloaded 12/27/06. FY06 Learning Plan, Human Resources, (last modified on 8/16/05). Downloaded 12/27/06. FY06 Learning Plan, Independent Evaluation Group, (last modified on 2/16/06). Downloaded 12/27/06. FY06 Learning Plan, Strategy, Finance and Risk Management, (last modified on 8/16/05). Downloaded 12/27/06. FY06 Learning Plan, World Bank Institute, (last modified on 4/6/06). Downloaded 12/27/06. FY06 Strategy and Performance Contract, Africa Region, (final as of 4/1/05). FY06 Strategy and Performance Contract, East Asia and Pacific Region, (final as of 4/1/05). FY06 Strategy and Performance Contract, Europe and Central Asia Region, (final as of 10/28/05). FY06 Strategy and Performance Contract, Latin America and Caribbean Region, (final as of 3/21/05). FY06 Strategy and Performance Contract, Middle East and North Africa Region, (final as of 3/21/05). FY06 Strategy and Performance Contract, South Asia Region, (final as of 3/31/05). FY06 Strategy and Performance Contract, Environmentally and Socially Sustainable Development, (final as of 4/1/05). 23 FY06 Strategy and Performance Contract, Financial Sector, (final as of 3/31/05). FY06 Strategy and Performance Contract, Human Development, (final as of 3/18/05). FY06 Strategy and Performance Contract, Infrastructure, (final as of 3/31/05). FY06 Strategy and Performance Contract, Information Solutions Group, (final as of 3/24/05). FY06 Strategy and Performance Contract, Operations Policy and Country Services, (final as of 3/18/05). FY06 Strategy and Performance Contract, Poverty Reduction and Economic Management, (final as of 3/23/05). FY06 Strategy and Performance Contract, Private Sector Development, (final as of 3/31/05). FY06 Strategy and Performance Contract, External Affairs, Communications and UN Affairs, (final as of 3/31/05). FY06 Strategy and Performance Contract, Human Resources, (final as of 3/31/05). FY06 Strategy and Performance Contract, Operations Evaluation Department, (final as of 3/31/06). FY06 Strategy and Performance Contract, Strategy, Finance and Risk Management, (final as of 4/1/05). FY06 Strategy and Performance Contract, World Bank Institute, (final as of 3/28/05). FY07 Learning and Knowledge Plan Toolkit (April 21, 2006). FY07 Strategy and Performance Contract, Africa Region, (final as of 8/15/06). FY07 Strategy and Performance Contract, East Asia and Pacific Region, (final as of 8/15/06). FY07 Strategy and Performance Contract, Europe and Central Asia Region, (final as of 8/15/06). FY07 Strategy and Performance Contract, Latin America and Caribbean Region, (final as of 8/15/06). FY07 Strategy and Performance Contract, Middle East and North Africa Region, (final as of 8/15/06). 24 FY07 Strategy and Performance Contract, South Asia Region, (final as of 8/15/06). Gelfond Group. Staff Survey 2005: World Bank Group 2005 Demographic Analysis Detailed Report. Gelfond Group. Staff Survey 2005: Overall Results. Gelfond Group. Staff Survey 2005: AFR Summary of Results. Gelfond Group. Staff Survey 2005: EAP Summary of Results. Gelfond Group. Staff Survey 2005: ECA Summary of Results. Gelfond Group. Staff Survey 2005. LCR Summary of Results. Gelfond Group. Staff Survey 2005. MNA Summary of Results. Gelfond Group. Staff Survey 2005. SAR Summary of Results. Gelfond Group. Staff Survey 2005. ESSD Summary of Results. Gelfond Group. Staff Survey 2005. FSE Summary of Results. Gelfond Group. Staff Survey 2005. HDN Summary of Results. Gelfond Group. Staff Survey 2005. INF Summary of Results. Gelfond Group. Staff Survey 2005. ISG Summary of Results. Gelfond Group. Staff Survey 2005. HRS Summary of Results. Gelfond Group. Staff Survey 2005. OPCS Summary of Results. Gelfond Group. Staff Survey 2005. PREM Summary of Results. Gelfond Group. Staff Survey 2005. PSD Summary of Results. Gelfond Group. Staff Survey 2005. EXT Summary of Results. Gelfond Group. Staff Survey 2005. IEG Summary of Results. Gelfond Group. Staff Survey 2005. SFR Summary of Results. Gelfond Group. Staff Survey 2005. WBI Summary of Results. 25 Gorjestani, Nicolas. September 2005. Capacity Enhancement through Knowledge Transfer: A Behavioral Framework for Reflection, Action and Results. Knowledge and Learning Group, Africa Region. IBRD Quarterly Staffing Report, FY05 Q4. July 15, 2005. World Bank HR Strategic Staffing. Mehari, Seifu and Lillian Foo. July 26, 2006. ESSD Staff Knowledge Sharing and Learning Needs Assessment FY06. Roberts, Dawn, Izlem Yenice, and Sara Kruse. 2007. Evaluation of the Environmentally and Socially Sustainable Development Network's Staff Learning Program in FY05 and FY06. Report No. EG07-127. Washington, DC: World Bank Institute. Roberts, Dawn, Izlem Yenice, Sara Kruse, and James Fremming 2007. Evaluation of the Infrastructure Network's Staff Learning Program in FY05 and FY06. Report No. EG07-123. Washington, DC: World Bank Institute. Staff Learning First Quarter Review FY06. October 22, 2005. Acting Chief Learning Officer and Knowledge and Learning Board. Staff Learning Second Quarter Review FY06. March 15, 2006. Acting Chief Learning Officer and Knowledge and Learning Board. Strong, Bruce. January 19, 2006. Knowledge Expenditure Review: The World Bank. The Coherence Group. 26 APPENDIX C: MANAGERS' ASCRIPTIONS OF DECISION-MAKING AUTHORITY FOR LEARNING (PRO-RATED VOTES) Decided by: VP/ Regional Regional Learning Manager or Sector Manage- Coordi- K&L resource use for: Staff Director Board ment Team nator Creating short learning High Medium Medium Low Low activities (e.g., BBLs, short (17.7) (9.7) (5) (2) (0.7) seminars) Creating courses (face to face Medium High Medium Low Medium or online) or workshops (½ ­ (4.5) (13.5) (7.5) (1) (9.5) 10+ days) Creating Advisory Services Medium Low High Low Low (e.g., help desks) (4.5) (2.5) (14.2) (1.2) (0.7) Creating knowledge products Medium High Medium Low Low (e.g., discussion papers, (7) (19) (9) (1) (1) toolkits, databases, video, debriefing) Creating knowledge storage High High Medium Medium Medium and retrieval (e.g., creation of (11.7) (10.7) (4) (5.7) (3) websites) Disseminating knowledge High High Low Medium Low (e.g., presenting papers at (10.5) (15.5) (1.5) (4.0) (0) conferences) Attending short learning High Medium Low Low Low activities (e.g., BBLs, short (32) (6) (0) (1) (0) seminars) Attending longer learning High High Low Low Low activities (½ ­ 10+ days) (17) (19) (0) (1) (1) Taking online learning (e.g., High High Low Low Low Fundamentals of Bank (25.5) (10.5) (0) (0.5) (0.5) Operations) Acquiring on-the-job learning High High Medium Low Low (through cross-support, being (16) (20) (3) (0) (0) mentored) Taking institutionally mandated Medium High Medium Medium Low courses (e.g., TLAP) (9.2) (18.2) (3) (4.2) (1.5) Note: Managers were asked "Who decides how Knowledge and Learning resources are used? Please indicate the MOST influential person or group and the SECOND MOST influential person or group for each decision listed below." Each manager was accorded two votes per decision (that is, no distinction was made between the most influential and the second most influential). Votes of managers who cast only one vote were doubled (e.g., if a manager voted for one decision maker, that decision maker was accorded two votes). Votes of managers who cast more than two votes were weighted by the inverse of the number of votes they cast (e.g., if a manager voted for four decision makers, each decision maker was accorded 0.5 votes (2/4)); "other" is omitted. High is defined as 10.5 votes or more; medium is defined as 3.0 < 10.5; low is defined as < 3. Three managers did not complete the question. 27 APPENDIX D1: REGIONAL LEARNING RECEIVED FOR RAISING AWARENESS (IN HOURS) This appendix presents the number of hours of learning received by Regional staff members (with regular, open, or fixed term contract) in FY06 and recorded in the LMS for the topics that received over half a percent of the total number of observable learning. Duration in hours Agriculture and Rural Development 8,225 Procurement 7,962 Ethics 6,714 Economic Growth 6,032 Water Supply and Sanitation 4,121 Communication for Bank Audiences 3,765 Operational Processes and Skills 3,557 Transport 3,416 Bank Orientation 3,296 Country Assistance Startegy 3,212 Private Sector Development 3,014 Energy 2,789 Governance/Anticorruption 2,378 Public Sector Development 2,246 Urban Development 2,220 Information Technology for Bank Staff 2,071 Networking 2,031 Information Technology for IT Staff 1,824 Administrative and Operational Support Skills 1,588 Social Protection and Labor 1,492 Human Resources For Bank Managers and Staff 1,360 Capital Markets and Capital Flows 1,013 Labor Markets 959 Health, Nutrition and Population 925 Financial Management 912 Safeguards 887 Economic Statistics and Data 780 Education 728 Monitoring and Evaluation 525 Bank Financial Products 485 0 1,000 2,000 3,000 4,000 5,000 6,000 7,000 8,000 9,000 28 Hours of Regional learning received for raising awareness by topic and Region Topics AFR EAP ECA LCR MNA SAR Total Agriculture and Rural Development 2,065 2,165 1,165 1,315 683 832 8,225 Procurement 2,928 616 1,954 984 552 928 7,962 Ethics 2,536 6 272 242 64 3,594 6,714 Economic Growth 1,744 848 1,424 784 512 720 6,032 Water Supply and Sanitation 1,077 578 632 555 563 718 4,121 Communication for Bank Audiences 1,464 432 768 549 168 384 3,765 Operational Processes and Skills 593 615 908 116 334 992 3,557 Transport 1,245 489 582 394 173 532 3,416 Bank Orientation 768 800 368 608 104 648 3,296 Country Assistance Strategies 672 147 609 1,451 202 132 3,212 Private Sector Development 1,438 319 329 212 131 586 3,014 Energy 660 400 685 276 300 469 2,789 Governance/Anticorruption 182 953 524 60 42 618 2,378 Public Sector Development 859 424 596 88 160 120 2,246 Urban Development 486 252 966 258 12 246 2,220 Information Technology for Bank Staff 439 430 434 358 185 227 2,071 Networking 427 328 543 292 209 233 2,031 Information Technology for IT Staff 410 126 489 650 84 66 1,824 Administrative and Operational Support Skills 218 488 440 107 24 312 1,588 Social Protection and Labor 360 140 592 170 96 134 1,492 Human Resources for Bank Managers and Staff 400 80 720 80 80 1,360 Capital Markets and Capital Flows 62 235 259 227 77 154 1,013 Labor Markets 390 22 204 47 195 102 959 Health, Nutrition and Population 297 15 149 51 155 259 925 Financial Management 155 181 202 170 86 118 912 Safeguards 223 203 100 177 108 76 887 Economic Statistics and Data 304 70 200 72 40 94 780 Education 211 245 36 113 40 83 728 Monitoring and Evaluation 107 188 70 75 33 52 525 Bank Financial Products 91 48 99 154 58 36 485 29 Hours of Regional learning received for raising awareness by topic and grade group Topics GA-GD GE GF-GG GH+ Total Agriculture and Rural Development 1,007 154 4,964 2,100 8,225 Procurement 2,836 976 3,526 624 7,962 Ethics 2,882 1,414 2,090 328 6,714 Economic Growth 192 352 3,904 1,584 6,032 Water Supply and Sanitation 74 260 3,085 703 4,121 Communication for Bank Audiences 1,317 840 1,560 48 3,765 Operational Processes and Skills 1,351 680 1,517 9 3,557 Transport 33 143 2,626 613 3,416 Bank Orientation 960 464 1,824 48 3,296 Country Assistance Strategies 480 830 1,690 213 3,212 Private Sector Development 194 104 2,273 443 3,014 Energy 98 173 1,916 603 2,789 Governance/Anticorruption 88 107 1,559 624 2,378 Public Sector Development 40 131 1,993 82 2,246 Urban Development 486 246 1,476 12 2,220 Information Technology for Bank Staff 767 154 1,014 137 2,071 Networking 1,525 274 198 35 2,031 Information Technology for IT Staff 787 515 522 1,824 Administrative and Operational Support Skills 1,533 51 5 1,588 Social Protection and Labor 9 17 1,268 198 1,492 Human Resources for Bank Managers and Staff 800 480 80 1,360 Capital Markets and Capital Flows 43 147 582 241 1,013 Labor Markets 5 919 35 959 Health, Nutrition and Population 138 5 736 47 925 Financial Management 4 16 715 177 912 Safeguards 195 72 551 69 887 Economic Statistics and Data 128 170 444 38 780 Education 9 20 527 172 728 Monitoring and Evaluation 9 42 344 130 525 Bank Financial Products 111 26 292 56 485 30 Hours of Regional learning received for raising awareness by topic and years in the Bank (as of December 31, 2005) Years in the Bank Topics 0-3 4-10 11-20 21+ n.a. Total Agriculture and Rural Development 2,092 3,082 2,040 693 318 8,225 Procurement 2,064 3,278 2,060 488 72 7,962 Ethics 1,824 2,690 1,558 304 338 6,714 Economic Growth 1,904 2,304 1,120 336 368 6,032 Water Supply and Sanitation 1,216 1,590 923 170 222 4,121 Communication for Bank Audiences 1,824 1,398 315 57 171 3,765 Operational Processes and Skills 2,004 719 374 136 325 3,557 Transport 1,432 860 560 200 365 3,416 Bank Orientation 1,800 480 24 992 3,296 Country Assistance Strategies 1,912 570 212 238 280 3,212 Private Sector Development 1,139 1,150 612 64 50 3,014 Energy 652 1,115 530 194 300 2,789 Governance/Anticorruption 698 860 559 190 71 2,378 Public Sector Development 1,222 609 152 55 209 2,246 Urban Development 972 984 12 252 2,220 Information Technology for Bank Staff 545 742 370 158 256 2,071 Networking 243 825 505 418 41 2,031 Information Technology for IT Staff 492 862 216 184 72 1,824 Administrative and Operational Support Skills 830 305 136 32 286 1,588 Social Protection and Labor 725 418 251 47 51 1,492 Human Resources for Bank Managers and Staff 400 480 400 80 1,360 Capital Markets and Capital Flows 448 404 136 24 2 1,013 Labor Markets 337 548 68 6 959 Health, Nutrition and Population 303 442 156 3 21 925 Financial Management 361 399 94 52 6 912 Safeguards 294 309 180 20 84 887 Economic Statistics and Data 510 138 76 4 52 780 Education 189 352 156 28 3 728 Monitoring and Evaluation 155 176 92 49 53 525 Bank Financial Products 211 206 51 2 16 485 Note: The "years in the Bank" categories follow the breaks and labels provided by the LMS. 31 APPENDIX D2: REGIONAL LEARNING RECEIVED FOR BUILDING INDIVIDUAL CAPABILITIES (IN HOURS) Duration in hours Procurement 13,477 Operational Processes and Skills 13,071 Administrative and Operational Support Skills 9,482 Information Technology for Bank Staff 6,345 Written Communication 5,775 Information Technology for IT Staff 4,000 Monitoring and Evaluation 2,949 Economic Statistics and Data 2,593 Health 2,403 Portfolio Management 2,064 Languages 1,955 Communication for Bank Audiences 1,884 Social Protection and Labor 1,732 Client Partner Engagement Skills 1,669 Trust Funds 1,504 Transport 1,419 Poverty Measurement 1,364 Urban Development 1,282 Resource Management 1,115 Gender 1,080 World Bank Forms 1,069 Presentations 1,005 Education 944 Bank Financial Products 820 Communication for External Audiences 688 Leadership 496 Information Technology Skills 480 0 2,000 4,000 6,000 8,000 10,000 12,000 14,000 16,000 32 Hours of Regional learning received for building individual capabilities by topic and Region Topics AFR EAP ECA LCR MNA SAR Total Procurement 4,957 1,600 1,612 2,145 1,894 1,270 13,477 Operational Processes and Skills 8,303 374 2,147 551 424 1,273 13,071 Administrative and Operational Support Skills 1,925 3,152 3,039 598 536 232 9,482 Information Technology for Bank Staff 1,237 1,040 2,347 802 835 85 6,345 Written Communication 1,441 1,324 1,104 564 124 1,219 5,775 Information Technology for IT Staff 1,906 668 946 684 152 124 4,480 Monitoring and Evaluation 638 184 966 593 343 226 2,949 Economic Statistics and Data 652 231 458 814 288 150 2,593 Health 1,240 320 112 443 288 2,403 Portfolio Management 96 32 1,056 144 728 8 2,064 Languages 421 15 68 1,301 150 1,955 Communication for Bank Audiences 688 96 252 584 216 48 1,884 Social Protection and Labor 722 160 247 42 240 320 1,732 Client Partner Engagement Skills 210 231 97 715 22 395 1,669 Trust Funds 352 320 268 104 76 384 1,504 Transport 604 195 228 124 64 204 1,419 Poverty Measurement 468 50 276 192 214 164 1,364 Urban Development 424 183 114 231 164 166 1,282 Resource Management 8 848 241 18 1,115 Gender 1,080 1,080 World Bank Forms 119 360 64 4 523 1,069 Presentations 263 312 244 63 64 59 1,005 Education 240 224 160 160 80 80 944 Bank Financial Products 260 64 176 144 64 112 820 Communication for External Audiences 120 320 48 72 112 16 688 Leadership 120 8 192 88 40 48 496 33 Hours of Regional learning received for building individual capabilities by topic and grade group Topics GA-GD GE GF-GG GH+ Total Procurement 3,339 1,911 7,790 437 13,477 Operational Processes and Skills 5,772 1,254 5,816 229 13,071 Administrative and Operational Support Skills 9,405 69 8 9,482 Information Technology for Bank Staff 3,761 1,625 928 32 6,345 Written Communication 2,341 650 2,678 106 5,775 Information Technology for IT Staff 2,228 1,906 346 4,480 Monitoring and Evaluation 366 350 2,019 215 2,949 Economic Statistics and Data 479 357 1,713 44 2,593 Health 280 80 1,803 240 2,403 Portfolio Management 160 576 1,248 80 2,064 Languages 446 343 1,165 1 1,955 Communication for Bank Audiences 360 336 1,084 104 1,884 Social Protection and Labor 80 40 1,490 122 1,732 Client Partner Engagement Skills 301 173 1,147 48 1,669 Trust Funds 304 112 956 132 1,504 Transport 67 44 1,248 60 1,419 Poverty Measurement 40 162 1,096 66 1,364 Urban Development 62 27 933 260 1,282 Resource Management 888 182 45 1,115 Gender 1,080 1,080 World Bank Forms 987 13 66 4 1,069 Presentations 626 76 303 1,005 Education 944 944 Bank Financial Products 24 16 668 112 820 Communication for External Audiences 136 40 256 256 688 Leadership 496 496 34 Hours of Regional learning received for building individual capabilities by topic and years in the Bank (as of December 31, 2005) Years in the Bank Topics 0-3 4-10 11-20 21+ n.a. Total Procurement 6,369 4,344 1,409 424 932 13,477 Operational Processes and Skills 5,896 4,453 1,473 632 617 13,071 Administrative and Operational Support Skills 3,146 4,069 1,670 417 180 9,482 Information Technology for Bank Staff 1,479 3,370 1,084 242 171 6,345 Written Communication 2,625 2,372 453 20 306 5,775 Information Technology for IT Staff 616 3,360 353 114 37 4,480 Monitoring and Evaluation 1,022 1,380 258 169 122 2,949 Economic Statistics and Data 1,355 669 235 144 191 2,593 Health 1,035 1,048 120 120 80 2,403 Portfolio Management 1,000 976 40 48 2,064 Languages 1,364 550 41 1,955 Communication for Bank Audiences 784 636 200 96 168 1,884 Social Protection and Labor 646 843 162 40 40 1,732 Client Partner Engagement Skills 623 551 208 93 195 1,669 Trust Funds 496 420 332 80 176 1,504 Transport 784 387 36 48 164 1,419 Poverty Measurement 626 560 24 40 114 1,364 Urban Development 398 428 341 66 49 1,282 Resource Management 81 1,034 1,115 Gender 360 720 1,080 World Bank Forms 707 203 142 18 1,069 Presentations 331 415 195 56 8 1,005 Education 288 448 80 48 80 944 Bank Financial Products 316 308 80 64 52 820 Communication for External Audiences 192 328 112 48 8 688 Leadership 80 232 168 16 496 Note: The "years in the Bank" categories follow the breaks and labels provided by the LMS. 35 APPENDIX E: LOGISTIC REGRESSION ANALYSIS OF LMS DATA FOR SIX REGIONS, FY06 The following analysis is based on all LMS data for staff working in the six Regional VPUs. Activities were coded into four broad categories based on their objectives: raising staff awareness, building individual staff capabilities, building team effectiveness and developing leaders. Within each of these categories, specific objectives were identified (for example, ethics awareness, P&T skills) and further refinements made where appropriate. We then examined the probability of participating in various activities, according to staff Region, country office (vs. headquarters), grade group, and years at the Bank (table A.E.2). A further analysis was undertaken for activities for building individual capabilities specifically (table A.E.3). Regions differ in their attention to raising awareness, building individual capabilities, building team effectiveness and developing leaders. Staff in EAP, LCR and SAR are more likely and staff in MNA are less likely to participate in awareness activities than staff AFR or ECA. Staff in AFR and MNA are more likely to participate in activities to build skills compared with staff in other Regions. Staff in ECA are three times as likely to engage in activities to enhance team effectiveness than staff in AFR, those in MNA are half more likely to, while those in EAP and SAR are less likely to do so. Leadership development is largely unrelated to Region, but staff in LCR are 80 percent more likely to engaged in leadership development than staff in AFR, and staff in MNA are 70 percent more likely to do so. Among staff who participate in activities designed to build their capabilities, those working in AFR, MNA or SAR are more likely to focus on building operational skills, as compared with staff in EAP, ECA or LCR, for whom the odds are much lower. Staff working in SAR have lower odds of learning related to P&T skills, and staff in EAP, LCR and SAR have higher odds of learning related to communication, when compared with staff in other Regions. Country office staff, when compared with headquarters staff are half as likely to participate in awareness activities and over twice as likely to participate in those that build skills. CO staff are 40 percent more likely to participate in activities to develop leadership. Location is not related to the probability of participating in activities to build team effectiveness. Among staff who participate in activities designed to build capabilities, CO staff are twice as likely to focus on building Operational Skills compared with HQ staff. However, CO staff are significantly less likely to participate in activities related to other major areas of capability: P&T skills, resource management skills, administrative skills, or IT skills. Staff grade groups--and hence job descriptions--are strongly related to the odds of participating in various types of learning. Staff levels GE and above are significantly more likely to participate in learning activities to build awareness than are GA-GD staff, whereas GA-GD staff are significantly more likely to participate in learning activities to 36 build their skills. Staff levels GF and above are more likely to participate in learning activities to build team effectiveness and those in levels GG and above are more likely to participate in leadership development activities. Among staff who participate in activities designed to build capabilities, staff levels GE and above are much more likely to engage in learning about P&T skills, operational skills, and resource management, and less likely to engage in learning about administrative skills, IT skills or communications, as compared with those in levels GA-GD. Grade group by location interactions: Country office staff in levels GF and GG are twice as likely to engage in learning about operations, compared with other staff, and are about half as likely to engage in P&T and communications learning activities. Table A.E.1: Odds ratios for observed learning for selected staff capabilities by grade group and location, World Bank Regional staff, FY06 Odds Ratios (significance) Interaction term Operations P&T skills Communications CO x GE 0.94 0.20*** 1.27 CO x GF 2.05*** 0.53* 0.43*** CO x GG 2.12*** 0.56* 0.25*** CO x GH 0.84 1.06 0.57 * = p < .05; ** = p < .01; *** = p < .001 Note: After controlling for grade group, Region, years in the Bank, and location. 37 Table A.E.2: Odds ratios for observed learning by major purpose of learning, World Bank Regional staff, FY06 Raising Building Team Developing Awareness Capabilities Effectiveness Leaders Region AFR 1.00 1.00 1.00 1.00 EAP 1.67*** 0.62*** 0.70* 1.01 ECA 1.02 0.73*** 3.15*** 0.79 LCR 1.21*** 0.78*** 1.17 1.81** MNA 0.88* 1.02 1.55** 1.68* SAR 1.70*** 0.63*** 0.58** 0.86 Location Headquarters 1.00 1.00 1.00 1.00 Country Office 0.44*** 2.25*** 1.03 1.44* Grade Group GA-GD 1.00 1.00 1.00 1.00 GE 1.20** 0.83** 0.97 0.86 GF 1.34*** 0.70*** 1.66*** 1.64 GG 1.37*** 0.60*** 2.47*** 2.42** GH+ 1.58*** 0.29*** 3.29*** 39.32*** Years in the Bank 0-3 years 0.84*** 1.19*** 1.10 0.48* 4-10 years 0.85*** 1.12* 1.31* 1.09 11+ years 1.00 1.00 1.00 1.00 Number of cases 17292 17292 17292 17292 * = p<.05; ** = p< .01; *** = p< .001 38 Table A.E.3: Odds ratios for observed learning for selected staff capabilities, World Bank Regional staff, FY06 Opera- Adminis- Resource tional trative P&T Commu- Manage- Skills Skills Skills nications IT Skills ment Region AFR 1.00 1.00 1.00 1.00 1.00 1.00 EAP 0.59*** 0.66 1.23 2.74*** 1.01 1.40 ECA 0.63*** 0.53** 1.12 1.12 1.28** 1.15 LCR 0.69*** 0.87 0.99 1.72*** 1.22* 1.17 MNA 1.34** 0.28* 0.81 1.00 1.18 1.04 SAR 1.07 0.77 0.72* 1.32* 0.75 1.42 Location Headquarters 1.00 1.00 1.00 1.00 1.00 1.00 Country Office 2.21*** 0.63* 0.38*** 1.20* 0.78** 0.62*** Grade Group GA-GD 1.00 1.00 1.00 1.00 1.00 1.00 GE 3.70*** --- 5.45*** 0.58*** 0.70** 1.24 GF 2.52*** 0.02*** 17.69*** 0.82* 0.22*** 2.69*** GG 4.84*** 0.04*** 16.75*** 0.54*** 0.14*** 2.56*** GH+ 4.55*** 0.14** 18.84*** 0.73 0.13*** 1.89* Years in the Bank 0-3 years 1.30** 1.09 1.12 1.09 0.76* 0.85 4-10 years 0.99 0.84 1.00 1.15 1.29** 0.57*** 11+ years 1.00 1.00 1.00 1.00 1.00 1.00 Number of cases 7270 6450 7270 7270 7270 7270 * = p<.05; ** = p< .01; *** = p< .001 39 APPENDIX F: ORGANIZATIONAL CHART FOR REGIONAL LEARNING COORDINATORS ECAVP EAPVP EAPCO Regional Learning Coordinator Regional Learning Coordinator AFRVP LCRVP AFTQK LCSOS AFTRL LCSLU Regional Learning Regional Learning Coordinator Coordinator MNAVP SARVP MNACS SARSQ MNAOS SAROQ Regional Learning Regional Learning Coordinator Coordinator 40 APPENDIX G: SURVEY OF STAFF FROM SAMPLED REGIONAL UNITS Staff of sampled managers were asked: "Since July 1, 2005, what types of learning experiences did you have (other than World Bank training sessions)? Please check an option only if the World Bank sponsored, arranged, and/or organized at least part of it. (Please check all that apply.)" Out of 457 staff surveyed, 299 (65 percent) answered. Number of participant Response Type of learning modality responses rate a. Carrying out a developmental assignment or secondment 23 8% b. Participating in an immersion program (e.g., language 27 9% immersion, field immersion) c. Visiting other organizations or field visits 63 21% d. Learning on-the-job from a colleague (e.g., bringing a 117 39% Network sector expert on mission, getting a mentor or coach) e. Participating in a group peer learning program or 52 17% community of practice f. Learning with your team using real work through a facilitator 60 20% (e.g., multi-sectoral team learning) g. Using a help desk or consulting an expert who provides 77 26% advisory services h. Using a self-learning tool designed to help you perform a 13 4% specific task i. Reading a study, operational guide, and/or a knowledge 146 49% web site j. Being evaluated through a 360 degree assessment 24 8% k. Attending a non-World-Bank course (but paid in part by the 61 20% WB) l. Participating in a non-World-Bank conference or 48 16% professional association meeting (but paid in part by the WB) d or e or both. Learning on-the-job from a colleague (e.g., 137 46% bringing a Network sector expert on mission, getting a mentor or coach) and/or Participating in a group peer learning program or community of practice Source: Authors' analysis 41 APPENDIX H: LIST OF RECOMMENDATIONS FOR THE LMS This appendix highlights a series of issues and provides actionable recommendations related to making the Learning Management System a more useful tool to align staff learning strategically. The tables below are the result of a diagnostic of the LMS functions as of May and June 2007. This exercise uncovered other issues not listed here, as these are less relevant to the topic of this report. The appendix focuses on functions for two types of users of the LMS, i.e., learners and managers. When functions for course administrators affect the information for learners, the recommendations also covered the screens completed by course administrators. In each table the recommendations are organized around three types of actions: (a) providing relevant information to help managers and learners use the LMS strategically, (b) making the LMS more user-friendly, and (c) fixing malfunctions. Functions for learners and background functions for course administrators Issues Recommendations Providing relevant information clearly Target audience by functions: Replace the target audience field "HR manager flag" Learning activities are often designed around and "grade group" with the "function" with the people's functions. Yet, the LMS does not have a following values: managers (higher level), task team target audience field by function. Instead the LMS leaders, other higher level staff, office managers, and uses two proxy fields, grade groups and "HR ACS staff. manager flag." While both fields jointly provide an indication of the function, indicating the target audience directly by functions would be simpler and more relevant. For example, a course administrator may select GF-GG as a grade group to indicate that the course is for TTLs. GE TTLs may miss it in a search by grade, while GF non- TTLs may return this course in their search. Specify HQ/CO target audience differently: The Eliminate the HQ/CO target audience field. The LMS recently added a HQ/CO target audience country of delivery (a useful recent addition to the field. Identifying activities for CO staff is useful, LMS search) or the online delivery mode (if canned but the field is used inconsistently and can or with moderation on their time zone) are better confuse learners who search through this field. means of signaling to CO staff which activities are Some course administrators tag HQ activities both designed for them. HQ and CO, because CO staff are welcome to attend, if they happen to be in HQ. Learners in COs find these activities in searching for CO activities. Conversely, tagging HQ activities for HQ only may signal to CO based learner that s/he is not the target audience, even if s/he happens to be at HQ. CO is not a specific location. Bangkok sessions are irrelevant to Dakar-based staff. 42 Issues Recommendations Target individuals vs. intact teams: Add to the session screen: "Is this session for intact A major difference in target audiences is teams?" yes or no. individuals versus intact team. Yet, the LMS does not have a field distinguishing these types of Next to this question, add an information icon that audience. specifies: "Click yes if the session is specifically for participants who work together outside of the learning session. If participants need not be related to each other, click no." Make target information search precise: Make the search return only activities tagged All categories of target audience and course specifically with the selection criterion, i.e., the difficulty level are selected by the course course administrator did not select at least one of the administrator as multiple-choice options. Many options available in the field. activities are tagged with all choices available. In the course administrator's activity detail screen, Learners select a search criterion because it add an information icon next to the categorical target matters to them. They would expect to find only fields explaining the consequences of selecting all activities meeting their search criteria. However, choices available when learners search by this field. they also find activities for which all choices were tagged, making the list needlessly long and In the learner's advanced search screen, add an potentially misleading. For example, a learner information icon next to the categorical target fields who searched for introductory activities may explaining that specifying a criterion will only return assume that all activities returned by the search the activities that specifically met the criterion. If a are introductory, but some activities are tagged learner wants to find activities that may or may not with all levels of difficulty. meet the criterion, s/he should not search by this criterion. Rename the difficulty level categories: Rename the difficulty level categories "introductory," The activities have recently been classified in the "intermediate," and "advanced." LMS as "introductory," "advanced," and/or "cutting edge." However, "cutting edge" relates to a concept different from "introductory "and "advanced." A topic can be cutting edge and treated at an introductory level. Conversely, a long-established topic can be taught at an advanced level. Therefore, the categories are not clearly mutually exclusive. Enable search by duration: Duration matters to In the advanced search screen, add a field to enable learners who search for learning opportunities. the search of activities by duration ranges. Yet, the LMS search screens does not enables learners to search activities by duration. 43 Issues Recommendations Mandate clear learning objectives: The learning Make the learning objectives field mandatory. objectives field is often empty. Guidelines to fill it are vague: "specify learning objectives for this Guide the writing of results oriented learning learning activity. This field appears in the objectives by offering three text boxes to type the Catalog." objectives. Have the following read above each window (one sentence per window): When information is provided, it is frequently · Upon completion of the activity, the written in terms of process (e.g., "participants will participants will have greater awareness of: have the opportunity to discuss the issue") rather than targeted outcomes. Currently, the value of · Upon completion of the activity, the this important field is limited. participants will be able to: · Upon completion of the activity, the participants will have developed: Add the following instructions in the information icon next to the field: "Write the learning objectives in the relevant text box(es). If a text box does not apply to the activity, please leave it blank. Examples of completed learning objectives: Upon completion of the activity, the participants will have greater awareness of: (a) the risks associated with ignoring corruption, and (b) tools available to deal with corrupt environments. Upon completion of the activity, the participants will be able to: (a) recognize the prevailing types of corruption in a given environment, and (b) choose an appropriate response to the corrupt situation. Upon completion of the activity, the participants will have developed: (a) greater confidence in their ability to respond to corrupt situations; (b) an action plan to deal with the corruption environment affecting their project; (c) stronger ties with their teammates, and (d) contacts with a network of professionals with whom they could discuss corruption issues." 44 Issues Recommendations Clarify delivery modes: Delivery mode Interim fixes: information matters particularly to country office Rename the delivery mode options. "Face to face" staff. Yet, the LMS delivery mode information is should become "synchronous (face to face, not intuitive and not always available to the videoconference, webminars, moderated online)" and learners. "Face to face" means that everyone "online" should become "online without moderator." takes the session at the same time, including webminars and videoconferences. "Online" relates to activity content taken online in a self- Update the information help menu accordingly and paced fashion. The advanced search screen add it to the advanced search screen by the delivery lacks the icon explaining these definitions. mode field. Reading the explanatory icon leaves learners Rename the "enroll" link of blended activities "view wondering how moderated online activities are sessions." recorded in the LMS. Also, no field indicates the time zones covered by live moderators. Ultimate fixes: Create a field for course administrators to select all "Blended" means that the activity has both a applicable delivery modes among: required online and face to face components. · Face to face However, blended activities returned by a search · Videoconference do not display session information. Instead learners are expected to click "enroll" before · Online with moderator knowing the dates of the face to face component · Online without moderator of the activity. · Webminar · Other, specify: For activities "Online with moderator," add fields for the start and end time of live moderation (including the time zone). Let the learners search by this revised delivery mode field. Inform learner of session's exact location: In Add an "address/room number" field to the session FY06, the LMS did not let learners know the city information screen. and country of delivery of sessions. The location field was an open-ended field most often filled with a room number and not systematically searchable. Recently, this open ended location field has been replaced with city and country of delivery. This is great progress, particularly for CO staff. However, from the current LMS, learners do not know the precise location of a session, beyond city or country. Before enrolling, learners should be able to know the precise location of the session, notably when sessions take place out of the Bank's buildings. Inform users of sessions' schedule: Inform users of the exact session time before To help learners fit training in their work, several enrolling. activities use non-consecutive sessions. The LMS session information screen does not let the Fix the interface with the Lotus Notes calendar entry. learner know the exact session schedule, before enrolling. After enrolling, the learner receives an automatic calendar invitation by email that covers all hours between the start and end dates of the sessions, including the time not assigned for the session. 45 Issues Recommendations Expand the language field: The drop down list of Add "other, please specify" to the list of activity activity language only enables a selection languages. between Arabic, Chinese, English, French, Portuguese, Russian, and Spanish. If an activity Make the language field a "select all that apply" type is in another language, it cannot be indicated and of field. searched by this language. Also, the LMS does not enable course administrators to indicate sessions delivered in several languages, e.g., a joint client staff learning session in French and English with interpretation. Enable learners to flag unrecorded sessions: The Add a "WB session not in the catalog" button to the individual learning plan screen enables learners to individual learning plan screen. add external training and on-the-job learning opportunities to their learning records. However, Have this button open a dialog box asking for the learners cannot add internal sessions that they session title, dates, city of delivery, duration in hours, attended but that were not recorded in the LMS. estimated number of participants, and contact Learners who want to keep accurate records of person's name. their learning are left with the choice of recording missing internal learning sessions as external or not recording them at all. Once a participant completes the information, the CLO's office should receive a flag email prompting them to verify the information and have the course The LMS could take advantage of the fact that administrator retrospectively include the session in some learners keep their records accurate to the catalog and update the participants' records. The identify unrecorded learning sessions and have CLO's office could decide to follow-up only on them recorded retrospectively. It would take only activities that are over a certain duration, e.g., at one participant to flag a missing session and least three hours. prompt the session to be included in the catalog after the fact thereby improving the records of all participants. Recognize mentors by name in the LMS: The In the on-the-job learning activity dialog box filled by LMS has fields to identify the training providers of learners, add a field asking for the names of people internal and external activities, but not for on-the- who provided peer learning or mentoring. job activities. Staff who receive on-the-job training from a peer or mentor cannot include their Have this information appear on the records of the mentor's name to the LMS. Enabling this would mentors, their managers, and the learners. encourage staff to record some of the largely underreported peer learning and mentoring activities. This information should get to the mentors' learning records and to their managers for recognition. Making the LMS more user-friendly Send learners relevant search updates: The Enable learners to have the LMS notify them of current LMS does not enable learners to save upcoming sessions meeting their search criteria. searches. Learners interested in a topic for which no session is scheduled need to return to the LMS Enable learners to disable this function anytime. in the hope that a relevant session will be available at the time of their next search. This may induce learners to enroll in less relevant sessions available at the time of their search rather than waiting for more relevant sessions. 46 Issues Recommendations Enable search refinement: The LMS help menu in Enable learners to save searches. the activity view returned after a search says: "There are seven advanced catalog search fields Add a "Refine search" button on the search results that you can use to further refine your search by screen to enable learners to refine existing searches. topic, delivery type, location, date, language, activity type, and Region. After completing a search, you may further refine your results if Update the help menu to match the fields by which necessary by selecting the "Refine" link at the top one can search. of the search results panel." However, the "refine" link does not exist and search refinement is not possible. Learners need to select all search criteria from the start every time. Allow several selections in dropdown lists: The In the advanced search, enable multiple selections advanced search screen has several fields that for the fields that are currently in drop down lists with are conceptually designed for multiple selections, more than two options, notably: topic, language, i.e., topic, language, learning activity type, country delivered in, level, and sector. delivered by, country delivered in, level, sector, VPU, organization, time in Bank, and grade group. Yet, these fields appear in dropdown list allowing only one choice at a time. Enable exclusion of Brown Bag Lunches: BBLs Add a check box on the advanced search screen to are numerous and generally advertised by emails. allow users to exclude BBLs. Learners who look for in-depth knowledge cannot exclude BBLs from the list of activities returned in a search. Easily omit activity without sessions: Unless Add a check box to the advanced search screen to learners search by dates, the LMS returns all return "activity with scheduled sessions only." matching activities, including those without scheduled sessions. The screens do not For this search, count online activities as having intuitively let users search only activities with scheduled session. scheduled sessions. On top of the search result screen, add a button to Learners who browse by topic return all activities show or hide activities without sessions. and cannot select only those with scheduled sessions. Display search criteria in the results screen: When On top of each search results screen indicate the a search is conducted, for most fields the results criteria selected, akin to what is already done for the screen does not indicate what was searched. date fields. Enable the sorting of search results: The LMS Enable all searched activities to be displayed on one searches return activities sorted by alphabetic screen, regardless of length. order of the title. This sort order is of little value to learners, and there is no means to change it. Add the start date of the sessions to the activity search results screen. Also, the list of activities is split by screens of about fifteen activities. Longer lists cannot display Use the first session's start date as default sort order. on one screen. Place online activities after the activities with sessions. Place face to face activities without sessions at the end. Enable each column to be sorted (in ascending or descending order) by clicking on its header. 47 Issues Recommendations Display sessions with activity information: Display the list of sessions at the bottom of the Learners can either see activity information or activity description. session information. Both related pieces on information cannot be viewed together. Enable session view exit without enrolling: Create a "back" button from the session screen, and Often, though not always, learners who view the make it work. sessions of an activity are only offered the choice to enroll in a session. There is no exit button, and the Internet "back" button generally gives error messages. Then, learners need to rebuild their search from scratch, a needlessly time-consuming task. Highlight activity target and level better: If a In the learner's view of the activity, display "any" course administrator selected all available options instead of the list of each option, when the course in the target audience or difficulty level fields, administrator selected all available options. each option is displayed in the activity description viewed by learners. This makes the information on target audience and level long and confusing. Clarify the sector field: Both course administrators Spell out the acronyms of the sector field values on and learners are presented with a long and the screens of course administrators and learners. unclear list of sector's acronyms not spelled out, making the sector field hardly useful in its present Review and possibly shorten the list of options, form. notably those that refer to the same sector. The list can also be confusing, e.g., the options include URB and URBAN, or INF and INFRA without explaining the differences. Fixing malfunctions Find countries under the Region searched: Have the LMS advanced search return all activities The "country delivered in" field lists world delivered in the countries that make up the selected Regions. A learner searching for activities Region. delivered in a Region expects to find all activities delivered in the countries of the selected Region. Instead the search result omits all activities delivered in the countries of the Region-- potentially misleading country office staff. Place "rest of the world" after "South Asia:" In the Reorder the dropdown list option to place "South drop down list of the field "country delivered in" Asia" before "rest of the world." the option "rest of the world" comes before "South Asia." Fix the "exclude these words" field: The advanced Make the "exclude these words" field exclude--rather search field "search for all of the words" or than include--activities with these words. "search for any of the words" together with "exclude these words" returns only the activities that have both the words to include and those to exclude. The activities corresponding to the search are precisely those not returned by the search. 48 Functions for managers Issues Recommendations Providing relevant information Add learning content to manager reports: Managers Add to the LMS reports for managers the options need to know what their staff have learned, and to: what they need to learn. The LMS reports for 1. display the topics and learning themes of the managers are not providing information on the learning activities completed by staff ("learning content of their staff's learning activities. The theme" currently appears in the reports, but information currently available to managers in the without data) LMS reports is limited to the staff name, number of completed learning activities, and number of days 2. display the title of the leaning activity, with or spent on learning. without date of completion10 3. query and group the results by topics and learning themes, enabling multiple selections 4. query and group the results by title of a particular activity (e.g., FBO, TLAP) selected from the Catalog. Add staff profile to manager reports: Learning Add to the LMS reports for managers the following needs depend in part on functions and areas of variables for query, grouping and display: expertise. The current LMS reports for managers 1. Function, i.e., managers (higher level), task only provide information by grade, location, gender, team leaders, other higher level staff, office years in the Bank, appointment type, and HR managers, and ACS staff. (Each person could manager flag. be automatically coded using HR and SAP data.) 2. Sector mapping 3. Professional affiliation 4. Specialization. Display learning content in ILPs: Managers have Add topics and learning themes to the individual the option to view the learning plans of their staff learning plan's view. one person at a time. However, the individual learning plans are uneasy to read and do not provide information on the content by categories. Email suggested learning to managers: Managers Quarterly email the unit's manager suggested need to have a relevant menu of learning learning for each staff member based on: corresponding to the learning program recommended for each staff member.11 1. past learning (no course repetition, taking into account prerequisites), 2. staff profile (years in the Bank, grade, function, sector mapping, professional affiliation, specialization), and 3. available matching upcoming events for the next six months. Making the LMS more user-friendly Save default queries: To busy managers, the 1. Create a default query based on the manager's reports may look cumbersome to build. A manager unit, grade, and number of days spent in is most likely to query his unit. Yet, the default unit learning. value is blank. Once a manager builds a query 2. Enable a manager to save queries, including s/he is likely to monitor the same data in the future, saving a customized query as default. but the queries cannot be saved. 10The LMS should enable staff to exclude some training from their learning records. For example, some staff may not want to attend training about harassment, if managers receive this information. 11Excluding short-term temporaries, consultants and contractors. 49 Issues Recommendations Improve the ILP view: The Individual Learning 1. Enable ILP sorts by displayed fields. Plan's view is confusing. The records are 2. Put the date and location in separate fields to displayed in an unfriendly manner. Over time, ILPs enable sorting by date. display more and more records. Newer records appear at the bottom, sometimes hidden in the 3. Give the option to hide the location. The room collapsed section. number is an unimportant piece of information that clutters the ILP and managers' view of the ILP. 4. Enable records to show or hide based on filters akin to an Excel auto-filter function (e.g., planned events, events within specific dates) Expand the ILP comment length: The comment Do not limit the length of the "comment" field. sections of the ILP are limited to 256 characters. Make the tutorials more complete: The LMS tutorial Explain the reports in the tutorial. for managers stops short of explaining what information the reports provide (or do not provide, e.g., lack of information on learning content). Clarify the report's navigation: When a manager's Make the navigation of the report clearly indicate report appears on screen, users may think that only the content of the report from the start. the "main report" or "aggregate results by numbers" is available because the drop down menu only shows "main report." The arrows on top of the screen indicate "1/1+" instead of the actual number of pages. It is misleading. Only after clicking the yellow arrows showing the subsequent pages will the subsequent section "section 1" with individual records appear in the drop down menu, and the number of pages in the report be indicated. Fix the export function: The report format "MS Export numeric data as numeric data. Excel 97-2000" exports numeric data as text. Statistics cannot be directly computed on the Make the report format "MS Excel 97-2000 (Data number of staff, number of completed learning only)" export data in a spreadsheet format ready for activities, and number of days spent on learning. analysis, including all grouping variables indicated on the section "report by aggregate numbers" The report format "MS Excel 97-2000 (Data only)" associated with each individual's record. does not provide the data in a spreadsheet format ready for analysis. It only exports the section Put the data in the column corresponding to their "report by staff name." Empty rows separate variable names. records. Variables related to the query (unit queried, grouping variables) are not associated with each record. Variable names are not in the same columns as their data. Describe the report's selected content: The reports Describe on the report the exact parameters used do not indicate the exact choices made at the query to produce the report, i.e., the unit and start and time. Someone receiving the report cannot tell the end dates of the period queried, the grouping parameters of the report (i.e., start and end dates of variables, and the sort order. the data, grouping variables, sorting order). Automatically assign an export filename that Every report is exported with the same file name indicates the parameters queried. "CrystalReportsViewer." Fixing malfunctions Make the report internally consistent: The number Have both sections of the same report match. of staff in the "report by aggregate numbers" does not match the number of staff in the next section of the same report called "report by staff name." 50 Issues Recommendations Fix the sorting function: The report's sort order Make the report return what was requested. "number of days spent on learning (from high to low)" sorts the data by "number of completed Treat "number of completed learning activities" as a learning events (from high to low)" instead, and numeric value and sort them accordingly. vice versa. Also, the sorting is inaccurate because the report treats numbers as text. Therefore, "10" appears between "2" and "1." Label and report years in the Bank clearly: The Rename the ranges of years in the Bank to indicate labels used to reports the years in the Bank are not where the categories are cut--without gaps or continuous. Some people do not fit into any of the overlaps. report's year ranges. People with less than a year in the Bank are reported as having "01-02" years. Display the years of experience as of the end date It is unclear whether people with two and a half of the query and mention in the footnotes that the years are counted with "01-02" or with "03-05" data are as of the end date of the query. years in the Bank. Explain in a footnote what counts as years of The years of experience may match the actual experience, notably clarify how the following cases years of experience of staff as of today, but not as are counted: of the dates queried. 1. non-consecutive years The reports do not explain what is counted as years 2. short- and long-term appointments as of experience. consultants or temporaries 3. contractors 4. fixed term contracts, etc. Use the query's end date for grade groups: The Display the grade group as of the end date of the data on grade group is not always as of the dates query and mention in the footnotes that the data are queried. This is misleading when retrospective as of the end date of the query. queries are conducted. Fix query processing errors: The reports for Make the queries return data instead of error managers are slow and often return error messages. messages. Even when the "report by aggregate numbers" displays data, the subsequent "report by staff name" may display error messages. The reports time out quickly. The error messages associated with the internet navigation suggest refreshing the query, but this operation is not feasible and leads to other error messages. 51