The Impact of FY01 Formal Training Turning Knowledge and Reflection into Action in The World Bank Sukai Prom-Jackson Tena Malone Maria Bina Palmisano Wema Jackson Kategile WBI Evaluation Studies No. EG03-62 The World Bank Institute The World Bank Washington, DC September, 2002 Acknowledgements This report was prepared for the Learning Board under the overall guidance of Marlaine Lockheed, Manager, Evaluation Group. The team was led by Sukai Prom-Jackson. WBI Evaluation Studies are produced by the WBI Evaluation Group (WBIEG) to report evaluation results for staff, client, and joint learning events. An objective of the studies is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at http://www.worldbank.org/wbi/evaluation/puball.htm. Assistance in data collection was provided by various members of the WBIEG unit to include Tena Malone, Maria Bina Palmisano and Wema Jackson Kategile Vice President, WBI Ms. Frannie Léautier Chief Learning Officer Ms. Phyllis Pomerantz Manager, WBI Evaluation Group Ms. Marlaine Lockheed Task Manager Ms. Sukai Prom-Jackson ii ACRONYMS ACS Administrative and Client Support BO Bank Operations Programs (OPCS, FPS, QAG) BS Behavioral and Social Skills Programs (Communications, ACS, HR) DEC Development Economics (cross-network) ESSD Environment and Socially Sustainable Development Network FPS Financial Products and Services (cross-network) FSE Financial Sector HD Human Development HR Human Resources (cross-network) IT Information Technology LEG Legal Department (cross-network) OPCS Operational Policy and Country Support PREM Poverty Reduction and Economic Management PT Professional Technical Programs (PREM, HD, ESSD, DEC, Legal, RM) QAG Quality Assurance Group RM Resource Management WBI World Bank Institute WBIES World Bank Institute Evaluation and Scholarships Unit WBISD World Bank Institute Skills Development Group iii iv Table of Contents EXECUTIVE SUMMARY ix CHAPTER I PURPOSE OF THE STUDY 1 CHAPTER II EVALUATION OBJECTIVES 2 CHAPTER III METHODOLOGY 3 A. Population and Sample Studied 3 B. Data Collection Instrument and Method of Data Collection 4 C. Design for the Analysis 5 CHAPTER IV DESCRIPTION OF THE TRAINING QUALITY, USE OF 7 LEARNING, WORK ENVIRONMENT FACTORS, PERFORMANCE CHANGES, FACTORS THAT AFFECT USE AND PERFORMANCE A. Quality of FY 2001 Formal Training 7 B. Level of Use of Knowledge & Skills Acquired from Training, and Factors 8 that Affect Use C. Work Environment Conditions: Level Of Support in Using Knowledge & Skills 10 D. Types of Support Received From Managers and Peers 11 E. Factors That Impeded Use 12 F. Reported Impact of Use on Performance 13 G. Reported Impact on Bank Corporate Objectives 13 H. Support Needed to Enhance Performance and Productivity 15 I. Summary of Findings and Conclusions 16 CHAPTER V ANALYSIS OF THE RELATIONSHIPS AMONG TRAINING, 18 WORK ENVIRONMENT, AND LEVEL OF USE A. Core Hypotheses 18 B. How Much of The Level of Use, and The Performance Change Variances 18 Can Be Accounted For By (1) Training and (2) Work Environment C. Which of The Two (Training and/or Work Environment Is A Higher Predictor 19 of Level of Use D. What Specific Variables of The Training and Work Environment Factors 19 That are Assessed in The Study are Important Predictors of Us E. What Are The Dominant Relationships for The Different Bank Programs? 19 F. Recommendations for Improving Training by Learners 21 v G. Summary of Findings and Conclusion 21 LIST OF ANNEXES Annex A: Theoretical Model of Performance Change 23 Annex B: Impact Evaluation Questionnaire 25 Annex C: Recommendations for Improving Training And Increasing Learning Or 29 Impact on Performance or Productivity Annex D: How Impact Is Manifested ­ Description of Specific Ways in Which 31 Training Benefits The Bank TABLES Table 1: Number of Courses And Respondents Included in The Study, Across 3 and by Program Type Table 2: Demographics of Respondents, Across and by Program Type 4 Table 3: Quality of Training (All Respondents)Percentage of Respondents Rating 7 Training Above Average or High Table 4: What Percentage of This Knowledge and Skills Have You Used by Program? 9 Table 5: What Percentage of This Knowledge and Skills Have You Used on The Job? 10 Differences Among Learners Based on Reason for Taking Training Table 6: To What Degree Have You Had Support or Collaboration in Using 11 Your Knowledge and Skills From The Following. Table 7: Regression of Level of Use on Training and Environment Factors 19 by Program Table 8: Regression of Level of Use on Training and Environment Factors 20 on Level of Use (Specific Variables) by Programs FIGURES Figure 1: Percentage of Learners Rating "4" Or "5' (High) on Various Aspects xi of Training. Figure 2: Have You Used The Knowledge And Skills Acquired From The Course? 8 (Percentage Stating "Yes" by Training Program) Figure 3: Have You Used The Knowledge And Skills Acquired? (Percentage Stating 9 "Yes" by Reason for taking Training and by Training Program) Figure 4: Impact of Use on Performance 13 vi BOXES Box 1: Support Received From Managers 11 Box 2: Factors That Impeded Use 12 Box 3: Corporate Performance Criteria 14 Box 4: Manager Support Needed To Enhance Performance And Productivity 15 Box 5: Improving The Quality of Training Programs ­ Recommendations By Learners 22 vii viii EXECUTIVE SUMMARY Turning knowledge and reflection into action: The trademark of successful learning organizations Successful learning organizations turn knowledge and reflection into action. It is necessary for capacity building that an organization increase the knowledge and skill levels of its staff, but raising knowledge and skill levels is not in itself sufficient for business success or development. Staff use of the knowledge and skills acquired during training drives business success and innovation, and that in turn determines the value of investments made in training. This is especially true when there is a close linkage between the training provided and corporate or business goals. This study assesses World Bank staff use of the knowledge and skills acquired from training courses delivered in FY 2001. Staff use or the adoption/adaption of the knowledge and skills acquired from training provides an indication of behavior change. Behavior change results in performance improvement. It is affected by training quality, work environment conditions, institutional systems, and personal factors. The study also assesses the effect of training quality and work environment support on behavior change or the use of knowledge and skills acquired from training. OBJECTIVES AND METHODOLOGY The study uses information from an electronic questionnaire completed by 733, or 15 percent, of attendees at 81 training courses in FY2001. With the exception of Senior Staff at grades GH­GJ, the sample is demographically consistent with Bank staffing overall. This fact lends some confidence in the results, and likewise the evidence of concurrent validity provided by various stakeholders who have already used the results for improvements in policies and procedures. Nevertheless, the self-report and voluntary1 nature of response to the questionnaire, limit the extent to which generalizations should be made and suggests caution should be used when interpreting the findings. All respondents completed the questionnaire at between two and nine months after undergoing training Using descriptive and nonparametric statistics, the study analyzes: 1While most respondents attended an average of 2 of the courses included in the evaluation, they were requested to complete a questionnaire for only one of the two courses, to limit fatigue. When this approach is taken into consideration in defining the actual population for the study, the derived response rate is 25%. ix · the quality of FY 2001 formal training, in terms of its relevance, effectiveness in increasing knowledge and skills, and design features to facilitate the transfer of learning to the workplace · the level of use of the knowledge and skills acquired from the training · the impact of the use of that knowledge and those skills on performance (i.e., on productivity, quality of work, work relationships with Bank staff, and work relationships with Bank clients) · the impact of performance improvements on the Bank, its clients, and partners · the degree of support given to the use of the acquired knowledge and skills in the work environment (i.e., support from managers, peers in the work unit, and colleagues across the Bank; also the degree of support afforded by information/knowledge systems such as Websites, help desks, experts, and mentors) As noted above, the use of acquired knowledge and skills is a function not only of training and learning but also of work environment, personal, institutional, and systemic factors. This study describes these factors as predictors to determine the combined and separate effects of training quality and work environment conditions on the level of use of the knowledge and skills acquired. The analysis of work environment conditions is limited to analysis of support from managers, peers, colleagues, and information/knowledge systems. It does not include institutional incentives, policies, and accountability systems. Time and resource limitations in conducting this study also prevented the analysis of personal factors, which can be significant for behavior and performance change. Analyses were performed across all respondents to yield a Bank-wide result, and separately by the four programs represented in the Staff Learning Agenda. These programs are: · Professional Technical program (15 training courses and 96 respondents from courses sponsored by PREM, HD, ESSD, DEC, Legal, RM) · Behavioral and Social Skills program (22 courses and 195 respondents from courses sponsored by ACS, Communication, HR) · Bank Operations program (19 courses and 130 respondents from courses sponsored by OPCS, QAG, FP) · Function Skills or Information Technology program (12 IT courses and 208 respondents) IMPACT OF FORMAL TRAINING The study's findings on quality of training, use of knowledge acquired, impact on performance, and support from the work environment were summarized generally across all programs (Figure 1). An outline of the findings follows. Quality of training The quality of the FY01 formal training is below the Bank benchmark. 76 percent of respondents rated the formal training in which they participated as "above average" or "high" on (1) relevance to their job needs, (2) effectiveness in increasing x knowledge and skills, and (3) technical design (i.e., ability to facilitate the transfer of learning to the workplace). This result is below the Bank benchmark for quality training, which is set at 85 percent rating of "above average" or "high." Figure 1. Percentage of Learners rating "4"(above average) or "5' (high) on various aspects of training. 100% 83% 80% 80% 76% 59% 60% Learners of 42% 40% Percentage 20% 0% Training Quality* Use of Knowledge Used 50-100% of Performance Supportive Work knowledge acquired Change/High Environment Notes: *Training quality = Rating of 4 "above average" or 5 "high" on quality criteria (relevance of the training; effectiveness in increasing knowledge, and technical design for transfer of learning). Use of knowledge and skills or behavior change The level of use of the knowledge and skills acquired is high. · 83 percent of respondents have used the knowledge and skills acquired from training. · 80 percent of these users report having used more than 50 percent of the knowledge and skills acquired. This level of use is consistent across the four program types investigated. Impact on performance Using knowledge and skills acquired improved staff performance. · 59 percent of respondents indicated the impact on their performance of the use of acquired skills and knowledge as "above average" or "high" (i.e., its impact on productivity, quality of work, relationships with colleagues in the Bank, and relationships with clients). The highest impact is on the quality of work produced. · 25 percent of respondents indicate an "average" level of impact on performance. Impact on institutional objectives xi Using knowledge and skills helped staff meet institutional objectives. · The survey invited respondents to offer further observations of their own. Generally, these indicate a belief among respondents that their use of knowledge and skills has improved Bank business in several ways, including: better professional quality of service; improved participation and stronger partnerships; increased client responsiveness and selectivity; increased efficiency and innovation; better implementation of fiduciary responsibilities; increased professional excellence, honesty, and integrity; and improved relationships and teamwork on the job. Support from the work environment Support from the work environment in using knowledge and skills is low. · Only 42 percent of respondents indicated an "above average" or "high" degree of support from the work environment. The highest level of support came from managers or supervisors, and peers in the work unit. For Bank Operation learners, managers, peers in the work unit, as well as colleagues across the Bank provide equivalent levels of support. · Analysis of the descriptions of the types of support provided indicates that managers tend to provide mentoring and coaching on the strategic directions or alignments of work products. Peer coaching support is mainly task-focused and technical. IMPACT OF TRAINING AND WORKPLACECONDITIONS ON THE USE OF KNOWLEDGE AND SKILLS · Training and workplace factors, while being significant predictors of use or behavior change, account for only 25 percent of participants' reported behavior change with the remainder accounted for by factors not assessed in this study. · Of the two, training is a more important predictor of use than the work environment conditions. · Analysis of differences among programs indicates that this finding about training holds true especially for the Professional Technical and the Behavioral and Social Skills programs. For the latter, increasing knowledge and skills from the training is a most significant feature for behavior change or use. For the former, the finding suggest that active involvement in the learning process is important. This however was not based on a large enough sample size and needs to be confirmed in future studies. · For the Bank Operations, the work environmental conditions play a more significant role in the use of knowledge and skills acquired from training. The most significant aspect about the work environment is the knowledge and information systems that support continuous learning, including websites, help desks, experts, and mentors. For Information Technology program, training quality (i.e. relevance, and the active involvement in learning), and the information support systems of the work environment are important predictors of use of knowledge and skills. xii CONCLUSION AND RECOMMENDATIONS There is no Bank standard for an acceptable "level of use" of knowledge and skills acquired during training, but the findings of this survey indicate a generally impressive use and impact of the knowledge and skills acquired through formal training. A significant majority of learners, 83 percent, are "turning knowledge from training into action." Fifty-nine percent attest to an "above average" or "high" impact of training on their performance, with a further 25 percent indicating an "average" impact. This appears to be a reasonable pattern of impact for learners applying new knowledge and skills within a range of two and nine months after training. However, it is for the stakeholders in the learning agenda to decide if these findings indicate a satisfactoryreturn on training for the Bank. This needs to be discussed and a benchmark established for future studies. The survey findings suggest that behavior change and performance improvement are occurring in spite of two inhibiting factors: (1) training quality is below the Bank benchmark for quality; and (2) work environment support is inadequate. The findings, indicate that other factors play a significant role in the learner's use of knowledge and skills and his or her performance improvement. Personal factors--such as commitment to learning, capacity for action learning and meta-cognition, personal confidence, and willingness to take risks--may playa significant role. These attributes within the individual are important for the development of learning organizations and "communities of practice;" the Learning Board's action learning program in fact is seeking to develop such skills, and also to identify the institutional infrastructure that would facilitate active learning and risk taking. Lessons to be drawn, particularly from the pilot Multi-sectoral Team Learning program focused on action learning would be most significant for upgrading the learning culture of the Bank. The comments provided by survey respondents on the factors that impede their use of new knowledge and skills also highlight a considerable number of institutional factors that need to be included in future impact studies of this type. These factors include the opportunity to use the knowledge and skills acquired, the incentive system, institutional policies and procedures, social norms and affective atmosphere for work, and a general resistance among managers and colleagues against innovation. The current Management Training and Development Curriculum should examine these factors and if not already included in the curriculum, consider how to address such via the curriculum. The results of the analysis show that the training variables are a more significant predictor of staff use of acquired knowledge and skills than work environment support. This underscores the importance of ongoing efforts of the Learning Board to enhance the quality of training courses and to develop a pedagogic unit to support this goal. Considering some of the past efforts in proving pedagogic support, one key challenge is how to provide services that would be valued and used by content experts. The findings of this survey indicate that improvements in training quality would likely increase the value of training for the Bank, especially for the Professional Technical (via active involvement in the learning process) and the Behavioral and Social Skills training programs (by increasing the knowledge and skills acquired). This finding for Professional Technical has great xiii implications for the structuring of such training events to enhance the quality of interactive learning. It also suggests a focus of the evaluation system on a more systematic analysis of the active learning and the use of high level productive learning strategies that form part of interactive learning. Work environment conditions, while less significant than training, are also significant predictors of use and performance change. This is particularly true for the Bank Operations and Information Technology programs. The significant variable in this case is the support provided by knowledge systems such as Websites, help desks, experts, and mentors. It appears that increased investment in these areas would be beneficial for these programs. Another important finding of the analysis of environmental conditions is the support from managers, supervisors, and peers in the work unit to enhance the learner's use of knowledge and skills. (This support is not, however, a significant predictor of behavior change.) Managers and peers provide mentoring and coaching, respectively, two very important interventions in the Bank. Further study of the quality of the support and the mentoring and coaching provided could potentially reveal the ways in which managers, peers, and colleagues could significantly impact behavior change. Broad-based surveys of this type are useful in providing indices of behavior and performance change that can instruct corporate stock taking of the value of training. The Learning Board should consider how a larger and more representative study could provide detailed information about training quality (Level 1 Evaluation ) and the impact of training (Level 3 Evaluation) for the corporate scorecard. Another option is to include a selected number of impact indicators in the HR Staff Survey. Other considerations or lessons for impact studies of this type, designed to help guide policies and practices for staff learning and performance improvement, include the following: · identify a complementary data collection base, for data from managers or peers of learners, to validate the self-reported findings of the study · expand the work environment variables to include institutional variables · include an assessment of the quality of support provided by managers, peers, and colleagues · include an assessment of the personal attributes of learners · include the date of participation of learners, to permit the development of an understanding of the relationship between the stage of use of acquired knowledge and skills and their impact on performance and productivity · increase the response rate by continuing the survey once started and implement a rigorous and continuous follow-up procedure; also for a survey of this type where respondents were asked to select one of two courses for which to complete a questionnaire, a good tracking system would help establish a more appropriate basis for defining the response rate. · consider including questions fromthe benchmark study of the American Society for Training and Development, to provide data for comparison with outside organizations concerned with turning knowledge into action xiv 1. PURPOSE OF THE STUDY 1.1 Successful learning organizations turn knowledge and reflection into action. It is necessary for capacity building that an organization increases the knowledge and skill levels of its staff, but raising knowledge and skill levels is not in itself sufficient for business success or development. Staff use of the knowledge and skills acquired during training drives business success and innovation, and in turn determines the value of investments made in training. This is especially true when there is a close linkage between the training provided and corporate or business goals. 1.2 An interest in understanding how training and learning benefit corporate goals and objectives has developed in the Bank. The FY 2001 Accelerating Results Together process (ART) on Staff Learning highlighted the need for the staff evaluation system to go beyond Level 1 and Level 2 evaluations of training quality and learning acquired, to focus instead on the analysis and evaluation of the use of knowledge and skills and its effects on performance change and on Bank business or corporate goals. This evaluation study accordingly is focused on learner self-reported use of knowledge and skills acquired from formal training; the impact of this use on productivity, product quality, and work relationships; and the consequent impact on the Bank, its clients, and partners. 1.3 The ART process also underscored the importance of understanding the factors that affect behavior change and performance improvements, and of using those factors to help target improvements in staff learning and performance. It has been hypothesized that performance change is a function not only of the training quality and of the learning gained, but also of work environment factors such as support from managers, supervisors, peers, and information systems. This study analyzes these hypothesized relationships, using as a framework the model of change illustrated in Annex 1. The model was developed using both theoretical information and information developed from past WBIES Level 1 and Level 3 Evaluations of staff learning. It is multidimensional in nature and highlights the complex set of variables that mediate or contribute to performance change and institutional impact. Parts of the model that are noted as important predictors of performance change are not included in this study, but are acknowledged in the interpretation of the results. 1 2. EVALUATION OBJECTIVES 2.1 This study is divided into two parts. The first part aims to provide a description of the following: · the quality of the FY 2001 formal training · the level of use of the knowledge and skills acquired from training · the work environment support for the use of acquired knowledge and skills · the types of support received from managers and peers · the elements of training design and delivery and the features of the work environment that impede use · the impact of use on performance · the impact of use on the Bank's institutional objectives · the types of support needed to enhance performance and productivity 2.2 The information provided here constitutes a first broad-based, comprehensive, and integrated study of the value of training (i.e., its quality, its use, and its impact on performance and business goals). It also provides a basis for developing benchmarks for future evaluations of impact based on self-reports. 2.3 The second part of the studyanalyzes the associations among the factors examined in the first part of the report. It assesses the linkages or associations among these factors using training quality and work environment as the predictor factors, and behavior change (or level of use) and performance improvement as criterion factors. The focus of the study is to determine the combined and separate effects of the training and work environment conditions (predictor variables) on the use of the knowledge and skills acquired. The specific questions for investigation are: · How muchof the level-of-use variance can be accounted for by (1) training and (2) the work environment? · Which of the two factors - training and work environment, is a greater predictor of use? · What specific variables or dimensions of these two factors are important predictors of impact? · What is the pattern of relationships among these factors for the various Bank training programs identified in the study? 2.4 The study seeks to identify the relative contribution of training and work environment conditions to level of use and performance change. It also seeks to provide a preliminary basis for determining the specific aspects of training and work environment conditions that should be given priority for improvement, and that should be used to help in the application of limited resources. 2 3. METHODOLOGY A. POPULATION AND SAMPLE STUDIED 3.1 Undertaken in June 2001, the survey contacted the 5,067 participants in the 81 courses of the staff learning programs that were delivered between July 2000 and April 2001and that were included in the WBIES Level 1 Evaluation of FY 2001. Some of the participants (about one third) attended more than one of these courses and were asked to complete a questionnaire for one course. Seven hundred and thirty-three, or 15 percent, of participants responded (N=733/5067), representing 95 percent of the courses in the WBIES data system (n=81), (see Table 1). While there is a high level of representation at the level of the courses delivered, the small percentage of respondents and the use of self-reporting as the sole source of data mean that caution should be used in interpretation of the findings and the conclusions submitted in the study. Table 1. Number of courses and respondents included in the study, across and by Program Type (courses surveyed=81; participants surveyed=5,067) All courses & Professional Behavioral Bank Information Technical and Social Operations Technology respondents* Courses 75 15 22 19 12 Respondents 733 96 195 130 208 *All course and Respondents" include the overall number of courses and respondents studied, but not included in this table. 3.2 To support the generalizations that are made here, the study compared the gender and grade level representation of the sample with that of the Bank staff composition. The results, in Table 2, show that, consistent with the Bank's overall staffing, there are more female respondents than male. For the grade levels of the sample of the study, there is an equivalent representation for GA­GD, an overrepresentation of GE­GG, and an under-representation of GH­GJ. The results at best have greater validity for the GA­GD and GE­GG grades than for GH­GJ. The study also sought to establish the concurrent validity of the findings by asking key stakeholders to conceptually or empirically validate the results provided for their specific programs or courses ­ based on feedback from demanders of the training, evidence from managers, and alignments with Level 1 data results provided over the past two years. There was substantial validation resulting in some of the key suppliers using the results to change policies or procedures. 3 Table 2. Demographics of Respondents, across and by Program Type (courses surveyed=81; participants surveyed=5,067) World All Professio Bank Informati Bank respondent nal Behavioral on and Social Operatio Staff s* Technical ns Technolo gy Gender Male 48% 39% 38% 29% 48% 44% Female 52% 61% 62% 71% 52% 56% Grade GA-GD 37% 41% 26% 65% 22% 39% GE-GG 26% 50% 63% 29% 68% 52% GH-GJ 37% 9% 11% 5% 10% 9% Location Washington DC n.a. 86% 88% 81% 76% 96% Country n.a. 14% 12% 19% 24% 4% Office *All course and Respondents" include the overall number of courses and respondents studied, but not included in this table. B. DATA COLLECTION INSTRUMENT AND METHOD OFDATA COLLECTION 3.3 This study is limited to reported information from a questionnaire completed by the participants in the various courses. The questionnaire was sent electronically in June 2001 to participants, and responses also were recorded electronically. Reminders were sent in July and August. A more robust studywould use multiple measures to include manager or peer assessments of behavior and performance change and institutional impact, but this study was designed to merely provide, in an exploratory fashion, broad-based knowledge about (1) the level of use of knowledge as an index of behavior change resulting from training, (2) performance outcomes, (3) factors affecting use and performance change, and (4) the nature of the impact of behavior changes on Bank institutional objectives. The credibility of learner self-reported results is nonetheless supported by emerging empirical evidence from the American Society for Training and Development (ASTD) benchmarking study, which indicates a high correlation between learner ratings and ratings by their managers on effectiveness and impact criteria (Bassi and Ahlstrand, 2000).1 3.4 The two-page questionnaire used in the study is included in Annex 2. The design of the questionnaire was guided by the change model in Annex 1 and by both theoretical and empirical information on transfer of learning,2 on the process of behavior change in 1 Bassi, L., and A. Ahlstrand. 2000. "The 2000 ASTD Learning Outcomes Report: Second Annual Report on ASTD's Standards for Valuing Enterprises' Investments." Alexandria, VA: ASTD. 2.Garavaglia, Paul. 1993. "How to Ensure Transfer of Training." Training and Development, October 1993, pp.63­65. 4 innovation adoption,3 and on the information provided by Bank staff in past WBIES studies of the impact of training. The eighteen questions questionnaire assesses the areas listed below (the Cronbach's alpha for the reliability of the quantitative subscales is noted): · Demographic factors - gender, grade level, and reason for taking training; · Training quality: four questions rated, on a 1 ­ 5 scale (1=low, and 5 =high), the degree to which (a) the training was relevant to the participant's job needs, (b) the participant was actively involved in the learning process, (c) the training increased the participant's knowledge and skills, and (d) the training prepared the participant for using the new knowledge and skills. ( a=0.8933); · Participants' use of knowledge and skills: one question on whether or not the participant had used the knowledge and skills acquired from the course, and, if the response was "yes", a second question asking the participant to indicate the percentage of knowledge and skills used, and a third question asking to rate the frequency of use on a 1-5 scale; · Impact on participants' performance: four questions rated, on a 1 - 5 scale (1=low, and 5 =high), the degree to which the participant's use of the knowledge and skills had an impact on the participant's (a) productivity, (b) quality of work, (c) work relationships with Bank staff, and (d) work relationships with clients. (a=0.9084); · Impact of performance on the Bank, its clients, and partners (open-ended questions) · Work environmental support: four questions rated, on a scale 1 ­ 5 (1=low, and 5 =high), the degree to which the participant had support in using the knowledge and skills from (a) a manager or supervisor, (b) peers and team members in his or her unit, (c) colleagues across the Bank, and (d) support systems (websites, help desks, experts, mentors). (a=0.8770); · Comments about factors that impeded use, specific support needed to enhance performance, and recommendations for training and performance change (open-ended questions) C. DESIGN FOR ANALYSIS 3.5 As noted above, the study is based on a single-group design and a single data source--i.e., self-reported responses of participants who took training in FY 2001 and who responded to the questionnaire between two and nine months after training. 3.6 The study used percentages and chi-square analysis to analyze the responses. The results for training quality are evaluated against the WBIES quality benchmark (85 percent of respondents rating the training as above average or high on quality criteria). The results for the level of use, impact on performance, and levels of support from the work environment are described rather than evaluated, because of the absence of a Bank benchmark for impact. Together with the recent ASTD study on the impact of training, conducted using learner- reported information from 2,118 companies, this study should provide a basis for dialogue about an acceptable benchmark for use in future evaluations. 3.7 Learner responses to open-ended questions were content-analyzed and responses with a high frequency of occurrence are recorded. The study does not code the content analysis for 3Hall, G., et.al. 1979. "Levels of Use of the Innovation: A Framework for Analyzing Innovation Adoption." Levels of Use Manual. Texas: Center for Teacher Education. 5 use in making linkages with quantitative responses, a process that is rigorous and labor- intensive and that was not accommodated by the resources and time frame of this study. 3.8 Analysis of the causal linkages in impact evaluation was based on a logic model which purports that job performance or impact is driven by learner level of use of the knowledge and skills acquired (Annex 1). This factor is in turn driven by (1) training quality and learning effectiveness, (2) work environmental factors, and (3) personal attributes and traits. Simple linear regression is used to analyze the relationship between level of use and training, and work environment factors--the limitations of the study data precluded a more comprehensive analysis. The logic model for the study also identifies the importance of personal factors; while this study acknowledges the significance of these factors it does not assess them. 3.9 To analyze the degree to which the relationships under investigation would hold for the various programs of the Staff Learning Agenda, the programs were grouped according to their primary learning objectives, as follows: · Professional Technical program (PREM, HD, ESSD, DEC, Legal, RM) · Behavioral and Social Skills program (ACS, Communications, HR) · Bank Operations program (OPCS, QAG, FP) · Functional Skills or Information Technology program (IT) 6 4. DESCRIPTION OF TRAINING QUALITY, USE OF LEARNING, WORK ENVIRONMENT FACTORS, PERFORMANCE CHANGE, FACTORS THAT AFFECT USE AND PERFORMANCE. A. QUALITY OF THE FY 2001 FORMAL TRAINING 4.1 The quality of the training was assessed in terms of its relevance and effectiveness in increasing knowledge and skills, and for its design--specifically in terms of features that ensure the transfer of learning, such as its encouragement of active involvement in the learning process and preparation for use of the knowledge acquired. · 76 percent of respondents rated the training as above average or high on these quality criteria4 (Table 3). This finding is below the WBIES quality benchmark for training, which requires that 85 percent of respondents rate training as above average or high. Table 3. Quality of training (all respondents) Percentage of respondents rating training above average or high All Professional Behavioral Bank Information Respondents Technical and Social Operations Technology Training Outcomes The training was relevant to your job 79 75 84 83 74 needs The training increased your knowledge and 76 72 80 82 70 skills Transfer of Learning You were actively involved in the 83 84 86 87 78 learning process The training prepared you for using the new 67 61 69 70 66 knowledge and skills Overall Quality 76 73 80 81 72 4Note that this quality rating differs from the quality rating of the Level 1 evaluation (see WBIES Report to the Learning Board, July 2001) where 85% of the respondents rated the training to have above average or high quality. These ratings were made right after the course delivery in end-of-course questionnaires assessing identical quality criteria. 7 B. LEVEL OFUSEOF KNOWLEDGE AND SKILLS ACQUIRED FROM TRAINING, AND FACTORS THAT AFFECT THEIR USE 4.2 The analysis shows the following: · 83 percent of respondents have used the knowledge and skills acquired from training (Figure 2). Figure 2.Have you used the knowledge and skills acquired from the course? (Percentage stating "Yes" by Training Program) 100% 86% 84% 85% 83% 80% 69% Respondents of 60% % 40% Total Professional Behavioral Bank Information Respondents Operations Technical &Social Technology Skills Learners who participated in the training to improve their performance reported higher levels of use compared with learners who participated in training out of professional interest, for future assignments or careers, or to share knowledge and to network (Figure 3). The results are statistically significant (p <.001) for all program types except for the Behavioral and Social Skills program, where both groups were high users. 4.3 To further understand the level of use, learners were asked to indicate the amount of knowledge and skills they used. · 80 percent of users have used more than 50 percent of what they acquired from training; 53 percent have used between 75 percent and 100 percent of what they acquired from training 8 Figure 3.Have you used the knowledge and skills acquired? (Percentage stating "Yes" by reason for taking training and by training program) 100% 80% 60% 40% Percentages 20% 0% World ProfessionalBehavioral Bank Functional Bank Operations RespondentsTechnical &Social Skills Development Enhance Performance Other Reasons This pattern of result is consistent across the various programs analyzed in the study (Table 4). The Professional Technical (PT) Skills program develops general understanding and knowledge that is relevant but not directly applicable to the job (i.e. for adaption and not adoption), and was therefore expected to show much lower levels of use than the other programs. While use was lower as predicted, 75 percent of PT respondents nonetheless had used more than50 percent of the knowledge and skills acquired. Further analysis of the data shows that the level of use is higher for PT staff who participated in training to enhance their performance rather than for other reasons (Table 5). Table 4. What percentage of the knowledge and skills acquired have you used by Program? All Professional Behavioral Bank Information Respondents* Technical and Social Operations Technology % of knowledge and skills used 10% 5 8 5 6 5 25% 15 19 13 14 15 50% 27 26 32 30 20 75% 35 31 35 31 40 100% 18 15 16 19 20 50%+ 80 72 83 80 80 9 Table 5. What percentage of the knowledge and skills acquired have you used by Reasons and by Program? All Respondents* Professional Behavioral and Bank Operations Information Technical Social Technology % of knowle dge Enha Other* Other* Other* Enhanc Other* Other** Enhance* Enhance* Enhance* and nce* * * * e* * skills used 10% 5 6 9 7 4 4 6 6 4 8 25% 15 13 16 26 14 11 15 11 16 8 50% 25 33 24 30 33 30 28 39 15 39 75% 37 32 31 30 31 42 34 17 46 19 100% 18 16 20 7 18 13 17 28 19 25 50%+ 80 81 75 67 82 85 79 84 80 83 * Enhance= to enhance performance on the job; **Other=for professional interest, future assignment, for sharing knowledge C. WORK ENVIRONMENT CONDITIONS: LEVEL OF SUPPORT IN USING KNOWLEDGE AND SKILLS 4.4 The results show that: · 42 percent of respondents indicated an above average or high level of support from the work environment · The highest level of support came from managers and supervisors and from peers and team members in the work unit. Forty-six percent of respondents indicated above average or high levels of support from managers or supervisors; 47 percent indicated above average and high levels of support from peers and team members (Table 3). Analysis by program indicates that the most significant support for the Professional Technical and the Behavioral and Social Skills programs comes from managers and supervisors; for the Bank Operations and Information Technology programs, the most significant support is from peers and team members. Across the programs, support provided by colleagues across the Bank, is more significant for Bank Operations than for the other programs; support systems (Websites, help desks, experts, and mentors) are more significant for the Information Technology and the Bank Operations programs than for the Professional Technical and Behavioral and Social programs. 10 Table 6. To what degree have you had support or collaboration in using your knowledge and skills from the following:(Percentage of respondents rating above average or high) All Professional Behavioral Bank Information Respondents Technical and Social Operations Technology Manager or 46 47 49 50 42 supervisor Peers and team members in your 47 43 44 56 48 unit Colleagues across 37 31 34 51 33 the Bank Support systems (Websites, help desks, experts, 36 30 33 39 41 mentors) Total 42 38 40 49 41 D. TYPES OF SUPPORT RECEIVED FROM MANAGERS AND PEERS 4.5 Learners report that the different types of support received from managers and from peers complement each other, as follows: · Manager support is reported to include, besides coaching for performance improvement along the strategic directions of the Bank, mentoring and social-emotive support (e.g., acknowledgement, clarifications, directions and guidance, encouragement, and increased responsibility) · Peers and colleagues provide more technically oriented task support and coaching, in the form of continuous feedback in using knowledge; engagement in dialogue on new ideas; increased collaboration in the adoption of new ideas; peer review; and consultations (Box 1) Box 1: Support received from Managers Mentoring Encouragement Acknowledgements Increased responsibility Clarifications and engagement in discussions Increased confidence in ability to get things done Support received from peers, team members, and colleagues across the Bank Feedback on use Coaching and mentoring Brainstorming on new ideas Tips and clarifications of policy Increased collaboration Peer review and consultation Increased acceptance of work and new ideas 11 E. FACTORS THAT IMPEDED USE OF ACQUIRED KNOWLEDGE AND SKILLS 4.6 Learners were asked in an open-ended question to describe the factors that impeded their use of the knowledge and skills acquired through training. Analysis of the responses (restricted to those who reported that they have used the knowledge and skills acquired from training) shows that: · the factors that impede use are mainly work environmental factors, such as opportunity; the incentive system; the nature of institutional polices and procedures; social norms and affective atmosphere; and resistance to innovation. Some of the ways in which these factors impeded use, as identified by survey respondents, are listed in Box 2. These factors should be considered in future formal training and in the performance enhancement system of the Bank's Human Resources department. Box 2 Factors that impeded use of acquired knowledge and skills Training factors that impeded use Instructor was too fast to allow for mastery of skills Training did not teach me anything new Materials not comprehensive enough for reference and application Work environment factors that impededuse Opportunity to use Institutional policies and procedures · Lack of time to use or · Too many changes in programs at mid-stream practice skills · Varying interpretations of Bank requirements · Infrequent use of skills · Unnecessarily complex policy requirements · Time lag between training that need to be simpler and application was too long · Lack of well-defined quality standards · Current job does not require · Budget constraints knowledge and skills (changed jobs) Social norms and affective atmosphere · Perception that you are a troublemaker when Incentive system you speak up · No positive incentives for · Absence of a delegated authority and using knowledge and skills atmosphere of respect and trust · Demoralizing impact of · Harassment by manager unequal reward system · Negative incentives, whereby Resistance to adoption of innovation excellence in teamwork is · Defensive colleagues against innovation discouraged · Rigid and unsupportive manager · Lack of knowledge among managers on trained topics 12 F. REPORTED IMPACT OF USE ON PERFORMANCE Fifty-nine percent of respondents indicated that the knowledge and skills used have had an above average or high impact on their performance (defined as a combination of productivity, quality of work, and work relationships). It is interesting to note that, for all programs except Information Technology, the highest impact is on the qualityof work. For Information Technology, the impact is highest on productivity (Figure 4). Figure 4. Impact of use on performance (Percentage of respondents rating impact above average or high) 80% 60% 40% 20% 0% World Professional Technical Behavioral & Social Bank Operations Functional Skills BankRespondents Development 1. Your productivity 2. Quality of work 3. Work relationships with bank staff 4. Work relationships with clients G. REPORTED IMPACT ON BANK CORPORATE OR INSTITUTIONAL OBJECTIVES 4.7 Learners were asked in an open-ended question to describe the impact of their performance change on the Bank (their units, sectors, or networks), on clients, and on partners. Respondents indicated various ways in which this impact was manifested. Content analysis of the most frequently occurring responses shows the different types of effects to fall mainly into eight key areas of the Bank's corporate performance criteria (Box 3). 13 Box 3 Bank corporate performance criteria and learner descriptions of the impact of their performance change on the Bank, its clients, and partners Professional quality of service (advisory and analytical services; quality at entry; portfolio management) · Increased effectiveness and efficiency in implementing Bank policies and practices · Accuracy in work, products, responses, and information · More reliable and higher quality outputs · Improvements and higher quality in the way business is conducted with clients and products · Can articulate coherent strategies Participation and partnership (quality of country dialogue; aid coordination and co financing; participation with stakeholders) · Improved collaboration as a result of understanding roles/perspectives · Improved communications · Developed stronger partnerships Selectivity (focus on borrower ownership and capacity; adequacy of instruments; use of Bank's comparative advantage) · Client responsiveness · Able to assess, identify, and respond to client needs · Able to provide technical assistance · Can provide options and solutions to clients Efficiency and innovation (budget efficiency; business standards; adaptability; and responsiveness) · Greater independence and less supervision in doing work · Improved organization and task completion · Able to multitask for time management and organization · Need limited or no supervision in work · Use innovative approaches and strategies · Improved communication that translates into better relationships with manager, peers, and team members Prudence and probity (risk management; financial management; monitoring and reporting) · Improved project management · Implement Bank policies and procedures · Provide better communications of financial implications and agreements 14 Box 3 (cont'd) Bank corporate performance criteria and learner descriptions of the impact of their performance change on the Bank, its clients, and partners Knowledge management and dissemination · Able to provide accurate information · Can better manage requests and records · Greater accessibility to products/services Professional excellence, honesty, integrity, and commitment among Bank staff · Greater confidence in and ability to work with groups and teams · Reinforced teamwork · More proactive and responsive to team/managers · More knowledgeable and more efficient when working with team members Healthy work environment (vibrant, stimulating, caring, and sharing) · Improved relationships, sharing of knowledge, and increased collaboration with peers and team members · Assume greater responsibilities · Reinforced teamwork · Can advise and provide feedback and solutions H. SUPPORT NEEDED TO ENHANCE PERFORMANCE AND PRODUCTIVITY 4.8 An open-ended question requested respondents to identify the things they would like their managers to provide that they believe would enhance their performance. A selection of the requested support is listed in Box 4. Box 4 Manager support needed to enhance performance and productivity Management support · More time to perform tasks · Larger budget and redistribution of tasks and priorities within unit · Competent management support · Simpler policy requirements · More reasonable deadlines · Constructive criticism from manager 15 Technical assistance Team support · Technical support and help desk · Collaboration with team · Availability of a specialist or members advisors to answer questions · More advanced training · Easier access to information I. SUMMARY OF FINDINGS AND CONCLUSIONS 4.9 The following serve as the basis for the conclusions made in this study. · 76 percent of learners rated the training they attended to be above average or high on the key quality criteria of relevance, effectiveness, and transfer of learning. · 83 percent of learners have used the knowledge and skills acquired from training. · 80 percent of learners have used more than 50 percent of the knowledge and skills they acquired. · The pattern of use is consistent across the different types of learning programs (Professional Technical, Bank Operations, Behavioral and Social, and Information Technology). · 59 percent of learners assessed the impact of their use of knowledge and skills on their productivity, work quality, and relationships withcolleagues and clients to be above average or high. The highest impact for the Professional Technical, Bank Operations, and Behavioral and Social programs is on the quality of work; for Information Technology, the impact is greatest on productivity. · Respondents indicated that use and performance improvements benefit the Bank through a better quality of services provided; improved partnerships; increased responsiveness to clients; increased efficiency and innovation; and greater professional excellence, honesty, and team work. · 42 percent of respondents indicated that they have had an above average or high level of support from the work environment. Bank-wide, the highest level of support came from managers and peers in the work unit. · Managers and peers provide different types of support. Manager support comes primarily in the form of mentoring and strategically driven coaching to enhance the relevance of staff performance; support from peers and colleagues is mainly technical to enhance the quality of products. · The main impediments to use are the lack of opportunities to use the knowledge and skills acquired, inadequate incentive systems, institutional policies and procedures, an atmosphere that is non-affective, and resistance to innovation. · Learners indicated that their performance change could be further enhanced through greater manager support. Suggestions for the type of support needed include more time to do tasks and more reasonable deadlines; larger budgets and a fairer distribution of resources; competent management support and constructive criticism; simpler policy requirements; technical assistance from experts; and more collaborative teamwork. 16 4.10 Despite the findings that FY 2001 training was of average quality and environmental support poor, the level of use of the knowledge and skills acquired through training is impressive and the impact on performance change adequate. The reported impact on the Bank's corporate goals is also impressive. The overall impression is that the Bank got substantial value for its FY 2001 investments in training. Improvements in training and on the work environment nonetheless could enhance the performance of learners, with the consequent productivity gains significantly increasing the returns on training programs. 17 5. ANALYSIS OF THE RELATIONSHIPS AMONG TRAINING, WORK ENVIRONMENT, AND LEVEL OF USE A. CORE HYPOTHESES 5.1 The hypothesis at the heart of this study is that the level of use of knowledge and skills acquired through training is a function of (1) the quality of training, in terms of relevance to job needs, its design and therefore ability to facilitate the transfer of learning, and its ability to raise knowledge and skill levels; and (2) the work environment, specifically the nature of support from managers, supervisors, and peers, and the existence of information and knowledge systems that support continuous learning and action. 5.2 Individual and personal factors, while also important to the level of use, are not assessed here. The study also recognizes but does not assess the host of institutional factors (e.g., incentives, communication, rules of the game, policies, and accountabilities) that play a role in behavior change (see Annex 1). 5.3 Using regression analysis, the study seeks to determine the combined and separate effects on use of training and work environment. The key questions that guide the analysis are the following: · How much of the level-of-use or behavior change can be accounted for by (1) training and (2) work environment? · Which of two - training and work environment is a higher predictor of use? · What specific variables or dimensions of these two factors are important predictors of use? · What are the results for the different programs? B. HOW MUCHOF THE LEVEL-OF-USE VARIANCECAN BEACCOUNTED FOR BY (1) TRAINING AND (2) WORK ENVIRONMENT? 5.4 Simple regression analysis (see Table 7) shows that 25 percent of the level-of-use variance can be attributed to training and work environment factors (R2 = .250). Broken down by program, this figureranges from 35 percent for the Professional Technical programto 20 percent for Bank Operations. The evidence overall supports the underlying theory of this study, that training and work environment affect the use of acquired knowledge and skills. The evidence also strongly suggests that there are other major factors that contribute to the variance in use and behavior change. In the context of the World Bank, where the motivation, resourcefulness, initiative, internal drive or locus of control and the commitment of individual staff members are key criteria for success, future studies on the impact of training programs should seek to include such personal factors as probable important predictors of impact. 18 Table 7. Regression of level of use on training and environment factors, by program (dependent variable, level of use = 10%; 25%; 50%; 75%; 100%)5 All Professional Behavioral Bank Information Respondents Technical and Social Operations Technology Work Environment 0.270** 0.215** 0.225** 0.328** 0.349** Quality of Training 0.355** 0.532** 0.432** 0.249** 0.233** Total model R2 0.253 0.364 0.298 0.200 0.236 Total model R2adj 0.250 0.348 0.291 0.185 0.228 N (matched) 550 70 200 103 166 *p<.05; **p<.01 C. WHICH OF THE TWO - TRAINING OR WORK ENVIRONMENT, IS A HIGHER PREDICTOR OF USE? 5.5 The results presented in Table 7 show that both training and work environment are significant predictors of use. The quality of training is a stronger predictor of use compared with the work environment; the difference is more pronounced for the Professional Technical and Behavioral and Social Skills programs. For Bank Operations and Information Technology, however, work environment is the more significant predictor. Overall, this result underscores the importance of efforts within the Bank and by the Learning Board to improve the quality of training--particularly for the Professional Technical, and the Behavioral and Social training programs. D. WHAT SPECIFIC VARIABLES OR DIMENSIONS OFTHE TWO FACTORS ASSESSED IN THESTUDY ARE IMPORTANT PREDICTORS OF USE? 5.6 An assessment of the features of the training and work environment factors that are important predictors could assist the targeting of interventions in training and enhance the impact of knowledge and skill use on performance. Analysis across all cases show that within the work environment, the best predictors of use are support systems . (See Table 8 next page for Bankwide results.) In terms of training, the best predictors are - the relevance of training to job needs, active involvement in the learner process, and preparation for the use of acquired knowledge and skills (i.e., in terms of contacts, references, materials, and networks). E. WHAT ARE THEDOMINANT RELATIONSHIPS FOR THEDIFFERENT BANK PROGRAMS? 5.7 Analysis of the most significant predictors of use for each of the four Bank programs reveals some significant differences as follows. 5Categorical variable 19 · For Behavioral and Social Skills (BS) programs, the quality of training plays a more significant role than the work environment in the use of knowledge and skills. (See Table 7.) For BS learners, an increase in knowledge and skills and being actively in the learning process are most significant for eventual use or behavior change. (See Table 8.) · For the Information Technology program, both training and work environment are significant predictors for use. (Table 7) More specifically, the most important predictors are information support systems, the relevance of the training to job needs, and active involvement in the learning process (Table 6). · For the Professional Technical program, the quality of the training plays a more significant role than the work environment in the use of knowledge and skills acquired. (Table 7.) An attempt to analyze the specific features of training which impact use was hampered by the small number of cases available for this analysis (n=47). The result thus presented in Table 8 for the PT program is not robust enough for one to accept the findings presented on the significant role of active involvement in training as being most important for transfer of learning on the job. · For the Bank Operations program, the work environment is a more important predictor of use than training. The most significant predictor is the support system for continuous learning and use, comprising such elements as help desks, mentors, and Websites. Again, this latter result, while logically true, is not based on a large enough number of cases for one to place confidence in the finding. Table 8. Regression of level of use on training and environment factors (specific variables) by program (dependent variable level of use = 10%; 25%; 50%; 75%; 100%) Independent variables All Professio Behavior Bank Informati Respondent nal al and Operati on s Technical Social ons Technolo gy Work environment factors Support from 0.001 0.016 -0.033 0.160 0.220 managers/supervisors Support from peers and team 0.146* 0.281 0.116 -0.027 0.108 members Support from colleagues across -0.104 0.222 0.051 -0.087 -0.275 the Bank Support system 0.215** 0.097 0.088 0.347* 0.261* Training factors The training was relevant to your 0.147* -0.164 -0.068 0.176 0.299* job needs 20 You were actively involved in the learning process 0.195** 0.374** 0.218* 0.071 0.259* The training increased your 0.034 0.017 0.289* 0.47 -0.151 knowledge and skills The training prepared you for using the new knowledge and 0.199** 0.232 0.155 0.169 0.209* skills Grade -0.037 -0.008 -0.117 0.082 0.45 Gender -0.030 0.132 -0.039 -0.113 -0.006 Reasons 0.027 -0.035 0.057* 0.063 0.000 (1=enhance performance; 2 =other) R2 0.376 0.669 0.433 0.368 0.502 R2 adj 0.354 0.567 0.375 0.226 0.437 N (Matched) 330 47 118 60 96 * p <.05; ** p <.01 F. RECOMMENDATIONS FOR IMPROVING TRAINING BYLEA 5.8 The results overall highlight the significance of the work by the Learning Board to upgrade the quality of training programs. Open-ended comments and recommendations made by respondents to the survey provide further information that could be used to direct efforts to improve the quality of training programs (see Box 5 and Annex 3). The evidence provided in these comments is consistent with the recommendations made in the Level 1 Evaluation of Training. G. SUMMARY OF FINDINGS AND CONCLUSION 5.9 Training quality and work environment conditions account for 25 percent of the variance in the level of use. This finding raises questions about the other factors that account for the remaining 75 percent of the variance for use. Personal and institutional factors not included in the study are probable contributing factors, and should be included in future impact studies. 5.10 While both training and work environment are significant predictors, training quality is overall a greater predictor of use for the Bank programs. The significance of training underscores the importance of efforts by the Learning Board to enhance the quality of 21 training programs, with the programs most likely to benefit from better training being the Professional Technical and Behavioral and Social programs. Respondents to the survey additionally provide a number of recommendations that could be used to direct improvements to formal training. Box 5. Improving the quality of training programs: Recommendations from learners Training design to increase learning · Have a good mix of theory and practice · Provide more interactive exercises based on practical examples · Use real and practical experiences · Provide more advanced, technical, or analytical training · Increase the use of participant experiences and application of new skills to own or different situations · Provide an opportunity to review the materials before the training to increase learning · Use more case studies. Continue case study discussions (to provide tools for implementing, monitoring, and evaluating projects) · Increase the one-on-one or group interaction with the facilitator Training design to increase continuous learning and performance improvement · Develop materials for follow-up use · Provide follow-up e-learning or distance learning · Create an online forum for sharing experiences and asking questions · Provide a reference pamphlet for easy access once back on the job Provide follow-up or advanced training It is interesting to note that while managers/supervisors, peers in the work unit, and colleagues across the Bank provide high levels of support, they are not important predictors of use or behavior change. (Table 8 ­ Bankwide data.) Survey respondents report that these groups provide valuable mentoring and coaching, which within the context of the Bank are important interventions. The ongoing efforts of the Learning Board and HRLOE to provide training on coaching and mentoring for both Managers and supervisors or Task leaders should greatly enhance the quality of such efforts in FY 2002 and beyond. Comments from survey respondents on factors that they believe could enhance performance and productivity also suggest that managers need to provide more competent support, more funding, and simpler policy requirements, and that they should establish more reasonable deadlines. Increased collaboration among team members is also identified as potentially important for enhancing performance and productivity. 22 ANNEX A. THEORETICAL MODEL OFTRAINING, LEARNING, BEHAVIOR, ANDPERFORMANCE CHANGE Learner continues to be Learner motivation committed to learning to learn and change and business change ("active ("knowledge worker") learner") OUTPUTS PERFORMANCE OUTPUT CHANGE DEMONSTRATED VIA: BANK CORPORATE GOALS STRATEGY · Significant learning gains or · Improved · Professional quality of HIGH-QUALITY attainment of desired proficiency levels. BEHAVIOR CHANGE productivity service TRAINING AND VIA: · Improved · Participation and LEARNING · Confidence and High and appropriate quality of partnership PROGRAMS empowerment to level of use of the do work. portfolio FOR: components of the · Selectivity innovation · Reflection on · Improved · Learning learning and working · Efficiency and relationships innovation · Transfer of readiness to use and partnerships learning (1) LEVEL3 in development · Prudence and probity EVALUATION · Learning-to- · Development of · Knowledge learn and action networks, strategies management and resources for LEVEL 1 dissemination continuous EVALUATION learning · Professional excellence, honesty, LEVEL2 integrity, and EVALUATION commitment among Bank staff WORK ENVIRONMENT CONDITIONS HEALTHYWORK Manager and peer support (2) Institutional support systems for continuous learning and performance change (3) See footnotes next page Institutional incentives for 23 change and innovation (4) Footnotes: (1) Readiness to use is a function of: (i) the degree to which the training/learning program was designed to ensure transfer of learning; (ii) performer reflection on how it will use Knowledge and Skills and translation of this into an action plan or into performance contracts; (iii) performer development of plans to address concerns about adoption and the type of support needed for effective adoption Bank Staff identify the following from the WBIES Level 1 Evaluation system as factors important for enhancing use of their knowledge and skills: (2) Support from Managers: Accountability and incentives; budget; empowerment via challenging opportunities, mentoring, quality control standards, managers knowledge of concepts and innovation applied; support for professional development; encourage innovation; encourage knowledge sharing and joint learning. Support from Peers: Teaming; peer supervision or review: trust; collegiality; knowledge sharing; joint learning; appreciation or honest critique of innovation. (3) Support Systems for Continuous and Action Learning: Thematic groups and communities of practice system for sharing knowledge and good practices; collaboration in carrying out work; team work to make necessary adjustments for increased impact; access to quality information and websites; access to experts and help-desks; role of mentoring programs; role of apprenticeship programs; role of professional development internships and externships. (4). Institutional Mechanisms and Incentives: Presidential statements; Senior management directives; clear policy and guidelines about innovation, clear statement of expected standards; accountability and quality control structure; incentive structure for innovation or for quality products; access to support units 24 ANNEX B. IMPACT EVALUATION QUESTIONNAIRE (THE QUESTIONNAIREWASADMINISTERED ELECTRONICALLY VIALOTUS NOTES) 1. Office in: O Washington, D.C. 2. Your grade level: O GA-GD 3. Gender: O Female O Country Office O GE-GG O Male O Other: O GH-GJ 4. Your main reason for taking the course (Please choose only one). O To enhance performance in current assignment O Professional interest and growth O For a future assignment O To network and share information Training quality Low Average High 1 2 3 4 5 How would you rate the degree to which 5. The training was relevant to your job needs. O O O O O 6. You were actively involved in the learning process. O O O O O 7. The training increased your knowledge and skills. O O O O O 8. The training prepared you for using the new knowledge O O O O O and skills? (contacts, references, materials, networks)? Use of Knowledge and skills 9. Have you used the knowledge and skills acquired fromthe course? O Yes O No If yes, answer questions #10-18; if no, answer questions # 16-18. 10. What percentage of this knowledge and skills have you used on the job? 10% 25% 50% 75% 100% O O O O O N/A Low Average High 1 2 3 4 5 11. How would you rate the frequency with which you used the knowledge and skills? O O O O O 25 N/A Low Average High 1 2 3 4 5 Impact of use or performance change 12. To what degree has your use of the knowledge and skills had an impact on: (a) Your productivity O O O O O (b) Quality of work O O O O O (c) Work relationships with Bank staff O O O O O (d) Work relationships with clients O O O O O (e) Other effects (please specify and rate)________________ O O O O O Impact on Bank Business 13. Please describe the nature of this impact for : (a) The Bank (unit, sector, network)______________________________________________________________________________ (b) Your clients ___________________________________________________________________________________________ (c) Bank partners __________________________________________________________________________________________ (d) Other groups __________________________________________________________________________________________ Work environment support Low Average High 1 2 3 4 5 14. To what degree have you had support or collaboration in using your knowledge and skills from the following: (a) Manager or supervisor O O O O O (b) Peers and team members in your unit O O O O O (c) Colleagues across the Bank O O O O O (d) Support systems (access to Websites, help desks, experts, mentors) O O O O O 15. What types of support did you get from the above? 26 Information for improvement 16. What factors impeded your use of the knowledge and skills? 17. What type of support would you now need to enhance your performance and productivity? 18. What are your comments and recommendations about training in order to increase either learning gains or its impact on performance or productivity? Thank You 27 28 ANNEX C. RECOMMENDATIONS FOR IMPROVING TRAINING AND INCREASING LEARNING GAINS OR IMPACT ON PERFORMANCE OR PRODUCTIVITY Professional Technical Behavioral and Social Skills Bank Operations Information Technology Quality of training design to Quality of training design to Quality of training design to Quality of training design to increase learning increase increase learning increase learning · Have a good mix of theory learning · Provide more in-depth · Host training offsite and practice · Limit content coverage to reading materials to eliminate · Provide more interactive allotted time · Develop an appropriate distractions exercises based on · Allow for practice time to length for training and · Provide more focused practical examples test knowledge and skills number of topics training · Use real and practical · Increase the match between · Provide more hands-on · Provide more experiences announced objectives and and practical training practical handouts · Provide more advanced, course objectives · Use expert trainers and reference technical, or analytical · Distribute book in advance guide(s) training · Increase hands-on and role- · Provide one-on-one · Increase use of participant playing exercises Quality design to increase sessions experiences and · Use trainers who can continuous learning and application of new skills answer questions raised by performance improvement to own or different participants · Provide follow-up Quality design to increase situations · Hold training away from training continuous learning and · Provide an opportunity to the Bank for maximum performance improvement review the materials gain · Provide additional before the training to · Continue to use one-on-one and follow-up increase learning time with instructor for training · Use more case studies. direct feedback · Allow time to Continue case study · Allow more practices, practice/use new discussions (to provide exercises, simulations and skills tools for implementing, role plays during training · Offer organized, monitoring, and · Continue to use excellent how-to technical evaluating projects) trainers and instructors courses · Increase the one-on-one or group interaction with the Quality design to increase facilitator continuous learning and performance improvement · Provide follow-up training 29 Quality design to increase (for advanced skills or to continuous learning and practice what is learned) performance improvement · Provide coaching and · Develop useful materials mentoring in using for follow-up use knowledge and skills · Provide follow-up e- · Put class materials on the learning or distance Web learning · Provide opportunity to · Create an online forum for apply new knowledge and sharing experiences and skills asking questions · Provide a reference pamphlet for easy access once back on the job · Provide follow-up or advanced training 30 ANNEX D HOW IMPACT IS MANIFESTED: DESCRIPTION OFWAYS INWHICH TRAINING BENEFITS THE BANK Professional Technical Behavioral and Social Skills Bank Operations Information Technology Programs Programs (OPCS, QAG, FP) (HD, PREM, ESSD, DEC, RM, (Communications, ACS, HR) Legal) Impact on Better able to implement Bank More efficient in performing tasks More knowledgeable, thus Better quality of presentations and Bank unit, policies and providing direction more efficient when working documents sector, with team members network Advise colleagues and give Improved organizational skills; can Improved communications, feedback to manager multitask Better understanding of Bank's organization skills, and time work management Can provide more accurate Empowered with new approaches for information to supervisor training others Understand quality assessment More efficiency in expressing data on better spreadsheets Change the way things are More effective and confident when done--redesign data collection working across groups and in teams procedures Better project follow-up and Can articulate strategies and management describe institutional development aspects Improved analytical rigor Assume greater responsibilities Draft better correspondence to clients Can work without direct Increased confidence supervision More clarity in correspondence and communications Better quality of outputs Can diffuse conflicts easier and sooner Focus on more important points 31 Impact on Able to assess client needs Improved collaboration (based on Provide more accurate More responsive and supportive to Clients better understanding of respective information and responses clients Can advise clients or provide roles) technical assistance Improved collaboration and Improved collaboration support as a result of Can provide options and understanding roles and solutions to clients different perspectives Can provide accurate data and information Can better assess and monitor projects Impact on Can share with them knowledge Provide better and more partners about project monitoring and accurate communications evaluation Can provide information about bank policies to academia and partners Can provide more accurate information Information I provide is user- friendly Impact on Strengthened relations with civil Improved working relations Improved communications and other groups society with consultants relations wb21626 L:\The Impact of FY01 Formal Training Turning Knowledge Into Action.doc August 12, 2002 6:48 PM 32