49344 Bank StaffLearning Agenda: Quality of Formal Classroom Training FYOO Mid-Year Evaluation Report Sukai Prom Jackson Ray Rist Tena Malone Sean Biko Sankofa Martin Chika Hayashi Maria Bina Palmisano Bank Staff Learning Agenda: Quality of Formal Classroom Training FYOO Mid-Year Evaluation Report March 2000 Sukai Prom Jackson Ray Rist Tena Malone Sean Biko Sankofa Martin Chika Hayashi Maria Bina Palmisano WBI Evaluation Studies Number ESOO-49 World Bank Institute The World Bank Washington, D.C. Copyright © 2000 The International Bank for Reconstruction and DevelopmentfThe World Bank 1818 H Street, N.W. Washington, D.C. 20433, U.S.A. The World Bank enjoys copyright under protocol 2 of the Universal Copyright Convention. This material may nonetheless be copied for research, educational, or scholarly purposes only in the member countries of The World Bank. Material in this series is subject to revision. The findings, interpretations, and conclusions expressed in this document are entirely those of the author(s) and should not be attributed in any manner to The World Bank, to its affiliated organizations, or the members of its Board of Directors or the countries they represent. If this is reproduced or translated, WBI would appreciate a copy. TABLE OF CONTENTS Page List of Tables .......................................................................................................................... i List of Acronyms ................................................................................................................... ii Preface ................................................................................................................................... iii Executive Summary .............................................................................................................. v I. Description ofthe FYOO Training and Learning Programs for Bank Staff ................. 1 II. Evaluation Purpose .......................................................................................................... 3 III. Evaluation Objectives .................................................................................................... 3 IV. Data Source, Analysis, and Performance Standard .................................................... 4 V. Main Findings ................................................................................................................... 5 1. Doing The Right Things - Relevance and Appropriateness - Level 1 Data Analysis .................................................................................................. 5 2. Doing Things Right - Efficiency, Effectiveness and Satisfaction - Level 1 Evaluation ........................................................................................................ 8 3. Doing Things Right - Analysis of Effectiveness Using Level 2 Methodology ......... 10 4. Ensuring Sustainability of Learning Outcomes and Institutional Support for Use of Learning .......................................................................................................... 11 VI. Conclusions, Interpretations and Recommendations ................................................ 15 VII. Good Practices ............................................................................................................. 16 VIII. Next Steps in Evaluation: January - June 2000 .................................................... 17 IX. Annex ............................................................................................................................. 19 LIST OF TABLES Table A: Network Programs: Percentage of Respondents Providing Ratings of 4 or 5 on the Three Factors Assessing "Doing the Right Things" .......................................... 6 Table B: Cross-NetworkIWBISD Supported Programs: Percentage of Respondents Providing Ratings of 4 or 5 on the Three Factors Assessing "Doing the Right Things" ........ 7 Table C: Functional and Foundation SkillslWBISD Programs: Percentage of Respondents Providing Ratings of4 or 5 on the Three Factors Assessing "Doing the Right Things" ....................................................................................................... 8 Table D: Network Programs: Percentage of Respondents Providing Ratings of 4 or 5 on the Three Factors Assessing "Doing Things Right" ............................................... 9 Table E: Cross-NetworkIWBISD Programs: Percentage of Respondents Providing Ratings of 4 or 5 on the Three Factors Assessing "Doing Things Right" .............................. 9 Table F: Functional and Foundation SkillslWBISD Programs: Percentage of Respondents Providing Ratings of4 or 5 on the Three Factors Assessing "Doing Things Right"............................................................................................................ 10 Table G: List ofCoursesfor Level 2: Evaluation and Average % Score Based on Participant Performance on Objective Tests Given at the end ofthe Course .................. 11 Table H: Percentage of Respondents Providing Ratings of 4 or 5 on Potential Impact or Transfer of Learning to the Job ........................................................................................ 12 Table I: Network Programs: Percentage of Respondents Providing Ratings of4 or 5 on Three Factors Assessing "Ensuring Sustainability" ......................................................... 13 Table J: Cross-NetworkIWBISD Supported Programs: Percentage of Respondents Providing Ratings of 4 or 5 on Three Factors Assessing "Ensuring Sustainability" ............ 14 Table K: Functional and Foundation SkillslWBISD Programs: Percentage ofRespondents Providing Ratings of4 or 5 on Three Factors Assessing "Ensuring Sustainability" ...................................................................................................... 14 LIST OF ACRONYMS ACS Administrative and Client Support DEC Development Economics ESSD Environmentally and Socially Sustainable Development FPS Financial Products and Services FPSI Finance, Private Sector and Infrastructure HD Human Development IRL Institute for Research on Learning OCS Operational Core Services OED Operations Evaluation Department OECD Organization for Economic Cooperation and Development PREM Poverty Reduction and Economic Management QAG Quality Assurance Group SPSS Statistical Package for the Social Sciences WBI World Bank Institute WBIES World Bank Institute, Evaluation and Scholarships 1FT Information Technology COMM Communications HUB Region-Based Training of Country Office Staff ii PREFACE The Learning Agenda ofthe Bank: FYOO Evaluation Objectives For FYOO, the objectivesfor the evaluation ofthe Learning Agendafor Bank staff training are: · To provide information on the quality of the formal classroom training (Levelland Level 2 Evaluations), to complement this with more differentiated analyses (by sector, course types, and participant profiles), and to provide information on good practices. · To develop a framework and methodology for the evaluation of the impact of training on performance and productivity (Level 3 Evaluation) and to use this to carry out evaluations of five courses. · To expand the evaluation of FYOO beyond structured classroom training to include learning activities that are alternatives to the traditional classroom training. The two addressed in this years' evaluation are: Sector Weeks; and The learning activities and practices of Thematic Groups (such as clinics, intact team training, group work sessions, and other small group practices). The World Bank Institute Evaluation Unit (WBIES) has begun to conduct evaluations in all three areas. This report is focused on the first objective - the evaluation of structured classroom training delivered between July and December 1999. iii Bank StaffLearning Agenda, FYOO Midyear Evaluation - Executive Summary EXECUTIVE SUMMARY Evaluation Objectives The objectives are to determine whether: · Training programs are relevant and appropriate and are "Doing the Right Things" ofvalue for the Bank and staff; · The processes, outputs, and outcomes are technically optimal and likely to produce high impact on performance - "Doing Things Right"; and · The conditions are in place to ensure the sustainability of learning outcomes - "Ensuring Sustainability". Methodology The evaluation uses Levelland Level 2 Evaluation methodologies. One hundred and eighty-three training courses are included in the Level 1 Evaluation for a total of 1,989 participants. It is estimated that this represents between 25% and 40% of the participants of the courses included in this evaluation. To date, 20 courses have been identified as appropriate for inclusion in the Level 2 Evaluation. Data for seven of these courses is included in this evaluation. WBIES is working with the Network Learning Representatives to increase the number of Level 2 assessments from 20 to 30 courses. The analysis is made across all programs for a Bank-wide average, by Network and by Sector or Family within Network. Comparisons are made between the FY99 and FYOO data. Improvements made are highlighted while problem areas still remaining are delineated. The benchmark for judging quality based on Levell assessment is that 85% ofthe respondents to the questionnaire will award a score of 4 or 5 (i.e. good or excellent) on a five point scale. For Level 2, the performance indicator is that courses with post-test assessment will have an average score of 70%. Main Findings · On a Bank-wide basis, the relevance and appropriateness of the programs are good (84% of participants rated the courses as good or excellent). The efficiency, effectiveness, and level of satisfaction are also very good (86%). This represents a major improvement over the FY99 result that averaged 77%. The sustainability or likelihood for staff to use knowledge and skills acquired from the training is very good (85% for FYOO and 75% in FY99). The built-in design and delivery factors that enhance such transfer of learning on-the-job are also quite good (81 %). v Bank StaffLearning Agenda, FYOO Midyear Evaluation ~ Executive Summary · Contributing to these results is the improvement in the performance of PREM and FPSI with differences in scores between FY99 and FYOO ("difference scores") averaging 20% and 12%, respectively. Performance on key variables (assessing relevance, appropriateness, efficiency, and effectiveness) operate at the 80-85% range for PREM, and at the 75-80% range for FPSI. While it is true that the cases involved for PREM and FPSI are quite small, the trend of the performance observed is encouraging. The design and development factors accounting for the improvement should be fully understood and adopted. · It is also important to note that the improved results originate from including in this year's evaluation, the WBISD foundation and functional skills programs such as- the HUB Resident Mission Staff Training (90%), and the long standing Communications training (94%), both of which are well recognized in the Bank for their high quality. · The OCS and the Cross-Network programs have difference scores averaging 6% and 7% respectively across all variables of investigation. The rate of increase demonstrated by the two appears to have greater validity given the larger database for the two Networks. Both programs are also at a more mature stage of development when compared with PREM and FPSI and thus are less likely to yield massive rates of increase or major fluctuations in such increases. · An examination of the design and delivery variables assessed for this study shows sector-specific variation in areas needing improvement. It also shows that the two key areas that need a concerted effort in improvement by all training programs are: (i) the pacing of the various sessions, and (ii) the depth of coverage of the course content. · The results show that only 16 of the 183 courses in this evaluation have an above average rating of 85% on "pacing" and "depth of coverage". The good practices used by these courses are being analyzed by WBIES and will be shared with all stakeholders. · Two other areas still needing attention in the staff trammg courses are the application of concepts and principles to Bank work, and the use of case studies or group exercises or other interactive learning processes. Good practices in these areas are also being analyzed by WBIES. Recommendations for Program Improvements Poor "pacing" and "depth of coverage" appear to be affected by the following: a. Conflicting demands on the part of staff, i.e. (i) staff interest for courses with not more than two days, and (ii) staff need to understand course subject matter in greater detail. b. Heterogeneity in the level of knowledge or ability among participants, and inability of instructors or courses designers to manage heterogeneity. vi Bank StaffLearning Agenda, FYOO Midyear Evaluation - Executive Summary c. Inability on the part of instructors to engage participants in the learning process or to make maximum use of the higher levels of knowledge or ability existing in the training room. Improvements to be made in pacing and depth need to examine not only instructional methodology but also: (a) the nature of the participants and the degree to which they represent the target group for whom the course was designed and (b) the process for selecting participants into the various deliveries relative to defined prerequisites for participation. Improvements for application to Bank work and interactive learning via case studies and group exercises could both enhance the depth of learning as well as the practical usefulness of the courses. The good practices noted by participants in these areas include the use of case studies that: provide practical, hands-on analysis; are based on situations encountered in Bank-financed projects; are more complex; and allow for intensive analysis. The poor application of principles and concepts to Bank work is common in cases where the focus is on OECD experiences or technologies and where the presenters do not know the nature of Bank work or development work in general to fully discuss implications or application. The assumption that staff would be able to independently draw out the implications or automatically develop applications to their work is invalidated. Designers of courses must be prepared to include time to discuss or to explicitly outline application of course concepts and principles to Bank work. Precis on "what works, why and how" for Bank staff in these four areas are being prepared by WBIES and will be produced starting in March 2000. Next steps for Evaluation, January - June 2000 1. Development of the Levelland Level 2 Systems for Evaluation The following actions will be taken to enhance the validity and robustness ofthe evaluation findings and interpretations. · Increase response rate for Level I and improve on the external validity or generalization of the results ofthe evaluation system: Networks must make the completion of an end-of-course questionnaire and participation in a Level 2 assessment requirements for course participation. WBIES will pilot the use of an electronic system (Cold Fusion) for questionnaire completion and submission. This system will also increase efficiency in data collection and processing. · Provide differentiated analysis that would increase the internal validity of the evaluation: All Training Programs are to provide WBIES information requested on the Course Description form. · Enhance the interpretation ofthe quantitative analysis: WBIES will establish a coding system to integrate quantitative and qualitative information in the SPSS data files. VII Bank StaffLearning Agenda, FYOO Midyear Evaluation - Executive Summary · Increase the number of courses with Level 2 assessment: Network Representatives will provide WBIES with additional courses to be included in the Level 2 evaluation. WBIES will pilot the use of an electronic system for Level 2 assessments (pre- and post-testing). 2. Development of Level 3 Evaluation System The work started on Level 3 Evaluation will continue with a focus on the following factors: · Pilot the use of electronic system for collecting data via questionnaires with the HUB training program. · Pilot the use of an integrated system of questionnaires, interviews, observations, and product analyses as part ofthe Level 3 system development. · Pilot the integration of Level 3 and Level 4 Evaluation in the Quality at Entry course. 3. Good Practice Notes · Complete good practice notes on "What Works, Why, and How" and produce first series in March 2000. · Establish a web-site for the evaluation ofthe Bank staff learning agenda. 4. Sector Weeks · Develop questionnaires to assess eight sector weeks in collaboration with Networks. · Establish an electronic system for questionnaire completion or for follow-up purposes. 5. Thematic Groups as a Community of Practice · Refine the concept note for the monitoring and evaluation with the Network Learning Representatives and develop an action plan for FYOO and FYO1. · Work with the Institute for Research on Learning (IRL) to begin quantitative and qualitative/ethnographic analysis ofthe practices ofthe communities. V1l1 I DESCRIPTION OF FYOO TRAINING AND LEARNING PROGRAMS FOR BANK STAFF The Learning Agenda for Bank Staff is composed of a variety of interrelated and mutually reinforcing activities and practices designed to enhance staff learning and performance and the development of the Bank as a learning organization. Among the most significant practices are the following.) Formal classroom training courses or seminars: These are designed primarily to develop or enhance the depth of knowledge, skills, and strategies in the technical, core business, relational, and management domains of the Bank's work. This evaluation report is focused on these courses and seminars. Thematic Group Continuous Learning and Performance Enhancement Activities and Practices. These include formal and informal group activities designated as clinics, action- learning workshops, intact team work sessions, and integrated seminal modules. They are focused less on knowledge acquisition and more on: · Refining knowledge and practice of greater value to Bank clients via the sharing of information on good practices; · The empowering of staff members as part of a community of professionals via the development of learning-to-learn strategies, or via opportunities that highlight the contributions they make in the development of new ideas and approaches; · The development of products of greater value and integrity, and the achievement of a collective and coordinated impact on clients within a common sphere of influence via coordinated efforts and activities with colleagues across disciplines; and · Renewing knowledge and exploring alternatives to current practices via an examination of new developments in the field, of cutting edge knowledge and technologies as well as via the exploration of new goals and approaches for development and poverty reduction. 1In the absence ofcomplete information about the FYOO Network Training Programs (which are still being developed), WBIES is not in a position to report on the scope ofthese activities for FYOO. The Learning Representatives plan to provide this information to the KLC as a complement to this Report. Bank StaffLearning Agenda, FYOO Midyear Evaluation WBIES will start the evaluation of the activities of the Thematic groups in FYOO. A concept note has already been prepared to be discussed with the Networks. Individualized, Self-paced and Just-In-Time Learning for Performance or Professional Enhancement: These include activities designated as mentoring programs, apprenticeship programs, professional development grants programs, coaching systems, computerized desk-top modules, CD roms, video-tapes, hotlines, help-desks, and web-sites. These programs are not being evaluated by WBIES in FYOO. Sector Weeks: These are designed primarily to bring all members of a sector, their client counterparts, and partners to advance their cutting edge knowledge, to share knowledge, to reflect upon on-going policies and practices, to generate new ideas, and to bond and develop collegiality among sector members. About 12 Sector Weeks are planned for FYOO. WBIES anticipates evaluating a selected number of these Sector Weeks during FYOO at the request ofNetworks and Sector Boards. Other Learning Activities: These include Study Tours; Distance Learning Courses; and Joint Staff and Client Learning Seminars led by Networks (and not by WBI). These activities are not the focus ofthe FYOO evaluation by WBIES. 2 Bank StaffLearning Agenda, FYOO Midyear Evaluation II EVALUATION PURPOSE It is noted above that the focus of this report is on the evaluation of formal classroom training courses delivered between July and December 1999. The purpose of the evaluation is to provide information to: · Show trends in the performance ofthe Bank stafftraining program; · Guide the improvement of the quality of formal training during FYOO; · Highlight exemplary practices of high impact in the FYOO programs; and · Guide decisions for program planning in FYOO and FYO1. III EVALUATION OBJECTIVES This evaluation seeks to determine whether: · The training programs are relevant and appropriate and offer the greatest potential value for the Bank - Doing the right things; · The processes, outputs and outcomes are efficient, effective and satisfying and likely to produce high impact on learning and performance - Doing things right; · The conditions are in place to ensure the sustainability of learning outcomes and the transfer oflearning on the job - Ensuring sustainability. In addressing these areas, the evaluation will compare the performance of the training programs from FY99 and FYOO, and this will highlight major improvements in performance. Performance areas that still need to be strengthened to enhance course value and quality will also be identified. The data base for FYOO is more differentiated than the FY99 data and permits analysis of differences in performance among the various sectors within Networks. This information will allow program managers to make more targeted decisions. 3 Bank StaffLearning Agenda, FYOO Midyear Evaluation IV DATA SOURCE, ANALYSIS, AND PERFORMANCE STANDARD Level 1 - Data Analysis: The analysis will be based primarily on participant responses to Level 1 End-of-Course questionnaires which have been customized for various training types. In spite of varying formats and differing scopes of content coverage, these questionnaires all assess: (a) course outcomes (relevance, effectiveness, efficiency, potential impact, overall usefulness and satisfaction); and (b) course design and delivery features related to: content, structure, instructional methods, materials, and participant behaviors. For this reporting period, the analysis is based on responses provided by 1,989 respondents. There were 95 training courses scheduled and 183 delivered between July and December 1999. The benchmarkforjudging quality based on the Levell assessment is that85% of the respondents to the questionnaire will award a score of4 or 5 (i.e. good or excellent) on thefive point scale ofthe questionnaire. Level 2 - Data Analysis: The second data source is the performance of participants on objective tests measuring learning outcomes. Thus far, assessments have been completed for seven or 35% of the 202 training courses selected at the beginning of the year for Level 2 Evaluations in FYOO. While the ideal for WBIES is for both pre- and post-testing to derive gain scores, an analysis of the overall Level 2 assessment process for FYOO indicates that the post-test only methodology will be most feasible for the majority of the courses, given (a) the short duration of courses - average duration oftwo days, and (b) limitations in the current capacity to develop good tests for both pre- and post- purposes. The evaluation system thus established for FYOO is a criterion-referenced system based on post-test performance. < Discussions with various stakeholders indicate that an acceptable level of learning mastery is an overall average of 70% (across all participants who are tested). Using this as a basis,the performance indicator established for the evaluation of the training programs is that courses with a post-test assessment will have an average score of 70%. 2This number has now been increased to 30 courses for Level 2 assessment and evaluation. 4 Bank StaffLearning Agenda, FYOO Midyear Evaluation v MAIN FINDINGS The synthesis provided in this mid-year report is based on the prime factors that define - doing the right things, doing things right, and ensuring sustainability. The specific variables that define these factors are provided in the tables in the annexes and significant features of these findings are referred to in the main text below. Information in this report is provided for: (a) Networks and sectors for which data are made available (i.e. HD, PREM, FPSI, OCS3); and (b) the WBISD supported programs (i.e. cross-Network programs, HUB Country Office Staff Training, Communications, Information Technology, and the Orientation for new staft/. 1. Doing The Right Things - Relevance and Appropriateness -Levell Data Analysis The factors of the training program analyzed in assessing relevance and appropriateness were: · Linkage with the content of the Bank's corporate performance or business requirements; · Relevance to staff learning needs for job performance and professional growth; and · The application of appropriate principles for adult learning and performance improvement. Comparison FY99 and FYOO: The results for "Doing the right things" summarized in Table A below indicate that on a Bank-wide basis, 84% of the respondents rated the programs as good or excellent in their relevance or appropriateness. This is "good performance". It is nine percentage points above the FY99 result. It is also a mere one percentage point below the WBI benchmark for "very good" performance. 3The ESSD program is not included in this FYOO evaluation. It is currently being revised. 4The original plan of WBIES was to provide more differentiated analyses for the various course types and for differentiated populations. This would have included analyses for courses - with structured versus non-structured knowledge types; with technical versus social interpersonal objectives; of short versus longer duration; for specialists versus for general operational staff; with homogeneous versus heterogeneous groups; and for different populations. This information which is requested by WBIES via a Course Description Form has not been completed by the Networks. The request for demographic data on staff made via the Level I Questionnaire is also generally not honored. WBIES will work with the Learning Representatives for the rest of FYOO to develop a method for collecting this and other customized information desired by the Networks and other stakeholders. 5 Bank StaffLearning Agenda, FYOO Midyear Evaluation Table A: Network Programs Percentage ofRespondents Providing Ratings of 4 or 5 on the Three Factors Assessing "Doing the Right Things" Total All Bank ESSD HD PREM FPSI oes Network FY99 I FYOO FY99 I FYOO FY99 IFYOO FY99 I FYOO FY99 I FYOO FY99 I FYOO FY99 I FYOO # ofRespolldellis -- 1015 I 1989 159 I -13 I r 102 I 659 I 115 335 I r8 -16 I 659 Linkage with content of Bank & 77% /83% I - 63% 77% 186% 66% 180% 67% 175% 80% 183% 7l'Y. 181% Perfonnance tis· 6% NIA 19% 14% 8% 3% 11% I I Linkage with staff learning needs for 77% 185% - 65% 72% 182% 65% 1850;., 69% 177% 81% 186% 72'Y. 82% perfonnance tis· 8% NIA 10% 20% 8% 5% 10% I I Application of principles of adult I 70% 182% 54'Y. - 72% 180% 51% 83'Y., 56% 177% 73% 81% 63% 180% learning tis· 12% NIA 8% 32% 21% 8% 17% Total 75% I 84% 61% I - 74% I 83% 61% I 82% 64% I 76% 78% I 83% 69% J 81% tis" 9% NIA 9% 21% 12% 5% 12% *ds - differellce between the results ofFYOO and FY99. Unless mdlcated with a mmus sIgn, all difference scores show a positive change infavor ofFYOO. Network and Sector Performance: The pattern of increase from FY99 to FYOO also holds true for all the Networks. This is most remarkable in the cases of PREM, FPSI and HD: The three Networks have average difference scores of 21 %, 12% and9 %, respectively. The increase moves the HD and the PREM from average and poor performance to good performance. The performance is almost comparable to that of the OCS and Cross-Network programs. These two groups have average increases of 5% and 4%, respectively, basically maintaining their FY99 status of "good performance". Most significant of all the improvements observed are: (a) the performance of PREM and FPSI in their "application of the principles of adult learning" (difference scores of 32% and 21 %, respectively) and (b) the performance of PREM in its "linkage of courses with staff learning needs for performance on the job". The findings attest to the increased efforts by the two Networks in improving the relevance and appropriateness oftheir courses in FYOO. The results in Table AI in the Annex (which presents a more differentiated analysis of these factors), and Tables 2 and 3 also in the Annex (which provide results for the various sectors of the Networks), show that the following areas should be of focus in the continued efforts at program improvement. · All programs need to pay attention to "the depth of coverage of the various courses" (77% for all Bank). Table 3 shows that this applies to almost all of the sectors with the exception ofACS (94%), DEC (91%) and the HUB (88%). · HD needs to work on "case studies for interactive learning and knowledge construction (68%). · The OCS needs to pay particular attention to the pacing of instruction and to the logic in the sequencing ofthe sessions ofthe courses. 6 Bank StaffLearning Agenda. FYOO Midyear Evaluation Performance of Cross-Networks: The analysis of the results shown in Table B below indicates that the Cross-Network programs which are supported in their design and delivery by WBISD have made some improvements in their results (average difference score of 4% points). (See Table B below.) This improvement is true for the DEC and Financial Products and Services programs (FPS), even though the attainments made are still below the quality benchmark. Table B: Cross-NetworkIWBISD Supported Programs Percentage of Respondents Providing Ratings of 4 or 5 on the Three Factors Assessing "Doing the Right Things" Total Cross- All Bank Cross-Network Networks DEC I FPS I QAG I Legal I HR FY99 I FYOO ~I~I~I~I~I~I~I~I~I~ FY99 I FYOO # ojRe.,pol/dents 1015 I 1989 12 I 23 I 57 I 144 I I 11 I 52 I I 43 I 164 I 178 I Linkage with content of Bank & 77% 183% 83% 186% 81% 800/. - 75 I - I - 89% 65% 79% 180% 1 % Perfonnance ds' 6% 3% -1% N/A NJA N/A 1% Linkage with staff learning needs for 77% 185% 70% 182% - 77 I - I - 80% .185% 77% 181% 1 % 81% 77% perfonnance ds' 8% 12% 5% N/A N/A N/A 4% Application of principles of adult 70% 182% 85% 180% 73% 183% - 77 I - I - 72% 180% 1 % 80% 49% learning ds' 12% -5% 10% N/A NIA N/A 8% Total 75% I 84% 79% I 83% 78% I 82% - I 76% 83% I - 64% I - 76% I 80% ds' 9% 4% 4% N/A N/A N/A 4% Ods = difference between the results of FYOO and FY99. Unless indicated with a minus sign, all difference scores show a positive change infavor ofFYOO. Note: Legal and HR have FY99 data only. There is no FYOO data at this time. QAG is a new addition to the Cross-Network program and thus only has FYOO data. Performance of WBISD Functional and Foundation Knowledge and Skills: The FYOO evaluation includes an analysis of the performance of the WBISD programs. In FY99, only two of the programs - the ACS and the Orientation programs - were evaluated. The FY99 data for the Orientation Program is, however, not included since a different questionnaire was used in FY99. Table C below shows a high level of performance in FYOO surpassing the Bank benchmark in all programs except for the Orientation program which is rated at 81 %. 7 Bank StaffLearning Agenda. FYOO Midyear Evaluation Table C: Functional and Foundation Skills/WBISD Programs Percentage of Respondents Providing Ratings of 4 or 5 on the Three Factors Assessing "Doing the Right Things" All Bank WBISD TotalWBISD Information ACS Orientation Communication Technology I I I Leadership I HUB I I FY99 FYOO FY99 FYOO I FY99 FYOO FY FYOO FY99 FYOO FY99 FYOO FY99 FYOO 99 # ofRespondents 135 I 98 I 16- I 230 I r- I 16 -13 I 128 135 I JlI6 Linkage with content of Bank & I I 93% 194% - - - 89% - 85 - 75 - 91 - 87 1 % 1 % 1 % 1 % Perfonnance ds' 1% N/A N/A N/A N/A N/A N/A Linkage with staff learning needs for 91% 195% - 81 - I - 86 - 97 - 85 - 90 1 % 94% 1 % 1 % 1 % 1 % perfonnance ds' 4% N/A N/A N/A N/A N/A N/A I Application of principles of adult I 91% 89'Y. - - - I 90% - 85 - 91 - 93 - 1 % 1 % 1 % learning 1 90% ds' -2% N/A N/A N/A N/A N/A N/A Total 92% I 93% - I 81% - I 91% - I 85% - I 88% - I 90% - I 89% ds' 1% N/A N/A N/A N/A N/A N/A Ods = difference between the results of FYOO and FY99. Unless indicated with a minus sign. all difference scores show a positive change infavor ofFYOO. 2. Doing Things Right - Efficiency, Effectiveness and Satisfaction - Levell Evaluation Comparison ofFY99 and FYOO: The factors analyzed were: (a) delivery efficiency, (b) learning effectiveness, and (c) participant satisfaction. As indicated in Table D below, Bank performance is above the established standard of 85%, or of"very good" quality. It averages 86% for all three factors considered together. It is nine percentage points above the FY99 Total performance of 77%. Network Performance: The Networks made the most dramatic shift in performance with an average increase of 14% for FYOO (for 83% in FYOO). Table D shows this increase to come mainly from PREM (19% increase), FPSI (12%), and HD (11%). The OCS increased its performance by 6%. This increase, although less than that of the other Networks, is a more stable rate considering two factors: (i) the large number of cases included in the analysis for OCS; and (ii) the more advanced stage of curriculum development of the OCS program. An analysis of the specific variables for "doing things right" (Table Dl and Table 2 in Annex) suggests that for all programs, the trend of improvements in the training programs could be enhanced via major modifications in the pacing of the various sessions in almost all courses for all families. The pacing is generally too fast; instructors try to cover a large amount of subject matter within the short duration that characterizes the courses. In cases where there are major differences in knowledge levels among participants, the pace ends up being highly unsatisfactory - too fast for some, or too slow for others. The low level of performance for "pacing" is identical to that for depth of coverage (see above section on Doing Things Right) and suggests a need to examine both variables in tandem when considering improvements to be made. 8 Bank StaffLearning Agenda, FroO Midyear Evaluation Table D: Network Programs Percentage of Respondents Providing Ratings of 4 or 5 on the Three Factors Assessing "Doing Things Right" Total All Bank Network Networks HD I PREM I FPSI I OCS I ESSD FY99 I FYOO ~I~I~I~I~I~I~I~I~I~ FY99 I FYOO # ofRespondents 1015 I 1989 43 I 47 I 102 I 44 I 77 I 115 I 335 I 378 I 159 I 716 I 1116 Delivery I I I I I I I 77% 86% 81% 76'l'. 67% 90% 68% 78% 79% 83% 63% - 72% 82°1., Efficiency ds' 9% -5% 23% 10% 4% N/A 10% Learning I I I I 75% 85% 66% 81% 66% 78% 82% 1 89% - 66% 1 84% Effectiveness 65% .1 89% 52% ds' to% 24% 15'l'. 12% 7% N/A 18% Participants' I I I I I I 80% 87% 73% 87% 70% 89% 63% 1 79% 82% 89% 65% - 71% 86°1., Satisfaction ds' 7% 14% 19% 16% 7% N/A 15% Total 77'l'0 I 86% 73% I 84% 68% I 87% 66% I 78% 81% I 87% 60% I - 70% I 84% ds' 9% ll'l'. 19'1'. 12% 6% N/A 14% *ds = difference between the results ofFYOO and FY99. Unless indicated with a minus sign, all difference scores show a positive change in favor ofFYOO. Cross-Network Performance: The performance of the Cross-Network is shown in table E below. The results show that the programs are all above the benchmark in "doing things right". This applies even to the newly developed courses by QAG. Performance of WBISD Functional and Foundation Knowledge and Skills: These programs principally all perform above standard except for the Orientation program. (See Table F.) The evidence provided in Table 3 of the Annex suggests a need to re-examine its structure in making course adjustments, particularly the "logic in the sequencing of the vanous sessions" of a given course (72%), and the "pacing" of the instructional process (70%). Table E: Cross-NetworkIWBISD Supported Programs Percentage of Respondents Providing Ratings of 4 or 5 on the Three Factors Assessing "Doing Things Right" Total Cross- All Bank Cross-Network Networks DEC I FPS I QAG I Legal I HR FY99 I FYOO ~~~I~I~I~I~I~I~I~I~ FY99 I FYOO # ofRespondenls 1015 I 1989 12 I 23 I 57 I 144 I I 11 I 52 I I 43 I 164 I 178 Delivery I I I I I I 77% 86% 82% 94% 84% 85% - 86% 91% - 66% - 81% 86% Efficiency I ds' 9% 12% 1% N/A N/A N/A S% I I Learning I85% I 96% I85% I I 87% 75% 83% 74% - 91% 92% - - 80% Effectiveness 72% ds' 10% 13% 11% N/A N/A N/A N/A Participants' I I I I I I 80% 87% 83% 96% 87% 87% - 92% - - 83% 89% Satisfaction 1 100% 71% ds' 7·/. 13% 0 N/A NfA N/A 6% Total 77% I 86% 83% I 95% 82% I 86% - I 92% 92% I - 70% I - 82% I 87% ds' 9% 12% 4% N/A N/A N/A S% *ds = difference between the results ofFYOO and FY99. Unless indicated with a minus sign, all difference scores show a positive change in favor ofFYOO. 9 Bank StaffLearning Agenda, FYOO Midyear Evaluation Table F: Functional and Foundation Skills IWBISD Programs Percentage of Respondents Providing Ratings of 4 or 5 on the Three Factors Assessing "Doing Things Right" Information ACS Orientation Communication Technology Leadership HUB TotalWBISD FY99 I FYOO FY99 I FYOO FY99 I FYOO FY99 I FYOO FY99 I FYOO FY99 I FYOO FY99 I FYOO # ofRe.'pondenls 135 I 98 I 167 I 230 I 477 I 16 I 128 135 I 1116 Delivery I 94% 95% - I - I I 94% - 88% - 93% - I - I Efficiency 77% I 92% 90% ds· 10/. N/A N/A N/A N/A N/A N/A Learning I I 93% 94% - I - I I 95% - 84% - 83% - I - Effectiveness 79% I 94% 88% ds· 1% N/A N/A N/A N/A N/A N/A I 1 Participants' I I I I I 93% 94% - - 95'Yo - - - - 820/. - 93% Satisfaction 100% ds· 1% N/A N/A N/A N/A N/A N/A Total 93% I 94% - I 78% - I 95'Yo - I 86% - I 92% - I 89% - I 90% ds· 1% N/A N/A N/A N/A N/A N/A *ds = difference between the results ofFYOO and FY99. Unless indicated with a minus sign, all difference scores show a positive change in favor ofFYOO. 3. Doing Things Right: Analysis of Effectiveness Using Level 2 Evaluation Methodology A Level 2 evaluation offers a more objective analysis of learning effectiveness. For FYOO, a total of 30 courses have been identified for inclusion in the Level 2 Evaluation. Given the predominance of courses with only post-test data, the benchmark for evaluating program quality is that courses included in the Level 2 Evaluation will have an average score of70% based on the post-test score ofparticipants. An analysis of the level of performance mastery for the seven courses tested thus far reveals a Bank-wide average of 74%, with a score range of 61 % and 78%. As shown in Table G below, five or 71 % ofthe seven courses thus far evaluated have average scores at or above 70%. One ofthe seven courses misses this standard by two percentage points, scoring an average of 68%. The overall performance for FYOO is a great improvement from the FY99 results, where only 58% of the courses met the established criterion of quality performance. The difference between the two results is 15 percentage points. The more objective evidence from the Level 2 assessment correlates with the observed trend of improvements in FYOO noted in the Level 1 evaluation. JO Bank StaffLearning Agenda, FYOO Midyear Evaluation Table G: List ofCoursesfor Level 2 Evaluation and their Average % Score Based on Participant Performance on Correct Answers in Tests Measuring Learning Seven Courses with Level 2 Assessment by Network Average % Score Course Score Network Score Bank Total FPSI 61% · Micro-finance 61% OCS 78% · Project Cycle Management with the Logframe 72%. · Activity Base Costing 78%. · Performance Measurement 73%. · Training the Trainers (Procurement) 87%. Cross-Network (CN) 76% · Microeconomic Policy 84%. · Investment Operations Economic 68% Overall Bank Score 74% 4. Ensuring the Sustainability of Learning Outcomes and Institutional Support for Use ofLearning Training has value for the Bank when participants use the knowledge and skills they have acquired from the courses and improve their job performance and productivity. Information of this type is generally provided via a Level 3 Evaluation5. This Level I Evaluation, however, assesses the sustainability of the learning outcomes and their transfer effects via: · Participant ratings of the likelihood that they will use the knowledge or skills acquired from the training; · An analysis of key design and delivery features of training that have been demonstrated in the literature on training to impact transfer of learning; · An analysis of the type of management involvement that staff perceive as important in enhancing transfer of learning and performance improvement; · An analysis of other performance enhancement or support systems that staff perceive as important in enhancing their use of knowledge and skills from training. Participants indicated a very high likelihood to use the knowledge and skills acquired from the training. The Bank-wide average rating in FYOO is 85%. (See Table H) This level 5WBIES is starting a Level 3 Evaluation of selected courses in FYOO. 11 Bank StaffLearning Agenda, FYOO Midyear Evaluation of performance is true for PREM, OCS, ACS, and the WBISD programs. HD and FPSI have ratings of 75% and 74% respectively. The Cross-Networks have 89%. Tables 2 and 3 in the Annex show sector variations within Networks. The lowest performing sectors are: Energy and Mining - 64% (from FPSI); Education - 68% (from HD); FPS - 79% (from Cross-Network); PSG - 75% (from FPSI); Gender - 75% (from PREM); and the Orientation program - 81% (from WBISD). TableH: Percentage of Respondents Providing Ratings of 4 or 5 on Potential Impact or Transfer of Learning to the Job Networks Total All Bank Networks Networks HD I PREM I FPSI I OCS I ESSD FY99 I FYOO ~I~I~I~I~I~I~I~I~I~ -- FY99 I FYOO # ofRespondents 1015 I 1989 ./3 I r I lli2 I 659 I I 115 I 335 I r8 J 159 I -16 J 659 The likelihood to use knowledge 75% 85% 71% 75% 60% 841Yo 66% 74% 78% 86% 68% - 69% 80% and skills acquired from the training ds' 10% 12% 7% N/A N/A N/A 12% *ds = difference between the results ofFYOO and FY99. Unless indicated with a mmus sign, all difference scores show a positive change in favor ofFYOO. Cross-Network Programs Total Cross- All Bank Cross-Networks Networks DEC I FPS I QAG I Legal I HR FY99 I FYOO FY99 I FYOO I FY99 I FYOO I FY99 I FYOO I FY99 I FYOO I FY99 I FYOO FY99 I FYOO # ofRespondents 1015 I 1989 12 I 23 I 57 I 144 I I 11 I 52 I I 43 I 164 I 178 The likelihood to use knowledge 750/. 85% 75% 87% 72% 79% - 100% 78% - 84% - 77% 89% and skills acquired from the training ds' 10% 11% 7% N/A N/A N/A 12% *ds - difference between the results ofFYOO and FY99. Unless indicated With a minus sign, all difference scores show a positive change infavor ofFYOO. Functional and Foundation SkiIlslWBISD Programs Information ACS Orientation Communication Technology Leadership HUB Total WBISD FY99 I FYOO FY99 I FYOO FY99 I FYOO FY99 I FYOO FY99 I FYOO FY99 I FYOO FY99 IFYOO # ofRespondents 135 I 98 I 167 I 230 I 477 I 16 I 128 135 I 1116 The likelihood to use knowledge and skills 92% 97% - 81% 94% - 86% 100% - 87% 91% acquired from the training ds' 5% N/A N/A N/A N/A N/A N/A *ds - difference between the results of FYOO and FY99. Unless indicated With a minus sign, all difference scores show a positive change in favor of FYOO. 12 Bank StaffLearning Agenda. FYOO Midyear Evaluation The course design and delivery variables noted in Tables I and J below have an empirical linkage with the sustainability or longevity of learning acquired. The evidence shows that for FYOO and on a Bank-wide basis, 81 % of the participants rated the courses as good or excellent in attributes that enhance learning sustainability. This result is a marked improvement from FY99 when the ratings for these variables was 70%. The results also show that the programs with the lowest ratings are HD (67%), Orientation (68%), FPS (73%), and FPSI (77%). The qualitative analysis of comments and recommendations indicates that participants would like their managers to provide time for them to practice and master new knowledge and skills. They also want moral support in the form of trust, encouragement, and incentives to implement new ideas. The most desired support systems to enhance performance improvement are noted to include help-desks, hotlines, clinics, learning groups, access to quality web-sites and access to experts and resource persons. WBIES is currently involved in assessing such factors as part of the Level 3 Evaluation of a selected number of courses. It is also exploring the use of a Bank-wide survey to assess management support in transferring learning acquired and the value of support systems such as web-sites and help- desks. The information from these two sources will be used in the end-ofyear report. Table I: Network Programs Percentage ofRespondents Providing Ratings of 4 or 5 on Three Factors Assessing "Ensuring Sustainability" Total All Bank Network Networks HD I PREM I FPSI I OCS I ESSD FY99 I FYOO ~I~I~I~I~I~I~I~I~I~ FY99 1 FYOO -- # ojRespondents 1015 I 1989 -13 I r I 102 I -1-1 I I 115 J 335 I rs I 159 I -16 I 1116 Application to 77% 83% 70% 83% 62% 82% 64'Y. 73% 83% 83% 63% - 72% 82% Bank work ds' 6% 13% 20% 9% 0 N/A 10% Use of ease studies and group 65% 80% 72% 68% 38% 89% 48% 74% 64% 80% 58% - 66% 84% problem solving exercises ds· 15°/. -40;0 51'Yo 26% 16% N/A 18% Materials for follow-up learning 69% 81% 69% 50% 53% 80% 52% 84% 78% 80% 45% - 71% 86% or job application ds· 12% -HI% 27'Y. 32% 2% N/A IS'Y. Total 70% I 81% 70% I 67% 51% I 84% 55% I 77% 75% 1 81% 55% I - 70% I 84% ds' 11% ..3% 33% 22% 6% N/A 14% *ds - difference between the results 0/FYOO and FY99. Unless mdlcated with a mmus sign. all difference scores show a positive change in/avor o/FYOO. 13 Bank StaffLearning Agenda, FYOO Midyear Evaluation Table J: Cross-NetworkIWBISD Supported Programs Percentage of Respondents Providing Ratings of 4 or 5 on Three Variables Assessing "Ensuring Sustainability" All Cross- All Bank Cross-Network Networks DEC I FPS I QAG I Legal I HR FY99 I FYOO FY99 I FYOOIFY99 IFYOO I FY99 I FYOO IFY99 I FYOO IFY99 IFYOO FY99 I FYOO # ofRe.lpondellls 1015 I 1989 12 I 23 I 5- I 1-1-1I I /I I 52 I I -13 I 16-1 I r8 Application to 77% 83'Y. 92% 96% 84% 82% - 80% 84'Y. - 60% - 80% 86% Bank work ds' 6% 4% -2% N/A N/A N/A 6'Y. Use of case studies and group 65% 80% 67% 91% 69% 66% - 91% 74% - 32% - 61% 83% problem solving exercises ds' 15% 24% -3% N/A N/A N/A 22% Materials for follow-up learning 69% 81% 83% 86% 71% 71% - 100% 96%, - 51% - 75% 86% or job application ds' 12% 3% 0 N/A N/A N/A 1l'Y. Total 70% I 81% 81% I 91% 75% I 73% - I 90% 85% I - 48% I - 72% I 85% ds' 11% 10% -2'Yo N/A N/A N/A 13% *ds = difference between the results ofFYOO and FY99. Unless indicated with a minus sign, all difference scores show a positive change in favor ofFYOO. Table K: Functional and Foundation SkillslWBISD Programs Percentage of Respondents Providing Ratings of 4 or 5 on Three Variables Assessing "Ensuring Sustainability" Information ACS Orientation Communication Technology Leadership HUB Total FY99 I FYOO FY99 I FYOO FY99 I FYOO FY99 I FYOO FY99 I FYOO FY99 IFYOO FY99 I FYOO # ofRespondellIs 135 I 98 I 167 I 230 I 477 I 16 I 128 135 I 1116 Application to 92% 94% - - - 88% - 84% - 87% - 91% - 88% Bank work ds* 20;' N/A N/A N/A N/A N/A N/A Use of case studies and 94% 91% - 68% - 90% - 80% - 83% - 94% - 84% group problem solving exercises ds* -3'Yo N/A Materials for follow-up 87% 85% - - - 84% - 86% - 85% - 85% - 85% learning or job application ds* -2% N/A N/A N/A N/A N/A N/A Total 91% 90% - I 68% - I 87% - 83% - I 84% - I 90% - I 85% ds' -1'Y. N/A N/A N/A N/A N/A N/A *ds - difference between the results ofFYOO and FY99. Unless indicated with a minus sign, all difference scores show a positive change in favor ofFYOO. 14 Bank StaffLearning Agenda, FYOO Midyear Evaluation VI CONCLUSIONS, INTERPRETATIONS, AND RECOMMENDATIONS Three major results ofthe study should be highlighted: 1. Improvements in performance from FY99. 2. Reduction in variation among Networks. 3. Major weaknesses still remaining among most programs in the "depth of content coverage" and in "the pacing ofthe various course sessions". A major contributor to the observed results is the remarkable improvement III the performance of PREM, FPSI and HD. How does one explain the observed trend in the results? While it is true that the numbers included in the study for the Network are not large enough to make definitive statements about the trend of performance, credit needs to be given to the efforts that have been made by these Networks in FYOO to scale down on their number of courses and to improve the quality ofthe structured classroom training. The OCS and the Cross-Network programs show an increase in performance which although less dramatic, is perhaps more stable and gives more valid information on the rate of improvement expected oftraining programs that have moved beyond the early mechanical stage of development to a more advanced stage characterized by minor adjustments and complementary pilots. Irrespective of the calculated rate of increases in performance, it is evident that the observed trend of improved performance is likely to be maintained when concerted efforts are directed at improving four key design features: the pacing of the various sessions of the courses; the depth of coverage; the application of concepts and principles to Bank work; and the use of case studies and group exercises for interactive learning. The FYOO evaluation included a significant number of WBISD programs designed to develop functional and foundation skills. The ratings for the program are primarily at or above the Bank benchmark, with the exception of the newly developed program, Orientation for New Staff. The high quality of these WBISD programs is linked to good needs assessments, planning, and the use of resident training experts in design and delivery, and is complemented with the use of outside consulting companies with expertise both in content, design and delivery. 15 Bank StaffLearning Agenda, FYOO Midyear Evaluation VII GOOD PRACTICES As one basis for guiding action in the areas of the Bank staff training program that need improvement, Table 5 in the Annex provides a list of all the courses included in this study together with information on their performance relative to the 85% benchmark on all the variables of the study. The table is significant in highlighting courses that consistently manifest high performance on most or all of the variables. Of significance for improving the training in the rest of FYOO are 15 training courses that have high scores (i.e. ratings of 85% and above) on the variables noted above to represent the areas of weakness in the Bank's training program. These areas are: the pacing of the various sessions; the depth of coverage; the application of concepts and principles to Bank work; and the use of case studies and group exercises for interactive learning. The task managers for these courses are being requested to share information about the practices they apply in the design, delivery and overall management of their training courses. WBIES is synthesizing this information and will produce precis highlighting "What Works, Why and How?" 16 Bank StaffLearning Agenda. FrOO Midyear Evaluation VIII NEXT STEPS IN EVALUATION JANUARY - JUNE 2000 The following outlines the continuation of the evaluation activities during the second half of FYOO. Evaluation of Formal Classroom Training 1. Levell Evaluation · Continue to provide Task Teams and Learning Representatives individual evaluation reports of courses within five to ten days after submission to WBIES. · Encourage training managers to complete the Course Description Form and Participant Demographic Profile to allow WBIES to conduct differentiated analysis by course type, population of participants and knowledge structure. · Coordinate with other units to establish an accessible database for monitoring information on courses planned actual deliveries and course types. · Pilot the use of electronic system (Cold Fusion) for questionnaire completion and submission. This process should remove the extensive amount of time spent by WBIES in cleaning up questionnaires that are incorrectly completed. 2. Level 2 Evaluation · Increase the number of courses with Level 2 Evaluation from 20 to 30. · Continue technical work in enhancing test validity and reliability. · Pilot the use ofan electronic system for Level 2 testing. 3. Level 3 Evaluation · Develop a framework and design for assessment and evaluation. · Develop an electronic system for questionnaire completion and submission (starting with the HUB training). · Develop an integrated system for using questionnaires, interviews, and product analysis in Level 3 Evaluation. · Develop a system for providing individual reports and for synthesis across reports. · Pilot test via a small scale survey the use of knowledge acquired on the job and the value of performance enhancement systems - managers, web-sites, hotlines, help-desks. etc. This will be done to complement the 5-10 cases studies of Level 3 Evaluations. 4. Good Practice Notes · Complete good practice notes on, "What Works, Why, and How?" with the first series appearing in March 2000. 17