WBI Evaluation Briefs REPORTING ON CLIENT AND STAFF LEARNING PROGRAMS Marlaine E. Lockheed, Manager WBI Evaluation Group September 2002 Annual Review of the Quality of Formal World Bank Staff Learning, FY02 The Challenge · Using their new skills and knowledge, staff will motivate Staff learning is increasingly important in the World Bank and institutional changes, enabling the Bank to better achieve its accounts for over 5 percent of the total administrative budget. development objectives and corporate priorities. This past fiscal year (FY02), stakeholder groups received · Staff will adopt and promote a culture of "ongoing learning." substantial resources ($68 million) to sponsor the delivery of hundreds of course offerings to Bank staff.1 Providing individuals Evaluation Method and Instrument with opportunities to acquire knowledge and skills, while ensuring The stakeholders nominated a non-random sample of courses to the quality of learning events, continues to be a challenge for the be evaluated by the WBI Evaluation Group in fiscal year 2002. Bank. The focus of this brief is on the latter: that is, how Bank Although the data were collected for all course participants, this learning programs fared according to training participants.2 study focuses on the responses of Bank Staff because they are the primary target for the formal staff learning programs. World Bank Intervention The World Bank provides learning opportunities to Bank staff to A standard evaluation form was used to collect self-report data build knowledge and capacity crucial to the Bank's mission of from the 1,387 Bank staff enrolled in 126 courses. Respondents development and poverty reduction. The learning agenda for Bank rated the quality of learning programs on eight dimensions. The Staff is composed of a variety of interrelated and mutually items measured opinions on the specific aspects of the training reinforcing programs designed to develop staff's depth of courses, as well as more comprehensive attitudes regarding the knowledge, skills and strategies in the technical, core business, overall value to participants. The Bank's goal is for 85 percent relational and management domains of the Bank's work. of participants to rate these items as a four or higher on a five- point response scale, where a five indicates the maximum Underlying Assumptions positive rating and a one is the extreme negative rating. In addition, the survey included two questions permitting The assumptions underlying formal staff learning are as follows:3 benchmarking to American Society of Training Development · Staff will implement the knowledge and skills acquired from (ASTD) standards.4 training to enhance "on-the-job" performance. Factor analysis revealed a single quality dimension underlying · Staff will receive the support of managers and peers in ten various items designed to measure course quality.5 Based on implementing their newly acquired knowledge and skills. this factor score, several regression models were estimated to 1. The staff learning under analysis was provided by Networks, Regions, WBI, Cross-Networks, and other stakeholder groups. 2. For the complete report, please see Fumika Ouchi and Violaine Le Rouzic (2002). "Annual Review of the Quality of Formal World Bank Staff Learning, FY02," Evaluation Group, World Bank Institute, October 2002. 3. Assumptions are based on "The Impact of Formal Training Programs for World Bank Staff, FY2001," WBI Evaluation Brief, February 2002. 4. "The 2001 ASTD Learning Outcome Report" conveys statistics on education and training programs for more than three thousand organizations. 5. The questions include: 1) To what extent did the training fulfill your learning needs?; 2) To what extent did the training achieve its announced objectives?; 3) How would you rate the choice of training content or subject matter?; 4) How would you rate the logic in the sequence of the content?; 5) How would you rate W O R L D B A N K I N S T I T U T E Promoting knowledge and learning for a better world assess the effects of individual and course Evaluation Results Bank-wide. Bank staff from theAfrica characteristics on participants' ratings of 1. The majority of evaluated courses Region were over-represented and the learning offerings. were professional and technical. those from IFC were underrepresented. Specifically, 53 percent were 3. Staff in grade levels GE and higher Participants and professional and technical, 17 percent were more likely to attend Response Rate concentrated on Bank operations, 16 professional and technical courses, Three thousand and sixteen people percent covered behavioral and social whereas staff in grades GA-GD participated in the FY02 training courses skills, 13 percent addressed participated primarily in behavioral evaluated by WBIEG using the "Level 1 information and technology and 1 and social skills courses. Forty-six Evaluation Questionnaire for Formal percent dealt with trust funds. percent of staff in GA-GD attended Training" approved by the Learning Board. 2. Respondents represented a variety of behavioral and social skills courses, Among them, 2,106 responded to the form, Bank groups including all Regions while 75 percent of staff in GH and for a response rate of 70 percent. One and Networks (Figure 1). However, higher attended professional and thousand three hundred and eighty-seven 65 percent of respondents worked in technical courses (Figure 2). respondents indicated that they were World Regional Vice Presidential Units, 4. The majority of participants (nearly Bank staff.6 compared to only 42 percent of staff 60 percent) selected learning Figure 1. Primary Area of Work inWorld Bank Percent of respondents and Bank staff 30 Respondents 20 Bank staff 10 0 AFR ECA EAP SAR Other LCR ISN MNA HD PREM ESD DEC WBI OPCS IFC PSI LEG FSE HRS MIGA Figure 2. Percent of Staff Enrolled in DifferentTypes of Courses, by Grade-level Percent of respondents 80 75 66 60 51 46 43 40 27 26 27 19 21 19 20 7 9 11 10 7 7 4 7 7 2 3 3 1 1 0 GA-GD GE GF GG GH+ Grade level Bank operations Behavioral and social skills Information technology Professional and technology Trust funds the materials used during the training?; 6) How would you rate the overall quality of the training?; 7) How would you rate the overall usefulness of the training?; 8) How would you rate your overall level of satisfaction with the training?; 9) My knowledge/skills increased as a result of this training; and 10) The knowledge/skills gained through this training are directly applicable to my job. 6. The remaining respondents included: short-term temporaries, consultants, individuals who do not work for the World Bank and those who did not indicate affiliation. Page 2 WBI Evaluation Briefs--September 2002 W O R L D B A N K I N S T I T U T E Promoting knowledge and learning for a better world programs primarily to enhance Figure 3. Percent of Respondents Rating Courses 4 or 5 onVarious their "performance in a current or Dimensions planned assignment." Other main Choice of content 88.7 motivators were "professional interest and growth" (34 percent) and Overall quality 88.6 "networking" (4 percent). Only 37 percent of respondents reported taking Announced objectives 87.4 a course because it was included in Overall usefulness 86.1 their Development Action Plan (DAP). Logic in sequence 86.1 5. The overall quality of formal Overall satisfaction 84.7 learning events exceeds the Bank's benchmark. More than 85 percent of Material used 84.1 respondents evaluated the overall quality (88 percent) and usefulness Learning needs met 82.9 (86 percent) of the learning offerings 0 20 40 60 80 100 positively. Moreover, the surveys indicate that the courses not only achieved their announced objectives, Figure 4. Participants' Ratings of Knowledge/Skill Increased, by Course but also delivered high quality content Content in a logical order (Figure 3). Behavioral and social skills 4.50 6. Courses met industry standards for increasing knowledge and skills and Information and technology 4.40 for being directly applicable to Bank operations 4.34 participants' jobs. On a five-point ASTD 4.21 scale, Bank staff 's self-assessments of how much they learned averaged 4.27 Professional and technical 4.14 for knowledge/skills and 4.15 for Trust funds 3.89 training utility. These results are comparable with average scores (4.21 1 2 3 4 5 for both items) reported in ASTD Mean ratings surveys across 3000 institutions. 7. Courses in behavioral and social Figure 5. Participants' Assessments of the Extent Learning was Directly skills were more successful than Applicable to Job, by Course Content professional and technical courses in enhancing trainees' perceived Behavioral and social skills 4.48 knowledge and skills in areas ASTD 4.21 relevant to their jobs. Figures 4 and 5 depict self-reported ratings of Bank operations 4.20 improvements in knowledge and skills Trust funds 4.12 as well as respondents'opinions of how Information and technology 4.05 applicable such expertise is to their Bank duties. Professional and technical 4.01 8. Several participant characteristics 1 2 3 4 5 were correlated with course quality Mean ratings WBI Evaluation Briefs--September 2002 Page 3 W O R L D B A N K I N S T I T U T E Promoting knowledge and learning for a better world ratings. Staff who work in the Network Anchors and other for the design of future learning programs, it is important to note Bank offices7 rated courses more positively than staff who that the generalizability of the findings are constrained to the data work primarily for Regions. On average, staff in grades GA- collected and are thus not necessarily applicable to courses GD rated courses more positively than did staff in grades GE sponsored by several of the stakeholder groups for which the and above. Learning that was agreed upon in participants' Evaluation Group was unable to obtain data during this cycle. DAPs was rated more favorably. Overall, based on this analysis, the report recommends that: 9. Several course characteristics were also associated with · Standard Learning Board Level-1 Evaluations should be overall respondent ratings. Longer courses were evaluated required for a subset of training offerings across all Staff more positively than shorter courses. Learning offerings Learning Courses and Regions. This would provide a more mostly designed and/or delivered by external organizations representative sample, and consequently, more generalizable were rated more favorably than courses designed and/or results. delivered by the World Bank. Respondents evaluated training · The Learning Board should take into consideration the specific that took place in headquarters more positively than they rated implications of the study regarding the perceived quality of courses offered in field offices, or in off-site locations in lengthier courses and learning programs that were mostly Washington, D.C. According to open-ended responses by designed and/or delivered by external providers. participants who attended some of the most highly rated courses, the features that "worked best" were participatory · Specific attention will need to be paid to improving the quality approaches that resulted in substantial class discussion, and relevance of professional and technical courses. practical examples and materials, and knowledgeable instructors who maintained an engaging manner. 10. Numerous course characteristics were not associated with perceived quality. Neither class size, course maturity AboutWBI (training offered repeatedly), nor grade-level diversity among The World Bank Institute (WBI) works to build the capacity of its client participants, were related to perceptions of course quality. countries for poverty reduction and sustainable development. It supports the Further, it made no difference whether courses were delivered World Bank's learning and knowledge agenda by delivering learning programs, providing policy services, facilitating action programs, supporting networks of via Distance Learning or as part of a Learning Week. professionals, and creating and managing initiatives for knowledge sharing. 11. Course content and stakeholder sponsorship were related The WBI Evaluation Group (WBIEG) works with the Institute's program to perceived overall course quality. leaders and with sector managers Bank-wide to prepare, process, and report evaluation results for staff, client, and joint learning events. WBIEG also offers · Courses focusing on Bank operations, behavioral and distance learning and face-to-face training in program evaluation. WBIEG Evaluation Briefs report on the evaluation results, lessons learned, and impact social skills, and information and technology received more of the Institute's major offerings. positive evaluations than courses that concentrated on Contacts professional and technical skills. WBI Evaluation Briefs · Overall, learning offerings sponsored byACS,AFR, DEC, Cristina Ling Chard, Evaluation Analyst 202.473.2310 or Cchard@worldbank.org FPS, ISN and WBISD elicited considerably more favorable Learning Board Level-1 Evaluations ratings than courses sponsored by other groups. Cristina Ling Chard, Evaluation Analyst 202.473.2310 or Cchard@worldbank.org Implications Visit our web site for more information on the Evaluation group of WBI This study demonstrates that respondents who participated in these and electronic copies of all Evaluation Briefs: http://www.worldbank.org/wbi/evaluation/index.html learning programs considered them to be useful and of high quality. Further information gleaned from the surveys suggests that courses could be improved through greater use of outsourcing and increased instructional time. While these findings are useful 7. DEC, HRS, LEG, WBI, IFC, and other Bank offices except Regional VP Units and Networks. Page 4 WBI Evaluation Briefs--September 2002