ANNUAL REVIEW OF THE QUALITY OF FORMAL WORLD BANK STAFF LEARNING FY02 Fumika Ouchi Violaine Le Rouzic WBI Evaluation Studies No. EG03-56 The World Bank Institute The World Bank Washington, D.C. September, 2002 ACKNOWLEDGEMENTS This report was prepared for the Learning Board World Bank Institute under the overall guidance of Marlaine Lockheed, Manager, Evaluation Group. The team was led by Violaine Le Rouzic. The authors gratefully acknowledge the contribution of Aynalem Bogale, Paul Date, and Katherine O'Connor who collected, analyzed and reported the data of the courses studied, as well as the technical comments of Nidhi Khattri and Jaime Quizon. WBI Evaluation Studies are produced by the WBI Evaluation Group (WBIEG) to report evaluation results for staff, client, and joint learning events. An objective of the studies is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at http://www.worldbank.org/wbi/evaluation/puball.htm. Vice President, WBI Ms. Frannie Léautier Chief Learning Officer Ms. Phyllis Pomerantz Manager, WBI Evaluation Group Ms. Marlaine Lockheed Task Manager Ms. Violaine Le Rouzic -ii- CONTENTS 1 Introduction 1 2 Which Courses were Evaluated? 3 3 Which Bank Staff Participated in Learning Programs? 5 4 Why Did Staff Participate in Learning Programs? 9 5 How Did Staff View the Quality of Staff Learning Programs? 11 Overall Perception of Course Quality 11 Perceived Gain in Knowledge and Skills Compared with Benchmark 12 Determinants of Perceived Course Quality 14 6 Summary and Conclusions 17 Annexes Annex 1. List of 146 FY02 Formal Staff Learning Courses evaluated by WBIEG 19 Annex 2. Distribution of Respondents by Affiliation and Grade 23 Annex 3. Main Reason for Taking Courses by Location of Staff, Affiliation and Grade; Respondent Reaction to the Balance of Time in the Courses 25 Annex 4. Factor Score Loadings 27 Annex 5-A. Logistic Regression Results of Benchmarking Questions: Analysis Using Stakeholder Group 29 Annex 5-B. Logistic Regression Results of Benchmarking Questions: Analysis Using Types of Course Content 31 Annex 6. Regression Results on Overall Perceived Quality 33 Annex 7. Description of Regression Variables 35 Annex 8. FY02 Standard Level 1 Evaluation Questionnaire for Formal Training 37 Figures 1. Distribution of Evaluations Across Stakeholder Group....................................................... 3 2. Type of Courses.................................................................................................................... 4 -iii- 3. Duration of Courses .............................................................................................................. 4 4. Primary Area of Work in World Bank Group ........................................................................ 5 5. Location of Work .................................................................................................................. 6 6. Grade Level........................................................................................................................... 6 7. Type of Courses Enrolled by Respondents......................................................................... 7 8. Main Reason for Course Taking............................................................................................. 9 9. Courses Agreed Upon in DAP?.......................................................................................... 10 10. Percent of Respondents Rating Courses 4 or 5................................................................ 12 11. Perceived Kowledge/Skill Increased, by Stakeholder Group......................................... 13 12. Perceived Knowledge/Skill Increased, by Course Content ............................................ 13 13. Knowledge/Skill Directly Applicable to Job, by Stakeholders...................................... 14 14. Knowledge/Skill Directly Applicable to Job, by Course Content.................................. 14 -iv- EXECUTIVE SUMMARY Staff learning is increasingly important in the World Bank, accounting for 5.1 percent of the total administrative budget. Staff learning is provided by Network, Regional and Cross-Network stakeholder groups. In FY02, these stakeholder groups received earmarked resources for training and delivered hundreds of course offerings to Bank staff. This report synthesizes the results from end-of- course evaluations provided by participants in 126 courses sampled across 17 stakeholder groups. It addresses three questions: 1. Which Bank staff participate in learning programs and why do they do so? 2. What is the quality of staff learning programs? 3. What course characteristics contribute to higher quality courses? Staff from all parts of the Bank, working at all levels and locations, participate in staff learning. Compared with the Bank as a whole, a higher share of staff learning participants come from Regional VPUs, work in field offices and work at levels GE-GF. The main reason given for participation is to enhance job performance, and over one-third of participants take courses called for in their Development Action Plan (DAP). However, another 40 percent take courses not mentioned in their DAPs. Overall, staffs view the quality of staff learning courses very positively, with the average course rating reaching that of the average for 3000 organizations internationally. Three characteristics of course participants were related to higher ratings of course quality: grade level, alignment with DAP, and work location. Staff in grades GA-GD rated courses more positively than did staff in higher grades. Staff who took courses that were agreed upon in their DAPs rated courses more favorably than did those who took courses not in their DAPs. Staff working in non-Regional VPUs rated courses more favorably than those working in Regional VPUs. Six course characteristics were related to higher ratings of course quality: stakeholder sponsorship, course duration, external course design and delivery, course location, share of Bank staff, and content type. Courses sponsored by ACS, AFR and WBISD were more highly rated than those sponsored by others. Longer courses, those designed and/or delivered by external organizations, those that were offered at the Bank Headquarters in Washington, DC, those in which a higher share of Bank staff were enrolled, and those that dealt with behavioral and social skills were more highly rated. -v- Five other course characteristics were unrelated to perceived course quality: course maturity, class size, level of diversity with respect to grade level, integration into a Learning Week, and delivery through Distance Learning. These findings have implications for course improvement through greater use of outsourcing and increased instructional times, and implications for improved efficiency through distance learning, wider target audiences for courses, and independence from a Learning Week. -vi- 1 INTRODUCTION 1.1 Staff learning is increasingly important in the World Bank, accounting for 5.1 percent of the total administrative budget. Staff learning is provided by Network, Regional and Cross-Network stakeholder groups. In FY02, the Learning Board mandated that 140 formal staff learning courses be evaluated at Level 1 by the WBI Evaluation Group; the distribution of the 146 courses eventually evaluated across 20 stakeholder groups is reported in Annex 1. Ninety percent of the learning events were offered in the second half of FY02. This report presents a summary of findings from 126 course offerings provided by 17 stakeholder groups that successfully utilized the standard evaluation form (Annex 8) and whose data were recorded by WBIEG. 1.2 Of 3,016 total participants in the 1281 FY02 courses having complete evaluations, 2,106 responded to the "Level 1 Evaluation Questionnaire for Formal Training" approved by the Learning Board (a response rate of 70 percent). Both Bank staff and non-Bank staff participated in learning programs (Basic statistics for all respondents are described in Box 1). The present study analyses the responses of the 1,387 Bank staff only, as they are the primary targets for formal staff learning programs. Box 1. Basic Statistics for Respondents who Participated in Staff Learning Events, FY02 All respondents (n=2,106): Staff 66%; STT/STC 8%; Other 1%, Client 20%; No answer 5% Among respondents who work for the World Bank (n = 1,639): HQ/Field offices HQ staff 64%; Field office staff 33%; No answer 3% Affiliation: Region 62%; Network 22%; Other 16%; No answer 1% Primary Area of Work AFR 24%; EAP 8%; ECA 12%; LCR 6%; MNA 4%; SAR 7%; ESD 3%; FSE 0.7%; HD 4%; ISN 6%; OPCS 2%; PREM 5%; PSI 2%; DEC 2%; HRS 0.4%; LEG 2%; WBI 3%; IFC 2%; MIGA 0%; Other 7%; No answer 1% Appointment Staff 85%; STT/STC 11%; Other 1%; No answer 4% ACS/Higher level ACS 28%; Higher level staff 67%; Other 1%; No answer 4% Grade GA-GD 27%; GE 13%; GF 21%; GG 18%; GH+ 4%; STT 0.7%; STC 10%; Other 0.5%; unspecified (e.g. secondment, etc) 2%; No answer 5% 1Two courses were delivered exclusively for clients and were excluded from the study. -1- 2 WHICH COURSES WERE EVALUATED? 2.1 Initially, the Learning Board allocated the share of 140 evaluations among 24 stakeholder groups: ten per Network or WBI; five per Region, ACS or DEC; and two per other groups (i.e., other Cross-Networks, RMC or QAG). As implemented, however, eight stakeholders received substantially more evaluations than planned and eight received substantially fewer, as a consequence of having nominated fewer for evaluation. Those who received none (LCR, MNA, SAR and QAG) did not fully nominate courses to be evaluated for which they were uniquely the sponsor or did not provide adequate and/or timely course information2. 146 courses were evaluated (Figure 1), although 18 used irregular forms whose results were not included in this report. Figure 1. Distribution of Evaluations Across Stakeholder Group 2.2 Among the 126 courses: (a) Sixty-seven were professional and technical courses, 21 were courses on Bank operations, 20 were on behavioral and social skills, 16 were on information and technology, and two were courses on Trust Funds (Figure 2). (b) Most (102) were designed by the World Bank Group. 2Two "Hub training courses" for MNA and SAR were sponsored by WBI and included in the study. -3- (c) About half (62) were delivered more than twice, while 20 were delivered for the second time, and 44 were delivered for the first time. (d) Twenty-six were delivered by organizations other than the World Bank Group. Figure 2. Type of Courses 2.3 Course duration varied from a half day to 13 days (Figure 3). Courses lasted from one day to 8 days for Bank operations, from half day to 13 days for behavioral and social skills, from one day to 4 days for information and technology, from half day to 5 days for professional and technical courses, and half day for Trust Fund courses. Figure 3. Duration of Courses -4- 3 WHICH BANK STAFF PARTICIPATED IN LEARNING PROGRAMS? 3.1 Evaluation respondents reflected all Regions, Network Anchors and Cross-Networks, but there were differences3: (a) Sixty-five percent of respondents worked in Regional Vice Presidential Units4, compared with 42 percent of staff, Bankwide (Figure 4). A higher share of respondents worked in Africa (twice as many), Europe and Central Asia, East Asia and the Pacific and South Asia, compared with Bank staff as a whole, and a much lower share worked in IFC. Figure 4. Primary Area of Work in World Bank Group (b) Thirty-six percent of the respondents worked in field offices, compared to 30.4 percent of staff, Bankwide (Figure 5). 3Based on the current distribution of World Bank staff as a whole (n=11,132) (Source: WBICA, June 2002) . 4Regional VP Units: AFR, EAP, ECA, LCR, MNA and SAR; Networks: ESD, FSE, HD, ISN, OPCS, PREM, PSI; Other: DEC, HRS, LEG, WBI, IFC, MIGA, and other offices. -5- Figure 5. Location of Work (c) Fewer participants were staff in grades GA-GD and GH or above (31 percent and 5 percent, respectively), compared with the Bank overall (34.4 percent and 13.4 percent, respectively). More participants were in grades GE-GF (40 percent), compared to 27 percent, Bank wide (Figure 6 and Annex 2). Figure 6. Grade Level 3.2 Staff in grades GE and higher levels were more likely to attend professional and technical courses, whereas staff in grades GA-GD primarily attended behavioral and social skills courses (Figure 7). -6- Figure 7. Type of Courses Enrolled by Respondents -7- 4 WHY DID STAFF PARTICIPATE IN LEARNING PROGRAMS? 4.1 The evaluation asked staff to indicate their main reason for taking the course. Most staff (nearly 60 percent) reported participating in learning programs to enhance their performance in a current or planned assignment (Figure 8); in addition: (a) About as many respondents (37 percent) took courses that were called for in their Development Action Plan (DAP), as took courses that were not called for in their DAP (41 percent) (Figure 9). (b) The reason for course taking did not vary by staff location (between HQ staff and field office staff), by affiliation (between Region, Network and other staff), or by grade level (GA-GD, GE, GF, GG, GH and above) (See Annex 3 Table 1). Figure 8. Main Reason for Course Taking -9- Figure 9. Courses Agreed Upon in DAP? -10- 5 HOW DID STAFF VIEW THE QUALITY OF STAFF LEARNING PROGRAMS? 5.1 The respondents rated the quality of the learning programs from low to high on a five-point scale in response to eight questions5. These questions were examined in terms of the percentage of higher ratings of 4 or 5 (the Bank target is 85 percent for a question). Respondents were also asked five questions regarding the use of time in the course. Two other questions allowed comparison with over 3000 institutions worldwide that participated in the ASTD benchmarking program. Although it was anticipated that the survey questions would assess different aspects of the learning program, a factor analysis identified a single factor only (Annex 4 presents the factor loadings). As a consequence, this section reports results for the overall rating of all questions, and detailed analysis of the benchmarking questions and the single factor score. Overall Perception of Course Quality 5.2 Overall, 85 percent of all participants were satisfied with the learning offerings, with the majority of respondents rating the overall quality (88 percent) and usefulness (86 percent) of the learning offerings 4 or 5 on a five-point scale (Figure 10). Respondents reported that the courses achieved their announced objectives, chose high quality training content or subject matter, and were logically sequenced. Overall, over 80 percent of respondents found the courses balanced with respect to attention to theoretical content, time for instructors' presentations and time for participation, although seventeen percent of respondents found the courses paid insufficient attention to practical content and sixteen percent found the pace of courses too fast (Annex 3 Table 2). 5The questions were: 1) To what extent did the training fulfill your learning needs?; 2) To what extent did the training achieve its announced objectives?; 3) How would you rate the choice of training content or subject matter?; 4) How would you rate the logic in the sequence of the content?; 5) How would you rate the materials used during the training?; 6) How would you rate the overall quality of the training?; 7) How would you rate the overall usefulness of the training?; 8) How would you rate your overall level of satisfaction with the training? -11- Figure 10. Percent of Respondents Rating Courses 4 or 5 Perceived Gain in Knowledge and Skills Compared with Benchmark 5.3 The course evaluations included two questions that assessed respondents' perceived knowledge/skill change and utility of training: (a) "My knowledge/skills increased as a result of this training" and (b) "The knowledge/skills gained through this training are directly applicable to my job." 5.4 On a five-point scale (from strongly disagree to strongly agree), the average score of all Bank staff was 4.27 for the first question and 4.15 for the second, compared with the average across 3000 institutions of 4.21 for both questions6. 5.5 Respondents in courses sponsored by ACS, WBISD, DEC, and AFR were more likely to report increase in knowledge/skills from the training, as compared with respondents from courses offered by the reference group (PREM)7 (Figure 11 and Annex 5-A). Respondents in courses sponsored by PSI and RMC were less likely to report increase. No standard Learning Board evaluations were received from FSE, ECA, LCR, MNA, SAR, EXT and QAG. 6The two questions were taken from "The 2001 ASTD Learning Outcome Report," a study on education and training programs of more than three thousand organizations. 7In the regression analyses, PREM was selected as the reference group because it had the largest number of cases. -12- Figure 11. Perceived Kowledge/Skill Increased, by Stakeholder Group 5.6 Respondents who participated in courses on behavioral and social skills were more likely to give a higher rating of 4 or 5 on a five-point scale on knowledge/skills change, compared to those who participated in professional and technical courses (Figure 12 and Annex 5-B). Those who participated in courses on Trust Funds were more likely to give lower ratings.8 Figure 12. Perceived Knowledge/Skill Increased, by Course Content 5.7 Respondents in courses sponsored by WBISD, ACS and AFR were more likely to give a higher rating of 4 or 5 on the course's applicability to their jobs than did respondents in the reference group (Figure 13 and Annex 5-A). 8 Subsequent to the analyses reported herein, 13 (10%) of the courses were recoded to the Bank operations content category: 5 behavioral and social, 6 professional and technical, and 2 trust fund courses. The overall effect in the results from recoding the data was substantively inconsequential. In the first logistic model presented in Annex 5-B, the impact of behavioral and social courses decreased from being marginally significant (p<.08) to no longer reaching statistical significance in predicting perceptions of knowledge gained. Conversely, in the second logistic model explaining opinions of training applicability to respondents' jobs, the effect of Bank operations increased in statistical significance (p<.04). In the OLS model presented in Annex 6, predicting perceived overall quality, there was no change. Therefore, taken as a whole recoding the course content for these 13 courses did not produce any new findings. A detailed account of the results is available upon request. -13- Figure 13. Knowledge/Skill Directly Applicable to Job, by Stakeholders 5.8 Respondents who participated in behavioral and social skills courses were more likely to give a higher rating of 4 or 5 on the course's applicability to their jobs, compared with those who participated in professional and technical courses (Figure 14 and Annex 5-B). Figure 14. Knowledge/Skill Directly Applicable to Job, by Course Content Determinants of Perceived Course Quality 5.9 A factor analysis was used to identify what underlying factors may have explained respondents' ratings of the questions on the quality of the learning programs. A single underlying factor was identified that represented conceptually the ten questions measured on a five-point scale. Based on this analysis, an average score of ten questions was computed for each respondent. Respondents were generally highly satisfied with the course quality; on average they awarded 4.27 on a five- point scale. Regression models were constructed to assess the effects of -14- various individual and course characteristics on these factor scores (See Annex 6). 5.10 Three characteristics of course participants were related to higher ratings of course quality: (a) Staff who worked in Network Anchors and those who worked in other Bank offices9 rated courses more positively than did those who worked in Regions. (b) Staff in grades GA-GD rated courses more positively than did staff in higher grades. (c) Staff who took courses that were agreed upon in their DAPs rated the courses more favorably than did those who took courses not included in their DAPs. 5.11 There was no significant difference between ratings given by HQ and field office staff. 5.12 Stakeholder sponsorship was related to respondent ratings of the overall quality of the course. Courses sponsored by ACS, AFR and WBISD, which generally dealt with Bank operations and behavioral and socials skills, received substantially higher ratings (about .75 of a standard deviation) than the reference group, PREM, which sponsored primarily professional and technical courses. Courses sponsored by DEC (professional and technical courses) by ISN (information and technology) and by FPS (Bank operations) were also rated higher (about 1/3 of a standard deviation) than courses sponsored by the reference stakeholder. 5.13 Six course characteristics were associated with higher ratings of quality: (a) Respondents rated longer courses more positively than shorter courses; each additional day was associated with .10 of a standard deviation higher rating. (b) Respondents rated courses designed and/or delivered by external organizations higher than they rated courses designed and/or delivered by the World Bank (about one-third of a standard deviation). (c) The higher the proportion of Bank participants in a course, the higher the ratings. (d) Respondents rated courses that took place in the World Bank Headquarters higher than they rated courses offered in field offices or in Washington, D.C. but outside of the Bank Headquarters (about one-third of a standard deviation). 9DEC, HRS, LEG, WBI, IFC, MIGA and other Bank offices except Regional VP Units and Networks. -15- (e) Respondents rated courses dealing with behavioral and social skills (about half of a standard deviation), Bank operations (about one third) and information technology (one quarter) higher than those dealing with the professional and technical skills. (f) In addition, the higher the response rate, the higher the overall ratings on courses, suggesting that better courses retained participants to the end of the course when the surveys were administered. 5.14 Five course characteristics were not associated with perceived quality: (a) Course maturity (repeated courses) (b) Class size (c) The level of grade level diversity among course participants (d) Courses delivered as part of a Learning Week (e) Courses delivered through Distance Learning 5.15 In addition, respondents in more highly rated10 courses identified features of the course that they considered "worked best": (a) Coverage: "Content of the course.", "Topics covered: this is one of the outstanding training I have attended" (b) Approaches: "Participatory approaches.", "Working in groups, sharing our ideas with the rest of the class." , "Share criticism and knowledge with other participants." , "Role plays." (c) Materials: "Practical examples.", "Using current working assignments in class." (d) Instructors: "The instructor's ability to engage the participants." "Instructors' energy levels plus obvious superior knowledge, experience and skill kept the whole thing going.", "The delivery from very knowledgeable facilitators who are very flexible and respectful, and have a good sense of humor." 10Three courses sponsored by ACS ("Building Your Skills for Working in a Team-based Environment" offered in December 2001, February and March 2002) and five courses sponsored by WBISD ("Organizing and Delivering Presentations," "Media Expertise Training," "Memo Writing Workshop," "Report Writing," and "Short Document Writing" offered in 2002). These courses had been repeated at least twice and had a duration of 2 or more days. -16- 6 SUMMARY AND CONCLUSIONS 6.1 In sum: (a) Staff learning programs offered by Regions, Network Anchors and Cross-Networks meet benchmark standards for relevance and usefulness, for both ACS and higher-level staff. (b) The courses are perceived as of high quality, with an appropriate balance between theory and practice. (c) In general, ACS staff rate the learning programs more favorably than do higher level staff, suggesting either that courses targeted toward ACS staff may be better designed and delivered or that ACS staff are more appreciative of staff learning opportunities. (d) Courses included in staff DAPs (that is, that have a clear business rationale) are rated higher than those not included in staff DAPs, suggesting that greater attention be paid to aligning courses with business needs. (e) Courses on behavioral and social skills, Bank operations, and information and technology are more likely to receive higher respondent ratings than professional and technical courses. (f) Longer courses are rated more highly than shorter courses, suggesting that instructional time is an important course design consideration. (g) Courses that are designed and/or delivered by other external organizations are rated higher than those that are designed and/or delivered by the World Bank, suggesting that more outsourcing be considered for staff learning. (h) Courses that took place in the World Bank Headquarters are rated higher than courses delivered in field offices or in Washington, D.C. but outside the Bank Headquarters, suggesting that participants value the opportunity to network while in HQ for courses. -17- (i) Courses whose designs included more participatory approaches and practical examples, and those taught by instructors who have good communication skills were viewed more positively. 6.2 At the same time, the lack of evaluations of courses sponsored by several stakeholder groups, most notably regional sponsors of staff learning in ECA, LCA, MNA and SAR, limit the generalizability of the findings. 6.3 We recommend that: (a) Standard Learning Board Level-1 evaluations be required for a sample of course offerings of all SLCs and Regions in FY03, with the selection of learning events delegated to WBIEG according to a stratified sampling frame agreed upon by the Learning Board. (b) To prevent inconsistencies between reporting of participant data and evaluation data when courses are jointly sponsored by WBI and other Bank stakeholders, jointly sponsored courses should be evaluated with the questionnaire corresponding to the initiator of the AIS, i.e., WBI form for WBI sponsored activities focused primarily on clients and Learning Board forms for courses sponsored by other stakeholders and focus primarily on staff. (c) The implications of the results are followed up by the Learning Board. -18- ANNEX 1 List of 146 FY02 Formal Staff Learning Courses evaluated by WBIEG Major Detailed Offering Title Start End Code Sponsor Sponsor Date Date Networks ACS Building Your Skills for Working in a Team-based 12/10/01 12/14/01 ACS212-05-015 Environment (Blended) Networks ACS Building Your Skills for Working in a Team-Based 02/11/02 02/15/02 ACS212-05-016 Environment (Blended) Networks ACS Building Your Skills for Working in a Team-Based 03/11/02 03/15/02 ACS212-05-017 Environment (Blended) Networks ACS Building Your Skills for Working in a Team Based 06/03/02 06/07/02 ACS212-05-018 Environment (Blended) Networks ACS Building Effective Presentations 06/25/02 06/25/02 ES#000-00-037 Networks ACS Quality at Entry and LOGFRAME 06/27/02 06/28/02 ES#000-00-036 Networks ESD Mainstreaming the Environment in PRSPs 01/24/02 01/24/02 ENV206-06-009 Networks ESD Designing Rural Poverty Reduction Strategies 02/05/02 02/05/02 RDA204-06-006 Networks ESD Introduction to Climate Change and Its Relevance 02/06/02 02/06/02 ENV206-07-010 to Bank Operations Networks ESD Public Expenditure Policy in the Rural Context 02/26/02 02/26/02 RDA204-05-005 Networks ESD Poverty and Environment Links 02/27/02 02/27/02 ENV206-08-011 Networks ESD Poverty Aspects of the Private Sector and Market 04/17/02 04/17/02 RDA-204-11-014 Development in Rural Areas Networks ESD Designing Rural Poverty Reduction Strategies 04/18/02 04/18/02 RDA204-06-012 Networks ESD Developing Agribusiness and Agro-Food Systems: 04/18/02 04/18/02 RDA204-12-015 The Impact on Poverty Networks ESD Rural Poverty Diagnostics 04/19/02 04/19/02 RDA204-04-013 Networks ESD A Workshop on OP 4.09 - The Safeguard Policy 05/15/02 05/15/02 SAF202-41-058 on Pest Management Networks FSE Distance Learning Course for Insurance 04/08/02 04/29/02 FIN210-03-009 Supervisors--Level 2 (the Philippines, Thailand, FIN210-03-011* and Vietnam) Networks FSE Curso de Aprendizaje a Distancia para 04/08/02 04/29/02 FIN210-02-006 Supervisores de Seguros--Nivel 2 (Argentina, FIN210-02-007 Bolivia and Nicaragua) FIN210-02-008* (Distance Learning Course for Insurance Supervisors--Level 2 LAC) Networks HD Quality at Entry 12/13/01 12/13/01 EDU220-04-004 Networks HD Effective Partnerships for Accelerating the 01/07/02 01/11/02 HNP206-25-025* Implementation of Reproductive Health Programs Networks HD Meeting the Education MDG: A New Policy and 03/04/02 03/04/02 EDU212-54-066 Financing Framework for Education for All (EFA) Networks HD Economic Analysis in the Education Sector 03/04/02 03/04/02 EDU221-06-010 Networks HD Textbooks Operational Guidelines and 03/04/02 03/05/02 EDU221-07-011 Procurement Day 1 Networks HD Textbooks Operational Guidelines and 03/04/02 03/05/02 EDU221-07-011 Procurement Day 2 Networks HD HD Cross-Sectoral: M&E Issues in the HD Sectors 03/04/02 03/04/02 SPL216-02-002 Networks HD Schooling Profiles: New Methodology to Estimate 03/05/02 03/05/02 EDU212-55-067 Schooling Coverage Networks HD HD Cross-Sectoral: Quality of Supervision (HD 03/05/02 03/05/02 SPL216-03-003 Week) Networks HD What's New in Targeting? From Formulas (proxy 03/05/02 03/05/02 SPL206-19-019 means testing) to Fellowship (community based) Networks HD Impact of HIV/AIDS on Education 03/05/02 03/05/02 EDU211-04-006 Networks HD Public Private Partnerships in Education 04/11/02 04/11/02 EDU221-13-017 Networks HD Public Expenditure Review (PER) Clinic: Social 04/29/02 05/03/02 SPL215-05-005 Sector Expenditure and Financing Reviews; HD - PRSP Training Networks HD Quality of ESW for the HD Sectors 05/14/02 05/14/02 EDU220-06-006 Networks HD Textbooks Operational Guidelines and 05/29/02 05/30/02 EDU221-07-020 Procurement Networks HD Assessing Policy Options for Teacher Training & 06/18/02 06/18/02 EDU221-19-025 Pay: A Two-part Hands-on Training Module Networks HD Protecting the Poor and User Fees for Health Care 06/26/02 06/26/02 SPL206-24-024 -19- Major Detailed Offering Title Start End Code Sponsor Sponsor Date Date Networks ISN XML- A comprehensive Hands-On Introduction 02/26/02 03/01/02 ES#000-00-005 Networks ISN HTML 4 - Level 1 03/01/02 03/01/02 ES#000-00-017 Networks ISN HTML 4 Level-2 03/06/02 03/06/02 ES#000-00-013 Networks ISN Dream weaver 4 - Level 2 03/12/02 03/12/02 ES#000-00-015 Networks ISN Photoshop 6 Level 1 03/14/02 03/15/02 ES#000-00-011 Networks ISN Java Programming 03/19/02 03/22/02 ES#000-00-014 Networks ISN XML - Level 1 03/22/02 03/22/02 ES#000-00-010 Networks ISN Enhancing Web Pages using JavaScript 03/26/02 03/26/02 ES#000-00-009 Networks ISN XML - Level 2 03/27/02 03/27/02 ES#000-00-008 Networks ISN HTML 4 - Level 2 04/03/02 04/03/02 ES#000-00-007 Networks ISN Photoshop 6.0 - Level 2 04/08/02 04/09/02 ES#000-00-018 Networks ISN Intermediate JavaScript 04/10/02 04/11/02 ES#000-00-019 Networks ISN XML - A Comprehensive Hands-On Introduction 05/14/02 05/17/02 ES#000-00-029 (Section B) Networks ISN XML - A Comprehensive Hands-On Introduction 05/14/02 05/17/02 ES#000-00-030 (Section A) Networks ISN Introduction to IT Project Management 06/03/02 06/04/02 ES#000-00-034 Networks OPCS Use of MS Project for Managing Projects 01/09/02 01/10/02 OPS216-26-017 Networks OPCS Basic Procurement 01/28/02 02/01/02 PRO201-02-019 Networks OPCS Basic Procurement 02/25/02 03/01/02 PRO201-02-035 Networks OPCS Selection of Consultants 03/04/02 03/06/02 PRO201-05-036* Networks OPCS Project Financial Management Module 1 03/05/02 03/06/02 OPS214-01-007 Networks OPCS Disbursement for FM Staff 03/06/02 03/06/02 OPS209-08-018 Networks OPCS Procurement for FM Staff 03/07/02 03/08/02 OPS209-07-017* Networks OPCS Public Sector Financial Management Seminar 03/18/02 03/19/02 OPS209-09-020 Networks OPCS Project Planning and Preparation 03/18/02 03/21/02 OPS216-06-094 Networks OPCS Client Capacity Building in Financial Management 03/21/02 03/21/02 OPS209-10-022* Networks OPCS Project Financial Management Module 1 03/26/02 03/27/02 OPS214-01-008 Networks OPCS Project Financial Management Module 2 03/28/02 03/29/02 OPS214-02-011 Networks OPCS Basic Procurement 04/08/02 04/12/02 PRO201-02-040 Networks OPCS Project Supervision and Implementation 05/13/02 05/16/02 OPS216-05-097 Networks OPCS Basic Procurement 06/10/02 06/14/02 PRO201-02-088 Networks OPCS Disbursement for Operational Staff GA-GD 06/10/02 06/11/02 OPS210-01-012 Networks OPCS Updates on International Accounting Standard 06/18/02 06/19/02 OPS208-01-014 Networks OPCS Séminaire sur les Approvisionnements, 06/24/02 06/28/02 ES#000-00-033* Décaissements et Gestion Financière Networks PREM Empirical Tools for Governance Analysis 06/17/02 06/18/02 GOV209-01-001 Networks PREM Economic Statistics 06/17/02 06/19/02 ECN207-04-008 Networks PREM Minute Writing 06/18/02 06/18/02 COM206-04-#38 Networks PREM Impact Evaluation Tools and Techniques 11/15/01 11/16/01 MAE202-03-005* Networks PREM Poverty Monitoring in the Context of Poverty 01/22/02 01/22/02 POV223-01-001* Reduction Strategies Networks PREM Economic Tools for Trade Policy Analysis: 02/20/02 02/20/02 ECN228-01-001 Quantifying the Effects of Trade Agreements Networks PREM Economic Tools for Trade Policy Analysis: 02/28/02 02/28/02 ECN228-01-002 Quantifying the Effects of Trade Agreements Networks PREM Dynamic Panel Econometrics I 03/25/02 03/29/02 ECN231-01-001 Networks PREM Tools for Implementing the Gender Strategy 04/05/02 04/05/02 GEN221-01-001 Networks PREM Integrating Gender into Operations 04/17/02 04/17/02 ES#000-00-027 Networks PREM Impact Evaluation Tools and Techniques 04/22/02 04/23/02 MAE202-03-009 Networks PREM Macroeconomic Modeling for Poverty Reduction 05/07/02 05/09/02 ECN206-05-011 Networks PREM Macro econometrics Using E-views 05/13/02 05/17/02 ECN222-06-007 Networks PREM International Macroeconomic Tools for Fiscal 05/13/02 05/17/02 ECN224-01-001 sustainability Analysis Networks PREM Standards & Agricultural Trade: The Development 05/16/02 05/16/02 ECN228-02-003 Dimension Networks PREM Public Expenditure Analysis and Management 05/21/02 05/23/02 PUB232-01-001 Core Course Networks PREM Macroeconomic Modeling Techniques and Tools 06/11/02 06/13/02 ECN206-05-012 Networks PREM Attacking Poverty: Building and Supporting PRSPs 06/12/02 06/14/02 POV231-17-018 Networks PREM A Comprehensive Approach to Administrative and 06/21/02 06/21/02 Civil Service Reform -20- Major Detailed Offering Title Start End Code Sponsor Sponsor Date Date Networks PSI Attracting FDI in Energy Sector through World 02/14/02 02/14/02 ES#000-00-003 Bank Guarantees: One-day Seminar for Africa & MNA Energy Task Team Leaders Networks PSI Highway Development and Management Model 04/08/02 04/09/02 IAT216-01-001-1* (HDM)-4 Training Networks PSI Transport Roads Economic Decision Model (RED) 04/10/02 04/10/02 IAT216-01-001-2 Training Networks PSI Assessing Damages and Needs After Disasters 05/15/02 05/16/02 URB208-04-004 Regions AFR Assuring the Quality of Bank Operations in Africa 02/04/02 02/12/02 ES#000-00-004 Regions AFR Team Leaders training Role of Public Expenditure 02/18/02 02/20/02 ES#000-00-020 Reviews in supporting PRSCs and Poverty Status Reporting Cape Town, SA Regions AFR Introducing the Social Protection Agenda in Africa 03/18/02 03/20/02 SPL215-03-003-1 Operations and ESW: Action Plan FY03-05-- Videoconferences for Country Group 1 (Anglophone) Eritrea, Ethiopia and Ghana Regions AFR Introducing the Social Protection Agenda in Africa 03/25/02 03/27/02 SPL215-03-003-2 Operations and ESW: Action Plan FY03-05-- Videoconferences for Country Group 2 (Anglophone) Zambia Regions AFR Introducing the Social Protection Agenda in Africa 03/25/02 03/27/02 SPL215-03-003-3 Operations and ESW: Action Plan FY03-05-- Videoconferences for Country Group 3 (francophone) - Burkina Faso, Guinea, Mauritania, and Niger Regions AFR Introducing the Social Protection Agenda in Africa 04/03/02 04/05/02 SPL215-03-003-5* Operations and ESW: Action Plan FY03-05-- Videoconferences for Country Group 5 (lusophone) Mozambique Regions AFR Introducing the Social Protection Agenda in Africa 04/03/02 04/05/02 SPL215-03-003-4 Operations and ESW: Action Plan FY03-05-- Videoconferences for Country Group 4 (francophone), Benin, Burundi and Madagascar Regions AFR Assuring the Quality of Bank Operations in Africa 04/09/02 04/16/02 ES#000-00-026 Regions AFR Introducing the Social Protection Agenda in Africa 04/22/02 04/26/02 SPL215-03-003-6 Operations and ESW: Action Plan FY03-05-- Training Workshop in Paris Regions AFR Assuring the Quality of Bank Operations in Africa 05/20/02 05/24/02 ES#000-00-031 Regions AFR Assuring the Quality of Bank Operations in Africa 06/25/02 07/02/02 ES#000-00-032 Regions EAP EAP Pilot - Integrated Operational and Technology 03/03/02 03/09/02 ES#000-00-002 Skills Training for ACS (Hanoi, Vietnam) Regions EAP EAP Integrated Technology and Operational Skills 04/20/02 04/26/02 ES#000-00-028 Training for ACS Regions ECA Project Cycle, Operational Processes & 06/03/02 06/03/02 ES#000-00-035* Information Management in ECA WBI WBI EAP Hub: Team Process Skills and Client 12/02/01 12/03/01 OPS225-05-006-1* Relationships WBI WBI EAP Advanced Operational Skills Hub Training 12/04/01 12/09/01 OPS225-05-006-2* WBI WBI MNA ACS Hub Training - Amman, Jordan 01/20/02 01/26/02 OPS225-09-010 WBI WBI Memo Writing Workshop: Part I 02/11/02 02/15/02 COM206-01-098 WBI WBI Organizing and Delivering Presentations 02/19/02 02/28/02 COM205-01-037 WBI WBI Media Expertise Training 02/25/02 02/27/02 COM205-03-038 WBI WBI SAR Hub Training Level 1 - Overall - Jaipur, India 03/03/02 03/15/02 OPS225-02-003 & OPS225-03-004 WBI WBI Report Writing: Part I 03/04/02 03/08/02 COM206-03-109 WBI WBI Short Document Writing 04/08/02 04/12/02 COM206-06-105 WBI WBI LCR Hub Training Level 2 - ACS - Mexico City 04/21/02 04/26/02 OPS225-08-009 Cross- BEI Integrity Awareness Seminar 12/13/01 12/13/01 HUM205-01-012 Networks Cross- CRN Dealing with Conflict for Managers 01/23/02 01/23/02 COM201-05-049 Networks Cross- CRN Dealing with Conflict 02/06/02 02/06/02 COM201-01-058 Networks Cross- DEC Poverty & Inequality Analysis--Module 3--Poverty 04/16/02 04/18/02 ECN230-03-003 Networks Measurement Analysis -21- Major Detailed Offering Title Start End Code Sponsor Sponsor Date Date Cross- DEC Poverty & Inequality Analysis--Module 4-- 05/06/02 05/08/02 ECN230-04-004 Networks Inequality and Pro-Poor Growth Cross- DEC Data Management 06/20/02 06/21/02 ECN205-05-013 Networks Cross- DEC Development Data Tools 06/27/02 06/27/02 ECN217-02-009 Networks Cross- DEC Introduction to Risk & Analysis in Project 11/07/01 11/08/01 OPS203-02-002* Networks Appraisal. Cross- DEC Basic Economics in a Global Perspective 02/04/02 02/07/02 ECN201-02-012 Networks Cross- DEC Development Data Tools 02/12/02 02/12/02 ECN217-02-010 Networks Cross- DEC Economic Statistics 02/19/02 02/21/02 ECN207-04-007 Networks Cross- DEC Macroeconomic Modeling for Poverty Reduction 02/25/02 02/27/02 ECN206-05-013 Networks Cross- DEC Poverty & Inequality Analysis--Module 1--Multi- 02/26/02 02/28/02 ECN230-01-001 Networks Topic Household Surveys Cross- DEC Data Management Course 03/04/02 03/05/02 ECN205-05-011 Networks Cross- DEC Poverty & Inequality Analysis--Module 2-- 03/13/02 03/13/02 ECN230-02-002 Networks Qualitative and Mixed Methods Cross- DEC Conflict & Development: Economic Analysis and 03/18/02 03/20/02 ECN220-01-002 Networks Operational Agenda Cross- DEC Basic Economics in a Global Perspective 03/25/02 03/28/02 ECN201-02-006 Networks Cross- DEC Basic Economics in a Global Perspective 06/04/02 06/07/02 ECN201-02-013 Networks Cross- EXT The Art & Science of Communication 11/13/01 11/16/01 COM203-02-003* Networks Cross- FPS New IBRD Loan and Hedging Products: General - 12/05/01 12/05/01 OPS201-02-007 Networks 1 day course Cross- FPS New IBRD Loan and Hedging Products: 05/08/02 05/10/02 OPS201-04-014 Networks Specialized Cross- HRS Process and Performing Consulting Skills for 10/29/01 12/10/01 HUM208-01-003 Networks Human Resources Professionals Cross- HRS HR Training for HR Country Office Administrators 10/29/01 11/09/01 HUM203-05-006* Networks Cross- HRS HR OCI/LSI Personal & Org Success 12/11/01 12/11/01 HUM209-02-002* Networks Cross- HRS HR OCI/LSI Personal & Org Success 12/12/01 12/12/01 HUM209-02-003* Networks Cross- LEG Legal Orientation Program 10/02/01 10/25/01 ES#000-00-001* Networks Cross- LEG The World Bank Lawyer as Writer 02/13/02 02/13/02 LAW204-00-018 Networks Cross- LEG International Trade Seminar: Legal Aspects - DL ­ 04/23/02 04/23/02 LAW208-03-003 Networks GATS Cross- LEG International Trade Seminar: Regional Trade 05/14/02 05/14/02 LAW208-06-006 Networks Arrangements Others RMC Trust Fund Training for Project Team Leaders 01/17/02 01/17/02 OPS217-07-022 Others RMC Trust Fund Training for Project Team Leader 02/19/02 02/19/02 OPS229-02-002 Note: Courses identified with asterisk (*) were not included in the present study. -22- ANNEX 2 Distribution of Respondents by Affiliation and Grade GA-GD GE GF GG GH+ No answer Unspecified Total AFR Count 94 27 95 105 23 4 348 % 27.0 7.8 27.3 30.2 6.6 1.1 100 EAP Count 66 8 28 11 2 3 1 119 % 55.5 6.7 23.5 9.2 1.7 2.5 0.8 100 ECA Count 57 30 35 25 13 5 1 166 % 34.3 18.1 21.1 15.1 7.8 3.0 0.6 100 LCR Count 35 16 19 15 1 2 88 % 39.8 18.2 21.6 17.0 1.1 2.3 100 MNA Count 36 5 11 11 1 2 66 % 54.5 7.6 16.7 16.7 1.5 3.0 100 SAR Count 37 8 26 31 7 1 110 % 33.6 7.3 23.6 28.2 6.4 0.9 100 ESD Count 5 6 8 4 5 3 31 % 16.1 19.4 25.8 12.9 16.1 9.7 100 FSE Count 5 2 2 1 10 % 50.0 20.0 20.0 10.0 100 HD Count 14 8 18 10 5 55 % 25.5 14.5 32.7 18.2 9.1 100 ISN Count 7 29 30 15 81 % 8.6 35.8 37.0 18.5 100 OPCS Count 3 9 5 10 1 28 % 10.7 32.1 17.9 35.7 3.6 100 PREM Count 10 7 15 10 5 1 3 51 % 19.6 13.7 29.4 19.6 9.8 2.0 5.9 100 PSI Count 5 3 8 8 1 25 % 20.0 12.0 32.0 32.0 4.0 100 DEC Count 2 9 10 8 2 31 % 6.5 29.0 32.3 25.8 6.5 100 HRS Count 3 1 4 % 75.0 25.0 100 LEG Count 9 1 3 4 1 1 19 % 47.4 5.3 15.8 21.1 5.3 5.3 100 WBI Count 9 4 11 5 1 1 31 % 29.0 12.9 35.5 16.1 3.2 3.2 100 IFC Count 11 6 6 3 1 27 % 40.7 22.2 22.2 11.1 3.7 100 MIGA Count 0 % 0 Other Count 27 25 18 15 2 1 2 90 % 30.0 27.8 20.0 16.7 2.2 1.1 2.2 100 No answer Count 2 2 3 7 % 28.6 28.6 42.9 100 Total Count 434 208 351 292 71 18 13 1387 % 31.3 15.0 25.3 21.1 5.1 1.3 0.9 100 -23- ANNEX 3 Table 1. Main Reason for Taking Courses by Location of Staff, Affiliation and Grade Location Affiliation Grade Level HQ Field Office Region Network Other GA-GD GE GF GG GH+ Performance 61.6% 59.5% 62.3% 59.9% 54.1% 60.8% 62.1% 62.2% 59.1% 52.2% Growth 35.3% 35.1% 32.5% 37.9% 44.3% 35.0% 36.5% 34.8% 36.6% 33.3% Network 3.2% 5.4% 5.2% 2.2% 1.5% 4.3% 1.5% 2.9% 4.3% 14.5% Total 100% 100% 100% 100% 100% 100% 100% 100% 100% 100% Table 2. Respondent Reaction to the Balance of Time in the Courses Attention to Attention to Instructors' theoretical content practical content presentations Your participation Pace of course Insufficient 0.5 1.5 0.4 1.3 1.5 Somewhat insufficient 6.6 15.3 9.1 12.7 6.9 Adequate 83.7 78.7 82.9 81.7 75.6 Somewhat excessive 8.6 4.2 7.1 4.2 14.1 Excessive 0.5 0.3 0.5 0.2 1.9 Total (%) 100 100 100 100 100 Observations 1124 1138 1134 1130 1123 -25- ANNEX 4 Factor Score Loadings Communalities Initial Extraction Q8 fulfill learning needs 1 0.727153 Q9 met objectives 1 0.698558 Q10 choice of content 1 0.656276 Q11 logic in sequence 1 0.577497 Q12 material used 1 0.507390 Q13 overall quality 1 0.743253 Q14 overall usefulness 1 0.753770 Q15 overall satisfaction 1 0.814652 Q16 knowledge/skills increased 1 0.584549 Q17 applicable to my job 1 0.364885 Extraction Method: Principal Component Analysis. Total Variance Explained Initial Eigenvalues Extraction Sums of Squared Loadings Component Total % of Variance Cumulative % Total % of Variance Cumulative % 1 6.427983 64.279833 64.279833 6.42798329 64.27983291 64.27983291 2 0.796595 7.965945 72.245778 3 0.619316 6.193156 78.438934 4 0.423535 4.235351 82.674286 5 0.398885 3.988846 86.663132 6 0.370589 3.705892 90.369024 7 0.311500 3.114996 93.484020 8 0.272836 2.728360 96.212380 9 0.217749 2.177493 98.389873 10 0.161013 1.610127 100.000000 Component Matrix Component Score Coefficient Matrix Component Component Q8 fulfill learning needs 0.852733 Q8 fulfill learning needs 0.132659 Q9 met objectives 0.835798 Q9 met objectives 0.130025 Q10 choice of content 0.810108 Q10 choice of content 0.126028 Q11 logic in sequence 0.759932 Q11 logic in sequence 0.118223 Q12 material used 0.712313 Q12 material used 0.110814 Q13 overall quality 0.862121 Q13 overall quality 0.134120 Q14 overall usefulness 0.868199 Q14 overall usefulness 0.135066 Q15 overall satisfaction 0.902581 Q15 overall satisfaction 0.140414 Q16 knowledge/skills increased 0.764558 Q16 knowledge/skills increased 0.118942 Q17 applicable to my job 0.604057 Q17 applicable to my job 0.093973 -27- ANNEX 5-A Logistic Regression Results of Benchmarking Questions: Analysis Using Stakeholder Groups Dependent variable = a) Higher ratings of "4" or "5" on knowledge/skill change; and b) Higher ratings of "4" or "5" on course applicability to respondent's job Dependent var=higher rating on knowledge/skill change Dependt var=higher rating on course's job applicability B S.E. Sig. Exp(B) B S.E. Sig. Exp(B) DMYMTVN2 -0.5500 0.4568 0.2285 0.5769 DMYMTVN2 -0.3137 0.4415 0.4773 0.7307 DMYMTVN3 -0.1923 0.2000 0.3363 0.8251 DMYMTVN3 -1.1300 0.1535 0.0000** 0.3230 DMYDAPYS 0.1663 0.2218 0.4534 1.1810 DMYDAPYS 0.3947 0.1826 0.0306** 1.4839 DMYOTHR 0.5756 0.2877 0.0455** 1.7781 DMYOTHR 0.2606 0.2032 0.1998 1.2976 DMYFIELD 0.0699 0.2258 0.7570 1.0724 DMYFIELD 0.1485 0.1870 0.4272 1.1601 DMYNETWK 0.2680 0.2732 0.3267 1.3073 DMYNETWK 0.3790 0.2174 0.0813 1.4609 DMYAFFLO 0.3249 0.3202 0.3102 1.3840 DMYAFFLO 0.1780 0.2365 0.4516 1.1949 DMYGAGD 0.4447 0.2715 0.1014 1.5601 DMYGAGD -0.2888 0.1963 0.1412 0.7491 DMYACS 1.3604 0.7847 0.0830* 3.8977 DMYACS 1.3501 0.4938 0.0063** 3.8578 DMYAFR 0.7750 0.4144 0.0614* 2.1707 DMYAFR 0.7106 0.3272 0.0299** 2.0352 DMYBEI -0.2314 1.1187 0.8361 0.7934 DMYBEI 0.5502 1.1138 0.6213 1.7336 DMYCRN -0.6666 0.6921 0.3355 0.5135 DMYCRN 0.6176 0.7936 0.4364 1.8544 DMYDEC 1.0024 0.5112 0.0499** 2.7247 DMYDEC 0.5463 0.3175 0.0853* 1.7269 DMYEAP 1.4288 1.0617 0.1784 4.1738 DMYEAP 0.5725 0.5167 0.2679 1.7726 DMYESD -0.0701 0.3641 0.8474 0.9323 DMYESD -0.1529 0.3037 0.6146 0.8582 DMYFPS 0.1954 0.7831 0.8029 1.2158 DMYFPS -0.7987 0.5058 0.1144 0.4499 DMYHD -0.1623 0.3275 0.6201 0.8502 DMYHD 0.1672 0.2836 0.5554 1.1820 DMYHRS -0.1295 1.1168 0.9077 0.8785 DMYHRS 5.2366 8.2194 0.5241 188.0362 DMYISN 0.4690 0.4270 0.2720 1.5984 DMYISN -0.1456 0.2947 0.6214 0.8645 DMYLEG 0.8414 1.0612 0.4278 2.3197 DMYLEG 0.1847 0.6245 0.7674 1.2029 DMYOPCS -0.0407 0.3214 0.8992 0.9601 DMYOPCS 0.2058 0.2667 0.4403 1.2285 DMYPSI -1.0787 0.4953 0.0294** 0.3400 DMYPSI -0.1806 0.5252 0.7309 0.8347 DMYRMC -0.8266 0.4551 0.0693* 0.4375 DMYRMC 0.1325 0.4956 0.7892 1.1417 DMYWBISD 1.4364 0.7632 0.0598* 4.2054 DMYWBISD 2.4712 0.7534 0.0010** 11.8367 Constant 1.6767 0.2674 0.0000** 5.3481 Constant 1.4373 0.2238 0.0000** 4.2095 -2 Log likelihood 823.506 -2 Log likelihood 1153.34 Cox & Snell R Square 0.046 Cox & Snell R Square 0.083 Observations 1326 Observations 1326 Reference group of sponsors=PREM * Significant at the 10% level ** Significant at the 5% level Note: See the description of variables in Annex 7. -29- ANNEX 5-B Logistic Regression Results of Benchmarking Questions: Analysis Using Types of Course Content Dependent variable = a) Higher ratings of "4" or "5" on knowledge/skill change; and b) Higher ratings of "4" or "5" on course applicability to respondent's job Dependent var=higher rating on knowledge/skill change Dependent var=course's job applicability B S.E. Sig. Exp(B) B S.E. Sig. Exp(B) DMYMTVN2 -0.2599 0.4325 0.5479 0.7711 DMYMTVN2 -0.0449 0.4261 0.9161 0.9561 DMYMTVN3 -0.1694 0.1975 0.3911 0.8442 DMYMTVN3 -1.0871 0.1517 0.0000** 0.3372 DMYDAPYS 0.1429 0.2155 0.5073 1.1536 DMYDAPYS 0.3785 0.1790 0.0344** 1.4601 DMYOTHR 0.6607 0.2834 0.0197** 1.9361 DMYOTHR 0.2694 0.1990 0.1759 1.3091 DMYFIELD 0.0772 0.2170 0.7222 1.0802 DMYFIELD 0.1826 0.1789 0.3074 1.2004 DMYNETWK 0.3019 0.2657 0.2557 1.3525 DMYNETWK 0.2889 0.2123 0.1735 1.3350 DMYAFFLO 0.3414 0.3032 0.2602 1.4069 DMYAFFLO 0.1125 0.2261 0.6188 1.1191 DMYGAGD 0.6717 0.2686 0.0124** 1.9577 DMYGAGD -0.2434 0.1855 0.1896 0.7840 DMYBO 0.3040 0.2500 0.2240 1.3552 DMYBO 0.2112 0.1953 0.2794 1.2352 DMYBS 0.6474 0.3751 0.0843* 1.9106 DMYBS 1.2998 0.3083 0.0000** 3.6684 DMYIT 0.5149 0.3905 0.1873 1.6735 DMYIT -0.1168 0.2558 0.6479 0.8897 DMYTF -0.8041 0.4202 0.0556* 0.4475 DMYTF 0.0615 0.4738 0.8968 1.0634 Constant 1.6156 0.1894 0.0000** 5.0307 Constant 1.5041 0.1655 0.0000** 4.5003 -2 Log likelihood 847.075 -2 Log likelihood 1171.419 Cox & Snell R Square 0.029 Cox & Snell R Square 0.070 Observations 1326 Observations 1323 Reference group of course content=Professional and technical * Significant at the 10% level ** Significant at the 5% level Note: See the description of variables in Annex 7. -31- ANNEX 6 Regression Results on Overall Perceived Quality Dependent variable = Respondent's factor scores Model 1 Model 2-A Model 2-B Model 3 B Sig. B Sig. B Sig. B Sig. (Constant) -0.3363 0.0000** (Constant) -0.3982 0.0000** (Constant) -0.4715 0.0000** (Constant) -1.0366 0.0000** DMYMTVN2 -0.0568 0.6849 DMYMTVN2 -0.0450 0.7445 DMYMTVN2 -0.2464 0.0804* DMYMTVN2 -0.2187 0.1182 DMYMTVN3 -0.0741 0.2030 DMYMTVN3 -0.0513 0.3743 DMYMTVN3 -0.0627 0.2733 DMYMTVN3 -0.0774 0.1649 DMYDAPYS 0.2695 0.0000** DMYDAPYS 0.1832 0.0037** DMYDAPYS 0.2148 0.0007** DMYDAPYS 0.1614 0.0100** DMYOTHR 0.1466 0.0507* DMYOTHR 0.1343 0.0693* DMYOTHR 0.1292 0.0818* DMYOTHR 0.1025 0.1565 DMYFIELD -0.0188 0.7648 DMYFIELD -0.0787 0.2163 DMYFIELD -0.0640 0.3254 DMYFIELD -0.0722 0.2649 DMYNETWK 0.1688 0.0206** DMYNETWK 0.1095 0.1498 DMYNETWK 0.1173 0.1244 DMYNETWK 0.0963 0.1953 DMYAFFLO 0.2028 0.0158** DMYAFFLO 0.1505 0.0708* DMYAFFLO 0.1572 0.0663* DMYAFFLO 0.1279 0.1244 GAGD 0.5729 0.0000** GAGD 0.3954 0.0000** GAGD 0.3755 0.0000** GAGD 0.3167 0.0000** DMYBO 0.3227 0.0000** DMYACS 0.7138 0.0000** DMYACS 0.4258 0.0037** DMYBS 0.4858 0.0000** DMYAFR 0.6101 0.0000** DMYAFR 0.4878 0.0077** DMYIT 0.2462 0.0131** DMYBEI 0.3648 0.3123 DMYBEI 0.4716 0.1879 DMYTF -0.2647 0.1314 DMYCRN 0.2429 0.3211 DMYCRN 0.3172 0.1992 DMYDEC 0.3575 0.0018** DMYDEC 0.3422 0.0024** DMYEAP 0.2815 0.1089 DMYEAP -0.1330 0.5187 DMYESD 0.0074 0.9510 DMYESD 0.2270 0.0610* DMYFPS 0.3770 0.0752* DMYFPS 0.3579 0.0865* DMYHD 0.0263 0.8126 DMYHD 0.3266 0.0067** DMYHRS 0.4032 0.2987 DMYHRS -0.0065 0.9867 DMYISN 0.2917 0.0107** DMYISN 0.2998 0.0922* DMYLEG 0.2786 0.2981 DMYLEG 0.9813 0.0025** DMYOPCS 0.1299 0.1941 DMYOPCS 0.0665 0.5104 DMYPSI 0.1060 0.6154 DMYPSI 0.3011 0.1490 DMYRMC -0.1978 0.2773 DMYRMC -0.0371 0.8401 DMYWBISD 0.7656 0.0000** DMYWBISD 0.5497 0.0003** DESIGNRC 0.3131 0.0130** Duration 0.1196 0.0000** %NonBank -0.0043 0.0730* Resp rate 0.0044 0.0289** DMTLCTN2 -0.3275 0.0221** DMYLCTN3 -0.4831 0.0118** R2 0.090 R2 0.118 R2 0.137 R2 0.185 Observations 1241 Observations 1241 Observations 1241 Observations 1241 Model 2-A: Reference group=professioanl and technical courses Model 2-B: Reference group=PREM * Significant at the 10% level ** Significant at the 5% level Note: See the description of variables in Annex 7. -33- ANNEX 7 Description of Regression Variables Variable Description Mean Std. Deviation Personal characteristics: DMYMTVN2 1=motivation network, 0=otherwise 0.04 0.19 DMYMTVN3 1=motivation growth, 0=otherwise 0.34 0.48 DMYDAPYS 1=course agreed in DAP, 0=otherwise 0.37 0.48 DMYOTHR 1=no DAP, 0=otherwise 0.18 0.39 DMYFIELD 1=field office, 0=otherwise 0.35 0.48 DMYNETWK 1=Network, 0=otherwise 0.20 0.40 DMYAFFLO 1=affiliation than Region/Network, 0=otherwise 0.15 0.35 GAGD 1=grade GA-GD, 0=otherwise 0.31 0.46 Sponsor groups (reference group=PREM): DMYACS 1=sponsor ACS, 0=otherwise 0.06 0.23 DMYAFR 1=sponsor AFR, 0=otherwise 0.11 0.31 DMYBEI 1=sponsor BEI, 0=otherwise 0.01 0.07 DMYCRN 1=sponsor CRN, 0=otherwise 0.01 0.11 DMYDEC 1=sponsor DEC, 0=otherwise 0.09 0.28 DMYEAP 1=sponsor EAP, 0=otherwise 0.03 0.17 DMYESD 1=sponsor ESD, 0=otherwise 0.07 0.26 DMYFPS 1=sponsor FPS, 0=otherwise 0.02 0.13 DMYHD 1=sponsor HD, 0=otherwise 0.10 0.30 DMYHRS 1=sponsor HRS, 0=otherwise 0.01 0.07 DMYISN 1=sponsor ISN, 0=otherwise 0.09 0.29 DMYLEG 1=sponsor LEG, 0=otherwise 0.01 0.12 DMYOPCS 1=sponsor OPCS, 0=otherwise 0.15 0.36 DMYPSI 1=sponsor PSI, 0=otherwise 0.02 0.13 DMYRMC 1=sponsor RMC, 0=otherwise 0.03 0.16 DMYWBISD 1=sponsor WBISD, 0=otherwise 0.06 0.23 Type of course content (ref. group=Professional & technical courses): DMYBO 1=Bank operations, 0=otherwise 0.25 0.43 DMYBS 1=Behavioral and skills, 0=otherwise 0.17 0.38 DMYIT 1=Information and technology, 0=otherwise 0.10 0.30 DMYTF 1=Trust funds, 0=otherwise 0.03 0.16 Other course characteristics: DESIGNRC 1=designed/delivered by other organizations, 0=by WB 0.13 0.34 Duration Length of course (in days) 2.92 2.21 %NonBank Percent of non-Bank staff in a course 7.12 15.38 Resp rate Response rate of course 78.40 20.05 DMYLCTN2 1=course location in field offices, 0=otherwise 0.15 0.36 DMYLCTN3 1=course location in DC but outside the WB HQ 0.07 0.26 Repeated 1=repeated more than twice, 0=otherwise 0.69 0.46 Class size Number of participants 25.81 16.08 LearningWk 1=course part of a learning week, 0=otherwise 0.12 0.33 Diversity Grade diversity among course participants 0.85 0.40 DMYDL 1=distance learning, 0=face-to-face 0.01 0.12 -35- ANNEX 8 FY02 Standard Level 1 Evaluation Questionnaire for Formal Training Please assess the training to help to improve future training. To answer, please fill the circle like this . If you wish to l change an answer, fully erase or draw an X over the unwanted mark and fill the circle indicating your preferred answer. Please fill only one circle per question. 1. Training title: ________________________________________ 2. Training date(s): ________________ 3. What was your main reason for taking this training? (Fill only one circle.) _ To enhance performance in current/planned assignment _ To network and share information _ For professional interest and growth _ Other, specify: ______________________________________________________________________________ 4. Do you work for the World _ Yes, at HQ _ Yes, in the Field _ No If no, go to Q8. Bank? 5. For what part of the World Bank Group do you mainly work? (Fill only one circle.) Regions Network Anchors Other _ AFR _ LCR _ ESD _ OPCS _ DEC _ IFC _ EAP _ MNA _ FSD _ PREM _ HRS _ MIGA _ ECA _ SAS _ HD _ PSI _ LEG _ Other, specify: __________ _ ISN _ WBI 6. What is your grade level? _ GA-GD _ GE _ GG _ STC _ STT _ GF _ GH or above _ Other, specify: ______ 7. Was this training agreed upon in your Development Action Plan (DAP)? _ Yes _ No _ I don't have a DAP _ I don't know Using the scale on the right, please rate each question below. Not at Very all much N/A 8. To what extent did the training fulfill your learning needs? _ _ _ _ _ _ 9. To what extent did the training achieve its announced objectives? _ _ _ _ _ _ Very Very poor good N/'A 10. How would you rate the choice of the training content or subject matter? _ _ _ _ _ _ 11. How would you rate the logic in the sequence of the content? _ _ _ _ _ _ 12. How would you rate the materials used during the training? _ _ _ _ _ _ Very Very low high N/A 13. How would you rate the overall quality of the training? _ _ _ _ _ _ 14. How would you rate the overall usefulness of the training? _ _ _ _ _ _ 15. How would you rate your overall level of satisfaction with the training? _ _ _ _ _ _ -37- Strongly Strongly disagree Neither agree 16. My knowledge/skills increased as a result of this training. _ _ _ _ _ 17. The knowledge/skills gained through this training are directly applicable to my job. _ _ _ _ _ A N d o e o N q p Please note the change in scale and Somewhat insufficient Somewhat excessive u adjust your rating pattern a ni i accordingly. Insufficient t Excessive o e n L K J K L a b c d e 18. To assess how balanced the training was, please rate each aspect below with respect to quantity. a. Attention to theoretical content _ _ _ _ _ _ b. Attention to practical content _ _ _ _ _ _ c. Time for instructors' presentations _ _ _ _ _ _ d. Time for your participation _ _ _ _ _ _ e. Pace of the training (insufficient = too slow; excessive=too fast) _ _ _ _ _ _ Please print or write clearly. 19. What knowledge/skills acquired from the training will you now apply on the job? (List the most important three.) a._____________________________________________________________________________________________ b._____________________________________________________________________________________________ c._____________________________________________________________________________________________ 20. What type of support would you need to apply the newly acquired knowledge/skills? ____________________________________________________________________________________________________________________________________________________________________________ ____________________________________________________________________________________________________________________________________________________________________________ ____________________________________________________________________________________________________________________________________________________________________________ 21. What worked best in this training? ____________________________________________________________________________________________________________________________________________________________________________ ____________________________________________________________________________________________________________________________________________________________________________ ____________________________________________________________________________________________________________________________________________________________________________ 22. What would you recommend to improve this training in the future? ____________________________________________________________________________________________________________________________________________________________________________ ____________________________________________________________________________________________________________________________________________________________________________ ____________________________________________________________________________________________________________________________________________________________________________ Thanks for completing the questionnaire. Please leave it in the questionnaire box before leaving or send it to "L1 Eval" MSN J4-401. -38-