39937 The Effectiveness and Impact of WBI FY01-05 Activities: Results from 25 Focus Countries Cristina M. Ling Heidi S. Zia Basab Dasgupta Izlem Yenice The Effectiveness and Impact of WBI FY01-05 Activities: Results from 25 Focus Countries Cristina M. Ling Heidi S. Zia Basab Dasgupta Izlem Yenice WBI Evaluation Studies No. EG07-126 The World Bank Institute The World Bank Washington, D.C. March 2007 Acknowledgments The World Bank Institute Evaluation Group (WBIEG) prepared this report under the direction of Richard Tobin. Cristina Ling, who served as task team leader for the evaluation, is the principal author. The author thanks Guy Darlan, Wema Kategile, and Nidhi Khattri who reviewed the study and offered suggestions for its improvement. The authors also thank Humberto S. Diaz for his assistance in formatting and graphics. WBIEG evaluates learning by staff of the World Bank and activities of the World Bank Institute (WBI). The Institute supports the World Bank's learning and knowledge agenda by providing learning programs and policy services in the areas of governance, knowledge for development, human development, environment and sustainable development, poverty reduction and economic management, and finance and private sector development. The findings, interpretations, and conclusions expressed in WBI Evaluation Studies are entirely those of the authors and do not necessarily represent the views of the World Bank Group, including WBI. WBI Evaluation Studies are available at http://www.worldbank.org/wbi/evaluation Suggested citation: Ling, Cristina M., Heidi S. Zia, Basab Dasgupta, and Izlem Yenice. 2007. The Effectiveness and Impact of WBI FY01-05 Activities: Results from 25 Focus Countries. Report No. EG07-126 Washington, DC: World Bank Institute. Vice President, World Bank Institute Frannie Léautier Manager, Institute Evaluation Group Richard Tobin Task Team Leader Cristina M. Ling ii Table of contents ACRONYMS AND ABBREVIATIONS...............................................................iv EXECUTIVE SUMMARY............................................................................. v INTRODUCTION......................................................................................1 Background.............................................................................................................. 1 Study objectives....................................................................................................... 1 Data and limitations................................................................................................. 1 Model of WBI's program effectiveness and impact................................................ 5 RESULTS............................................................................................. 6 WBI's learning program effectiveness in focus countries....................................... 6 WBI's learning program relevance to country needs in focus countries................. 8 Enhancing activity effectiveness through applying pedagogy ................................ 9 Factors unrelated to effectiveness.......................................................................... 11 The perceived impact of WBI's learning programs .............................................. 12 CONCLUSIONS..................................................................................... 15 REFERENCES....................................................................................... 17 APPENDIXES Appendix A: Survey questionnaire for participants from Brazil, Egypt, Russia, Sri Lanka, and Thailand...................................................................... 19 Appendix B: Survey questionnaire for participants from other 20 countries........... 26 Appendix C: Data summary and model results........................................................ 32 iii Acronyms and Abbreviations CF Country focus CRS Client Registration System GNI Gross National Income F2F Face to face FY Fiscal year K&S Knowledge and skills NGO Nongovernmental organizations RCT Regional Coordination Team WBI World Bank Institute WBIEG World Bank Institute Evaluation Group WBIEN World Bank Institute Environment and Sustainable Development WBIFP World Bank Institute Finance and Private Sector Development WBIHD World Bank Institute Human Development WBIKD World Bank Institute Knowledge for Development WBIPR World Bank Institute Poverty Reduction and Economic Management iv EXECUTIVE SUMMARY The analysis in this report, based on participants from 25 of WBI's focus countries over five fiscal years, shows that WBI's learning programs improved significantly, particularly in FY05. WBI's learning activities became more effective and demonstrated higher impact over time. Programs utilizing a country-applicable approach to pedagogy were deemed highly effective compared with activities that were not perceived to be relevant to participants' countries. Tailoring content to participants' countries increased learning program effectiveness. Likewise, developing action plans that clients can relate to their work and country also enhanced the effectiveness of WBI's learning programs in focus countries. Learning activities were most effective among certain groups. Participants from low-income countries rated WBI's programs significantly more effective than respondents from lower middle-income countries. Likewise, impact was significantly more likely to occur among participants from low-income countries in all areas than in lower-middle income countries. Although these results mirror previous findings that WBI has increased its focus on poorer areas (Quizon et al. 2005), the magnitude of the effect is small with limited practical consequences. WBI's learning programs in focus countries were particularly effective among government clients. Government employees rated WBI as significantly more effective than respondents from other sectors, such as nongovernmental organizations. Also, not surprisingly, respondents who considered themselves highly proficient in the language of instruction found the activity to be more effective than respondents who felt they were lacking proficiency. Impact was greatest among high-level officials. High-level officials were more likely than middle or junior-level respondents to use what they learned overall, and particularly in operational work. v vi INTRODUCTION Background 1. Past analyses of WBI's learning programs have explored the determinants of WBI's effectiveness and impact as a function of the Institute's learning activities and the participants who attended them (Quizon et al. 2005; Khattri et al. 2002). The previous studies documented the importance of key activity features such as utilizing action plans and country-relevant content in the effectiveness of WBI's programs. Likewise, the significance of participant characteristics (e.g., being in high-level positions and from low-income countries) were related to participants' subsequent utilization of their acquired skills and knowledge. 2. Moreover, an analysis (Quizon et al. 2005) based on participants from 12 focus countries between FY01 and FY03 suggested that improvement among the more recent activities may have occurred. However, given the limited time period under study, it was not feasible to determine whether any trends existed. Study objectives 3. The current work tests whether WBI's learning program effectiveness and impact changed over time in WBI's focus countries.1 This study is based on a regionally diverse sample with five time points, FY01-FY05. Thus, the study allows exploration of the relationships between activity and participant characteristics and WBI effectiveness and impact over time. This study addresses three questions: (a) Has the impact of WBI's learning programs changed over time? (b) What aspects of WBI activities enhance WBI's effectiveness? and (c) Where did WBI's learning programs have the largest impact and, among which participants did WBI succeed in providing learning that was later applied? Data and limitations 4. Within the 25 countries, the respondent sample includes data from participants who attended WBI learning events in FY01, 02, 03, 04, and 05. Surveys were administered to a random sample of respondents eight to 22 months after their attendance in FY03, 04, 05, and 06. Sampling was based on a random selection of participants within target countries.2 The final sample was dependent on: the nature of the data collection process in countries over time; budget constraints (i.e., resources to select 1WBI's other capacity-building functions and services, such as technical assistance and research and knowledge-generation work, are beyond the span of this study. 2Target countries were selected based on the following criteria: (a) whether they were a focus country at the time of data collection; (b) the total number of participants with contact information (target number of respondents was 200); and (c) regional representation (obtaining coverage from all the regions). 1 certain countries instead of the entire population); and limitations of WBI's Client Registration System (CRS) for maintaining current contact information for participants. Consequently, the number of respondents in each fiscal year varied. Further, within a given fiscal year, the countries in the sample do not represent both low-income and lower-middle income countries. 5. In the first stage, WBIEG collected data in five lower middle-income countries: Brazil, Egypt, Russia, Sri Lanka, and Thailand. In the second stage, the questionnaire was condensed and sharpened based on experience in the previous year (appendices A and B). WBIEG conducted country-focus evaluations in seven low-income countries: Burkina Faso, Guatemala, Kenya, Indonesia, Nigeria, Tajikistan, and Yemen. In the third stage, data were collected in a combination of low- and lower middle-income countries: Bolivia, China, Ghana, Ethiopia, India, Iran, Mexico, Morocco, Turkey, and Vietnam. Most recently, in the fourth stage, WBIEG collected data in the Philippines, Senegal, and Tanzania. 6. Table 1 summarizes the sample by country. The second and third columns illustrate the fiscal year of the activity that participants attended and the year in which they were surveyed. The next two columns show the total number of respondents from the country and the number of activities they attended. The final column shows the cooperation rate, which is an adjusted response rate indicating the proportion who responded out of those who were able to be contacted. 7. WBIEG hired local consultants to survey WBI alumni. Respondents were randomly selected based on data in the CRS. Much of the contact information for participants was not current because the activity they attended occurred months prior to the evaluation survey. Thus, consultants were often required to locate participants using inaccurate information. Therefore, in addition to the target participant list, WBIEG provided consultants with a list of replacement names and instructions to contact them only after exhausting all possible methods of reaching individuals on the original target list. The rate of cooperation depended on the competency and perseverance of the consultant as well as the local conditions in which they worked (i.e., infrastructure, geographic limitations). The average survey cooperation rate was 66 percent. 2 Table 1: Summary of data, sample size, and response rates, by country Learning WBIEG activities collected Total Total Cooperation Participant Country in FY data in FY respondentsa activities rateb AFR Region Burkina Faso FY02-03 FY04 137 40 51% Kenya FY02-03 FY04 133 41 92% Nigeria FY02-03 FY04 70 19 42% Ghana FY03-04 FY05 179 55 90% Ethiopia FY03-04 FY05 121 42 62% Senegal FY04-05 FY06 185 70 93% Tanzania FY04-05 FY06 40 26 22% EAP Region Thailand FY01-02 FY03 159 28 86% Indonesia FY02-03 FY04 114 30 75% China FY03-04 FY05 103 39 52% Vietnam FY03-04 FY05 190 36 95% Philippines FY04-05 FY06 92 34 49% ECA Region Russia FY01-02 FY03 217 55 91% Tajikistan FY02-03 FY04 57 18 88% Turkey FY03-04 FY05 121 38 61% LCR Region Brazil FY01-02 FY03 119 19 52% Guatemala FY02-03 FY04 45 15 54% Mexico FY03-04 FY05 61 17 32% Bolivia FY03-04 FY05 152 42 88% MNA Region Egypt FY01-02 FY03 174 23 84% Yemen FY02-03 FY04 28 13 54% Iran FY03-04 FY05 54 15 32% Morocco FY03-04 FY05 104 19 53% SAR Region Sri Lanka FY01-02 FY03 122 25 80% India FY03-04 FY05 146 45 75% aDifferences with previous reports reflect updates in the CRS because participant data were merged with the most current CRS database at the time of analysis. bThe cooperation rate is the number of completed surveys divided by the number of contactable respondents (excluding participants for whom the contact information was not valid and could not be traced, but including those who were contacted but refused to cooperate). 8. The sample potentially suffers from selection bias because it is based solely on participants for whom contact information is recorded. Such bias would likely occur only if there is a systematic element discriminating between respondents for whom information exists and those for whom it does not. The assumption is that the missing data problem is random. If bias exists, WBIEG assumes that it exists similarly across time and countries, and therefore does not affect key results regarding fiscal year and country income level, although it would prevent generalizations to the full population. 3 9. The second potential area of bias involves the correlation between time (fiscal year), country conditions, and income level.3 The effects of time and income status could be confounded with country conditions related to factors influencing WBI's effectiveness and impact. To correct for these concerns, WBIEG used estimated values based on multivariate models that consider time and countries' income levels. 10. A third possible limitation is that the report's unit of analysis is the individual. One school of thought argues that for capacity building to have an impact it can be measured only at the organizational level. The assumption in this report (and in the series of impact studies published by WBIEG) is that individual-level impact is a prerequisite to impact at higher levels. In other words, capacity-building takes place at three levels: the individual, organizational, and institutional levels. 11. In this analysis, WBIEG examined impact from the perspective of individuals within organizations. Impact is operationalized at the individual level as the success WBI had in arming participants with knowledge, skills, and strategies to apply within the organizations to which they belong. Thus, critics have argued that respondents may apply what they have learned every day and still not have an impact on the organization in which they work. While this may be true, the converse is not. Change at the organizational level can take place only if individuals within the organization initiate it. 12. Specifically, impact is defined as the reported frequency with which participants utilized their acquired knowledge within eight to 22 months after the activity's completion. The lag between the intervention and the evaluation is critical to allow sufficient time for impact to occur. In other words, participants must have the opportunity to utilize what they learned. Thus, clients were also asked to rate how frequently they applied and used their acquired skills in: · Raising public awareness in development issues, teaching, and research, · Organizing collective initiatives, · Implementing new practices within participants' work organizations, · Influencing legislation and regulations, and · Implementing country development strategies. 13. Country baseline indicators measuring WBI's relevance, effectiveness, and impact were developed for each of the 25 countries, and for WBI overall from FY01 to FY05.4 The surveys asked participants to rate the relevance of the activity to their country's needs as well as WBI's effectiveness in: · Raising their awareness and understanding of development issues important to their country, 3The sample of activities taking place in FY01 includes data collected primarily from lower middle-income country respondents. In contrast, the data for FY05 events includes respondents solely from low-income countries. 4See appendix C, figures C1-C3 for individual country indicators. 4 · Providing them with skills or knowledge, and · Helping them develop strategies or approaches to address the needs of their organization and country. Model of WBI's program effectiveness and impact 14. Previous studies (Quizon et al. 2005, 2004) show that for WBI to have an impact, it is first necessary for its programs to be rated as effective when participant selection is targeted and country conditions are conducive to change. Figure 1 illustrates a system of equations that captures the effects of WBI's activity features, participant characteristics, and country factors in predicting WBI's effectiveness and impact. (See appendix C for a description of key variables and summary statistics.) Figure 1: Two-stage model of the effectiveness and impact of WBI's learning programs Participant characteristics Activity features Language used (+)** Action plan (+)** Exogenous factors Govt. organization (+)* WBI managing unit (+ -)* Low-income country (+)* Relevance (+)** Stage 1: Effectiveness (+)** Exogenous factors Participant characteristics Low-income country (+)* High-level position (+)** FY01 (-)** Stage 2: Impact - utilization Note: = positive effect; - negative effect; * p < .05; ** p = .01 15. This model assumes that effectiveness is a necessary, but not sufficient condition for development impact. This model thus explores the conditions and catalysts necessary for effectiveness to translate into impact by using a two-stage model predicting effectiveness in the first stage and impact in the second stage. 16. The analysis examines WBI's effectiveness in three areas separately: (a) raising awareness, (b) building skills or knowledge, and (c) providing strategies and approaches for development at the organizational and country level. The analysis also examines overall effectiveness, as a composite measure comprised of effectiveness in three areas.5 5Overall effectiveness is the average score for the three key dimensions of effectiveness. 5 The model attempts to predict overall effectiveness and the various dimensions of effectiveness separately as a function of participant characteristics, activity features, and exogenous factors such as a country's economic status. The results show that each of the models predicting WBI's effectiveness fare well, especially given the measurement error associated with survey data (table C3).6 RESULTS WBI's learning program effectiveness in focus countries 17. In reviewing the summary statistics, it is important to keep in mind that the data are based on a series of evaluations conducted over multiple years. As a result, there is a correlation between time (fiscal year), country conditions, and country income level. The sample for activities taking place in FY01 includes data collected from respondents in Brazil, Egypt, Russia, Sri Lanka, and Thailand, all of which are classified as middle- income countries. In contrast, the data for FY05 events includes respondents primarily from low-income countries: Senegal, Tanzania, and Philippines. As a result, the effects of time and country income status can be confounded with country conditions related to factors influencing WBI's effectiveness and impact. To correct for these issues, WBIEG used estimated values based on models that control for country income level and fiscal year as well as activity and participant characteristics. The model results indicate that the effectiveness of WBI's learning programs increased significantly between FY01 and FY05.7 18. Figures 2a and 2b illustrate the actual and estimated annual average ratings for overall effectiveness, raising awareness and understanding of development issues, building knowledge or skills, and providing strategies and approaches for addressing organizational and country needs.8 Respondents' ratings were provided on a 7-point scale where 1 = "not effective at all," and 7 = "extremely effective." 19. WBI's learning programs in focus countries became more effective over time. The finding is consistent across all dimensions of effectiveness. Overall effectiveness, at least in the opinions of participants, increased by 16 percent in FY05 compared with FY01. The increase was 14 percent for raising awareness, 19 percent for skills and knowledge, and 13 percent for providing strategies and approaches. The differences were calculated using simulations based on the multivariate regression model to control for time and country income level.9 6The overall model of effectiveness was the most precise with an R2 of .29. The models predicting raising awareness, skills or knowledge, and strategy were strong with R2 of .22, .20, and .21, respectively. 7Table C3 illustrates the results for the model predicting WBI's effectiveness across the fiscal years under study. 8Estimated values are based on simulations setting country income level to low for all countries in the sample (see appendix C for details.) 9Rather than comparing the raw averages, or simply interpreting the beta coefficients from the model results to illustrate the effects of time, WBIEG used the model to calculate effectiveness ratings in the 6 Figure 2a: Mean effectiveness ratings in raising awareness, K&S, and development strategies, FY01-FY05 Extremely Fiscal Year 2001 effective 7 6 5.6 5.6 5.8 5.3 5 4.9 5 4.8 4.8 4 3 2 Not effective 1 at all Overall Awareness Knowledge & Strategies & skills approaches Areas of effectiveness Figure 2b: Estimated mean effectiveness ratings in raising awareness, K&S, and development strategies, FY01-FY05 Extremely Fiscal Year 2001 Fiscal Year 2005 effective 7 6 5.3 5.5 5.6 5.1 4.8 4.7 5 4.6 4.5 4 3 2 Not effective 1 at all Overall Awareness Knowledge & Strategies & skills approaches Areas of effectiveness respective fiscal years, while holding constant other explanatory factors such as activity features, participant characteristics, and country income. The simulated values are based on the regression model presented in table C3 and were calculated by holding all variables constant at the mean for a given time period, i.e., FY05 and FY01. The percentage increases are the difference between the predicted values for FY05 and FY01. Comparing the simulated effectiveness ratings is superior to comparing the raw averages because the estimates control for country income and other key variables influencing effectiveness ratings. 7 20. The analysis uncovered the key factors in WBI's learning programs that improved effectiveness. To test the robustness of the findings, WBIEG developed regressions predicting overall effectiveness separately by fiscal year.10 The results confirmed the consistency and the importance of the key activity characteristics ­ country relevance and action plans ­ reflected in previous studies (Quizon et al. 2005). In other words, the results point to lessons about how to improve learning activities. WBI's learning program relevance to country needs in focus countries 21. The increase in the effectiveness of WBI's learning programs over time can be explained in large part by the increase in WBI's relevance to participants' country needs in focus countries. Relevance was rated on a 7-point scale from 1, "not relevant at all" to 7, "extremely relevant." Figure 3 presents the average annual relevance ratings by fiscal year and depicts a substantial improvement in WBI's relevance to participants' specific country needs. A multivariate regression analysis controlling for country income confirmed the statistical significance (p<.0001) of the increases in relevance ratings in FY05 (10 percent), FY04 (6 percent), FY03 (6 percent), and FY02 (6 percent) compared with FY01 (see table C4). 22. Activities with learning content tailored to participants' countries enhanced WBI's effectiveness in raising awareness, building skills or knowledge, and providing strategies and approaches. According to the model's results, activities that were rated as highly relevant to participants' country needs were more effective (by 35 percent) than those that were not perceived to be relevant at all.11 In fact, relevance to participants' country needs was a key factor in explaining each of the dimensions of effectiveness. Activities that were rated highly relevant to the country (compared with those that were not relevant at all) were more effective in raising participants' awareness of development issues (41 percent), building skills and knowledge (35 percent), and providing strategies and approaches for addressing organizational and country needs (34 percent). 10Table C5 illustrates the results for FY02, FY03, and FY04, years in which low- and middle-income level countries were represented. 11The effect of relevance (beta coefficient =.35) can be interpreted as a 35 percent increase in effectiveness when all other variable are held constant because both the independent variable and the dependent variable range between zero and 1. The values for all survey items were recalibrated between zero (e.g., "not effective at all") and 1 (e.g., "extremely effective.") 8 Figure 3: Mean ratings of relevance to participants' country needs, by fiscal year Extremely relevant 7 6 5 4 5.6 5.7 5.7 5.9 3 5.2 2 Not relevant 1 at all FY01 FY02 FY03 FY04 FY05 Fiscal year Enhancing activity effectiveness through applying pedagogy 23. The increase in the effectiveness of WBI's learning programs in focus countries was also due to WBI's increased use of action plans. Action plans are work plans, strategy papers, or policy documents that allow participants to apply the skills and knowledge they learned in the activity.12 Confirming findings from previous studies (Quizon et al. 2005; Quizon, Ling, and Lockheed 2004; Khattri et al. 2002), the key pedagogical variable predicting WBI's effectiveness was participants' use of action plans during a learning event. As illustrated in figure 4, the proportion of respondents reporting that they developed an action plan increased between FY01 (24 percent) and FY05 (39 percent). In addition, developing strategies and plans to use in participants' organizations was an instrumental tool enhancing the effectiveness of the activity in all areas: developing strategies and approaches (6 percent); building skills and knowledge (6 percent); and raising awareness (5 percent). The use of action plans improved WBI's overall effectiveness by 6 percent.13 12Respondents were asked the following question: "During the learning activity, did you develop an action plan/strategy (e.g., work plans, strategy papers, policy documents, assessment of country needs assessment of sectoral needs) to apply the new ideas you learned?" 13The effect of action plan is consistent across all dimensions of effectiveness. When respondents reported using action plans in the activity they attended, they rated activity effectiveness 6 percent higher, on average. 9 Figure 4. Percentage of respondents who developed action plans during activities, by fiscal year Percent 100% 80% 60% 45 46 47 39 40% 24 20% 0% FY01 FY02 FY03 FY04 FY05 24. WBI's Participants from low-income countries rated WBI's effectiveness slightly higher than participants from lower middle-income countries (by 2.5 percent). In examining the various dimensions of effectiveness separately, WBIEG discovered that WBI's overall effectiveness in low-income countries was driven mostly by its effectiveness in raising participants' awareness of development issues. Participants from low-income countries were more likely than participants from lower middle-income countries to rate WBI as effective in raising awareness (by 3.5 percent). The results from running the models independently by fiscal year showed that respondents from low- income countries rated WBI's effectiveness in raising awareness higher than respondents from lower middle-income countries (see table C5). 25. WBI's learning programs were more effective among participants affiliated with governments as opposed to other types of organizations. WBI activities were 2 percent more effective among participants from government organizations compared with other organizations in the private sector, NGOs, and the media. This was true particularly in knowledge building and development strategies. No differences existed in terms of WBI's effectiveness in raising awareness. 26. Participants' self-reported levels of proficiency in the language of instruction were significantly related to WBI's effectiveness. Participants who rated themselves as highly proficient were 5 percent more likely to assess the activity as effective overall than respondents who did not rate themselves as highly proficient. Likewise, highly proficient respondents were also 6 percent more likely to rate the activity as effective in providing development strategies. 27. The increase in the perceived effectiveness of WBI's learning programs was mirrored by an increase in self-assessments of proficiency in the language of instruction. Figure 5 illustrates an increase in participants' average self-ratings of proficiency in the 10 language of instruction between FY01 and FY05. As WBI attracted participants with stronger skills in the language of instruction, WBI's events became more effective. Figure 5. Participants' mean self-ratings of proficiency in the language of instruction Highly proficient 7 6.58 5.81 5.67 5.95 5.81 6 5 4 3 2 Not 1 proficient FY01 FY02 FY03 FY04 FY05 at all 28. The mode of delivering an activity was related to WBI's effectiveness in developing strategies and approaches but not to other areas of effectiveness. Face-to- face events were especially effective in developing strategies and approaches. Activities using videoconferences that did not include a blended component were rated as less effective than activities delivered face to face in terms of developing strategies and approaches (by 3 percent). 29. Delivery mode did not influence WBI's effectiveness in the areas of raising awareness and knowledge building. Face-to-face activities were not more effective than distance learning events in raising awareness and knowledge building. 30. Participants' in high-level positions were more likely than those in less senior positions to apply what they learned to create development strategies. Respondents in high level positions were 3 percent more likely to report utilization of WBI-acquired skills and knowledge in creating development strategies for their organizations and country. This result is intuitive because junior-level respondents are not likely to work on developing strategies for their organizations. No differences existed between high- and junior-level respondents in effectiveness in raising awareness and building their skills and knowledge. Factors unrelated to effectiveness 31. Activity location did not affect WBI's effectiveness. Activities held solely for participants from the country were not deemed more effective than activities in other countries. This was surprising because activities in participants' countries were loosely defined as reflecting a country-focused approach. In other words, events that were held for participants from several countries were expected to be more general in content and therefore less applicable to participants' specific country situations. 11 32. There were some differences in effectiveness between learning providers within WBI (figure 6). For example, WBIKD was less effective than WBIEN in overall effectiveness and development strategies. One possible explanation is that a key pedagogical tool for activity effectiveness, action planning, was not used as frequently in WBIKD's activities compared with WBIEN events.14 WBIPR's events were rated higher than WBIEN's in knowledge building. This may be due to activity content in that participants in WBIPR's activities were rated higher than WBIEN's activities in relevance to their country needs.15 Figure 6. Participants' mean ratings of effectiveness, by thematic group, FY01-FY05 Overall effectiveness Strategy Knowledge Extremely effective 7 6 5.5 5.5 5.2 5.1 5.0 5 4.9 4.8 4.8 4.2 4 3 2 Not effective 1 at all WBIKD WBIEN WBIPR The perceived impact of WBI's learning programs 33. Based on the assumption that impact would occur only if WBI activities were effective, impact was operationalized as participants' use of skills and knowledge acquired at WBI events. The key explanatory variables included predicted overall effectiveness (from the first-stage equation), respondent characteristics, and activity features. Table C6 presents the results from the second-stage regression predicting overall utilization, and its two components, academic and operational use. Utilization was measured as participants' ratings of how frequently they used the skills and knowledge they acquired on a 7-point scale, where 1 = not at all, and 7 = very often. Figures 7a and 7b illustrate the actual and estimated average ratings.16 34. Participants' overall use of WBI-acquired skills and knowledge increased significantly over time. Between FY01 and FY05, overall usefulness increased by 30 14Thirty-two percent of respondents reported developing action plans in WBIKD's events compared with 47 percent of respondents in WBIEN's events. 15The average rating of relevance to country needs was 5.67 among respondents in WBIPR activities versus 5.32 among respondents in WBIEN events. 16The estimated values for use of WBI-acquired skills and knowledge for FY01 and FY05 are based on the model controlling for country income, participant characteristics, and activity factors in table C6. 12 percent.17 This was true for both academic and operational use. Academic use includes research, teaching, and raising awareness. Operational use includes the implementation of country development strategies and new practices within work organizations, influencing legislation and regulation, and organizing collective initiatives. WBI's impact was larger in areas related to academe than in operational uses. Impact in both areas increased over time. Academic use was 28 percent higher in FY05 than in FY01. Likewise, operational use was 32 percent higher in FY05 than in FY01. Figure 7a: WBI's mean ratings of impact: overall use, academic use, and operational use, FY01-05 Fiscal Year 2001 Fiscal Year 2005 Very often 7 6 5.0 5.1 4.9 5 3.8 3.9 4 3.7 3 2 Not at all 1 Overall Academic Operational Areas of impact Figure 7b: WBI's estimated mean ratings of impact: overall use, academic use, and operational use, FY01-05 Fiscal Year 2001 Fiscal Year 2005 Very often 7 6 4.8 4.9 4.7 5 3.7 3.8 4 3.6 3 2 Not at all 1 Overall Academic Operational Areas of impact 17The percentage increase is based on robust findings presented in the model in table C6 and the simulated percentage change between FY01 and FY05 in overall use. The difference between the two is significant at the p<.01 level. 13 35. Activity effectiveness increased participants' use of skills and knowledge acquired from WBI events they attended. As demonstrated in previous studies, predicted overall effectiveness was the most important and significant variable in explaining overall usefulness, knowledge building, and operational usefulness (Quizon et al. 2005; Quizon, Ling and Lockheed 2004; Ling and Arango 2004, Quizon 2004; and Quizon and Ling 2003). Perceived activity effectiveness (estimated from stage 1) explained nearly three quarters of the variation (74 percent) in participants' use of the skills and knowledge acquired at WBI events. As expected, the more effective participants rated WBI programs, the more likely and more frequently they were to use what they learned from WBI. 36. WBI had a stronger impact among participants from low-income countries than respondents from lower-middle income countries. Respondents from low- income countries more frequently utilized their WBI-acquired skills and knowledge in all areas: overall, academic, and operational use, compared with respondents from lower- middle income countries. However, the magnitude of the effect was small (1 percent) compared with WBI effectiveness, the primary explanatory variable in the model. In other words, although the effect of low-income country is statistically significant, the practical consequences are limited. 37. Participants in high-level positions were more likely than those in middle and junior-level positions to utilize WBI acquired skills and knowledge in operational work. Confirming previously reported findings (Quizon et al. 2005), participants in high-level positions were 5 percent more likely than those in middle and junior-level positions to apply the knowledge they obtained from WBI overall. High-level professionals were particularly likely to use what they learned in operations: organizing collective initiatives, implementing new practices within their work organization, influencing legislation and regulations, and implementing country development strategies. No differences existed according to professional level in participants' use of skills and knowledge in the academic areas, including research, teaching, and raising public awareness about development issues. 38. The impact of WBI's learning programs in focus countries in FY05 was significantly greater than in FY01. Participants were significantly more likely to report using what they learned in FY05 compared with participants in FY01 in all areas of impact: overall use, academic use, and operational use. The comparison between FY05 (after the introduction of the country-focus approach) and FY01 (before that approach), suggests the possibility of WBI's country-focus approach having begun to have an impact on the ground. Given that WBI's Regional Coordination Team (RCT) did not begin implementing country-focus initiatives until FY04, it is surprising to see potential indications of improvement already in FY05. It is important not to draw any definitive conclusions from the positive effect of the FY05 variable, however, because this effect captures all the unmeasured attributes that may have improved with time but are not in the regression model (e.g., better targeting and improved program content.) 14 CONCLUSIONS 39. This comprehensive evaluation of the World Bank Institute's FY01-FY05 learning programs for participants from 25 focus countries demonstrates significant improvement over time. WBI's learning programs in focus countries have become more effective and demonstrated greater impact over the years, especially in FY05. This suggests that shortly after the launch of WBI's country-focus approach, significant improvements in WBI's learning programs began to surface. Although the RCT was established in FY03, and did not begin implementing CF policies until FY04, there is already evidence of results in FY05 compared with FY01 (before WBI's country focus approach was developed). 40. Utilizing a country-focused approach in pedagogy also improved program effectiveness. Activities with learning content tailored to the target audience's countries were effective in raising participants' awareness, building their skills and knowledge, and providing them with new strategies and approaches to address the needs of their countries and organizations. 41. In addition to focusing on participants' country needs, WBI targeted some high- level officials from low-income countries. WBI's learning programs were most effective, and had the largest impact among participants from low-income countries. Impact was greatest among high level officials conducting operational work such as organizing collective initiatives, implementing new practices in work organizations, influencing legislation and regulation, and building country development strategies. 15 16 REFERENCES Bardini, Mark D., Victoria Gunnarsson, Elina M. Manjieva, and Yekaterina Narozhnaya. 2003. The Impact of WBI Activities, FY01-02, on Participants from Russia: A Baseline Assessment. WBI Evaluation Studies, EG04-81. Eckert, William, Graca Sousa, and Victoria Gunnarsson. 2004. The Impact of WBI Activities, FY01-02 on Participants from Brazil: A Baseline Assessment. WBI Evaluation Studies, EG04-82. Khattri, Nidhi, Jaime Quizon, Mark Bardini, William Eckert, Sukai Prom-Jackson, Heidi Zia, Maurya West Meiers, Zhengfang Shi, Maria B. Palmisano, Roman Novojilov, Shreyasi Jha, and Denis Nikitin. 2002. Impact Evaluations of WBI Client Programs, FY00-01. WBI Evaluation Studies, EG03-63. Khattri, Nidhi, Shreyasi Jha, and Denis Nikitin. 2004. The Impact of WBI Activities, FY02-03, on Participants from Kenya: A Baseline Assessment. WBI Evaluation Studies, EG04-89. Khattri, Nidhi, Peter Bachrach, and Tao Jiang. 2003. The Impact of WBI Activities, FY01- 02, on Participants from Sri Lanka: A Baseline Assessment. WBI Evaluation Studies, EG04-73. Le Rouzic, Violaine. 2004. The Impact of WBI Activities, FY02-03, on Participants from Burkina Faso: A Baseline Assessment. WBI Evaluation Studies, EG05-102. Ling, Cristina, and Diana J. Arango. 2004. The Impacts of FY02-03 WBI Activities on Nigerian Participants. WBI Evaluation Studies, EG04-87. Liu, Chaoying, Jonathan Richer, and Laurie Choi. 2004. Country Focus Evaluation of WBI Activities in FY02-03, Guatemala: A Baseline Assessment. WBI Evaluation Studies, EG05-93. Quizon, Jamie, B. 2002. "Measuring, Monitoring and Managing for Development Results: An Approach to Monitoring and Evaluation in World Bank Institute (WBI) Country Focused Activities." WBIEG Working Paper (final draft). ________. 2004. Evaluation of FY02-03 WBI Activities with Indonesian Participants: Relevance, Effectiveness and Impact. WBI Evaluation Studies, EG05-96. 17 Quizon, Jaime, and Cristina Ling. 2003. The Impact of WBI Activities, FY01-02, on Participants from Thailand: A Baseline Assessment. WBI Evaluation Studies, EG04-77. Quizon, Jaime, Cristina Ling, and Marlaine E. Lockheed. 2004. The Effectiveness and Use in FY03 of WBI FY01-02 Activities: A Baseline Assessment in Five Countries. WBI Evaluation Studies, EG04-86. Quizon, Jaime, Nidhi Khattri, Heidi Zia, and Victoria Gunnarsson. 2005. An Evaluation of the Effectiveness and Outcomes of WBI FY01-03 Activities: Results from 12 Focus Countries. WBI Evaluation Studies, EG05-108. Zia, Heidi, S., and Shreyasi Jha. 2004. The Impact of WBI Activities, FY02-03, on Participants from Tajikistan: A Baseline Assessment. WBI Evaluation Studies, EG05-95. Zia, Heidi, S., and Shreyasi Jha. 2004. The Impact of WBI Activities, FY02-03, on Participants from Yemen: A Baseline Assessment. WBI Evaluation Studies, EG04-92. Zia, Heidi, S., Mustapha Kamel Al-Sayyid, Sahar Tawila, and Victoria Gunnarsson. 2003. The Impact of WBI Activities, FY01-02, on Participants in Egypt: A Baseline Assessment. WBI Evaluation Studies, EG04-78. World Bank. 2004. World Development Report: Making Services Work for Poor People. Washington, DC: The World Bank. ________. 2006. World Development Report: Equity and Development. Washington, DC: The World Bank. World Bank Institute. 2003. "Pedagogy for Effectiveness and Impact: Evidence Regarding Action Learning and Course Duration." WBI Evaluation Brief. 18 APPENDIX A: SURVEY QUESTIONNAIRE FOR PARTICIPANTS FROM BRAZIL, EGYPT, RUSSIA, SRI LANKA, AND THAILAND (Activities Held in FY01-02) W O R L D B A N K I N S T I T U T E Promoting Knowledge and learning for a better world Evaluation Participant Questionnaire I. Usefulness of the WBI Learning Activity 1. Thinking about the WBI learning activity you attended, would you say that the activity has been effective or ineffective in the areas noted below. Not Areas effective 2 3 4 5 6 Extremely Don't at all Effective Know a. Raising your awareness and understanding of the development issues important to your country b. Updating or refining the knowledge or skills you already had c. Providing you with new knowledge or skills d. Helping you get to know people interested in the subject matter of the learning activity e. Providing you with strategies or approaches to address the development needs of your country f. Providing you with strategies or approaches to address the needs of your organization 2. Thinking about the WBI learning activity you attended, what was the main theme or area it addressed? 3. Did the WBI learning activity you attended lead to any overall changes in that Don't specific area? (If you answer "No," or "Don't Know," skip to question #5.) Yes No know 4. How would you rate the nature of the overall change in that specific area? Strong negative Strong positive Do change No change change Not Know 1 2 3 4 5 6 7 8 19 5. Thinking about the WBI activity you attended, how often have you used the skills and knowledge you acquired in the following areas? Please mark "Not Applicable" if you do not work in the given area. Areas Not at all 2 3 4 5 6 Very Often N/A a. Conducting research b. Developing country development strategies c. Implementing country development strategies. d. Changing or influencing legislation or regulations e. Teaching f. Raising others' awareness in development issues g. Organizing collective community initiatives h. Implementing new practices within your work organization 6. How helpful or unhelpful are the following factors in actually using the new knowledge or skills that you acquired from the WBI learning activity? Not Neither Factors helpful 2 3 helpful nor 5 6 Extremely Don't at all unhelpful Helpful Know a. Practices and procedures in your work organization b. Resources and funding availability in your work organization c. Incentive system in your work organization d. Your supervisor e. Your colleagues 7. How helpful or unhelpful are the following factors in implementing the ideas covered in the learning activity? Not Neither Items helpful 2 3 helpful nor 5 6 Extremely Don't at all unhelpful Helpful Know a. Your country's policies b. Social groups in your country c. Political groups in your country d. Your country's general mood of "readiness" for reform and innovation 20 8. How has the WBI activity that you attended, influenced or led to changes in the following areas? (Please mark N/A if the particular area is not relevant to the activity.) Areas Negative Positive Influence 2 3 4 5 6 Influence N/A a. Research methodology or publications b. Legislation or regulations c. Teaching materials for courses d. Consensus building for change e. Community-based initiatives f. Work practices in your organization g. Other (Please specify) 9. Have the issues raised in WBI learning activities been discussed at work, with local partners, government officials or NGOs? Never discussed Discussed Don't at all thoroughly Know 1 2 3 4 5 6 7 10. Thinking back to the time immediately following the learning activity you attended, were you aware of any follow-up activities (e.g., meetings, workshops, or E-mail Yes No discussion groups) arranged by participants? (If you answer "No," skip to question #15) 11. Did you participate in any of these follow-up activities? Yes No 12. In how many of these follow-up activities did you participate? 13. Are these activities still taking place? Yes No 14. If not, why? II. Relevance of WBI Learning Activities to the Needs of Your Country Recall, you are being asked to think about the activity you attended between the summer of 2000 and the summer of 2002. 15. Was the activity you attended designed specifically for participants from your Yes No Don't country? know 21 16. Thinking about the WBI learning activity, to what degree were the topics covered in the activity relevant to your country's specific needs? Extremely Extremely Don't Irrelevant Relevant Know 1 2 3 4 5 6 7 8 (First review the development goals under column "A." Next, under Column "B," indicate with an "X" the relevant goal(s), then rate the extent to which the activity addressed key issues in reaching that goal(s) under Column "C.") A B C Mark an "x" If you marked "X," please rate the extent to which here if WBI the activity addressed key issues in achieving that Activity was goal. related to each goal Goals for Development Mark with an "X' Low 2 3 4 5 6 High N/A a. Eradicate Extreme Poverty b. Achieve universal primary education c. Promote gender equality and empower women d. Reduce child mortality e. Improve maternal health f. Combat HIV/AIDS, malaria, and other diseases g. Ensure environmental sustainability h. Develop global partnerships for development i. Ensure water sanitation and supply j. Improve investment climate and finance k. Promote trade III. Comparison of WBI Learning Activities with Similar Activities Offered by Other Organizations 18. Thinking about the WBI learning activity you attended, are you aware of any learning activities with similar content offered by organizations other than WBI, in your country? (If Yes No you answer "No," skip to question #21.): 22 19. If yes, please provide the name(s) of the organization(s): 1. 2. 3. 20. In comparing the activities offered by WBI and non-WBI, would you say that, overall, the non-WBI learning activities were more effective or less effective than WBI training? Much About Much Do more the less Not effective Same effective Know 1 2 3 4 5 6 7 8 IV. Characteristics of the Learning Event and Your Background Now, we would like to ask you to think about some of the things that took place during the activity you attended and to recall some of the logistics. 21. How many of your colleagues, or others with whom you work closely (either within or outside of your organization), attended the same activity? None 1 to 2 3 to 4 5 to 10 Over 10 22. During the learning activity, did you develop an action plan/strategy (e.g., work plans, strategy papers, policy documents, assessment of country needs, assessment of sectoral needs) to apply the new ideas you learned? (If you answer Yes No "No," please skip to question #25) 23. If yes, did you work as a team with your colleagues to produce the action plan? Yes No 24. Did you actually use parts or all of the action plan in your work? Yes No 25. To your knowledge was the learning activity a part of a series of related activities? Yes No 26. If yes, did you participate in other parts of the series? Yes No 27. Were you provided with the contact information of other participants in the activity you attended such as email addresses, telephone numbers or mailing Yes No addresses? 28. If yes, please indicate whether the contact information was: Not useful Extremely Do Not at all useful Know 1 2 3 4 5 6 7 8 23 29. After the WBI learning activity, were there any follow-up contacts made by the Yes No organizers? 30. If yes, please indicate the nature of the follow-up: Face-to-Face Web-based E-mail Other (please specify) meetings/activities discussions discussion __________________ 31. After the learning activity, did you receive any newsletters related to the learning Yes No activity? 32. If yes, please indicate whether it was: Not Do useful at Extremely Not all useful Know 1 2 3 4 5 6 7 8 33. In retrospect, what aspect(s) of the learning activity did you find most useful for your work, organization, or country? 34. In retrospect, what aspect(s) of the learning activity did you find least useful for your work, organization, or country? 35. What was the primary language used during the learning activity? (For each item below, select one answer on the Not scale, where 1=Not proficient at all and 7=Highly proficient 2 3 4 5 6 Highly proficiency.) at all Proficient 36. At the time of the learning activity, what was your level of proficiency in the language of instruction of the learning activity? 37. At the time of the learning activity, what was your level of proficiency in the technical terminology utilized in the learning activity 38. How long was the learning activity? Please indicate number of days 39. What proportion of the learning activity did you attend? A quarter Less More All or or less than half Half than almost half all 24 40. At the time of the learning activity, which of the following best characterizes the organization in which you worked? (Select one.) University /Research Institution Donor Agency Non-Governmental Organization (NGO) (not-for- profit) National/Central Government Private Sector (for profit) Provincial/Regional Government Other ____________________________________ Local Government 41. At the time of the learning activity, which of the following best describes the primary type of work you did? (Select one.) Research Management/Administration Teaching Policymaking / Legislative Research / Teaching Provide Services (e.g. financial, health, etc.) Other ____________________________________ 42. At the time of the learning activity, how would you best characterize your position in your organization? Top management (e.g. Minister, Deputy Minister, Top government official) Senior management/administration (e.g. Department Head, Division Head) Middle management/administration (e.g. Program Manager, Project Leader) Senior professional/technical/research staff Junior professional/technical/research staff Other (Please specify) ____________________________________ 43. Please indicate your gender Male Female 44. In which year were you born? (Please fill in the year in the boxes provided.) 1 9 Thank you for your feedback. We greatly appreciate your cooperation. 25 APPENDIX B: SURVEY QUESTIONNAIRE FOR PARTICIPANTS FROM OTHER 20 COUNTRIES (Activities between FY02-03 and FY04-05) World Bank Institute (WBI) Country Focus Evaluation Questionnaire Questionnaire to be completed by former participants I. Relevance of the Activity The activity that you are asked to evaluate is mentioned on the first page of this questionnaire. 1. Since the end of the activity, to what degree has the activity been relevant to your work? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 2. To what degree have the topics covered in the activity been relevant to your country's needs? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 3. Was the activity designed specifically for participants from your country? Yes No Don't know 4. Was the activity related to the country development goals listed below? a. Eradicate extreme poverty Yes No Don't know b. Achieve universal primary education Yes No Don't know c. Promote gender equality and empower women Yes No Don't know d. Reduce child mortality Yes No Don't know e. Improve maternal health Yes No Don't know f. Combat HIV/AIDS, malaria, and other diseases Yes No Don't know g. Ensure environmental sustainability Yes No Don't know h. Develop global partnerships for development Yes No Don't know i. Ensure water sanitation and supply Yes No Don't know j. Improve investment climate and finance Yes No Don't know k. Promote trade Yes No Don't know 26 II. Usefulness of the Activity 5. Please rate the degree of effectiveness of the activity in each area noted below. (If the area was not an objective of the activity, please mark "not applicable.") Not Areas effective Extremely Not at all effective applicable 1 2 3 4 5 6 7 NA a. Raising your awareness and understanding of the development issues important to your country b. Providing you with knowledge or skills c. Helping you better understand your role as an agent of change in your country's development d. Helping you develop strategies or approaches to address the needs of your organization e. Helping you develop strategies or approaches to address the needs of your country f. Helping you develop contacts, develop partnerships and build coalitions in the field 6. How would you rate the change--brought by the activity--in the main topic or issue it addressed? Strong Strong negative positive change No change change Don't know 1 2 3 4 5 6 7 DK 7. How often have you used the skills and knowledge you acquired in the activity for the following purposes? (If you have not worked in the given area since this activity, please mark "Not applicable.") Purposes Not Very Not at all often applicable 1 2 3 4 5 6 7 NA a. Conducting research b. Teaching c. Raising public awareness in development issues d. Implementing new practices within your work organization e. Organizing collective initiatives f. Influencing legislation and regulation g. Implementing country development strategies 27 8. To what extent did the following factors help or hurt the process of using the knowledge/skills that you acquired at the activity? Factors Greatly Neither helped Greatly Not hurt nor hurt helped applicable 1 2 3 4 5 6 7 NA a. Your work environment (e.g., work procedures, colleagues, incentive system, funding, etc.) b. Your county's development environment (e.g., country policies, social groups, political groups, readiness for reform, etc.) 9. How has the activity influenced or led to changes in the following areas? (If the area is not relevant to the activity, please mark "Not applicable.") Areas Negative No Positive Not influence influence influence applicable 1 2 3 4 5 6 7 NA a. Research b. Teaching c. Public awareness in development issues d. New practices within your work organization e. Collective initiatives f. Legislation and regulation g. Country development strategies 10. Since the activity, have you discussed the issues raised in the activity, at work, with local partners, government officials, NGOs, or in the media? Never Thoroughly discussed discussed 1 2 3 4 5 6 7 III. Comparison of the WBI Activity with Similar Activities Offered by Other Organizations 11. Did you participate in any similar learning activities offered by other (NON-WBI) organizations in your country? (If no, please skip to question 14.) Yes No 12. If yes, please provide the name(s) of the organization(s): 1. ................................................................................................................................................................................................................... 2. ................................................................................................................................................................................................................... 3. ................................................................................................................................................................................................................... 28 13. How would you rate the usefulness of the WBI activity compared with NON-WBI activities? WBI WBI much less much more No useful About the same useful opinion 1 2 3 4 5 6 7 IV. Characteristics of the WBI Activity, its Follow-up and Your Background 14. How would you describe the type of the WBI learning activity that you attended? Video Sessions (Distance Class room Mix of Video and Web-based Learning) (Face to Face) Face to Face Conference Learning Study tour 1 2 3 4 5 6 15. How effective was this type of learning activity in helping you learn? Not effective at Extremely No all effective opinion 1 2 3 4 5 6 7 16. During the WBI activity, did you develop an action plan/strategy (e.g., work plans, strategy papers, or policy documents) to apply the skills and knowledge you learned? (If no, please mark "no" below, then skip to question 18.) Yes No 17. If yes, did you use part or all of the action plan in your work? Yes No 18. Were you provided with the contact information of other participants in the activity, such as e-mail addresses, telephone numbers or mailing addresses? (If no, please mark "no" below, then skip to question 20.) Yes No 19. If yes, how did you use it? Used it to Used it to Other uses Never used it continue activity organize joint follow- (Please specify briefly) related discussions up activities _____________ 20. Was the language of instruction used during the activity the same language you use at work? Yes No 29 21. At the time of the activity, what was your level of proficiency in the language of instruction? Not proficient Highly at all proficient 1 2 3 4 5 6 7 22. At the time of the activity, what was your level of proficiency in the technical terminology used in the activity? Not proficient Highly at all proficient 1 2 3 4 5 6 7 23. After the activity, did WBI contact you for follow-up issues regarding the activity? Yes No 24 After the activity, did YOU contact WBI for follow-up issues or questions on the content of the activity? (If no, please skip to question #23) Yes No 25. If yes, please rate WBI's helpfulness in addressing your issues. WBI I did not responded, WBI responded have WBI did but was not and was follow-up not helpful extremely requests respond at all helpful for WBI 0 1 2 3 4 5 6 7 NA 26. Which of the following best describes the organization in which you have worked the longest since the activity? (Select one.) University/research institution National/central government Non-governmental organization (not-for-profit) Provincial/regional government Media Local/municipal government Private sector Other, specify: _________________ 27. Which of the following best describes the primary type of work you have done the longest since the activity? (Select one.) Research Teaching Policymaking/legislation Provision of services (e.g., financial, health, etc) Management/administration Other, specify: __________________ 30 28. How would you best describe the level of the position you have held the longest since the activity? Highest level (e.g., Minister, Deputy Minister, Top Government Official, Full Professor, President of an organization) Senior level (e.g., Department Head, Division Head, Associate Professor, Senior Researcher) Middle level (e.g., Program Manager, Project Leader, Assistant Professor, Technical Expert) Junior level (e.g., Research associate, Ph.D. level graduate student, Technical Specialist) Entry level (e.g., Intern, assistant) Other, Please specify:______________________________________________ 29. What is your gender? Male Female Thank you for your feedback. We appreciate very much your cooperation. 31 APPENDIX C: DATA SUMMARY AND MODEL RESULTS Table C1. Variable Descriptions Independent Variable Descriptions and Data Sources (Data Source in Parenthesis) Participant Characteristics Position (ref: Entry/junior evel) Dummy taking a 1 if the job position within the participant's organization is mid-level Mid-level position such as program manager, project leader, assistant professor or technical expert (survey). Dummy taking a 1 if the job position within the participant's organization is senior-level Senior-level position such as department head, division head, associate professor or senior researcher (survey). Top-level position Dummy taking a 1 if the job position within the participant's organization is top-level such as minister, top government official, full professor or CEO (survey). Organization (Ref: other) Academic Dummy taking 1 if the participant represents a research or teaching organization (survey). Government Dummy taking 1 if the participant represents government organization (survey). Language proficiency Level of proficiency in the language of instruction of the learning activity 1=not proficient at all, 7=highly proficient (survey). Terminology proficiency Level of proficiency in the language of instruction of the learning activity 1=not proficient at all, 7=highly proficient (survey). WBI Policy Variables/Activity Features Follow-up Dummy taking a 1 of the participant perceived that he/she was contacted by the activity organizer after attending the activity (survey). Action plan Dummy variable taking a 1 of the participant reported that the activity used action planning (survey). Relevance Participant's ratings on relevance of the activity to her country's need Focus Dummy taking a 1 if the activity location is same as participant's country Mode of Delivery (ref: Classroom/F2F) Blended/Multi-media Dummy taking a 1 if the activity used a blended/multi-media mode (CRS). VC/Global dialog Dummy taking a 1 if the activity was a videoconference of global dialog (CRS). Web-based/ Dummy taking a 1 if the activity used a web-based or electronic computer format to Computer/Electronic access the activity (CRS). Division or WBI thematic program (ref: WBIEN) WBIFP Dummy taking a 1 if the activity falls under WBIFP (CRS). WBIHD Dummy taking a 1 if the activity falls under WBIHD (CRS). WBIPR Dummy taking a 1 if the activity falls under WBIPR (CRS). WBIKD Dummy taking a 1 if the activity falls under WBIKD (CRS). WBIR Dummy taking a 1 if the activity falls under WBIPR (CRS). (Table C1 continues on next page.) 32 (Table C1 continued..) Independent Variable Descriptions and Data Sources (Data Source in Parenthesis) Exogenous Factors GNI per capita Gross National Income of the country of the participant (World Bank Country at A Glance). Time of delivery (reference: FY05) Delivered in FY01 Dummy taking a 1 if the activity was delivered in 2001. Delivered in FY02 Dummy taking a 1 if the activity was delivered in 2002. Delivered in FY03 Dummy taking a 1 if the activity was delivered in 2003. Delivered in FY04 Dummy taking a 1 if the activity was delivered in 2004. Facilitators and Barriers Organizational factors level of helpfulness of organizational factors such as practices, procedures and incentive system in the organization, available resources and funding, supervisors and peers in helping the participant apply the skills and knowledge acquired in the activity, 1=not helpful at all, 4=neither helpful nor unhelpful, 7=extremely helpful (survey). Country level factors level of helpfulness of country level factors such as country policies, social and political groups and the country's readiness for reform and innovation in helping the participant apply the skills and knowledge acquired in the activity, 1=not helpful at all, 4=neither helpful nor unhelpful, 7=extremely helpful (survey). Dependent Variable Description And Data Sources (Data Source In Parenthesis) Activity Effectiveness Participant's perception of the extent of effectiveness of the activity attended in three areas and overall. scale: 1=not effective at all, 7=extremely effective (survey) Overall effectiveness average of all areas of activity effectiveness that coincide for the first round and second round of surveys Raising awareness Effectiveness in raising awareness and understanding of country development issues Building new skills and Effectiveness in building new skills and knowledge knowledge Providing strategies and Effectiveness in providing strategies and approaches for addressing organizational approaches or country needs Use of Skills and Participant's report of the frequency of use of skills and knowledge acquired in the knowledge activity in two areas and overall scale: 1=not at all, 7=very often (survey) Overall use Average of all areas of use of skills and knowledge that coincide for the first round and second round of surveys Academic use Use in research, teaching, raising awareness Operational use Use in implementing practices, developing strategies, influencing legislation, organizing initiatives 33 Baseline indicators of relevance, effectiveness, and impact by country Figure C1. WBI's relevance to country needs, by GNI/capita18 Extremely relevant 7 6.0 6.3 6.1 6.0 6.0 6.2 6 5.9 6.0 5.8 5.8 5.6 5.7 5.6 5.6 5.9 5.6 5.6 5.3 5.4 5.4 5.4 5.1 5.4 4.9 4.8 5 4.9 4 3 2 Not relevant 1 at all o s t , ci sia ka o n ey Fas Iran Kenya ssia Turk Brazil xicoe Ethiopia Burkina Nigeria Ghana Vietnam emenY India Lan China Egyp Islam M Tajikistan Tanzania Senegal Bolivia country Thailand Indone Sri Philippine 25 meann Morocc Guatemala Ru Federation $90 $180 $220 $290 $290 $320 $360 $480 $490 $530 $550 $710 $851 $890$1080$1100$1237$1320$1470$1740$1980$2000$2140$2790$2830$6230 Figure C2. Ratings on overall effectiveness, by country GNI/ capita Extremely effective 7 6 5.6 5.5 5.4 5.4 5.7 5.7 5.6 5.8 5.4 5.7 5.7 5.6 5.5 5.0 5.2 5.4 5.1 5.3 4.9 5.1 5 4.7 4.8 4.7 4.7 4.7 4.9 4 3 2 Not effective 1 s at all o sia ytr o t , ci n ey Fas ssia Ghana Bolivia Islam Turk Brazil xicoe Ethiopia Burkina Nigeria Kenya Vietnam emenY India Lanka China coun Egyp Iran M Tajikistan Tanzania Senegal Indone Sri Morocc Philippine 25 mean Guatemal a Thailand Ru Federation $90 $180 $220 $290 $290 $320 $360 $480 $490 $530 $550 $710 $851 $890 $1080 $1100 $1237 $1320 $1470 $1740 $1980 $2000 $2140 $2790 $2830 $6230 18GNI is based on the year in which the activity took place. 34 Figure C3. Ratings of overall use of acquired K&S, by country GNI/capita Very 7 often 6 5.5 5.3 5.2 4.8 5.1 5.5 5.0 4.6 4.9 5 4.8 4.8 4.7 4.6 4.6 4.7 4.6 4.7 4.7 4.4 4.2 4.4 4.3 4.4 4.2 4 3.8 3.3 3 2 Not 1 o t at all o s , sia ntry n ci n Fas India ocix Ghana Bolivia China Egyp Iran ssia Brazil Ethiopia Burkina Nigeria Kenya Vietnam emenY Lankai ouc mea Islam urkeyT Me Tajikistan Tanzania Senegal Ru Indone Sr Morocc Philippine 25 Thailand Guatemala Federation $90 $180 $220 $290 $290 $320 $360 $480 $490 $530 $550 $710 $851 $890$1080$1100$1237$1320$1470$1740$1980$2000$2140$2790$2830$6230 35 Table C2. Descriptive statistics summarizing key variables Variable Mean Std Dev A. Individual ratings (1= not at all, 7=extremely) Relevance 5.59 1.26 Overall Effectiveness 5.16 1.06 Awareness 5.37 1.28 Knowledge 5.36 1.29 Strategy 5.02 1.23 Overall use 4.55 1.09 Academic Use 4.64 1.16 Operational Use 4.48 1.24 B. Participant characteristics 1. Position (Ref: entry/junior) Mid-level position 0.30 0.46 Senior-level position 0.33 0.47 High-level position 0.10 0.30 2. Work organization (Ref: Other organizations) Academic 0.17 0.37 Government 0.40 0.49 Proficiency in language of instruction 6.02 1.24 Technical terminology 5.61 1.27 3. GNI per capita (Ref: Lower middle-income countries) Low-income countries 0.44 0.50 C. Activity features Follow up 0.26 0.44 Action plan 0.43 0.49 Focus country 0.43 0.49 1. Mode of delivery (Ref: F2F) Blended learning 0.03 0.18 Distance learning 0.22 0.41 Electronic learning 0.07 0.25 2. WBI managing division (Ref: WBIEN) WBIHD 0.18 0.39 WBIK 0.01 0.09 WBIPR 0.15 0.35 WBIEN 0.17 0.38 WBIFP 0.25 0.43 WBIR 0.06 0.24 3. Fiscal year (Ref: FY05) FY01 0.13 0.34 FY02 0.21 0.41 FY03 0.32 0.46 FY04 0.28 0.45 36 Table C3. Ordinary least squares regression results predicting effectiveness of WBI learning activities, FY01-05 Variable Overall effects Awareness Knowledge Strategy N 2,923 2,923 2,923 2,923 R2 0.291 0.220 0.199 0.209 Intercept 0.375** 0.374** 0.415** 0.368** A. Participant characteristics 1. Position Mid-level position 0.012 0.002 -0.003 0.021* Senior-level position 0.000 -0.007 -0.014 0.009 High-level position 0.010 -0.015 -0.024 0.034** 2. Work Organization Academic 0.001 0.012 0.012 0.005 Government 0.017** 0.012 0.020* 0.016* Low-income countries 0.025** 0.035** 0.014 0.017* Proficiency in language of instruction 0.047** 0.039 0.036 0.063** Technical terminology 0.003 -0.024 0.018 -0.011 Focus country -0.002 0.013 0.007 -0.009 B. Activity features 1. Mode of delivery Blended learning -0.011 -0.016 -0.012 -0.011 Distance learning -0.012 -0.001 -0.013 -0.027** Electronic learning -0.025* -0.001 -0.020 -0.043** 2. Division WBIHD -0.002 -0.012 0.014 -0.014 WBIKD -0.068* -0.040 -0.039 -0.082* WBIPR 0.013 -0.001 0.033** 0.005 WBIEN -0.009 -0.007 0.005 -0.023* WBIFP -0.003 0.032 -0.020 -0.020 3. Fiscal year FY01 -0.052** -0.026 -0.066** -0.031 FY02 -0.039** -0.022 -0.053** -0.033 FY03 -0.035* -0.014 -0.026 -0.042* FY04 -0.029* -0.013 -0.029 -0.032 Follow up 0.021** 0.011 0.015 0.028** Action plan 0.058** 0.049** 0.057** 0.059** Relevance 0.353** 0.411** 0.346** 0.341** Note: ** p<.01; * p<.05 37 Explanation of simulation methodology used for comparing effectiveness and impact in FY01 and FY05 Potential bias: There is a potential bias in comparing data in FY01 and FY05 without controlling for country income level differences. The problem is that in FY01 data exist primarily for participants from middle-income countries, and in FY05 data exist solely for participants from low-income countries. A direct comparison of the raw averages in the two years would not tell us whether the effect is due to country income level differences or time. The challenge in comparing the two fiscal years is that we need to parcel out the effects of country income level. Solution: The report makes comparisons between FY01 and FY05 based on the values simulated by the regression model utilizing the entire sample of respondent data, controlling for a variety of factors related to effectiveness and impact including activity and participant characteristics and country GNI. Methodology: We use the power from the model based on all the data to estimate what the effectiveness ratings would be if we had data in all countries. We simulated the relationship between country income on the one hand and effectiveness and utilization ratings on the other, based on the model holding all variables constant. Using the regression results in Table C3 we calculated estimated mean ratings of effectiveness and utilization, based on the entire database of participant responses between FY01-FY05. 19 Based on these estimated values, we calculated the percentage change. After comparing the percentage change at the various levels simulated for country-income level, we used the most conservative estimate of change which was when country income level was set to low income status. The advantage to using this approach is that it takes into account the relationship of the variables across all years of data. Therefore, we can draw conclusions and make generalizations about effectiveness and utilization in FY01 and FY05. 19 Technically, this means we use the coefficients from the full model to calculate estimated values for effectiveness and utilization, holding country income level and other variables constant. The coefficients used in the calculations are general values that are not based on any specific fiscal year; they are the result of 5 years of data that capture the overall relationship between the dependent and independent variables over the 5 year duration. 38 Table C4. Ordinary least squares regression results predicting relevance of WBI's learning activities, FY01-05 Variables Beta Coefficient N 2922 R2 0.028 Intercept 0.696* Fiscal Years FY02 0.056* FY03 0.057* FY04 0.063* FY05 0.092* Low-income countries 0.042* Note: * p<.01 39 Table C5. Ordinary least squares regression results predicting effectiveness of WBI learning activities, by fiscal year Variable FY04 FY03 FY02 N 829 923 627 R2 0.310 0.360 0.290 Intercept 0.269** 0.315** 0.381** Participant characteristics Position Mid-level position 0.019 0.015 -0.016 Senior-level position -0.005 -0.006 0.005 High-level position 0.013 -0.021 0.036 Work organization Academic 0.003 -0.013 0.033 Government 0.025* -0.007 0.034* Low-income countries 0.014 0.045 0.032 Proficiency in language of instruction 0.095** 0.041 -0.001 Technical terminology 0.034 0.004 -0.001 Focus country 0.017 -0.028** 0.013 Mode of delivery Blended learning 0.034 0.097 -0.027 Distance learning -0.029* -0.009 Electronic learning -0.001 -0.015* -0.127 WBI managing division WBIHD 0.017 -0.021 -0.018 WBIKD 0.120 -0.103** WBIPR 0.018 0.015 0.006 WBIFP -0.020 -0.021 0.014 WBIR -0.050 0.017 Follow up 0.020 0.039** 0.007 Action plan 0.032** 0.058** 0.096 Relevance 0.376** 0.405** 0.310 Note: ** p<.01; * p<.05 40 Table C6. Two-stage least squares regression results predicting WBI impact (utilization of WBI-acquired K&S) Variables Overall use Academic use Operational use N 2923 2923 2923 R2 0.290 0.210 0.260 Intercept 0.102** 0.124** 0.086** Predicted effectiveness 0.736** 0.728** 0.740** Position Reference: Other (junior and entry level) Mid-level position 0.010 0.004 0.015 Senior-level position 0.007 0.004 0.009 High-level position 0.040** 0.022 0.048** Fiscal Year FY01 -0.080** -0.077** -0.086** FY02 -0.016 -0.008 -0.020 FY03 0.011 0.001 0.010 FY04 -0.001 -0.001 0.003 Follow-up 0.004 0.008 0.007 Low-income countries 0.014** 0.014** 0.014** Note: ** p<.01; * p<.05 41