31310 The Impact of WBI Activities, FY02-03, on Participants from Yemen: A Baseline Assessment Heidi S. Zia Shreyasi Jha WBI Evaluation Studies EG04-92 The World Bank Institute The World Bank Washington, D.C. July, 2004 ACKNOWLEDGEMENTS This report was prepared for the World Bank Institute (WBI) under the overall guidance of Marlaine Lockheed, Manager, WBI Evaluation Group. The team was led by Heidi Zia of the Institute Evaluation Group (IEG). The authors thanks Mahmood A. Ayub, Country Director of Djibouti, Egypt, and Yemen, and Robert E. Hindle his support and his valuable guidance; and Yahia Ahmed Raja'a for administering the participant surveys. We would also like Humberto S. Diaz for his help with formatting this document. WBI Evaluation Studies are produced by the WBI Evaluation Group (WBIEG) to report evaluation results for staff, client, and joint learning events. An objective of the studies is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at: http://www.worldbank.org/wbi/evaluation/puball.htm. Vice President, World Bank Institute Ms. Frannie Léautier Manager, Institute Evaluation Group Ms. Marlaine Lockheed Task Manager Ms. Heidi Zia ii ACRONYMS WBI World Bank Institute FY03 Fiscal Year 2003 FY03-04 Fiscal Year 2003 to 2004 CAS Country Assistance Strategies AFR Africa Region EAP East Asia and Pacific Region ECA Europe and Central Asia Region LCR Latin America and the Caribbean Region MNA Middle East and North Africa Region SAR South Asia Region MDGs Millennium Development Goals OLS Ordinary Least Squares NGOs None Government Organizations WHO World Health Organization UNICEF United Nation Children's Fund CRS Client Registration System iii iv TABLE OF CONTENTS ACKNOWLEDGEMENTS ................................................................................................ ii ACRONYMS..................................................................................................................... iii TABLE OF CONTENTS.....................................................................................................v EXECUTIVE SUMMARY .............................................................................................. vii Methodology ................................................................................................................ vii Main Messages of the Evaluation............................................................................... viii Findings from the CRS & WBI Participants............................................................... viii Findings from Counterparts ...........................................................................................ix Key Findings from Focus Group Meetings with Participants........................................ix Recommendations ...........................................................................................................x 1. INTRODUCTION ...................................................................................................... 1 Organization of the Evaluation...................................................................................... 2 2. EVALUATION QUESTIONS AND METHODOLOGY.......................................... 3 Objectives....................................................................................................................... 3 The Participant Survey................................................................................................... 5 Participant Survey Response Rates................................................................................ 6 3. PARTICIPANTS AND ACTIVITIES........................................................................ 7 Participants' Characteristics........................................................................................... 7 Features of the WBI Learning Activities ....................................................................... 9 Design ................................................................................................................... 10 Activity Follow-up................................................................................................ 11 4. RELEVANCE OF WBI LEARNING ACTIVITIES ............................................... 12 Alignment with CAS .................................................................................................... 12 Relevance for Yemen................................................................................................... 12 Activity Alignment with Millennium Development Goals.......................................... 14 "Country Focus"........................................................................................................... 15 v 5. EFFECTIVENESS OF THE WBI LEARNING ACTIVITIES................................ 17 Overall Effectiveness ................................................................................................... 17 Correlates of Effectiveness........................................................................................... 19 6. USEFULNESS OF THE WBI LEARNING ACTIVITIES...................................... 20 Rating of Usefulness .................................................................................................... 20 Predictors of Use of New Knowledge and Skills......................................................... 22 Facilitators and Barriers to the Utilization of Learning ............................................... 24 Comparing WBI to Similar Non-WBI Activities........................................................ 25 7. IMPACT OF WBI LEARNING ACTIVITIES ........................................................ 26 Gauging Impact............................................................................................................ 26 Predictors of Impact of WBI Learning Activities ........................................................ 28 8. LISTENING TO OUR COUNTERPARTS.............................................................. 30 Overall View of Effectiveness ..................................................................................... 30 WBI Outcomes............................................................................................................. 31 Responsiveness to Operations ...................................................................................... 31 9. LISTENING TO PARTICIPANTS .......................................................................... 32 Views on Relevance and Usefulness............................................................................ 32 Measures to Increase Impact in Yemen....................................................................... 33 10. CONCLUSIONS AND RECOMMENDATIONS ............................................... 35 Recommendations ........................................................................................................ 38 ANNEX Annex 1: World Bank Institute (WBI) Country Focus Evaluation Questionnaire ...... 41 Annex 2: List of FY02-03 Activities............................................................................ 48 Annex 3: Focus Group Questions ................................................................................ 49 Annex 4: Variable Description and Summary Statistics - Regression Variables ........ 50 vi EXECUTIVE SUMMARY World Bank Institute (WBI) programs are organized around priority themes or initiatives linked to the World Bank's corporate aims and the Millennium Development Goals. Over the past two years, however, WBI modified its strategy and structure by adding a country pillar to the previous thematic and knowledge/learning pillars. In line with the new country focus approach, in FY03, evaluations focused on the following five countries: Brazil, Egypt, Russia, Sri Lanka, and Thailand. In FY04, evaluations have been carried out in Burkina Faso, Guatemala, Indonesia, Kenya, Nigeria, Tajikistan, and Yemen. This study reports the findings of the retrospective study of WBI's Yemen program and will serve as the baseline for the prospective evaluation work in that country. The evaluation adopts the conceptual model of estimating relevance, effectiveness and impact used in the study, "Impact of WBI Activities, FY02-03, on Participants from Egypt: A Baseline Assessment".1 This evaluation was undertaken to answer the following key questions: · What is the relevance of WBI learning activities to the country's needs? · What is the effectiveness of WBI learning? · What is the impact of WBI learning activities? · What features of WBI Programs affects effectiveness and impact? METHODOLOGY The evaluation was based on several sources and types of information, including participant questionnaires, participant focus group discussions, interviews with Operational Staff working on Yemen (in Washington and Sana'a) as well as their counterparts in the Yemeni government, and relevant Operational and WBI documents. Based on the Client Registration System, the participant list had originally identified 65 Yemeni attending a total of 13 WBI activities. Excluding invalid entries, the adjusted response rate was 54 percent.2 1 Zia, Heidi, et al., "Impact of WBI Activities, FY02-03, on Participants from Egypt: A baseline Assessment" (October, 2003) 2 Excluding invalid entries, the number of total eligible respondents is reduced to 62, of which 28 were successfully interviewed. The limited number of observations for the quantitative analysis underlines the need to interpret the empirical results with caution. Perhaps, a better assessment of WBI's role in Yemen is the qualitative analysis of interviews with counterparts and focus groups meetings. vii MAIN MESSAGES OF THE EVALUATION Findings from participant surveys, focus groups, and interviews with counterparts convey three simple messages: · WBI has done too little in Yemen; and in order to make a difference, it needs to start offering local offerings in Arabic; · WBI is uniquely positioned in Yemen; there is no other learning institute like WBI in Yemen; · Both the Yemeni participants and our Operational counterparts welcome WBI's country focused approach to Yemen. These messages stem from the findings ­ listed below - from participants, counterparts and focus groups: FINDINGS FROM THE CRS & WBI PARTICIPANTS · Except for one video-conference offering, none of the activities attended by Yemeni participants took place in Yemen3. Half of the activities took place in Arabic speaking countries (Lebanon, Egypt, Jordan and Tunisia) and half in Europe and North America. The language of instruction was English in eleven of the activities and French in the remaining two. None of the activities were instructed in Arabic. · In FY02-03, 75 percent (10 out of total 13) of WBI's activities were aligned with Yemen's CAS, addressing all four CAS priorities/ objectives ­ private sector development, water resource management, human capital and governance. · Although most WBI activities were aligned with the CAS, the percentage of Yemeni participants to the total participants of a typical activity was only 7 percent. · Two-thirds of the participants reported that there was no follow-up by WBI (68 percent) and only 15 percent of the participants contacted WBI for follow-up. · Forty six percent of all participants perceived the topics covered to be highly relevant to the needs of their country while 54 percent perceived it only moderately relevant. The perceived relevance of WBI learning activities to the needs of the Yemeni participants' work is perceived to be high by 56 percent of the participants and moderate by 41 percent of the participants. · Perceived effectiveness of the WBI activities in Yemen was slightly above the average (5.1 in a scale of 1-7). Participants found WBI activities to be mostly effective in: 1) raising awareness/understanding of development issues, 2) providing knowledge and skills, and 3) helping participants develop 3The video conferenceing from Washington to Yemen, Egypt, Jordan, Lebanon, Gaza and West Bank. viii approaches/strategies to address the needs of their organization. Participants found WBI activities to be least effective in: 1) providing strategies/approaches to address the needs of their country and 2) providing networking opportunities. · Although, almost 90 percent of the participants reported that they discussed the issues raised during the WBI learning activities afterwards, either occasionally or thoroughly, at work; they gave overall usage a very low rating (3.63 on a scale from 1-7). The most consistent determinants of usage are perception of course effectiveness and being a senior manager. · Nearly 70 percent of participants indicated that the WBI learning activity they attended led to an overall change in the specific area of the activity. When applicable, WBI learning activities have the strongest positive influence/ impact on country development strategy, teaching and raising public awareness. The positive influence is least in the area of legislation or regulation and developing collective initiatives. FINDINGSFROM COUNTERPARTS · According to almost all of the Operational counterparts (and their counterparts in the Yemeni government) interviewed, WBI has not been effective in Yemen. They all emphasized, however, that WBI is uniquely positioned in Yemen but it needs to expand its activities and develop local partnerships in order to make a difference in Yemen. · Interviewees agreed that in order to be more effective, WBI must offer local offerings in Arabic on a more regular basis since language is seen as a major barrier for the participants. To make the courses accessible, WBI needs to offer courses locally. Operational colleagues in the field stressed that in future WBI needs to deliver more `tailored training activities' focusing on areas that are most relevant locally ­ governance, poverty reduction and private sector development. They expressed the view that "Yemen is on the frontline and WBI is having no impact on Yemen". · During FY02-03 too little was done to assess the alignment of WBI activities with the country needs. The areas of weakness in Yemen, as described in the CAS, are water, human capital, private sector development and governance. WBI courses do target these areas, but "the courses are too few with limited Yemeni participants". KEY FINDINGSFROM FOCUS GROUP MEETINGS WITH PARTICIPANTS · Participants identified four key obstacles that hinder the use of knowledge and skills of WBI activities in Yemen: 1) Language; 2) Rigid and hierarchical work environment; 2) General resistance to change which involves changing the legal framework and procedures; and 4) Lack of team training. As one participant said, "To apply new concepts, one must change the existing ix procedures, and for that we need to build alliances, and to build alliances we need to be trained together". · Focus group participants had several messages for WBI task managers to improve the effectiveness and impact of WBI activities in Yemen: 1) offer more courses in local language or improve translation facility during courses; 2) Offer more courses locally; 3) Replace regional offerings with country focused activities; 4) Offer team training as opposed to individual training; 5) Encourage and support participants to disseminate their new knowledge and skills; and 6) Provide systematic follow-ups. RECOMMENDATIONS To improve the institutional impact and effectiveness of WBI activities in FY04 and beyond, and in short to make a difference, WBI's presence in Yemen should, in the first instance, be expanded; and second, become much more country-specific. This involves adopting a systematic approach to capacity building in Yemen. Some recommendations follow. · WBI must increase its offerings, provide a systematic set of interventions, hold activities locally in Yemen, target participants in teams and cohorts, and offer its courses in Arabic. · Learning activities must be designed with the country's needs and constraints in mind. To better assess the local needs and constraints of the Yemenis and move toward this country-focused approach, WBI should bridge the perceived "disconnect" with Operations, and align its learning programs with Operational project. · WBI should establish an institutionally agreed upon approach to systematic follow-ups to its learning activities. It has to design activities that build knowledge and skills at continuous levels, carefully targeting cohorts of participants, maintaining on-going contact with participants, and if possible promote training of trainers. x 1. INTRODUCTION 1.1 World Bank Institute (WBI) programs are organized around priority themes or initiatives linked to the World Bank's corporate aims and the Millennium Development Goals. Over the past two years, however, in order to respond more effectively to the challenges of achieving the Millennium Development Goals and in support of the World Bank's corporate aims, WBI modified its strategy and structure by adding a country pillar to the previous thematic and knowledge/learning pillars. 1.2 In FY02, the Institute adopted a "country focus" for its activities, linking them directly to the countries' own poverty reduction strategies and to the Bank's country assistance strategies. Training and other learning products continue to be the main element in WBI's package of assistance, but they are increasingly conceived as part of a menu of services which includes diagnostics and advisory work, strengthening of learning and service delivery organizations, and consulting on capacity enhancement issues. This new country focus is aimed at increasing the efficacy and impact of the WBI assistance and to ensure that these results are sustainable. 1.3 In line with the country focus approach, a list of 45 priority countries was developed for country focus work in response to the new themes and directions for poverty reduction, agreed at the Strategic Forum 2001. From among this list, 12 focus countries were selected for intensive, multi-sectoral, capacity-building activities over the next three years. In FY03, the evaluations focused on the following five countries: Brazil, Egypt, Russia, Sri Lanka, and Thailand. In FY04, evaluations have been carried out in Burkina Faso, Guatemala, Indonesia, Kenya, Nigeria, Tajikistan, and Yemen. These countries, representing all six regions, were selected on the basis of three criteria: their readiness as clients; WBI's relationship with the countries and in-country partners; and the "rapid results" collaboration mode (or the overall discipline with translating intent/learning into action). Table 1 lists the selected focus countries in FY03 and FY04. Table 1.1: Focus Countries Selected for Evaluation Region FY04 FY03 AFR Burkina Faso, Kenya, and Nigeria --- EAP Indonesia Thailand ECA Tajikistan Russia LCR Guatemala Brazil MNA Yemen Egypt SAR --- Sri Lanka 1.4 The overall objectives of WBIEG (WBI Evaluation Group) in WBI's country- focused efforts are three fold: (1) to examine the outcomes and impact of WBI programs at the individual, institutional and country levels; (2) to establish a baseline for further monitoring and evaluation activities; and (3) to form recommendations for improvement of the WBI programs. 1 ORGANIZATION OF THE EVALUATION 1.5 Building upon the experience of the first set of country-focused impact evaluations, this evaluation will focus on FY02 and FY03 WBI learning events for participants in Yemen. The evaluation strategy is based on WBI's new direction in providing services that are more country-focused and aligned to the needs of the country. 1.6 These findings are presented in the following nine chapters, which discuss the following issues on the basis of responses to questionnaire, as follows: Chapter 2: Evaluation questions and methodology, Chapter 3: Participants characteristics and features of WBI learning activities, Chapter 4: Relevance of WBI learning activities to Yemen's needs, Chapter 5: Effectiveness of WBI learning activities, Chapter 6: Usefulness of the WBI learning activities, Chapter 7: Impact of WBI learning activities, Chapter 8: Listening to Counterparts Chapter 9: Listening to Participants Chapter 10: Conclusions and Recommendations. 2 2. EVALUATION QUESTIONS AND METHODOLOGY 2.1 In FY02-03, 65 Yemenis participated in 13 WBI-sponsored activities; seven of the activities, attended by nearly three-quarters of the participants were in the HD Sector.4 Four of the activities were conferences, including MDF-4 2.2 Except for one video-conference activity from Washington, none of the 13 WBI activities with Yemeni participants were held in Yemen. Half of the activities took place in Arabic speaking countries (Lebanon, Egypt, Jordan and Tunisia) and half in Europe and North America. The language of instruction was English in eleven of the activities and French in the remaining two. None of the activities were instructed in Arabic. 2.3 Table 2.1 below lists the programs and number of activities with Yemeni participants. Health and Population was the largest program with 5 activities. The second largest program over the course the two years under study was Attacking Poverty with two offerings and 7 Yemeni participants. The remaining programs had six or fewer participants5. Table 2.1: WBI Programs and Number of Activities with Participants from Yemen Number of Number of Activities Yemeni Name of Program (FY02 and FY03) with Yemenis Participants Health and Population 5 37 Attacking Poverty 2 7 Education Program 1 6 Social Protection Learning Program 1 5 Market Solutions for Development 1 5 Financial Sector s 1 2 Knowledge for Development 1 2 Sustainable Development: Environmental Management 1 1 OBJECTIVES 2.4 The evaluation was undertaken to answer the following key questions: · What is the relevance of WBI learning activities to the country's needs? · What is the effectiveness of WBI learning? · What is the impact of WBI learning activities? · What features of WBI Programs are related to effectiveness and impact? 4Source of Yemeni participants by activity/program is the CRS database. 5Annex 3 provides the list of WBI activities with Yemeni participants in FY02-03, scheduled country of the activity and number of Yemeni participants. 3 2.5 Specifically, impact is measured by the degree to which the activity influenced or led to changes in the areas of research, training, legislation and regulation, and country development strategies. It measures the contribution of WBI learning programs/activities to building in-country capacity. Effectiveness is the degree to which an activity raised participant awareness, increased their knowledge, and helped them develop strategies to address country needs. In addition to the above questions, the evaluation also seeks to identify factors that relate to the effectiveness, use and impact of WBI interventions. 2.6 Table 2.2 lists the main evaluation questions, along with their sub-questions and sources of information pertaining to each question6. The evaluation was based on several sources and types of information, including: participant questionnaires, participant focus group discussions, interviews with Operational Staff working on Yemen (from Washington D.C. and Sana'a) and their counterparts in the Yemeni government, and relevant Operational and WBI documents. Table 2.2: Evaluation Questions and Source of Information Evaluation Questions Source I. What is the relevance of WBI learning activities to the country's needs? To what degree are the topics covered by WBI Participant Surveys & Interviews w/ learning activities relevant to the country's Country Staff specific needs? Do WBI learning activities address the Participant Surveys & Interviews w/ country's current issues and needs? Country Staff II. What is the effectiveness of WBI learning? How effective are WBI learning events? Participant Surveys III. What is the impact of WBI learning activities? To what degree are the participants utilizing the Participant Surveys & Interviews knowledge/skills they learned through WBI with Country Staff learning activities/products? In what areas? How useful is the activity? Has the activity influence or contributed to Participant Surveys & Interviews w/ changes in the relevant sector/area? Country Staff IV. What features of WBI programs are related to effectiveness and impact? What are the country specific facilitators and Participant Surveys & Interviews w/ barriers to utilizing the knowledge/skills? Country Staff What features of WBI programs are related to Participant Surveys & Interviews w/ effectiveness? Country Staff 2.7 To answer the evaluation questions, the study adopts (with some modifications) the conceptual model of estimating effectiveness and impact used in 2003 by Zia et al.. In 6The unit of analysis is the participant. 4 the conceptual model, the ultimate impact of the WBI learning activities are measured by two indicators: · The extent to which learning is used in the areas of teaching/research, policy/ legislation, at work and, in raising others' awareness of development issues; · The extent to which these learning activities have influenced teaching/research, policy legislation and, work practices at the level of organizations. 2.8 Impact of the WBI learning activities is determined by four sets of variables: · Characteristics of participants in WBI learning activities together with main features of these learning activities. This set of variables is conceptualized to have a direct effect on impact. In addition, they have an indirect effect on impact as they are considered determinants of perceived relevance and effectiveness; · Course characteristics such as level of attendance and design for the participants; · Perceived relevance of WBI learning activities and effectiveness of these activities, and; · Facilitators and barriers at the country level and at the level of the work organization. THE PARTICIPANT SURVEY 2.9 Questionnaires were administered to former participants in WBI-sponsored activities, Operational task managers and WBI staff. Focus group discussions and documentary analysis were utilized to probe the potential impact of the learning activities on sectoral policies, programs, and practices. Interviews were carried out with partners in the field to examine in more detail WBI's contributions and comparative advantages in enhancing country capacity. 2.10 The structured questionnaire (Annex 1) administered to former participants in WBI-sponsored activities comprised four main sections: · Section I: Relevance of the WBI learning activities to the needs of the country as perceived by the participants and also as they relate to the Millennium Development Goals. · Section II: Usefulness of WBI learning activities encompassing effectiveness, overall change and area-specific change led by these activities, use of learning in different areas, facilitators and barriers to the use of learning. · Section III: Comparison of WBI learning activities with similar activities offered by other organizations. · Section IV: Characteristics of the learning event and of the participant. 5 PARTICIPANT SURVEY RESPONSE RATES 2.11 The participant list had originally 65 participants from Yemen attending a total of 13 WBI activities in 9 programs (Annex 2 gives the list of activities offered for Yemeni participants in FY02-03). Given the small population size, we chose the entire population for the purposes of this study. Excluding invalid entries, the number of total eligible respondents is reduced to 52, of which 28 were successfully interviewed. The adjusted response rate was 54 percent.7 7Invalid entries comprised repetitions, participants who had moved from Sana'a with no lead and participants who did not attend the learning activity. 6 3. PARTICIPANTS AND ACTIVITIES 3.1 This chapter describes the participants' characteristics and the features of WBI learning activities they attended. The CRS database was used extensively to complete and complement survey findings on both participants and activities. PARTICIPANTS' CHARACTERISTICS 3.2 The participants' survey collected basic information on the individual characteristics of the respondents, namely: · Demographic characteristics: gender, proficiency in the language of instruction, and proficiency in the terminology used in the learning activity; · Characteristics related to work: type of organization, type of work, position at work; · Interest in the learning activity: contact WBI for follow-up issues or questions on content of the activity. 3.3 Most participants of most activities were male. Of all 28 Yemeni respondents, around one-third of them (36 percent) are females. According to the CRS, this percentage of female respondents mirrors the actual percent of Yemeni female participants. However, the distribution of female participants was highly uneven across the activities under study. Only three of the 13 WBI activities included female participants; two of the three activities had only one female participant and the third ­ a gender related activity ­ had sixteen. 3.4 Proficiency in the language and terminology of instruction was relatively high, but not consistent across offerings. The language of instruction was English for eleven of the thirteen activities.8 In survey reports, a little over one-half of the participants (54 percent) indicated high proficiency in the language of instruction while the majority of the remaining one-third reported a moderate level of proficiency (Figure 3.1). Contrary to the survey findings, focus group findings reported offerings that suffered from language as a barrier9. Since the focus groups represented individual activities, we conclude that for at least some offerings language proficiency was an issue. The distribution of the participants by the level of proficiency in the terminology used in the activity is slightly different: 44 percent reported high proficiency, 52 percent moderate and 4 percent indicated a low level (Figures 3.2). 8The other two offerings were held in Tunisia and France and the language of instruction was French. 9See Chapter 9 for focus group findings on language barriers. 7 Figure 3.1: Percent distribution of Figure 3.2: Percent distribution of participants by proficiency in the participants by proficiency in language of instruction technical terminology Low Low 7% 4% High 44% High Medium 54% 39% Medium 52% 3.5 Most participants worked with government organizations. As shown in figure 3.3 nearly two-thirds of the participants (65 percent) work with the government. Of these, the largest share (57 percent) work for the national government, followed by an equal number of participants who work for either regional/provincial governments or for local governments (4 percent for each category). The second largest categories of participants are affiliated to universities or research institutions (11 percent) and non-governmental organizations (11 percent). Participants from the private sector contribute only 4 percent each to the total number participating in the designated learning activities. Figure 3.3: Percent distribution of participants by type of organization 60 57 50 40 30 20 11 11 11 10 4 4 4 0 National/ University/ NGO/ non- Other Private Sector Provincial/ Local/ Central Govt Research profit regional govt municipal Institution govt 3.6 Most participants were involved in administrative or managerial work (Figure 3.4). The highest percent of participants (53 percent) are predominantly involved in management/administrative jobs, followed by an equal number of participants who are involved in services and policy-making/legislation (18 percent in each category). Only 7 percent and 4 percent of total participants are involved in research and teaching, respectively. 8 Figure 3.4: Percent distribution of participants by type of work Research Services 7% 18% Policy- making/legislation Teaching 18% 4% Management/admi nistration 53% Figure 3.5: Percent distribution of participants by position at work Junior 11% Highest 26% Middle 26% Senior 37% 3.7 Over fifty percent of the participants were senior managers or senior staff (Figure 3.5). WBI activities tend to include individuals in relatively high positions, perhaps those in a position to make decisions. In Yemen, participants belonging to the highest management position and senior managers/professionals together comprised nearly two- thirds of all participants. Junior professionals/technical staff and middle managers represent 11 percent and 26 percent respectively of all participants. FEATURES OF THE WBI LEARNING ACTIVITIES 3.8 Participants in the survey provided information concerning 13 WBI learning activities offered between the summer of 2001 and the summer of 2003 (Annex 2 lists the course offerings). The activity information falls under two general categories ­ activity design and follow-up: Design: · Duration of the activity 9 · Whether the activity was specially designed for participants from Yemen; · Whether it involved development of an action plan/strategy to apply the knowledge provided; · Mode of instruction (e.g. video sessions, face-to-face, web-based, etc.) Follow-up: · Whether the participants were provided with a contact list of participants; · Whether and how did the participants use the contact information, and; · WBI's helpfulness in addressing participants' follow-up issues/questions. 3.9 These design and follow-up information are used later in the report as potential determinants of activity relevance, effectiveness, usefulness and impact. Design 3.10 The majority of the learning activities extended over 3-5 days (68 percent) and a third of the activities lasted between 6 to 17 days (32 percent). There were no activities shorter than 3 days. Except for one video-conference offering, none of the activities took place in Yemen10. Figure 3.6: Percent distribution of learning activities by duration 80 68 70 60 50 32 40 30 20 10 0 3to5days 8to17days 3.11 Only about 15 percent of the participants reported "impressions" that the activity they attended seemed to be specially designed for participants from Yemen. The fact that participants did not view WBI activities as country-specific, is not a surprising finding, given that WBI's course designs were designed for the region and were not country- focused. 10One activity titled "Flagship Gender in Program and Project Monitoring and Evaluation" mentions Yemen as its scheduled country (Annex 2), but this activity was a video conference from Washington to Yemen, Egypt, Jordan, Lebanon, Gaza and West Bank 10 3.12 WBI's adoption of country-focused approach is welcomed by focus group participants who repeated the importance of tailoring courses to the unique needs and position of Yemen in the Middle East. As one focus group participant noted, "There is a need for tailored programs for Yemen. It is great to have the international experience. But to make change, we need to have a program tailored for Yemen." Participants agreed with one participant from the private sector who said, "the activities were not created with the Yemenis in mind." 3.13 Participants developed action plans in about one half of the learning activities. The participants developed action plan/strategy to apply the knowledge provided in only 52 percent of these learning activities. 3.14 The mode of instruction of learning activities was almost equally composed of one the four types: classroom (face to face), conference, video session (for distance learning activities) or a mix of face-to-face and video sessions. Fact to face mode of instruction was more common than any other type (nearly 29 percent). Activity Follow-up 3.15 The contact information provided was considered moderately useful. To gauge the sustainability of the learning activities, participants were asked if: (a) they were provided with a contact list of other participants and (b) they used the contact list (to either continue discussion or organize joint follow-up activities). Contact information of other participants was provided to 48 percent of the participants. 3.16 Follow-up contacts were not used extensively. Of those participants who were provided with contact information , a large proportion of them (41 percent) never used the contact information, 42 percent used the contact information to organize discussions and 17 percent used the information to organize follow-up activities. 3.17 Two-thirds of the participants reported that there was no follow-up by WBI (68 percent). In focus group meetings, participants agreed WBI courses are "informative, but not enough to produce a change. To create change or impact , future WBI learning activities must have systematic follow-ups". Only 15 percent of the participants contacted WBI to follow-up on the issues or questions on the content of the learning activity. Table 3.1: Percent distribution of participants based on their response on activity follow-up Yes No Did WBI follow -up? 32 68 Did Participants contact WBI for follow -up? 15 85 11 4. RELEVANCE OF WBI LEARNING ACTIVITIES 4.1 We examined the relevance of WBI learning activities to the country's needs and to the needs of the participants' work in four ways: · Alignment of the activities to the Country Assistance Strategy (CAS); · Participants' assessment of the learning activities' relevance for Yemen's development needs; · Relevance to WBI activities in Yemen to the MDGs and the Bank's corporate priorities; and · Empirical country focus assessment ALIGNMENTWITH CAS 4.2 In FY02-03, 75 percent (10 out of total 13 activities) of WBI's activities were aligned with Yemen's CAS priorities. Table 4.1 below lists the CAS objectives along with the number of activities and percent of Yemeni participants for each objective. 4.3 Although most WBI activities were aligned with CAS, it is important to note that the percentage of Yemeni participants to the total participants of a typical activity was only 7 percent. Table 4.1: CAS Objectives and Percent Participants from Yemen, FY02-03 Participants from Yemen as a percent of CAS Objectives N of activities total participants Promoting Investment and Private Sector 1 9 Development Environment ­ Water Resources Management 1 2 Human Capital - Health and Education 5 13 Governance 3 3 Total: 10 Average:7% RELEVANCE FOR YEMEN 4.4 All Yemeni participants in WBI learning activities that took place between the summer of 2001 and the summer of 2003 were asked about the degree of relevance of the topics covered in the activity to Yemen's specific needs and to their work. Relevance is measured on a 7-point scale ranging from "Not relevant at all" to "Extremely Relevant".11 1.1 11 Later in the report, for analytical purposes, responses related to perceived relevance of the topics covered were regrouped into the following three categories: Not relevant at all (scale points 1 and 2); Moderately relevant (scale points 3,4 and 5), and Extremely relevant (scale points 6 and 7). 12 4.5 On average, the relevance of WBI learning activities to the specific needs of Yemen was perceived as moderately high with a mean rating of 5.3. This is comparable to the average rating of relevance in the five countries studied in FY03, 5.4.12 Fewer than half (46 percent) of all Yemeni respondents perceived the topics covered to be highly relevant to the needs of their country; and the rest (54 percent) perceived only a moderate degree of relevance. None of the respondents perceived the activities to be irrelevant to their country. 4.6 The relevance of WBI learning to the participants' work was almost the same with mean rating of 5.2. The perceived relevance of WBI learning activities to the needs of the Yemeni participants' work is perceived to be high by 56 percent of the participants and moderate by 41 percent of the participants. Only 3 percent of Yemeni participants perceived the topics covered by WBI learning activities to be irrelevant to their work . The distribution of perceived relevance of learning activities to participants' work and Yemen's needs is shown in Figure 4.1. Figure 4.1: Percent distribution of participants by the level of perceived relevance of learning activities to Yemen's specific needs and participant's work 100% 18 25 80% 7 32 21 6 60% 5 4 40% 18 21 3 2 18 20% 29 11 0% 4 4 Relevance to work Relevance to country needs 4.7 Participants from university/research institutions and the private sector perceive relevance of WBI activities as high. The relationship between perceived relevance of the topics covered to a participants' work and the type of organization that a participant works for is significant. Participants from the private sector and university of research institutions perceive the topics covered in the WBI learning activity to be highly relevant to the development needs of Yemen. In contrast, participants from government agencies (national, regional or local governments) perceive the relevance of WBI activities to their 12See para. 1.3. The five countries are Brazil, Egypt, Russia, Sri Lanka, and Thailand. 13 work as moderate. In drawing these conclusions, we should be cautious because of our limited sample size. 4.8 The distribution of perceived relevance to Yemen's specific needs by the type of participant's organization is presented in Figure 4.2. Participants from the private sector perceived the WBI learning activities to be highly relevant to their country's specific needs while participants from local and regional/provincial governments ranked the activities as being only moderately relevant to their country's needs. The distribution of perceived relevance of WBI activities to Yemen's specific needs is almost identical to the perceived relevance of activities to participant's work. Figure 4.2: Percent distribution of perceived relevance of learning activity to Yemen's needs by the type of participant's organization 0 0 100% 80% 33 50 67 60% 100 100 100 40% 67 20% 50 33 0% Local/ Provincial/ Private NGO/ non- University/ National/ municipal regional g Sector profit Research Central Govt govt 0 Moderate High ACTIVITY ALIGNMENTWITH MILLENNIUM DEVELOPMENT GOALS 4.9 The participant survey investigated whether the developmental goals of WBI learning activities were closely related to the ten WB Millennium Development Goals (MDGs) and corporate priorities, and if so, the extent to which the key issues in achieving each goal were addressed. The development goals are presented in Table 4.2. Table 4.2: Millennium Development Goals and Corporate Priorities Eradicate Extreme Poverty Achieve Universal Primary Education Promote Gender Equality and Empower Women Reduce Child Mortality Improve Maternal Health Combat HIV/AIDS, Malaria, and Other Diseases Ensure Environmental Sustainability Develop Global Partnerships for Development Ensure Water Sanitation and Supply Improve Investment Climate and Finance Promote Trade 14 4.10 WBI learning activities are most related to four MDGs: (a) developing global partnerships, (b) improving investment climate, (c) promoting gender equality, and (d) eradicating extreme poverty.13 Over-half of the participants reported that the activities they attended are related to these four goals. The relationship between activity content and the remaining development goals and corporate priorities such as combating AID/HIV, water sanitation, and education is very low. Figure 4.3: Percent distribution of participants by whether the learning activity they attended is related to each of the MDGs 100% 29 28 25 35 80% 39 67 67 63 58 58 60% 78 40% 71 72 75 61 65 20% 33 33 38 42 42 22 0% Global Investment Gender Poverty Child Mortality Maternal Environment Trade Education Water HIV/AIDS Partnerships Climate Equality Health Sanitation Not related Related "COUNTRY FOCUS" 4.11 In order to determine whether an activity met the WBI FY03 objective of being "country-focused", we assessed the following aspects of the activity14: · Whether the activity took place in Yemen; · Share of participants who were Yemenis; · Activity alignment with Yemen's CAS priorities; · Activity perceived as being designed especially for Yemenis; and · Activity perceived as being relevant to Yemen's needs. 4.12 At the activity level, we conclude that the WBI activities with the Yemeni participants' were not country focused. While nearly 75 percent of WBI's activities (10 out of total 13 activities) were aligned with Yemen's CAS, none of the activities took place in Yemen (except for video-conference that was targeted to several other countries), and only 7 percent of participants of a typical activity were Yemenis. 13 14These criteria are based on FY01-02 Eygpt Report (Zia, et. al.) 15 4.13 At the participant level, only 14 percent of the questionnaire respondent perceived the activity they attended as designed for Yemeni participants. This is not surprising given that there were no activities in Yemen and the average activity had only 7 percent Yemeni participants. However, almost all of the participants reported that the activity they attended was either highly relevant (46 percent) or moderately relevant (54 percent) to Yemen's unique needs15. 4.14 In sum, the WBI activities for Yemen did not meet three of the five conditions for being country focused. Although, from the participants' perspectives, majority of them responded that WBI's activities are relevant to their work needs and to the needs of their country, at the activity level, there was no sign of an intended effort towards a country- focused program. In the following sections we further evaluate whether perceived country-focus is related to higher effectiveness, usefulness, or impact of the WBI courses. 15Moderately relevant is defined as a ranking of 3,4, or 5 on a scale of 1-7; and highly relevant is a ranking of 6 or 7 on a scale of 1-7. 16 5. EFFECTIVENESS OF THE WBI LEARNING ACTIVITIES 5.1 The survey instrument aimed at measuring six different aspects of effectiveness anticipated for the WBI learning activities: · Raising participants' awareness/understanding of development issues; · Providing participants with knowledge/skills; · Helping participants understand their role in their country' development; · Providing strategies/approaches to address the needs of the participants' organizations; · Providing strategies/approaches to address the development needs of the country; and · Helping in networking among people with common interests. OVERALL EFFECTIVENESS 5.2 Perceived effectiveness of the learning activities was medium to high. Effectiveness of the learning activity with regard to each aspect is measured using a 7- point scale. Responses are then averaged over the six areas and regrouped into three categories: low effectiveness (below 3), moderate effectiveness (between 3-5) and, high effectiveness (above 5). On the average, effectiveness was perceived as medium to high with a mean of 5.1. This rating is slightly higher than the rating based on the average of our five countries which was 5.0. The distribution of respondents by the three effectiveness categories is depicted in Figure 5.1. Figure 5.1: Percent distribution of participants by level of effectiveness Low effectiveness 3% Moderate High effectiveness effectiveness 52% 45% High effectiveness Moderate effectiveness Low effectiveness 5.3 Participants responses regarding the level of effectiveness of the learning activities suggests that these activities seem to be relatively most effective in the areas of: raising awareness/understanding of development issues, providing knowledge or skills, and providing strategies/approaches to address the needs of the participants' organization. 17 On the other hand, WBI learning activities are least effective in providing networking opportunities or in helping participants understand their roles in their country's development. However, as shown in Figure 5.2, the overall differences in the distribution of the level of effectiveness in the designated six areas are not large. This is emphasized by the very close values of the means and the standard deviations of all six variables. Figure 5.2: Percent distribution of participants by the level of effectiveness of WBI learning activities in distinct areas 100% 80% 32 57 54 52 48 45 60% 40% 64 48 50 20% 39 46 48 0% 4 3 5 4 Participant's Knowledge/ skills Strategies/ Understand role in Strategies/ Networking awareness/ approaches for country's development approaches for understanding organization country Aspects of Effectiveness Ineffective Moderately effective Extremely effective 5.4 Focus group discussions of effectiveness did not distinguish between government and private sector participants. All participants indicated that WBI activities were most effective in raising awareness and understanding about development issues and in providing exposure to new skills and knowledge. One focus group participant expressed the view that during the activities they "learned about issues that they had never heard about before". However, most participants agreed that in order to be more effective, the activities need to be targeted specifically towards to needs of Yemen. In addition, participants also revealed that the knowledge provided was too technical and could not be applied easily. 5.5 Factor analysis was applied to the data to reduce the dimensionality of the measurement of the level of effectiveness of the WBI learning activities. Results of the factor analysis emphasize the significant correlations and the minor differences between the levels of effectiveness in the six designated areas. The analysis concludes by extracting one factor only that summarizes the overall level of effectiveness for each participant16. To measure overall effectiveness we use the average of the six effectiveness areas. Averaging the areas is possible because we have complete data on effectiveness. 16 We also performed an alpha reliability test to determine the reliability of using one measure of effectiveness. The Cronbach alpha coefficient is: 0.89, indicating a 0.89 proportion of the true score variance in effectiveness is captured by sum of variance of Items 5a- Items 5f on the participant survey questionnaire. 18 CORRELATES OF EFFECTIVENESS 5.6 Due to the limited number of observations and missing data, a fully specified multivariate regression with all variables of interest could not be performed. Instead, we performed a series of ordinary least square regressions and pair-wise correlations between effectiveness and different aspects of participant and activity characteristics. The correlations and regressions revealed that perceived relevance is highly correlated with activity effectiveness. Due to the limited number of observations for our regression analysis, we do not report the results. However, below are the scatter plots of two of the significant correlates of effectiveness. 5.7 Activities perceived as relevant are also perceived as effective. Scatter plots of the relationship between effectiveness and relevance to work and relevance of country needs showed that there were only a few outliers and there exists a high degree of correlation between relevance to work and relevance to country needs and perceived effectiveness of activities. Figures 5.3 and 5.4 show the scatter plot between perceived relevance and effectiveness. The coefficient of correlation between perceived effectiveness and relevance to work is 0.70*** and between perceived effectiveness and relevance to Yemen's needs is 0.73***. Figure 5.3: Scatter plot of effectiveness and relevance to work 8 7 6 Work to 5 4 3 Relevance 2 1 0 0 1 2 3 4 5 6 7 8 Effectiveness Figure 5.4: Scatter plot of effectiveness and relevance to country needs 8 7 6 Country to 5 4 Needs 3 2 Relevance 1 0 0 1 2 3 4 5 6 7 8 Effectiveness 19 6. USEFULNESS OF THE WBI LEARNING ACTIVITIES 6.1 This chapter discusses the respondents' rating of usefulness based on the frequency to which they used the knowledge and skills acquired in the activity. It also makes an attempt to ascertain the predictors of use, determine its facilitators and barriers, and compare the usefulness of WBI activities to other non-WBI learning activities. RATINGOF USEFULNESS 6.2 One measure of usefulness in this study is participants' development of action plans in training and their subsequent usefulness at work. All participants were asked if they had developed an action plan/strategy to apply the new ideas they learned during the activity and if they did, whether they have actually used it at work. Overall, 53 percent of the participants did develop such an action plan during the activity; 38 percent of the participants actually used part/all of the action plan at work as presented in Figure 6.1. Figure 6.1: Percent distribution of participants by whether they had developed an action plan and used it at work 38% 47% 15% Did not develop action plan developed action and used at work developed action plan did not use at work 6.3 To gauge participants' usefulness of WBI activities, participants were also asked if the issues raised in WBI learning activities have been discussed at work, with local partners, government officials, NGO's or in the media. Figure 6.2 shows that about a tenth of the participants (11 percent) never discussed the issues raised during the WBI learning activities afterwards at work, 59 percent discussed them occasionally and 30 percent discussed these issues thoroughly at work. 20 Figure 6.2: Percent distribution of participants by whether or not they have discussed the issues raised during the activity (at work, with local partners, govt. officials, NGOs, or in the media) 30% 59% 11% Discussed ocassionally Discussed thoroughly Never discussed 6.4 The participants were asked to evaluate their usage of knowledge and skills from the course in seven different areas: · Conducting research; · Teaching · Raising public awareness in development issues; · Implementing new practices within your work organization; · Organizing collective community-based initiatives; · Changing/influencing legislation, and · Implementing country development strategies. 6.5 Participants could choose to rate the level of usage on a scale from 1-7 with 1 being "Use Not At All" and 7 being "Use Very Often". On the average overall usage (sum of each individuals rating in all seven areas) was rated low with a mean of 3.6 on a scale from 1-7. 6.6 Participants identified relatively few areas for applying their newly acquired knowledge and skills. Figure 6.3 displays the percent of participants who declared specific areas to be "Not Applicable" for using the knowledge and skills acquired during the WBI learning activity. Teaching and conducting research come at the forefront of these areas designated as "Not Applicable" for over 40 percent of the participants; these were followed by changing/influencing legislation (36 percent), organizing collective initiatives (24 percent) and finally raising public awareness of development issues, implementing new practices within work organization and implementing country strategy (15 percent). 21 Figure 6.3: Percent participants who declared a specific area 'not applicable' for using the knowledge acquired in the learning activity 46 50 42 36 40 30 24 20 15 15 15 10 0 Teaching Research Changing Collective Raising New Implementing legislation initiatives awareness practices at co. str. work 6.7 When the area of use of the knowledge/skills acquired in the WBI learning activity is considered applicable, it seems that the majority of participants use these knowledge/skills only occasionally as presented in Figure 6.4. Participants are likely to use these knowledge/skills relatively very often in the areas of teaching, implementing new practices within their work organizations, raising public awareness in development issues, and in conducting research. On the other hand, participants are least likely to use the knowledge/skills acquired in the two areas of changing or influencing legislation/regulations and organizing collective initiatives. Figure 6.4: Percent distribution of participants by the level of use of the knowledge/skills from WBI activities when applicable 100% 25 21 80% 50 50 45 40 39 60% 56 40% 47 79 43 57 50 55 20% 13 19 0% 7 4 Teaching New practices Raising Research Implementing Changing Collective at work awareness co. str. legislation initiatives Not at all Occasionally Very often PREDICTORSOF USEOF NEWKNOWLEDGE AND SKILLS 6.8 In order to generate a smaller group of variables to describe usage we attempted factor analysis. However, because of the large number of missing (non-applicable) observations factor analysis was not possible.17 Instead pair wise correlations indicated two sub-groups of usage: 17A large number of observations were rated as "Not Applicable". We recoded the "NA" to the average usage in the relevant category (institutional or country) to increase the sample size. Additionally, in 22 · Institutional Use: The first factor summarizes the level of use of the new knowledge/skills in the areas research, teaching, raising others' awareness of development issues, generating new practices within their organization or improving work practices, and by developing collective initiatives to enhance the development process; and · Country Use: The second factor has significantly large loadings in the areas of preparing or implementing country development strategies and influencing legislations and regulations. 6.9 Alpha reliability coefficients for the above two categories is also high (alpha for institutional use = 0.82 and alpha for country use = 0.73) indicating that these two categories account for a large part of the variation in overall usage. 6.10 Again, due to the limited number of observations and missing data, a fully specified multivariate regression with all variables of interest could not be performed. Instead, we performed a series of ordinary least square regressions and pair-wise correlations between two aspects of use and different aspects of participant and activity characteristics. These regressions and correlations revealed that both institutional use and country use are highly correlated with activity effectiveness. Due to the limited number of observations for our regression analysis, we do not report the results. However, below are the scatter plots of some of the significant determinants of effectiveness. 6.11 Perceived activity effectiveness is significantly correlated with usefulness. Scatter plots of effectiveness and the two aspects of use, country and institutional, showed that there were not many outliers and there exists a strong relationship between both types of use and perceived course effectiveness. Figures 6.5 and 6.6 show the scatter plot between institutional use and effectiveness and country use and effectiveness. The coefficient of correlation between perceived effectiveness and institutional use is 0.75** and between perceived effectiveness and country use is 0.55**. Figure 6.5: Scatter plot of institutional use and perceived effectiveness 8 6 7) Effectiveness to 4 (1 2 Perceived 0 0 2 4 6 8 Institutional Use (1 to 7) creating an aggregate measure of the variable, in each factor the ratings were summed up in order to control for wide usage (use many dimensions) as well as high usage (use each dimension highly). 23 Figure 6.6: Scatter plot of country use and perceived effectiveness (1 8 7 6 7) 5 Effectiveness to 4 3 2 1 Perceived 0 0 2 4 6 8 Country Use (1 to 7) FACILITATORS AND BARRIERS TO THE UTILIZATIONOF LEARNING 6.12 The participant survey examined the extent to which two factors related to the work organization and to their country, have been helpful or unhelpful in actually using the new knowledge/skills acquired from WBI learning activities. These factors are: · Work Environment: work procedures, colleagues, incentive system and funding, · Country Development Environment: macro factors such as country policies, social and political groups, readiness for reforms, etc. 6.13 Work environment factors and country development factors are rated neither helpful nor unhelpful in implementing the acquired knowledge. The distribution of the participants' ratings is displayed in Figure 6.7. In all, practices and procedures in the work organization and work colleagues are more likely to be considered facilitators than barriers to the use of the new knowledge/skills acquired during the learning activity. 6.14 As in the case of work factors, country environmental factors are more likely to be considered facilitators than barriers to use of new knowledge/skills. In general, however, these macro factors are perceived as neutral in terms of facilitating or impeding the implementation of knowledge and skills acquired during learning activities. Figure 6.7: Participant ratings of helpfulness of factors to use the knowledge/skills 100% 8 7 80% 60% 56 59 40% 20% 36 33 0% Work Environment Country Environment Helpful Neither helpful nor unhelpful Unhelpful 24 COMPARING WBI TO SIMILAR NON-WBI ACTIVITIES 6.15 Of all 27 participants surveyed, eleven (40 percent) participated in learning activities with similar content offered by organizations, other than WBI, in Yemen. The fact that in surveys respondents reported that there are other non-WBI related training in Yemen is peculiar. Among the institutions/organizations listed in participant surveys are: British Council, Notre Dame University, WHO, UNICEF, and the Women National Council. Both Operational counterparts in Sana'a and focus group attendees indicated that there is no other learning institute, similar to WBI, active in Yemen. 6.16 The usefulness of WBI activities in higher than the usefulness of non-WBI activities. Ten of the eleven respondents who had participated in similar non-WBI training were able to compare the usefulness of these non-WBI activities to similar ones offered by WBI. The majority of those (64 percent) reported the usefulness for WBI activities is much higher than similar non-WBI learning activities. An equal number of participants (18 percent) replied that WBI activities and similar non-WBI activities have about the same level of usefulness or that WBI activities are less useful than similar non- WBI activities. The comparison to non-WBI activities is graphically illustrated in Figure 6.8 Figure 6.8: Percent aware of other similar activities who reported that the non-WBI activities are: About the same 18% More useful 64% Less useful 18% 25 7. IMPACT OF WBI LEARNING ACTIVITIES 7.1 This chapter will discuss "impact" defined as the influence of, or change ignited by WBI learning activities. We first make an attempt to gauge impact and then to ascertain its determinants. GAUGING IMPACT 7.2 Over two-thirds of the participants indicated that WBI's activities led to an overall change. In the survey, each participant was asked whether the WBI learning activity he/she attended, had led to any overall changes in the specific area of the activity. About 70 percent affirmed the occurrence of such an overall change, 19 percent reported that the activity did not lead to any change and 11 percent did not know. The distribution is displayed in Figure 7.1. Figure 7.1: Percent distribution of participants' responses concerning overall impact of the learning activity Don't Know 11% Change 70% No Change 19% 7.3 The impact of the WBI learning activities was rated above average. All participants who confirmed an overall change at the policy level in relation to the activities they attended reported that the change was positive. None of the participants reported that the change brought about by the activity was negative. 7.4 Using a 7-point scale, the survey also investigated the impact of the WBI learning activities in terms of negatively/positively influencing six specific areas: · Research; · Teaching; · Public awareness in development issues; · New work practices in the participants' organization; · Collective initiatives; · Legislation and regulation; · Country development strategies 26 Figure 7.2: Percent of 'Not Applicable' corresponding to each of six areas related to anticipated impact of WBI learning activities Teaching 50 Research 37 Country devt str 31 Legislations and regulation 31 Collective initiative 28 Awareness 22 New work practices 11 0 10 20 30 40 50 60 7.5 Although the participants rated the overall change brought about by the activity as being positive, when impact was decomposed into seven specific areas as described above, the sum of each participant's rating was below mid-point with a mean of 3.9 on a scale from 1-7. This rating is significantly lower that the average of our five countries' rating for impact, rated at 5.2. A preliminary analysis of the data revealed that the WBI activity was not applicable to each of the specific areas of interest as depicted in Figure 7.2. 7.6 Figure 7.3 displays the percent of responses indicating the direction of influence (positive/negative/no influence) of the WBI learning activity attended, among applicable cases only. Figure 7.3: Percent distribution of responses indicating positive/negative influence of the learning activity among applicable cases only 0 100% 6 4 6 8 90% 14 18 16 80% 33 28 70% 60% 50% 94 92 40% 86 82 80 30% 67 67 20% 10% 0% Country devt Teaching Awareness Research New work Collective Legislations str practices initiative and regulation Positive No influence Negative 7.7 WBI learning activities have a strong positive influence (when applicable) on country development strategy, teaching and raising public awareness. Our findings in the chapter on usefulness showed that knowledge and skills acquired during the activity is most useful in research and raising public awareness. Therefore, it is not surprising that 27 participants expressed that these were the two areas with high positive impact. The least positive influence is in the areas of legislation and community-based initiatives. PREDICTORSOF IMPACTOF WBI LEARNING ACTIVITIES 7.8 Factor analysis could not be utilized for the purpose of reducing the different dimensions of the perceived influence of the learning activity due to the large number of cases that are "Not Applicable" to each individual dimension. Therefore, similar to the indices developed for use in Chapter 6, pair wise correlations indicated two sub-groups of perceived impact: · Institutional Impact: The first factor summarizes the level of use of the new knowledge/skills in the areas research, teaching, raising others' awareness of development issues, generating new practices within their organization or improving work practices, and by developing collective initiatives to enhance the development process; and · Country Impact: The second factor has significantly large loadings in the areas of influencing legislations and regulations and preparing or implementing country development strategies. 7.9 Alpha reliability coefficients for the above two categories is also high (alpha for institutional impact = 0.71 and alpha for country impact = 0.93) indicating that these two categories account for a large part of the variation in overall impact. Scatter plots of the relationships between two aspects of use and the two aspects of impact: country and institutional revealed that there exists a strong linear relationship between country use and country impact and between institutional use and institutional impact. 7.10 As in previous sections, a fully specified multivariate regression with all variables of interest could not be performed due to the limited number of observations and missing data. Instead, we performed a series of ordinary least square regressions and pair-wise correlations between two aspects of use and two aspects of impact. These correlations and regressions revealed that both institutional use and country use are highly correlated with institutional and country impact. Due to the limited number of observations for our regression analysis, we do not report the results. However, below are the scatter plots of the most significant determinant of impact: use of new knowledge and skills. 7.11 Course usefulness is highly correlated with impact of WBI learning activities on participants' work and country. Scatter plots of the two aspects of impact and use, institutional and country, showed that there were not many outliers and there exists a strong relationship between use and impact. Figures 7.4 and 7.5 show the scatter plot between institutional use and institutional impact and country use and county impact. The coefficient of correlation between institutional use and institutional impact is 0.74** and between country use and country impact is 0.78**. 28 Figure 7.4: Scatter plot of institutional use and nstitutional impact 7) 8 to (1 6 use 4 2 Institutional 0 0 2 4 6 8 Institutional Impact (1 to 7) Figure 7.5: Scatter plot of country impact and country use 8 7) to 6 (1 use 4 2 Country 0 0 2 4 6 8 Country Impact (1 to 7) 29 8. LISTENING TO OUR COUNTERPARTS 8.1 In order to gauge WBI's link to Operations and assess its responsiveness to country needs, several key Operational Counterparts were interviewed in the field. At the suggestion of Yemen's country manager, interviews were also held with government officials in the Ministry of Planning and Ministry of Finance who worked closely with our Operational counterparts. Operational counterparts (and their counterparts in the Yemeni government) were asked the following questions: · To what extent are WBI learning events aligned with the CAS? · What was the extent of your involvement in WBI activities during FY01-02? · Do you know of any outcomes that resulted from WBI learning activities? · What are the main barriers to applying new knowledge toward change in Yemen? · Is WBI uniquely positioned to deliver learning activities for capacity building in Yemen? · What could have been done differently by WBI to better serve the learning needs of Yemen? · Taking a forward looking approach, what are some ways that WBI can assist the country in moving closer its development goals? 8.2 Overall, the answers to these questions were not positive for FY02 and FY03. Summary highlights of the interviews are reported in the following sections. OVERALL VIEWOF EFFECTIVENESS 8.3 Counterparts in the field held a strong view that WBI has not been effective in Yemen. They emphasized that WBI has "simply not worked enough in Yemen". Interviewees noted that WBI activities for the Yemenis have been "insufficient, ad-hoc and sporadic". 8.4 Field office staff noted that "to see results, WBI must have a systematic set of interventions". Interviewees agreed that in order to be more effective, WBI must offer local offerings in Arabic on a more regular basis since language is seen as a major barrier to the use of knowledge and skills. They also noted the need for local courses. One interviewee noted that "since September 11, obtaining visas has become a very big obstacle for attending events out of the country". Except for one video conference offering (which was not targeted to Yemenis alone) there were no offerings that took place in Yemen. 30 WBI OUTCOMES 8.5 WBI Outcomes are not evident to Operational staff. None of the interviewees could pinpoint a specific outcome of a WBI program in Yemen during FY02-03. All interviewees expressed the view that WBI has not been active in Yemen and is not having an impact on Yemen. The primary reason cited for lack of impact of WBI activities was the fact that WBI activities tend to be supply driven not demand driven. As a result, there is lack of ownership and participation on part of the government. 8.6 Operational staff indicate that compared to many other MNA countries the Yemeni are open and receptive to new knowledge. One field staff indicated that what has constrained action on the implementation of WBI's results has not been the lack of willingness or `demand for new knowledge' on part of Yemenis but the lack of learning activities. "The Yemenis are receptive to new knowledge. They are open to attending training, and applying new knowledge. The problem is a supply issue. The demand is there." 8.7 WBI is uniquely positioned in Yemen, but it needs to develop local partnerships. Government officials in the Ministry of Finance said they view WBI as uniquely positioned in Yemen. They emphasized, however, that "WBI's outcomes would benefit from developing partnerships with local institutions such as the Arab Institute for Planning in Kuwait that specializes in short-term courses". Through local partnerships in the regions, WBI can expand its activities in Yemen and design activities that reflect the needs of the region more closely. 8.8 Language is the most important barrier to the use of new knowledge. Almost all interviewees in the field indicated that one of the most important impediments to the effectiveness of WBI learning activities is language. "There is no point in doing any training in Yemen unless you have Arabic speaking presenters. We need people who can speak to the region and understand the region." RESPONSIVENESS TO OPERATIONS 8.9 Operational staff at the Headquarters are disconnected from WBI. Interviewees at the headquarters could only allude to some "ad hoc" discussion with colleagues in WBI and involvement in administrative issues. Field Operational staff in Yemen also expressed that they only had limited communication with WBI. 8.10 Operational staff note that WBI needs to be more flexible and responsive to local demands. One interviewee stressed that WBI needs to deliver more `tailored training activities' focusing on areas that are most relevant locally ­ governance, poverty reduction and private sector development. 31 9. LISTENING TO PARTICIPANTS 9.1 Two focus group meetings were held in Sana'a in February 2004 to hear first hand from participants their views about: · Relevance and usefulness of WBI activities for the Yemenis; · Country specific barriers to use of their new knowledge and skills; and · Practical measures that WBI can take to increase impact of our programs in Yemen. 9.2 Focus groups were activity specific, and selection of the activities was based on the location of the activity and the number of Yemeni participants. Since there were no activities held locally in Yemen, the three activities with largest number of participants were chosen. VIEWS ON RELEVANCE AND USEFULNESS 9.3 Participants regard the WBI learning activities useful but not enough. Overall, the two groups indicated that they found the courses relevant to their personal, organizational and country learning needs. They indicated that the training was "useful but by no means enough". The most useful aspects were new technical knowledge, exposure to what other countries do and sharing experiences, and provision of references that are available for the future. New knowledge learned during the activity was not easily applicable to their work. 9.4 Both focus groups shared the same views about elements of WBI activities that make them useful or not. Features that made WBI activities useful were their ability to: (a) Expose the participants to new knowledge and concepts that "we had never heard of before" (b) Discuss relevant case studies and lessons learned from other developing countries. Features that reduced usefulness were: (a) WBI activities were "insufficient, ad-hoc and sporadic" (b) New concepts that were taught were too technical and could not be applied easily (either topics were too general or toot technical); (c) Programs were not targeted to Yemen's specific needs; and (d) Lack of follow-up in a meaningful way. 32 9.5 Participants were asked to describe the specific barriers that could hinder the use of participants' new knowledge and skills. 9.6 Only on country specific barrier was noted: a rigid and hierarchical work environment. The procedures in the work environment are very difficult to change making it difficult to implement new ideas. Participants suggested programs that are not only tailored to Yemen's needs but also to Yemen's constraints in implementing change. 9.7 Other barriers noted by focus group members were related to training design and need to be addressed by WBI: (a) Language is a barrier: Language is a major impediment for the Yemenis. Participants indicated that the lack proficiency in the language of instruction made it difficult to understand the concepts thoroughly. One participant said (in Arabic), "How do you expect us to use something that we have not understood word of?" Her colleague clarified that they attended a session where "50 percent of the participants spoke no English and 30 percent spoke some English". (b) Rigid and hierarchical work environment: The rules, regulations and procedures in the work environment are very difficult to change making it difficult to implement new ideas. Participants suggested programs that are not only tailored to Yemen's needs but also to Yemen's constraints in implementing change. (c) Lack of team training: Participants all agreed that in order for them to use the knowledge and skills, they should be in an environment that is at least familiar with the new thinking. They suggested "training in teams". As one participant said, "To apply new concepts, one must change the existing procedures, and for that we need to build alliances, and to build alliances we need to be trained together". (d) Courses are not offered locally: Except for one video-conference offering, none of the courses attended by Yemeni participants were not offered locally. As a result, it was extremely difficult for them to participate in WBI training programs. High costs coupled with difficulties in obtaining visas post-September 11 make it very difficult for Yemenis to participate, much less, use the training. MEASURES TO INCREASE IMPACT IN YEMEN 9.8 Focus group participants offered several practical measures that WBI can take to increase its effectiveness and make a difference: (a) Offer more courses in local language or improve translation facility during courses: lack of proficiency in language of instruction was cited as a major obstacle to learning in Yemen. Translation was poor and participants indicated that simply using a dictionary was not enough. Therefore, there is need for either more courses in Arabic or better translation facilities at the venues. 33 (b) Offer more courses locally: Strict budget constraints in the public sector limit travel for training. For this reason it is suggested that upcoming training targeted to government employees will be most practical if held in the country. (c) Replace regional offerings with country focused activities: There is a need for tailored programs for Yemen. "It is great to have the international experience. But to make change, we need to have a program tailored for Yemen." Participants emphasized that WBI's regional approach, by definition, fails to offer solutions within the constraints of Yemen. (d) Offer team training as opposed to individual training: In the words of one participant, "The procedures in the work environment are very difficult to change, and to apply certain concepts the system in place needs to be changed. To bring about change, there needs to be a critical mass of people who push the same agenda at different levels of an organization." Therefore there is need to train people from different organizations and different positions so as to foster alliances that can promote change. Therefore, as another participant noted, "invitations should be extended to teams, versus individuals. (e) Support training of trainers: In the words of one participant "new knowledge can not make a difference when it is applied by one person in the entire organization". For it to be applicable institutionally, the new knowledge should be shared with colleagues who had not had the opportunity to attend the activity. WBI could be very helpful in this respect, by building a component in the training that prepares the participants for making a presentation when they go back to their offices. (f) WBI training needs systematic follow-ups: Past training has had no follow-up. Participants agreed that their WBI training was "informative but not enough to produce real changes in their existing practice at an organizational and/or even an individual level. To create any sort of change or impact, future WBI learning activities must have systematic follow-ups." 34 10. CONCLUSIONS AND RECOMMENDATIONS 10.1 Based on FY 02-03 data collected from participant surveys, participant focus group discussions, interviews with Operational Staff working on Yemen (from Washington and Sana'a) and their counterparts in the Yemeni government, and relevant Operational and WBI documents, the following conclusions and recommendations can be drawn for each of the evaluation questions. 10.2 Before discussing the conclusions in detail, it should be noted that there are noticeable differences in the findings based on quantitative and qualitative instruments. Given the limited sample size for the quantitative analysis, we stress the need to interpret the quantitative findings with caution and give greater importance to the findings from focus groups and Operational staff interviews in drawing out conclusions. Question 1. What is the Relevance of WBI Learning Activities to the Country's Needs? 10.3 This evaluation examined the relevance of WBI learning activities to Yemen's needs and to the needs of the needs of the participants' work in four ways: alignment of activities to CAS, participants' assessment of activity relevance, activity relevance to MDGs and the Bank's corporate priorities; and empirical country focus assessment. 10.4 In FY02-03, 75 percent (10 out of total 13 activities) of WBI's activities were aligned with Yemen's CAS priorities. Although most WBI activities were aligned with CAS, Yemeni participants comprised only 7 percent of the total participants of a typical activity. Moreover, except for one video-conference activity from Washington, none of the activities that Yemenis participants attended took place in Yemen. 10.5 On average, the relevance of WBI learning activities to the specific needs of Yemen was perceived as moderate high, with a mean rating of 5.3. Less than half (46 percent) of all participants perceived the topics covered to be highly relevant to the needs of their country; and the rest (54 percent) perceived only a moderate degree of relevance. None of the participants perceived the activities to be irrelevant to their country. 10.6 The relevance of WBI learning to the participants' work was almost the same: a mean rating of 5.2. The relevance of WBI learning activities to the needs of the Yemeni participants' work was perceived to be high by 56 percent of the participants and moderate by 41 percent of the participants. 10.7 WBI activities for Yemeni participants were relevant to four of the MDGs and corporate priorities: developing global partnerships, improving investment climate, promoting gender equality and eradicating extreme poverty. Over one-half of the participants reported that the activities they attended are related to these four goals. Relationship between activity content and water sanitation, one of the main priorities for the country, is extremely low. 35 10.8 In examining the conditions of being country focused, this evaluation finds that the WBI activities for Yemen did not meet three of the five conditions for being country focused. Although from the participants' perspectives, WBI's activities are relevant to their work, at the activity level, there was no sign of an intended effort towards a country focused program. Question 2: What is the Effectiveness of WBI Learning Activities? 10.9 There are significant differences in the findings on effectiveness from participant surveys and from focus group and Operational staff interviews. Based on participant surveys, we find that overall effectiveness is slightly above the mid-point with a mean of 5.1 (on a scale of 1-low to 7-high). 10.10 Participants responses regarding the level of effectiveness of the learning activities suggests that these activities seem to be relatively most effective in the areas of: raising awareness/understanding of development issues, providing knowledge or skills, and providing strategies/approaches to address the needs of the participants' organization. Participants rated WBI activities least effective in providing strategies/approaches to address Yemen's unique needs or in providing networking opportunities. 10.11 Interviews with Operational counterparts and their counterparts in the Yemeni government reveal that WBI has not been effective in Yemen. In two focus groups held in Sana'a, both groups indicated that they found the courses to be "useful but by not means enough". Field office staff noted that WBI activities for the Yemenis have been "insufficient, ad-hoc and sporadic". 10.12 Though activities were insufficient, the focus group participants said they were useful in exposing them to new knowledge and concepts that "they had never heard before" and to case studies from other developing countries. They noted three features of WBI activities reduced usefulness: (a)new concepts were either too general or too technical, (b)programs were not targeted to Yemen's specific needs; and (c) there was no meaningful follow-up. 10.13 Almost all interviewees in the field indicated that one of the most important impediments to the effectiveness of WBI learning activities is language. "There is no point in doing any training in Yemen unless you have Arabic speaking presenters. We need people who can speak to the region and understand the region." Question 3. What is the Impact of WBI Learning Activities? 10.14 Overall usage of new knowledge and skills was rated low with a mean of 3.6 on a scale of 1 low ­7 high. Participants identified relatively few areas for applying their newly acquired knowledge and skills. Participants used these knowledge/skills relatively often in the areas of teaching, implementing new practices within their work organizations, raising public awareness in development issues, and in conducting research. On the other hand, participants the knowledge/skills acquired least in the two areas of changing or influencing legislation/regulations and organizing collective initiatives. 36 10.15 Asked whether the WBI learning activity they attended had led to any overall changes in the specific area of the activity, seventy percent of the participants affirmed the occurrence of a positive change. Although participants rated the overall change brought about by the activity as positive, when impact was decomposed into seven specific areas (i.e. teaching, research, strategy development, etc...)the sum of each participants' rating was modest with a mean of 4 (on a scale of 1-7). 10.16 WBI outcomes are not evident to Operational staff. None of the interviewees could pinpoint a specific outcome of a WBI program in Yemen during FY02-03. All interviewees expressed the view that WBI has not been active in Yemen and is not having an impact on Yemen. The primary reason cited for lack of impact of WBI activities was the fact that WBI activities tend to be supply driven not demand driven. As a result, there is lack of ownership and participation on part of the government. 10.17 One field staff indicated that what has constrained action on the implementation of WBI's results has not been the lack of willingness or `demand for new knowledge' on part of Yemenis but the lack of learning activities. "The Yemenis are receptive to new knowledge. They are open to attending training, and applying new knowledge. The problem is a supply issue. The demand is there." 10.18 WBI is uniquely positioned in the overall national training effort in Yemen. There are very few institutions offering similar activities. Both participants and counterparts in Operations and Yemeni government emphasized that WBI is unique in providing access to (and linkages with) alternative experience from around the world however it needs to develop local partnerships and offer more `tailored training activities' focusing on areas that are most relevant locally. 10.19 It is also crucial that task managers ensure the proficiency of participants in the language of instruction and in the terminology. Activities in Yemen should primarily be conducted in Arabic. If conducting the activities in Arabic is not viable, then proficiency of participants in the language of instruction and in the terminology should indeed be considered as a pre-condition for relevance. Task managers should consider it equally important to try to plan activities that are located locally in Yemen. Task managers need to be more responsive to local needs and plan courses focusing on areas that are most relevant locally. Question 4. What Affects Effectiveness and Impact? 10.20 This evaluation sought to identify the facilitators and barriers to using new knowledge in Yemen. While survey results did not reveal any significant facilitators or barriers, focus groups indicated that the most prominent barrier to impact and change is a rigid and hierarcial work environment. The procedures in the work environment are very difficult to change, making it difficult to implement new ideas. For this reason, participants suggested programs that that are not only tailored to Yemen's needs, but also to Yemen's constraints in implementing change. 10.21 Interestingly local field office staff spoke of no real barriers. On the contrary, they noted that the Yemenis' receptiveness to new knowledge facilitates the effectiveness of WBI learning programs. One field staff indicated "The Yemenis are receptive to new 37 knowledge. They are open to attending training, and applying new knowledge. The problem is a supply issue. The demand is there." 10.22 Given small sample size, quantitative analysis was limited. However, these support the conclusion that activities relevant to country needs are also more effective. 10.23 Perceived relevance is highly correlated with activity effectiveness, with a coefficient of correlation of 0.70**. We also found that use of the new knowledge is highly correlated with impact, with a correlation coefficient between country use and country impact of 0.78**. RECOMMENDATIONS 10.24 To improve the institutional impact and effectiveness of WBI activities in FY04 and beyond, and in short to make a difference, WBI's presence in Yemen should, in the first instance, be expanded; and second, become much more country-specific. This involves adopting a systematic approach to capacity building in Yemen. Some recommendations follow. · WBI must increase its offerings, provide a systematic set of interventions, hold activities locally in Yemen, target participants in teams and cohorts, and offer its courses in Arabic. · Learning activities must be designed with the country's needs and constraints in mind. To better assess the local needs and constraints of the Yemenis and move toward this country-focused approach, WBI should bridge the perceived "disconnect" with Operations, and align its learning programs with Operational project. · The Institute should establish an institutionally agreed upon approach to systematic follow-ups to its learning activities. It has to design activities that build knowledge and skills at continuous levels, carefully targeting cohorts of participants, and maintaining on-going contact with participants, and if possible promote training of trainers. 38 ANNEXES 39 40 ANNEX 1: WORLDBANK INSTITUTE (WBI) COUNTRYFOCUSEVALUATION QUESTIONNAIRE Instructions WBI had the pleasure to have you participate in the following learning activity: Title: _____________________________________________________________ Held from: ________________________ to: ________________________ In: _______________________________________________________________ Getting your opinion of the above-mentioned activity--now that you have had time to reflect on it--is very important to help WBI improve its programs. For this, we ask you to complete this questionnaire. The questionnaire has four sections and should take approximately 20 minutes to complete. · Section 1 asks about the relevance of the activity. · Section 2 asks about the usefulness of the activity. · Section 3 asks you to compare this activity with similar learning activities offered by other organizations. · Section 4 asks about the characteristics of the activity, its follow-up and your background. We need your honest feedback. Please keep in mind that your responses will be kept confidential, and will be used for the sole purpose of improving WBI programs. If you have any questions about the questionnaire please call or send a message to Ms. Heidi Zia at hzia@worldbank.org . Thank you for agreeing to complete this questionnaire! ID:_________________ 41 World Bank Institute (WBI) Country Focus Evaluation Questionnaire I. Relevance of the Activity 1. Since the end of the activity, to what degree has the activity been relevant to your work? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 s s s s s s s 2. To what degree have the topics covered in the activity been relevant to your country's needs? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 s s s s s s s 3. Was the activity designed specifically for participants from your country? s Yes s No s Don't now 4. Was the activity related to the country development goals listed below? a. Eradicate extreme poverty Don't now s Yes s No s b. Achieve universal primary education Don't now s Yes s No s c. Promote gender equality and empower women Don't now s Yes s No s d. Reduce child mortality Don't now s Yes s No s e. Improve maternal health Don't now s Yes s No s f. Combat HIV/AIDS, malaria, and other diseases Don't now s Yes s No s g. Ensure environmental sustainability Don't now s Yes s No s h. Develop global partnerships for development Don't now s Yes s No s i. Ensure water sanitation and supply Don't now s Yes s No s j. Improve investment climate and finance Don't now s Yes s No s k. Promote trade Don't now s Yes s No s 42 II. Usefulness of the Activity 5. Please rate the degree of effectiveness of the activity in each area noted below. (If the area was not an objective of the activity, please mark "not applicable.") Not Areas Extremely Not effective at all effective applicable 1 2 3 4 5 6 7 NA a. Raising your awareness and understanding of the development issues important to your country s s s s s s s s b. Providing you with knowledge or skills s s s s s s s s c. Helping you better understand your role as an agent of change in your country's development s s s s s s s s d. Helping you develop strategies or approaches to address the needs of your organization s s s s s s s s e. Helping you develop strategies or approaches to address the needs of your country s s s s s s s s f. Helping you develop contacts, develop partnerships and build coalitions in the field s s s s s s s s 6. How would you rate the change--brought by the activity--in the main topic or issue it addressed? Strong negative No Change Strong positive Don't change change Know 1 2 3 4 5 6 7 DK s s s s s s s s 7. How often have you used the knowledge and skills you acquired in the activity for the following purposes? (If you have not worked in the given area since this activity, please mark "Not applicable.") Purposes Not Very Not at all often applicable 1 2 3 4 5 6 7 NA a. Conducting research s s s s s s s s b. Teaching s s s s s s s s c. Raising public awareness in development issues s s s s s s s s d. Implementing new practices within your work organization s s s s s s s s e. Organizing collective initiatives s s s s s s s s f. Influencing legislation and regulation s s s s s s s s g. Implementing country development strategies s s s s s s s s 8. To what extent did the following factors help or hurt the process of using the knowledge/skills that you acquired at the activity? Factors Greatly Neither helped Greatly Not hurt nor hurt helped applicable 1 2 3 4 5 6 7 NA a. Your work environment (e.g., work procedures, colleagues, incentive system, funding, s s s s s s s s etc.) 43 Factors Greatly Neither helped Greatly Not hurt nor hurt helped applicable 1 2 3 4 5 6 7 NA b. Your county's development environment (e.g., country policies, social groups, political groups, readiness for reform, etc.) s s s s s s s s 9. How has the activity influenced or led to changes in the following areas? (If the area is not relevant to the activity, please mark "Not applicable.") Areas Negative No Positive Not influence influence influence applicable 1 2 3 4 5 6 7 NA a. Research s s s s s s s s b. Teaching s s s s s s s s c. Public awareness in development s s s s s s s s issues d. New practices within your work s s s s s s s s organization e. Collective s s s s s s s s initiatives f. Legislation and s s s s s s s s regulation g. Country development s s s s s s s s strategies 10. Since the activity, have you discussed the issues raised in the activity, at work, with local partners, government officials, NGOs, or in the media? Never Thoroughly discussed discussed 1 2 3 4 5 6 7 s s s s s s s III. Comparison of the WBI Activity with Similar Activities Offered by Other Organizations 11. Did you participate in any similar learning activities offered by other (NON-WBI) organizations in your country? (If no, please skip to question 14.) s Yes s No 12. If yes, please provide the name(s) of the organization(s): 1. 2. 3. 44 13. How would you rate the usefulness of the WBI activity compared to NON-WBI activities? WBI About the same WBI No much less much more opinion useful useful 1 2 3 4 5 6 7 s s s s s s s s IV. Characteristics of the WBI Activity, its Follow-up and Your Background 14. How would you describe the type of the WBI learning activity that you attended? Mix of Video and Video Sessions Class room Face to Conference Web-based Study tour (Distance Learning) (Face to Face) Face Learning 1 2 3 4 5 6 s s s s s s 15. How effective was this type of learning activity in helping you learn? Extremely No Not effective at all effective opinion 1 2 3 4 5 6 7 s s s s s s s s 16. During the WBI activity, did you develop an action plan/strategy (e.g., work plans, strategy papers, or policy documents) to apply the knowledge and skills you learned? (If no, please mark "no" below, then skip to question 18.) s Yes s No 17. If yes, did you use part or all of the action plan in your work? s Yes s No 18. Were you provided with the contact information of other participants in the activity, such as e-mail addresses, telephone numbers or mailing addresses? (If no, please mark "no" below, then skip to question 20.) s Yes s No 19. If yes, how did you use it? Used it to organize Used it to joint continue activity follow-up Other uses Never used it related discussions activities (Please specify briefly) s s s s_____________________________ 20. Was the language of instruction used during the activity the same language you use at work? s Yes s No 45 21. At the time of the activity, what was your level of proficiency in the language of instruction? Not proficient at all Highly proficient 1 2 3 4 5 6 7 s s s s s s s 22. At the time of the activity, what was your level of proficiency in the technical terminology used in the activity? Not proficient at all Highly proficient 1 2 3 4 5 6 7 s s s s s s s 23. After the activity, did WBI contact you for follow-up issues regarding the activity? s Yes s No 24 After the activity, did YOU contact WBI for follow-up issues or questions on the content of the activity? (If no, please skip to question #23) s Yes s No 25. If yes, please rate WBI's helpfulness in addressing your issues. WBI I did not responded, have WBI did but wasnot follow-up not helpful WBI responded and was requests respond at all extremely helpful for WBI 0 1 2 3 4 5 6 7 NA s s s s s s s s s 26. Which of the following best describes the organization in which you have worked the longest since the activity? (Select one.) s University/research institution s National/central government s Non-governmental organization(not- s for-profit) Provincial/regional government s Media s Local/municipal government s Private sector s Other, specify: ____________________________ 27. Which of the following best describes the primary type of work you have done the longest since the activity? (Select one.) s Research s Teaching s Policymaking/legislation s Provision of services (e.g., financial, health, etc) s Management/administration s Other, specify: ____________________________ 46 28. How would you best describe the level of the position you have held the longest since the activity? s Highest level (e.g., Minister, Deputy Minister, Top Government Official, Full Professor, President of an organization) s Senior level (e.g., Department Head, Division Head, Associate Professor, Senior Researcher) s Middle level (e.g., Program Manager, Project Leader, Assistant Professor, Technical Expert) s Junior Level (e.g., Research associate, Ph.D. level graduate student, Technical Specialist) s Entry level (e.g., Intern, assistant) s Other, Please specify: __________________________________________________________________ 29. What is your gender? s Male s Female Thank you for your feedback. We appreciate very much your cooperation. 47 ANNEX 2: LIST OF FY02-03 ACTIVITIES Session Title Yemenis Schedule Country Program Title Adapting to Change: Core Course on Population- Reproductive Health and Health Sector Reform 4 United States of America Health and Population Conference and Policy Services for the MNA Region 2 France Knowledge for Development Flagship Capacity Building in MENA - 1st Regional Course 13 Lebanon Health and Population Flagship Course on Immunization: Coping with Health Sector Reform and a Changing Global Context 2 Hungary Health and Population Flagship Gender in Program and Project Monitoring and Evaluation 16 Yemen18 Health and Population Frontiers in Infrastructure Finance 5 Egypt Market Solutions for Development Higher Education Conference - MENA 6 France Education Program Integrated Water Resources Management 1 United States Of America Sustainable Development Introduction to Poverty Monitoring and Evaluation 5 Egypt Attacking Poverty Mediterranean Development Forum 4 2 Jordan Pension Reform in the Middle East and North Africa 5 Tunisia Social Protection Learning Public Pension Fund Management Conference - World Bank 2 United States Of America Financial Sector Learning Reproductive Health and Health Sector Reform Course UNFPA 2 Italy Health and Population 18 This activity was a video conference from Washington to Yemen, Egypt, Jordan, Lebanon, Gaza and West Bank ANNEX 3: FOCUSGROUPQUESTIONS Focus Group Questions 1. What did you gain from this activity? 2. What were the most useful aspects of the activity? What were the least useful aspects of the activity? Explain. 3. Can you offer specific examples of how you applied what you learned in the activity? Probe 1: Have these led to any changes in your organization, or changes in policies and practices in your country? Probe 2: What were these changes? 4. What were some of the obstacles that you encountered in implementing the new knowledge and skills you acquired? Examples? 5. Were the knowledge and skills you acquired available from another source ­ either internationally or locally? Who are the prominent non-WBI players in your country? 6. How can we improve WBI activities in the future? 7. Lastly, would you be willing to pay for this course? 49 ANNEX 4: VARIABLEDESCRIPTION ANDSUMMARYSTATISTICS - REGRESSION VARIABLES Variables Description N Mean Std. Dev. Min Max Dependent variables Relevance to participant's Degree of relevance of the activity to work needs 28 5.18 1.42 2 7 work (1=irrelevant, 7=relevant) Relevance to country Degree of relevance of the activity to Yemen's 28 5.36 1.25 3 7 needs needs (1=irrelevant, 7=relevant) Effectiveness Degree of effectiveness of the activity (1=extremely 28 5.28 1.12 2.75 7 low, 7=extremely high) Overall use Sum of usage of knowledge and skills (1=low, 26 4.86 1.09 2.67 7 7=high) Institutional Use Sum of usage in participant's institution (1=low, 26 4.96 1.06 3 7 7=high) Country Use Sum of usage in Yemen (1=low, 7=high) 24 4.71 1.34 1.50 7 Overall Impact Overall influence/change/impact of the activity 27 5.56 0.94 3.57 7 (1=negative, 7=positive) Institutional Impact Influence/change/impact of the activity in 25 5.57 0.89 4 7 participant's institution (1=negative, 7=positive) Country Impact Influence/change/impact of the activity in Yemen 21 5.43 1.38 1.5 7 (1=negative, 7=positive) Demographics Gender Dummy if the participant is male 28 0.64 0.49 0 1 Language Proficiency Degree of proficiency of the language of instruction 27 5.25 1.29 2 7 Technical Proficiency Degree of proficiency of the terminology used in 28 5.28 1. 2 7 instruction Type of organization University/Research Dummy if the participant represents a university or 28 0.11 0.31 0 1 Institution research institution NGOs Dummy if the participant represents a NGO 28 0.11 0.31 0 1 Media Dummy if the participant represents the media 28 0.00 0.00 0 0 Private Sector Dummy if the participant represents the private 28 0.04 0.19 0 1 sector National Govt Dummy if the participant represents the national 28 0.57 0.50 0 1 govt Regional Govt Dummy if the participant represents the 28 0.04 0.19 0 1 regional/provincial govt Local Govt Dummy if the participant represents the 28 0.04 0.19 0 1 local/municipal govt Type of work Research Dummy if the participant mainly conducts research 28 0.07 0.26 0 1 and teaching Policy/legislation Dummy if the participant mainly involved in 28 0.18 0.39 0 1 policymaking/legislation Management/admin Dummy if the participant mainly involved in 28 0.54 0.51 0 1 management/administrative jobs Teaching Dummy if the participant mainly involved in teaching 28 0.04 0.19 0 1 Services Dummy if the participant mainly involved in services 28 0.18 0.39 0 1 Position Highest level Dummy if the participant is at the highest level 28 0.25 0.44 0 1 (Annex 4 continues on next page.) 50 (Annex 4 continued.) Variables Description N Mean Std. Dev. Min Max Senior level Dummy if the participant is at the senior level 28 0.36 0.49 0 1 Middle level Dummy if the participant is at the middle level 28 0.25 0.44 0 1 Junior level Dummy if the participant is at the junior level 28 0.11 0.31 0 1 Entry level Dummy if the participant is at the entry level 28 0.00 0.00 0 0 Course characteristics Action Plan Dummy whether or not an action plan was 27 0.52 0.51 0.00 1 developed Design Dummy if the activity was perceived to be des igned 21 0.19 0.40 0.00 1 specifically for Yemeni Contact Info. Dummy if contact information of other participants 27 0.48 0.51 0.00 1 were provided Video Dummy if the mode of delivery was video sessions 28 0.25 0.44 0 1 Face-to-face Dummy if the mode of delivery was face-to-face 28 0.29 0.46 0 1 Mix of video and face-to- Dummy if the mode of delivery was mix of video and 28 0.21 0.42 0 1 face face-to-face Conference Dummy if the mode of delivery was conference 28 0.25 0.44 0 1 Web-based Dummy if the mode of delivery was web-based 28 0.00 0.00 0 0 Study tour Dummy if the mode of delivery was study tour 28 0.00 0.00 0 0 Environmental Factors Work Environment Degree of helpfulness of organizational factors 24 5.04 1.23 2.00 7 (1=barrier, 7=facilitator) Country Environment Degree of helpfulness of country level factors 25 5.28 1.21 3.00 7 (1=barrier, 7=facilitator) 51