Impact Evaluation of WBI Sector & Thematic Programs in FY02-03 Poverty and Growth Chaoying Liu Shreyasi Jha Stephan Van Praet Impact Evaluation of WBI Sector & Thematic Programs in FY02-03 Poverty and Growth Chaoying Liu Shreyasi Jha Stephan Van Praet WBI Evaluation Studies No. EG05-105 The World Bank Institute The World Bank Washington, D.C. December 2004 ACKNOWLEDGEMENTS This evaluation report was prepared for WBI under the overall guidance of Marlaine E. Lockheed, Manager (retired in August 2004) and Nidhi Khattri (Acting Manager), World Bank Institute Evaluation Group (IEG). It was prepared by Chaoying Liu, Shreyasi Jha, and Stephan Van Praet. The report benefited greatly from discussions with Kene Ezemenari (AFR) and Soniya Carvalho (OED), who served as peer reviewers. Humberto S. Diaz contributed to the document production. This evaluation was partially financed by the Belgian Directorate General for Development Cooperation. WBI Evaluation Studies are produced by IEG to report evaluation results for staff, client, and joint learning events. An objective of the studies is to release findings quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at: http://info.worldbank.org/etools/WBIEG/publications/index.cfm?pg=getPubs&category= Publications&Intro=yes&instructions=no&showDetails=no&ID= Vice President, World Bank Institute Ms. Frannie Léautier Manager, Institute Evaluation Group (Acting) Ms. Nidhi Khattri Task Team Leader Ms. Chaoying Liu ii ACRONYMS ADB Asian Development Bank AFCB African Capacity Building Foundation APP Attacking Poverty Program CAR Central African Republic CAS Country Assistance Strategy CENA Capacity Enhancement Needs Assessment CPB Country Program Brief DEG Development Economics Group DFID Department for International Development FY Fiscal Year GDLN Global Distance Learning Network ICT Information and Communication Technology IEG Institute Evaluation Group IMF International Monetary Fund JICA Japan International Cooperation Agency MDG Millennium Development Goals NGO Non-Governmental Organization PAC Poverty Analysis Community PAI Poverty Analysis Initiative PGP Poverty and Growth Program PREM Poverty Reduction and Economic Management Network PRS Poverty Reduction Strategy PRSP Poverty Reduction Strategy Papers RCET Regional Capacity Enhancement Team TTL Task Team Leader UNDP United Nations Development Program USAID United States Agency for International Development WBI World Bank Institute iii iv TABLE OF CONTENTS ACKNOWLEDGEMENTS .............................................................................................ii ACRONYMS....................................................................................................................iii TABLE OF CONTENTS ................................................................................................. v EXECUTIVE SUMMARY ............................................................................................vii Evaluation Methods...................................................................................................vii Key Findings.............................................................................................................viii Lessons Learned and Recommendations.................................................................... ix 1. INTRODUCTION ..................................................................................................... 1 Evaluation Context and Objectives ............................................................................. 1 The Poverty and Growth Program............................................................................... 2 2. EVALUATION METHODOLOGY........................................................................ 4 Evaluation Questions and Scope ................................................................................. 4 Data Collection............................................................................................................ 6 3. RELEVANCE OF THE PGP ................................................................................. 12 Participant Views on the Program Relevance ........................................................... 12 Bank Staff and Partner Views on Program Relevance.............................................. 14 WBI PGP TTLs Interview......................................................................................... 15 Summary.................................................................................................................... 16 4. EFFECTIVENESS OF THE PGP ......................................................................... 17 5. USE OF LEARNING CONTENT.......................................................................... 21 Participant Use of Learning Content ......................................................................... 21 Facilitators and Barriers to Use ................................................................................. 26 Bank Staff and Partner Views on Barriers to Use ..................................................... 28 Summary.................................................................................................................... 29 6. IMPACT OF THE PGP.......................................................................................... 30 v Participant Perceptions of the Activity Impact.......................................................... 30 Bank Staff and Partner Views on Impact .................................................................. 34 Summary.................................................................................................................... 36 7. FACTORS INFLUENCING PROGRAM OUTCOME....................................... 37 Analytic Strategies..................................................................................................... 38 Results........................................................................................................................ 40 Summary.................................................................................................................... 42 8. SUSTAINABILITY OF THE PGP........................................................................ 44 Contact Information and Activity Follow-up ............................................................ 44 WBI's Comparative Advantage................................................................................. 45 Summary.................................................................................................................... 46 9. CONCLUSIONS AND RECOMMENDATIONS ................................................ 47 Conclusions................................................................................................................ 47 Lessons Learned and Recommendations................................................................... 49 ANNEXES ....................................................................................................................... 51 Annex 1: List of Sampled Events/Activities ............................................................. 53 Annex 2: Participant Survey...................................................................................... 54 Annex 3: Activities Represented by Survey Respondents ........................................ 59 Annex 4: Focus Group Questions.............................................................................. 61 Annex 5: Instruments for Individual Interviews........................................................ 62 Annex 6: Variables Description................................................................................. 66 Annex 7: Results of Multiple Regression Analyses ­ Basic Models ........................ 68 Annex 8: Path Analysis of Effects of Perception Variables on Use.......................... 70 REFERENCES................................................................................................................ 72 vi EXECUTIVE SUMMARY The Poverty and Growth Program (PGP) was one of 16 World Bank Institute (WBI) programs in operation during FY02 and FY03.1 The objective of the program was to help reduce poverty in client countries by building capacity of individuals and countries in preparation for and effective implementation of Poverty Reduction Strategy Papers (PRSP). In FY02 the PGP delivered more than 60 activities to approximately 3,000 participants, and in FY03, 36 activities to more than 1,800 participants. Approximately one-half of the activities were seminars, 30 percent were courses, and 20 percent were conferences. Nearly 80 percent of the events used a face-to-face delivery mode, and partners were involved with two-thirds of the activities. This report examines the extent to which the FY02 and FY03 PGP achieved its objectives. EVALUATION METHODS For this evaluation, a sample of PGP participants was contacted during FY04 and asked to assess the specific activity they had attended -- between July 2001 and June 2003 -- for its relevance, effectiveness, and impact on their work and their country. Evaluation information was gathered from 261 participants from 14 countries who participated in 25 of the FY02 and FY03 PGP activities. Structured focus groups were held with a subset of participants in Burkina Faso, Ethiopia, Nigeria, Tajikistan, and Indonesia. Additionally, individual interviews were held with six Bank Operations staff, seven WBI PGP Task Team Leaders (TTLs), and 12 activity partners. Key program documents were also reviewed. From this combination of qualitative and quantitative approaches, the evaluation addressed five questions: (1) What is the relevance of the program? (2) What is the effectiveness of the program? (3) What is the impact of the program? (4) What factors influence the program outcome? (5) What is the sustainability of the program? 1. PGP was called the Attacking Poverty Program (APP) through FY03. vii KEY FINDINGS The evaluation concluded that the PGP has supported client country poverty reduction strategies by building the capacity of individuals to design and implement a PRSP. In particular, the program has successfully addressed the needs of low-income countries. These conclusions are based on the following key findings: Activities of the PGP were rated more positively by participants coming from low- income countries than by participants from middle-income countries.2 Participants from low-income countries rated the program approximately 10 percent more relevant to their work and to their countries compared with participants from middle-income countries. Twice as many participants from low-income countries -- more than one-half -- reported high use of learning content compared with one-quarter of participants from middle-income countries. And nearly 40 percent of participants from low-income countries compared with less than 20 percent of participants from middle-income countries reported that the PGP learning activity resulted in positive changes in their countries. Overall, the PGP was rated as moderately relevant, effective, useful, and having positive impact. Participants rated three of the six effectiveness dimensions more highly than the others: (a) providing new knowledge and skills to participants, (b) raising awareness of development issues important to participant countries, and (c) helping participants understand their role as agents of change for development. These areas were also those in which the participants most frequently used what they had learned in the activity. Participants were nearly unanimous in reporting that the activity positively "influenced or led to changes" in the areas of teaching, research, and raising awareness. A minority (20 percent), however, reported that the activity had a negative impact on "implementing country development strategies." Within the PGP, activities with certain design and delivery features received higher ratings for program outcomes. These activities: (a) were held within the participant countries; (b) used action plans during the event; (c) were shorter than average (less than seven days); and (d) were perceived as relevant to participants and their countries. Although many of the quantitative and qualitative results were consistent, a few were not, particularly for language of instruction and partner involvement. All stakeholders interviewed recommended that more activities be offered in the participants' local language; however, the correlation between holding an activity in the participants' work language and the activity's perceived effectiveness was negative. Similarly, the analysis shows that partner involvement in PGP events did not yield a higher outcome rating, although stakeholders underscored the importance of working with partners. Among the PGP participants, some reported making comparatively more use of the learning content. The higher users: (a) came from low-income countries; (b) were more familiar with technical terminology; and (c) perceived the events as more relevant to their 2. Participants refer to participant-respondents. viii specific needs. Lower use was attributed to barriers existing in participants' work and country environments, such as instability, low financial and technological capacity, and the influence of cultural perceptions. Systematic follow-up with past participants was lacking. FY02-03 PGP participants reported receiving little systematic WBI follow-up. We note, however, that since FY03, many initiatives and strategies have been designed and implemented to ensure the sustainability of the program as part of the WBI country focus initiative. LESSONS LEARNED AND RECOMMENDATIONS According to the analysis and in conjunction with information collected from the various stakeholders, the evaluation offers several lessons and recommendations. A country-focus approach is key to program success. · The higher rating from low-income country participants indicates the appropriateness of the program design for the PRSP countries. The program needs to place more emphasis on strengthening organizational capacity within these countries, as well as on ensuring intensive and widespread training within those organizations. · While continuing to focus on low-income countries, the program might also consider revising or redesigning program content for participants from middle- income countries who might benefit from greater sensitization to poverty reduction and country-wide inequality issues. The PGP could improve its overall modest impact by enhancing its design and implementation in four ways. · Increase the relevance of activities to the needs of participating countries; · Prepare action plans during learning events; · Deliver learning events in participant countries; and · Select participants with a high level of proficiency in technical terminology. Long-term sustainability is a major challenge. · Long-term sustainability of the PGP impact is a major challenge. The new WBI country focus approach has presented a great opportunity for the program to address and meet the challenge by establishing in-country partnerships and designing long-term sustainability strategies at a relatively early stage of the program. ix x 1. INTRODUCTION EVALUATION CONTEXT AND OBJECTIVES 1.1 Poverty reduction remains the key global challenge. By building individual and institutional capacity in client countries, WBI aims to contribute to this mission. Through both traditional and distance learning methods, WBI and its partners deliver knowledge- based options to policymakers, technical experts, business and community leaders, and civil society stakeholders; foster analytical and networking skills to help client countries make sound decisions; design effective socioeconomic policies and programs; and unleash the productive potential of client country societies. 1.2 The WBI Evaluation Group (IEG) conducted the first impact evaluation of thematic and sector programs in FY02­03, focusing on the six largest of 15 programs delivered in FY00­01. In FY04, four WBI thematic learning programs implemented in FY02­03 were evaluated (Table 1). The Poverty and Growth Program (PGP) is one of the programs included in the FY04 evaluation. Table 1: Programs selected for evaluation FY04 FY02 Community Empowerment and Social Inclusion Decentralization Poverty and Growth Education Social Protection Good Governance City and Urban Management Health and Population --- Natural Resource Management --- Knowledge for Development 1.3 Building upon findings and experience of the first impact evaluation in FY02, the FY04 evaluation was designed to: · Examine the outcomes and impact of the PGP at the individual, institutional, and country levels; and · Form some recommendations that can be used to improve WBI programs. 1.4 This chapter presents a brief overview of the poverty program. Chapter 2 provides a description of the evaluation methodology, including its key evaluation questions, scope, and data collection strategies. Chapters 3 through 9 present the major findings and recommendations of the evaluation as follows: · Chapter 3: Relevance of the program 1 · Chapter 4: Effectiveness of the program · Chapter 5: Utilization of the program · Chapter 6: Impact of the program · Chapter 7: Factors influencing program outcome · Chapter 8: Sustainability of the program · Chapter 9: Conclusions and recommendations THE POVERTY AND GROWTH PROGRAM3 Background and design 1.5 Early in 2000, the Poverty and Growth Program (known as Attacking Poverty Program until 2004) was established by WBI in order to increase the focus on poverty in its learning programs. The PGP was conceived as a coalition, seeking to draw together WBI staff, other Bank experts, and key external players in the fight against poverty. The overall objective of the program was to support the preparation for, and effective implementation of Poverty Reduction Strategy Papers (PRSP) through a comprehensive learning package. This would be achieved through activities aimed at building the capacities of: · National PRSP country teams and other stakeholders to successfully prepare, implement, and monitor the PRSP; and · Local and regional institutions involved in poverty related activities. Furthermore, the PGP will seek to develop a support system for such centers by encouraging the creation of "communities of practice", linking expert bodies both within and across regions. 1.6 The PGP included social debates, analytical and applied skills training, and policy courses and workshops. All activities are designed according to the following four key principles: (1) Demand driven -- initiatives were designed in consultation with national authorities and World Bank country teams. (2) Country specific -- educational materials designed to meet the specific needs and demands of individual countries. 3. Information presented in this section is derived from several sources, including, the Attacking Poverty Program Financial and Progress Report FY02/03, the WBI Annual Report FY02/03, the WBI PGP Website, the PGP Newsletters, and the PGP Program Brief. 2 (3) Medium term -- a horizon of three years enabled WBI to develop capacities with appropriate phasing, and to increase the relative responsibilities of partners over time, thus ensuring local ownership and sustainability of the program. (4) A coordinated response -- the PGP aimed to coordinate its efforts with donors and other regional and international partners. 1.7 The target audiences for the PGP were all stakeholders involved in the design and implementation of PRSPs in client countries. They included the national PRSP teams, government officials, civil society organizations, private sectors, media, parliament, and research and policy institutes. 1.8 The program became an important channel through which national PRSP teams requested and received support from WBI. In addition, the PGP intended to assume a lead role in coordinating WBI's poverty reduction programs with other development agencies. 1.9 In FY04 the Attacking Poverty Program changed its name to the Poverty and Growth Program (PGP), reflecting a deepening and broadening of the program, within which was formed the Poverty Analysis Initiative (PAI), the PRSP Design and Implementation Initiative (PRSDI), and the Macroeconomics and Policy Assessment Skills Program (MPASP). New modules were developed to address the following issues: mainstreaming gender in poverty, poverty and social impact analysis, trade integration, disability, and development. Program implementation 1.10 In FY02 the PGP and its partners delivered more than 60 learning activities to more than 3,000 participants from more than 20 countries. More than one-half of the beneficiaries came from sub-Saharan Africa and about one-third were nominated by PRSP teams. More than 60 percent of the activities were delivered via distance learning.4 1.11 In FY03, the PGP delivered 36 activities to 1,956 participants from 97 countries. A total of 6,302 training days were delivered via face-to-face, and 3,107 distance learning. Civil society was the biggest group of beneficiaries with 929 participants from academia, the media, the private sector, and Non-Governmental Organizations (NGOs), followed by ministries and government officials with 723 participants, including three Ministers. Bilateral donors and other international organizations had 170 participants in the PGP activities in FY03.5 4. Progress Report for the World Bank's Fiscal 2002, Reporting Period: July 1, 2001 - June 30, 2002, Attacking Poverty. 5. FY 03 Report to Donors (July 1, 2002 ­ June 30, 2003). 3 2. EVALUATION METHODOLOGY EVALUATION QUESTIONS AND SCOPE Evaluation questions 2.1 The evaluation was designed to address these five questions. (1) What is the relevance of the program? (2) What is the effectiveness of the program? (3) What is the impact of the program? (4) What factors influence the program outcome? (5) What is the sustainability of the program? 2.2 Table 2 summarizes the main evaluation questions, sub-questions, and the key information sources. Table 2: Evaluation questions and sources of data Evaluation Questions Sources of data Q 1 What is the relevance of the program? Participant survey Q 2 What is the effectiveness of the program? Participant survey · How effective are the PGP programs in enhancing participants' awareness, Participant focus groups knowledge, skills, strategies, and networking? Q 3 What is the impact of the program? Participant survey · What is the impact of poverty program on its participants as measured by use of Participant focus groups learning? Partner interview · What is the impact of the poverty program on the ground? Bank staff interview · What are facilitators or barriers to the utilization? Q 4 What are the factors that influence the program's effectiveness and impact? Participant survey · What features of participants are related to the effectiveness and impact of the CRS program? · What features of learning activities are related to the effectiveness and impact of the program (features include: length of the course, delivery modes, instruction language, action plan, follow-up, etc)? Q 5 What is the sustainability of the program? Participant survey · What effort did the program make to enhance and ensure the sustainability and WBI TTLs interview built capacity? Partner interview · What is the WBI comparative advantage? 4 Evaluation scope, sampling procedure, and results 2.3 The current evaluation was designed to focus on the PGP learning activities delivered in FY02 and FY03 lasting longer than one day. The starting point of the sampling was at the country level and the participant level, not at the activity level (event). With limited resources, this approach allowed us to: (1) cover a wide variety of activities within the program (e.g., core courses, seminars, workshops, study tours, forums) and delivery modes (face-to-face and distance learning); (2) achieve a diverse participant group including policymakers, academics, media personnel, and other civil society members; and (3) integrate implementation with IEG's concurrent country retrospective impact evaluation, which focuses on the countries that WBI will likely concentrate on in the future. 2.4 Our sampling procedure included the following steps: (1) creating a list of eligible program participants (those who attended 1+ day events in FY02­03) who had at least one contact point (address, phone, or email); (2) including all individuals who were to be surveyed for the country focus evaluation in FY04; (3) including individuals from the countries that were to be reached by other three thematic programs (Community Empowerment, Urban Management and Social Protection) in FY04; and (4) drawing 30-50 percent random samples of program participants from countries with high proportions of participants.6 2.5 This procedure yielded a list of 424 participants7 from 14 countries and covered 27 learning events. The sample represented 24 percent of the participants reached by the PGP during FY02-03. In seven of the 14 countries, the eligible participant population was relatively small, so we included the entire eligible population in the survey. Columns 1 and 2 in Table 3 show the sample size, while Annex 1 lists activities covered by the evaluation. 6. Although some of these countries are not candidates for FY04 country focus evaluations, these countries are often the WBI priority countries, and were thus within our resource and technical capacity. Technical capacity refers to trained consultants on the ground. 7. Excluding one-day activities. 5 Table 3: Program population, sample, and survey response rate Sample (including Countries PGP population replacements) Response Rate (%) Focus groups Low-income countries Burkina Faso 58 58 11 (19) FG Ethiopia 51 51 28 (55) FG Ghana 39 39 30 (77) India 37 37 33 (89) Nigeria 23 23 18 (78) FG Kenya 32 12 12 (100) Tajikistan 42 42 27 (54) FG Yemen 5 5 3 (60) Middle-income countries China 7 7 3 (43) Indonesia 73 73 36 (49) FG Macedonia 9 9 8 (89) Serbia & Montenegro 14 14 13 (93) Thailand 29 29 26 (90) Yugoslavia 71 26 14 (54) Other countries (not included in the sample) 1323 Total 1821 426 261 (68) DATA COLLECTION 2.6 The evaluation used a combination of quantitative and qualitative methods to collect information for the analysis, including: (1) A quantitative survey of a sample of participants who attended at least one PGP event; (2) Focus group discussions with participants of select PGP activities from select countries; (3) Qualitative interviews with Operations/country office staff, WBI PGP task team leaders (TTLs), and WBI local partners; and (4) A desktop review of program documents and event features. 6 2.7 Data collection was carried out between December 2003 and March 2004. This section provides a summary of the instruments/procedures, scope, and the result of each data collection activity. Participant survey Instrument 2.8 A uniform survey instrument was used across the thematic programs studied in FY02 and in this evaluation (Annex 2). Specifically, the survey instrument seeks information from course participants about: · Their perceptions of WBI activities in terms of relevance and effectiveness; · Their use of knowledge and skills learned from the WBI intervention; and · Their perceived impact of the activity at the organization and/or country levels. 2.9 In addition, participants were asked whether the language of instruction was the same language they use at work, and their assessment of their level of proficiency in the language of instruction and in technical terminology used during the learning event. Response rate and characteristics of the learning events and respondents 2.10 With around 13 percent replacement, 426 participants from 14 countries were surveyed. A total of 261 participants completed the survey questionnaires, making a response rate of 68 percent (Table 3). · Nearly 33 percent of the sampled participants were from the Africa Region, followed by 27 percent from East Asia and 13 percent from South Asia, and 25 percent from Europe and Central Asia. Only 1 percent of the participants were from the Middle East and North Africa. · Sixty-two percent of the respondents were from low-income countries (N = 161), while 38 percent of the participants were from middle-income countries (N = 100). · Sixty percent of the survey respondents were male and 40 percent were female. Approximately 51 percent worked in government at various levels (Figure 1). · The participants were mostly senior, top- and mid-level staff (Figure 2). · While only 55 percent of the participants responded that the language of instruction was the same as the language they used at work, nearly 80 percent said that they were highly proficient (rating of 5.70 on a scale of 1 to 7) in the language of instruction and 76 percent said they were highly familiar with the technical terminology used during the event. There is a high degree of correlation between the level of familiarity with technical terminology and 7 seniority -- i.e., participants in senior positions were likely to be more familiar with the technical terminology (coefficient of correlation =-0.22, P = 0.00). Figure 1: Composition of survey respondents by type of organization Other Entry level Highest level 6% 6% 11% Junior level 14% Senior level Middle level 32% 31% Figure 2: Composition of survey respondents by position 10% 3% 2% 17% 15% 40% 11% 2% University/Research NGO Media Private Sector National Govt Provincial govt Local govt Other 2.11 These respondents represent 25 learning events that are listed in Annex 3A. Annex 3B also provides Level 1 scores for 18 of these courses and shows that the average scores of most events were higher than the average across all WBI events in the same study periods. 2.12 Moreover, according to the WBI Client Record System (CRS) and participant survey, around 65 percent of the learning activities (N = 171) lasted less than seven days and 35 percent of the activities (N = 90) lasted longer than 7 days. Courses appear to be longer (Mean = 7.61 days) than conferences (Mean = 3.66 days), and seminars (Mean = 4.58 days). Other key features of the learning events attended by survey respondents are summarized in Table 4. 8 Table 4: Features of learning events attended by survey respondents Feature variables Number (%) Type of activity Conference 48 (18.3%) Course 84 (32.0%) Seminar 108 (42.2%) Location In-country 101 (39.0%) Outside country 160 (61.0%) Delivery mode Face-to-face 187 (76.6%) Distance learning 3 (1.3%) Blended 57 (23.3%) Product line Skill building 197 (75.1%) Knowledge exchange 14 (5.7%) Policy services 32 (13.7%) Other features Partner involvement 135 (55.5%) Action learning 129 (53.1%) Representation of the respondents 2.13 To better understand the final survey sample, comparisons were made between survey respondents and the remaining population. Table 5 shows that, overall, there are some differences between two groups with respect to available demographic variables. More surveyed participants attended events outside the country and events that were delivered using mixed modes. Table 5: Comparison of survey respondents with remaining study population Respondents Remaining population Variables N=261 N=165 2 Location of learning event In-country 101 (39.0%) 112 (66.8%) 5.85*** Outside country 160 (61.0%) 54 (33.1%) Delivery mode Face-to-face 187 (76.6%) 139 (83.3%) 1.14 Distance learning 3 (1.3%) 7 (3.9%) 8.19*** Blended 57 (23.3%) 19 (12.8%) 7.77*** Sector representation Academic/research 44 (16.9%) 21 (13.3%) 1.73* NGO/Private 72 (27.8%) 43 (27.8%) 0.09 Government 132 (50.9%) 80 (47.5%) 2.19* Other 11 (4.8%) 20 (12.4%) 6.71*** ***p<=0.001, **p<=0.05, *p<=0.10 9 Focus group discussions 2.14 Outcome measurement not only considered quantitative aspects such as the number of activities, participants, and training days, it was also based on a qualitative measurement of the learning events (e.g., relevance and added value of courses for beneficiaries; strategic position of participants; probability/capacity of implementing new knowledge and skills). These qualitative measurements required a very precise methodological approach. Formal (questionnaires after the course) and informal (demand assessment) evaluations were good indicators, but they needed to be combined with more specific verification techniques. Focus group discussions and individual interviews provided additional information for thorough qualitative research. The purpose of focus group discussions was to obtain more detailed information from participants, to allow participants to elaborate on their feedback about the effectiveness of training courses, and to solicit participant recommendations for improving WBI learning activities. 2.15 Five focus group discussions were conducted with participants who attended PGP learning activities in Burkina Faso, Ethiopia, Nigeria, Tajikistan, and Indonesia.8 Each focus group had 8-12 participants. Annex 4 provides the focus group protocol and questionnaire. Individual interviews 2.16 Semi-structured interviews were conducted with the following people: · Six Bank Operations staff in the Africa Region; · Seven PGP TTLs/managers and WBI staff; and · Twelve representatives of donors, local partners, and international organizations who were involved in financing and implementing the PGP activities, mostly in Burkina Faso, Ethiopia and Washington DC. 2.17 These interviews took place between February and April 2004 in Washington and the country offices. All were conducted face-to-face and each lasted 30-60 minutes. Questions focused on the alignment of the program with the Country Assistance Strategy (CAS) and PRS process; the relevance of the activities to the development needs of the client countries; the organizational or policy impact of the program, and the barriers to implementation of new knowledge and skills. Though asked to focus specifically on FY02­03 PGP activities, interviewees also commented on the overall functioning of the PGP and WBI in general. These general comments and remarks have been integrated into this evaluation report on the basis of their relevance to lessons learned and formulation of recommendations. Additionally, suggestions and recommendations for quality improve- 8. The reasons for selecting these five countries out of the 14 were: (a) they represented the highest proportion of participants in the program; (b) these countries were the overlap countries with the FY04 country focus evaluation; and (c) Burkina Faso and Ethiopia were the countries of interest for the Belgian Trust Fund that provided financial support for this study. 10 ment of WBI services were elicited. The interview questions appear in Annex 5, together with the list of the organizations represented by interviewees. Desktop review 2.18 A desktop review was conducted to obtain information on the program design and implementation. Information was obtained from: · The WBI PGP Web site, the PGP Newsletters, and the PGP Program Brief; · The WBI Annual Reports FY02­03; · The Draft Country Program Briefs of Burkina Faso and Ethiopia; and · The WBI Client Record Systems (CRS). 11 3. RELEVANCE OF THE PGP 3.1 As reviewed earlier, the PGP was designed using four principles: demand-driven, country-specific, medium-term, and a coordinated response. Before beginning examination of program effectiveness and impact, we first looked at the degree to which PGP activities reflected those four principles and the relevance of the PGP to participant work and their country needs. We attempted to answer these questions via a participant survey, focus group discussions, and interviews with stakeholders. PARTICIPANT VIEWS ON THE PROGRAM RELEVANCE Overall views 3.2 Survey participants were asked about if the program was designed specifically for their country, the relevance of PGP activity topics to their country's needs and to the kind of work they do. Relevance is measured on a 7-point scale ranging from "Not relevant at all" to "Extremely Relevant". 3.3 Nearly 51 percent of the participants indicated that the activity they attended was not specifically designed for participants from their country, while 15 percent responded that they had no opinion. 3.4 Although many participants did not feel that the activity was designed specifically for their country, participants provided moderately high ratings on activity relevance. On a 7-point scale, the mean rating was 5.40 on relevance to their country's specific needs and 5.82 on relevance to participants' work. Since there is a correlation between relevance to country needs and relevance to work (Coefficient of correlation = 0.39, p = 0.00), we created one composite measure by taking the mean of the two separate aspects of relevance to simplify our analysis. The mean rating of the composite measure of relevance was 5.60 (SD = 1.09). 3.5 Looking closely, two-thirds of the participants perceived the activity they attended to be highly relevant to their country (ranking at 6 or 7 on a 7-point scale). Only two percent perceived the topics covered by WBI learning activities to be irrelevant. Participants said relevance of the activity to their work was lower. About 50 percent considered the activity to be highly relevant (ranking at 6 or 7 on a scale of 1 to 7), and four percent considered it to be irrelevant to the work they do. This feedback may suggest that while the activities targeted the broader issues of participants' country-specific needs, a considerable number of the participants did not feel that the content directly addressed their daily tasks. Nevertheless, they did take away some specific knowledge or skills that could be applied to their jobs. 12 Views from different participant groups 3.6 We examined participant perceptions according to their demographic characteristics and the features of the learning events that they attended. Using the aggregated relevance measure, we saw statistically significant variations in the mean rating across different participant groups in terms of demographic characteristics and course features. · Compared to participants from middle-income countries, participants from low- income countries gave a significantly higher mean rating to relevance (Mean = 5.76 vs. 5.36, t = 2.87, p = 0.004). According to the mean score distribution, there is a 10 percent increase in perceived relevance between participants from low-income countries (at ratings higher than 6, Figure 3) and participants from middle-income countries. Figure 3: Distribution of ratings for perceived program relevance, by country status Low income n=161 Middle income n=100 40 35 34 sesn 30 25 23 spo 20 re of 15 %10 5 0 1-2 >2<=3 >3<=4 >4<=5 >5<=6 >6<=7 Mean rating score 1-7 from low to extreme relevance · Likewise, while comparing participants who work at different levels of government, we found that participants working in university/research, media, and the private sector gave a higher mean rating on relevance (Mean = 5.45 vs. 5.76, t = 2.36, p = 0.02). · Participants with higher positions perceived the relevance of the WBI poverty program to be higher than those with middle or junior positions (Mean = 5.77 vs. 5.48, t = 4.50, p = 0.04). · There were statistically significant variations in perceived relevance among participants with proficiency in technical terminology. Participants who responded that they were highly proficient (with rankings of 6 or 7) in technical 13 terminology perceived the activities to be more relevant than participants who were either moderately proficient (with rankings at 4 or 5) or less proficient (with rankings of 1 to 3) (f = 12.07, p = 0.00). There were no variations in the ratings between genders. Message from focus groups 3.7 Focus group discussants from Burkina Faso, Ethiopia, Nigeria, Indonesia, and Tajikistan agreed on the overall relevance of the WBI programs in terms of personal, professional, or research activities. The relevance for country development needs was confirmed. All focus group discussants agreed on the high quality of content in the learning events and their potential for future networking. 3.8 Focus group participants from Nigeria said that the activities they participated in were highly relevant to their work as social researchers. The programs were not only germane to their personal research interests, they were also relevant to the focus of their institute (place of work) and to the country's development needs. 3.9 In Indonesia, many participants from regional governments claimed the content of WBI events was important, however the selection criteria for participants may not have ensured that the "correct" people received the training. In using criteria (a) ability to speak English; (b) belongs to the unit for which the training is relevant; and (c) position as echelon 2-3 in the government civil service, it is likely that only some participants, particularly from outer regions, were chosen to attend the workshop in Jakarta (presumably with government per diems) as a "reward". This was implied during the discussion although no participant explicitly admitted to this. BANK STAFF AND PARTNER VIEWS ON PROGRAM RELEVANCE 3.10 While participants considered the PGP to be relevant to their work and the needs of their country, Operations staff had a somewhat different view of the program relevance. According to Operations staff in Burkina Faso, WBI programs, including the PGP, were less country-specific. 3.11 Operations staff and partners were also asked about the alignment of the PGP with the CAS in Burkina Faso. One Operations staff member rated the alignment of the Country Assistance Strategy designed in FY02 and FY03 with WBI activities (using a scale of 1 to 7), at 6 for content and 3 for implementation. One partner (donor agency) gave a rating of 5 to alignment with the capacity enhancement needs of Burkina Faso, and another gave a rating of 4.5. Greater alignment with the CAS was confirmed for content, methodology, and implementation. WBI themes are relevant to actual needs, but perhaps more from a World Bank perspective than from the perspective of the local population. Overall, both Operations counterparts and donor partners indicated that WBI activities in the PGP were largely supply-driven. 14 3.12 By contrast, interviews with Bank Operations staff9 and partners in Ethiopia revealed that the FY02­03 activities were aligned relatively well with the ongoing PRSP process. The second Africa Forum on poverty reduction strategies provided great opportunities for South-to-South contact. These initiatives need to be further encouraged and stimulated. A purely national approach will not respond to such demands for this kind of learning. The impact evaluation course was extremely useful in Ethiopia for the PRSP core team and related Ministries. The PGP course for journalists and civil society was deemed as a good start, but interviewees mentioned the need to target more participants from civil society, journalists, and policymakers. 3.13 In Tajikistan, almost all interviewees agreed that WBI was ineffective and that WBI activities tended to be "totally supply-driven". Most interviewees agreed that the "quality and content of WBI activities is an issue", and there appeared to be disjunction between local needs and challenges and the content of WBI activities. 3.14 Focusing on a client country was considered as a priority rather than a global view. An increase in information sharing between the WB country office and WBI remains desirable, especially since Burkina Faso is a focus country. A more country- specific approach could be highly beneficial. Since country office staff working at sectoral levels know the real problems and challenges, they could make use of the WBI products, even in the execution of their regular tasks. 3.15 Operations staff recommended that constructive engagement with WBI be continued to further enhance capacity. It was suggested that more learning events should be developed and provided by local institutions, think tanks, and universities. They also suggested that WBI should try to avoid too many themes; it was better to focus on some themes in depth. A thorough analysis of client needs should be a priority. WBI PGP TTLS INTERVIEW 3.16 Interviews with the WBI PGP TTLs revealed that considerable efforts had been made to make the program relevant to the needs of client countries. In the past few years, the PGP has been primarily demand-driven. In particular, in FY02 a Needs Assessment questionnaire (seeking interest on topics for more than two years) was sent to several PRSP/low-income countries, based on which courses were designed and delivered during the past years. Since then, the PGP has become even more demand-driven (because the demand is not brought in by Regional Capacity Enhancement Teams (RCET), but double checked with needs assessment/CAS/PRSP, as well as with Poverty Coordinators, Sector Managers, and Country Directors/Program coordinators. 3.17 Initially, content of the poverty program concentrated on individual learning support and focused on training people; now the PGP also focuses on the environment in which the new knowledge will be applied. This evolution toward a real capacity building approach marks the most significant change for the program. Formerly, the thematic groups were central to the structure of the program, and WBI was organized as if it were 9. The Country Director of the World Bank office in Ethiopia was the WBI PGP team leader. 15 a university. WBI must not limit itself to training; it is also necessary to help create an environment where it is possible to use new knowledge. Since WBI has moved to a capacity building organization and decentralization via country-based budgeting, this will enable us to tackle contextual environmental issues. 3.18 In the TTL interviews, the agenda for WBI activities had more impact when the CAS was given thoughtful consideration. Knowledge must be relevant and available for the client governments at the right moment. The timing of the training is crucial -- "when you come in at the wrong moment, impact is compromised". The content of learning events was not always customized to country needs, government officials were not always interested in general policy issues, and they seemed to be more interested in detailed and specific issues. It is very important to select participants carefully and to offer a country specific course: "you need to know your audience if you want to get a real impact". 3.19 Now that WBI has moved toward a country-focused approach, up to 80-90 percent of PGP activities are country-based, and more field office staff are participating in WBI activities. PGP has also embarked on a number of pilot programs, such as activities with intensive multi-year engagements with specific institutions. New audiences such as NGOs are now involved, and specific themes such as the integration of youth and child issues in the PRSP are being emphasized. SUMMARY 3.20 Both participant surveys and focus groups indicated that PGP activities were moderately relevant to their work and country. Participants from low-income countries found the courses, on average, to be more relevant than those from middle-income countries. This may reflect that (a) course material and content were geared more heavily toward low-income countries, which can be seen as evidence of a demand-driven approach; and (b) the program was more responsive to the PRSP initiative, which is geared to low-income countries. Given the clear focus of the PGP on low-income and PRSP countries, it is therefore not surprising to see a lower relevance score from participants from middle-income countries. 3.21 Throughout the study period, the PGP took several new initiatives reflective of the four program principals -- needs assessment, involvement in operational tasks and civil society, and community organization. Because the organizational structure of WBI is not always explicit, Operations staff and local donor partners may not be fully aware of some new approaches and initiatives. This may be the reason that most of the Operations staff and partners interviewed consider the PGP, among other WBI activities, to be largely supply-driven instead of demand-driven. 16 4. EFFECTIVENESS OF THE PGP 4.1 Survey participants were asked to assess the program effectiveness in six different aspects: · Raising participant awareness/understanding of development issues; · Providing participants with knowledge/skills; · Helping participants understand their role in their country's development; · Providing strategies/approaches to address the needs of participant organizations; · Providing strategies/approaches to address the development needs of the country; and · Helping to network among people with common interests. Overall views 4.2 Using an aggregated measure of effectiveness,10 participants provided an overall average rating of 5.33 (SD = 1.05), indicating that the activities were considered to be moderately effective. 4.3 The overall average rating of the PGP was computed by combining the six dimensions of effectiveness measured on a 7-point scale11 with 1 denoting "not at all effective" to 7 denoting "extremely effective." When the results were disaggregated (Figure 4), three activities of the PGP were considered to be highly effective by a majority of the respondents. The participants gave high ratings to providing knowledge and skills (Mean = 5.75), raising awareness (Mean = 5.51), and helping understand the participant's role as agent (Mean = 5.34). Helping participants network (develop contacts and partnership) was on the low end of effectiveness. 10. Results of the factor analysis emphasize the significant correlations and the minor differences between the levels of effectiveness in the six designated areas (Annex 6A). The results of the analysis show that only one factor summarizes the overall level of effectiveness. 11. The responses were categorized into low effectiveness (ratings 1 and 2), moderate effectiveness (ratings 3-5) and high effectiveness (ratings 6 and 7). 17 Figure 4: Participant ratings of effectiveness by six aspects of effectiveness 100% 80% 55 51 48 46 45 63 60% 40% 48 49 42 46 48 20% 34 0% 3 3 3 4 6 6 Providing knowledge/ Raising awareness/ Understand role in devt Strategy/ approaches for Strategy/ approaches for Networking skills understanding process country organizations Ineffective Moderately effective Extremely effective Views of different participant groups 4.4 Comparisons of different participant groups show that there is no significant variation in perceived effectiveness across gender, organization and position. Variation does exist, however, between participants coming from different country categories. Participants from low-income countries perceived the activities to be more effective (Mean = 5.45) than participants from middle-income countries (Mean = 5.13). The difference in the perceived effectiveness between participants from low- and middle- income countries was statistically significant according to the t-test (t =-2.34, p = 0.002). Nearly 10 percent of participants from low-income countries rated effectiveness higher (over 6) than participants from middle-income countries (Figure 5). Figure 5: Perceived program effectiveness, by country status (mean score) Low income n=161 Middle income n=100 50 e 40 30 26 respons of 20 17 %10 0 >2<=3 >3<=4 >4<=5 >5<=6 >6<=7 Mean rating score 1-7 from low to high effect 18 4.5 Similar to the relevance rating, participants who said they were highly familiar with course terminology rated the event's effectiveness higher than those with lower familiarity (f = 7.89, p = 0.000). 4.6 Participant perceptions of effectiveness seem to be less related to features of the events they attended. As summarized in Table 6, there is no variation in perceived effectiveness across course features such as mode of delivery (face-to-face vs. distance learning), types of activities (conference vs. courses and seminars), action learning, and location. Participants who attended activities in which action plan development took place gave a higher effectiveness rating. Surprisingly, participants who attended activities in which partners were not involved in delivery and design rated the effectiveness significantly higher. Because we did not ask participants for their views on partner involvement in the survey, it is difficult to confirm if this lower score can be attributed directly to the partners. Table 6: Participant ratings of effectiveness, by course features (mean score) Course features Yes No t In-country event 5.26 5.38 -0.86 Language of instruction same as participant's work language 5.28 5.41 -0.98 Action learning 5.29 5.37 -0.29 Face-to-face 5.36 5.26 1.05 Conference 5.30 5.34 -0.24 Partners were involved in course design and implementation 5.26 5.51 -2.33** Use of action plans during the course 5.54 5.18 2.71** Follow-up 5.51 5.28 1.53 ***p< = 0.01, **p< = 0.05, *p< = 0.10 Message from focus groups 4.7 During focus group discussions, numerous gains and benefits directly related to PGP learning events were cited. According to participants, PGP learning events helped them: · To become more aware of poverty issues such as: - the function of stakeholders in the PRSP process; - new concepts such as sustainability, risk management, vulnerability, and transparency; - the skills and tools necessary to reduce poverty; and - the dynamics of social exclusion. · To build knowledge and develop skills in the area of poverty reduction, including: 19 - a change in approach from "top down" to "bottom up" that allows for dialogue between government and civil society; - improved poverty measurement and modeling; - thorough research activities, such as a study of the links between the development of health services, health policies, and macroeconomic programs; - writing policy memorandums; - improved use of the Web and reference texts for research; and - networking. Summary 4.8 Overall, both participant surveys and focus group results suggest that the PGP activities were moderately effective. The PGP was seen as particularly effective in influencing individual knowledge and awareness. Participants highly familiar with terminology and participants from low-income countries are more likely to rate effectiveness highly. Among all course features, developing an action plan during the learning event enhances effectiveness the most. 20 5. USE OF LEARNING CONTENT 5.1 In the previous two chapters we saw that survey respondents considered the relevance and effectiveness of the PGP as moderate. Next we look at use. PARTICIPANT USE OF LEARNING CONTENT 5.2 Using the survey, we asked participants to report on how often, if at all, they used the content they learned from the PGP in the following seven areas: · Research; · Teaching; · Raising public awareness of development issues; · Implementing new practices at work; · Organizing collective initiatives; · Influencing legislation and regulation; and · Implementing country development strategies. Overall utilization 5.3 Between 62 and 93 percent of respondents said they used learning content when applicable. In general, the mean rating of use was moderate. On average, 74 percent of participants exercised at least one dimension of use. A vast majority of 93 percent considered teaching to be a component of their jobs, but relatively few, 62 percent, said they had implemented new practices at work. 5.4 Among participants who reported occasional use of course content, the level of use ranged from 47 percent for teaching to 38 percent for implementing country development strategy (Figure 6). More notably, between 8 and 26 percent made little to no use of the knowledge and skills for a given area.12 Looking closely, use was seen strongest in teaching (Mean = 5.13) and raising public awareness (Mean = 5.05), followed by research applications (Mean = 4.73) --the areas reflecting largely individual work and responsibility. 12. Responses on a 7-point scale, with 1 denoting "no use" and 7 denoting "high use", were re-categorized as follows: very often (ratings of 6 and 7), occasionally (ratings of 3 to 5) and not at all (ratings of 1 and 2). 21 Figure 6: Participant ratings of frequency of use, by areas of use 100% 80% 38 37 34 47 46 45 39 60% 40 40% 39 48 46 50 46 47 20% 26 15 16 8 8 13 13 0% Teaching Research Awareness New work Country devt Collective Legislations practices str initiative and regulation Notatall Ocasionally Very often Use by different participant groups 5.5 The level of use of learning content varied for participants with different demographic and course characteristics: · Male participants reported a higher mean for use than female participants (t = 2.16, p = 0.03). Participants working in government sectors (either national or provincial/local) made less use of the knowledge and skills in their given areas of work (Figure 7). · The mean rating of participants from government organizations was 4.48 compared to 5.06 for participants in other organizations. The difference in the mean rating was statistically significant (t = 3.56, p = 0.00). Similarly, participants holding high positions in their organizations tended to make better use of the content than those holding lower positions (Figure 8). These observations may be explained by the fact that most senior or government positions in low-income countries were held by males. 22 Figure 7: Participant ratings of use, by type of organization (mean score) N=248 Government 4.48 (n=132) Academic/research (n=44) 4.98 NGO/Private (n=72) 5.12 1 2 3 4 5 6 7 Not use at all to frequently use Figure 8: Participants' ratings of use, by position (mean score) N=234 Highest level (n=28) 5.5 Senior level (n=77) 4.8 Junior level (n=79) 4.6 Middle level (n=35) 4.6 Entry level (n=15) 4.2 1 2 3 4 5 6 7 Not use at all to frequently use · On average, participants from low-income countries reported use approximately 20 percent more than participants from middle-income countries (Mean = 5.08 vs. 4.26). This difference is statistically significant (t = 4.96, p = 0.000). A notable portion of participants (52 percent) from low-income countries were high and frequent users, compared to 25 percent from middle-income countries (Figure 9). This huge difference may again confirm that the PGP was specifically designed to meet the needs of low-income countries. · There is a significant gap in use between participants with different levels of familiarity with course terminology. Participants with high familiarity used course content about 17 percent more than that the low-moderate familiarity group (Mean = 4.96 vs. 4.29, t = 3.95, p = 0.00). 23 Figure 9: Distribution of mean scores for use by country status Low income n=161 Middle income n=100 35 30 31 sesn 25 20 20 21 spoer 15 of % 10 5 5 0 1-2 >2<=3 >3<=4 >4<=5 >5<=6 >6<=7 Mean rating score 1-7 from low to high use · In addition, Table 7 shows that participants highly rated those courses with learning features that used a face-to-face delivery mode. Participants attending the conferences or courses/events that used action plans reported significantly higher use than those who attended either courses or seminars. Again, participants attending events involving partners reported fewer instances of use than participants who attended events not involving partners. Table 7: Participants' ratings of use, by course features (mean score) Course features Yes No t In-country event 4.65 4.83 -1.07 Language of instruction same as participant's work language 4.69 4.89 -1.17 Action learning 4.87 4.64 1.67 Face-to-face 4.81 4.51 2.28** Conference 5.37 4.67 3.29*** Partners involved in course design and implementation 4.51 5.08 -3.85*** Use of action plans during the course 4.95 4.63 1.89* Follow-up 4.90 4.74 0.85 ***p< = 0.01, **p< = 0.05, *p< = 0.10 Message from the focus groups 5.6 Focus group discussions revealed very similar findings across countries. Participants from Burkina Faso, Ethiopia, Nigeria, and Indonesia said that the most common use of WBI learning content was (a) concept and approach; (b) knowledge and 24 techniques; (c) dissemination; and (d) networking. These examples were offered by participants: · Adapting a participatory approach to revise the PRSP at both the governmental and civil society levels; · Applying community-driven concepts to institutional reorganization initiatives; · Carrying out empirical research, e.g., building dynamic models and poverty measurements; · Organizing training and seminars for further dissemination of course content; · Using Web-based material and reading materials as references; and · Building contacts and networking with colleagues from other departments and countries. 5.7 There seemed to be a difference in the aspects of use depending on a participant's organization. In Indonesia, participants from the central office did not find the discussion of data collection approaches/techniques very useful. On the other hand, these discussions were very useful to attendees from regional offices (whose main job was to collect data). In contrast, the participants from regional offices mentioned that sessions on data analysis were not immediately useful to them (though these sessions were valued by researchers from the central office). 5.8 In addition to the common use reported in other countries, in Tajikistan all participants agreed that the useful aspects were to: · Provide comparative development experience from other countries; · Emphasize the role of civil society in monitoring and evaluation of programs; and · Provide helpful references that could be used after the training. 5.9 Furthermore, participants indicated that the most useful aspect of the activities to their country is "bringing together NGOs and the government; until then the government gave no value to NGOs". one participant said, "...It was the first time that Tajikistan discussed involving the civil society in decision-making". 5.10 These two focus groups in Tajikistan also shared views on the elements of WBI activities that make them either useful or difficult to apply. Features that made WBI activities useful were their ability to: (a) open a dialogue about the role of civil society in decision-making and monitoring of government programs; and (b) expose participants to relevant case studies and lessons learned from other developing countries. Features that discouraged use were: (a) WBI's lack of follow-up; (b) WBI's lack of support for 25 implementation of PRSP; and (c) the lack of comparable levels of knowledge and experience locally. FACILITATORS AND BARRIERS TO USE 5.11 The participant survey explored the extent to which two specific environmental factors helped or hindered use: · Work environment -- work procedures, colleagues, incentive system, and funding; and · Country development environment -- macro factors such as political status, country policies, social and political groups, readiness for reforms, etc. 5.12 Participant ratings show that neither of these two environmental factors hindered use. Only 7 percent or fewer found environmental factors to be obstacles. In fact, nearly 40 percent of participants noted that these factors were helpful in using the knowledge and skills they gained from the learning activity (Figure 10). Figure 10: Participant ratings of helpfulness of environmental factors 100% 80% 54 55 60% 40% 40 38 20% 0% 6 7 Work Environment Country Environment Unhelpful Helpful Neither helpful nor unhelpful 5.13 In focus group discussions, participants were asked to identify facilitators and barriers to the use of knowledge and skills acquired in WBI activities. There were many similarities across different countries. In Burkina Faso and Ethiopia, the focus group discussants cited several barriers. Some were linked to the program, but most were external factors. Factors linked to the programs that had a direct or indirect effect on the level of learning and implementation of new knowledge and skills included: · Transport costs for persons coming from up-country limited their fully participation in the program. · Modalities of the program were not ideal for a decentralized approach. 26 · Selection process of participants influenced the probability of implementation of knowledge. Motivated and interested persons who participated in the events on the basis of willingness may be more willing to implement new knowledge than participants who were required to be there. · Expectation that further support for implementation would come from WBI. The new concepts were not institutionalized, so it was difficult to implement them without additional support. There was no focal person to contact and guide them in this matter. 5.14 External factors with a negative influence on the implementation of new knowledge and skills, and consequently on the institutional impact of the programs, included: · Instability of organizational environment - Lack of trained staff to implement new knowledge and skills.; - Changing positions and locations; and - Administrators who are overloaded with operational matters and therefore have little time for innovative strategic thinking. · Low financial and technological capacity - Limited access to computers and inefficient Internet services; - Lack of access to accurate data and the absence of information systems; and - Insufficient budget and financial capacity. · Influence of perception and culture - Leading officials or superiors in public services were reluctant to implement new approaches as they prefer secure and traditional approaches/methods. The acceptance of change is a slow and delicate process and needed approval from senior civil servants and political authorities. - The "gender" approach is often present in documents and statements, but on the ground, nothing changes. 5.15 The findings in Burkina Faso and Ethiopia were mirrored by findings in Tajikistan and Indonesia. Specific barriers in Tajikistan were: · Resistance to change in the system. As also noted by Operations counterparts, all focus group participants agreed that there is an overwhelming resistance to 27 change in governments and organizations. Local governments were reluctant to change procedures in the work environment, making it difficult to implement new ideas. · Lack of adequate/reliable data. Lack of available data on basic macro indicators and demographics of the poor hindered implementation of new knowledge and skills acquired in WBI training. Participants particularly complained about the lack of reliable and consistent data on the socioeconomic indicators in Tajikistan. BANK STAFF AND PARTNER VIEWS ON BARRIERS TO USE Bank staff views 5.16 In Ethiopia, Operations staff who were interviewed mentioned that a real constraint to implementation of new knowledge was the low capacity of the administration. "We need to define clearly who we want to train. In general, `the best' are selected, but this may not always be the right approach. Results may be better at the mid-level." 5.17 Likewise, the WBI TTLs interviewed mentioned difficulties in finding accurate and appropriate data, limited access to appropriate software, as well as changes in career paths of trainees and political constraints within the client countries. They also mentioned that mobility of trained staff is an important barrier for implementation of new knowledge and may negatively affect the impact of training. Partner views 5.18 Local partners echoed the findings from focus groups and Bank staff interviews about barriers to use of knowledge and skills. In Burkina Faso, partners mentioned several barriers, such as a lack of financial support for training. Financial accessibility to new knowledge was limited (80 percent barrier), and donors were frequently asked to finance participant training. Partners insisted that mechanisms are needed to fund the training of civil servants. Without external support, NGOs can't participate as they do not have the required funds. 5.19 Cultural features were also mentioned as a barrier. In Africa, an individual has to follow the group and innovations are seen as destabilizing. Time and investments are needed to influence decisionmakers so that they can understand the advantages of changes. It was said that the highest political levels need to be convinced, and that they should give successful examples. Decisionmakers often have no confidence in the local younger generation and their capacity. 5.20 New technologies were, however, very attractive and have a potential in African oral and visual culture. Local decisionmakers are stuck between local and international cultures. New technologies can be highly useful as a bridge. 28 5.21 Corruption was also mentioned as a great risk for real changes. Good governance and a fair and functioning justice system are needed to reach the development goals. Partners who were interviewed also appreciated the practical use of software. However, although courses provided a lot of information on different programs, the tools needed to make use of the software were not provided. More operational and specific tools were needed for strategy development and implementation. 5.22 In addition, local partners in Ethiopia cited relatively high fees as a barrier. As one interviewee noted:..."An average of US$ 200 per hour is asked for satellite connection costs. With a maximum of 40 participants per course and some courses lasting 16 hours, the financial cost of participation is considerable." 5.23 Finally, the limited number of libraries or information centers in the country and a very high rate of illiteracy were barriers. Technology remains a luxury in Ethiopia. At the information center in the World Bank office, six computers were available for public use and served about 800 people each month. Access time was limited so that as many people as possible had a chance to use the Internet. SUMMARY 5.24 As seen in the participant survey, the majority of survey respondents used knowledge and skills gained from the WBI activity at a moderate level. Participants from low-income countries reported significantly high use compared to those from middle- income countries. Use was largely concentrated on concepts and knowledge, and the event content was related to individual working responsibility, such as teaching, raising public awareness, and carrying out research activities. Although the survey did not identify any specific barriers to use, participants, Operations staff, and partners pointed out a number of barriers, including instability of organizational environment, low financial and technological capacity, and the influences of perception and culture. Clearly, these barriers are the challenges that the program faced in achieving its objectives in those countries. 29 6. IMPACT OF THE PGP 6.1 Thus far, the analysis has dealt mainly with participant perceptions of PGP relevance and effectiveness, as well as the extent to which they have used program content. In this chapter we attempt to determine the impact the poverty program has had not only on participants, but also at the institutional and country levels. PARTICIPANT PERCEPTIONS OF THE ACTIVITY IMPACT Overall participant perceived impact 6.2 In order to measure changes at the institutional and country levels, we asked participants to rate the degree to which PGP activities influenced or brought changes in the specific area addressed. Among the 91 percent who reported that the learning activity brought changes in the sector or area addressed by the activity, the vast majority (87 percent) rated the changes higher than 5 on a scale of 1 to 7 (1 to 2 = Negative, 3 to 5 = Neutral, 6 to 7 = Positive change). 6.3 More specifically, we asked participants to report their perception of changes resulting from the PGP in the following seven areas: · Research; · Teaching; · Raising public awareness about development issues; · Implementing new practices at work; · Organizing collective initiatives; · Influencing legislation and regulation; and · Implementing country development strategies. 6.4 Through factor analysis we identified one factor that summarizes the overall measure of impact across seven areas. 6.5 The majority of participants considered the activities to be very effective in each individual dimension of impact (Figure 11). Similar to the results for use, the most notable changes, according to participants, were teaching, followed by research and raising awareness. Comparatively, the observed impact appeared to be weak in terms of "Organizing collective initiatives" and "Influencing legislation/regulation". Some 30 participants (19 percent) said that WBI events led to negative changes in the area of "Implementing country development strategies". Figure 11: Participant ratings of the activity g impact 100% 80% 61 70 60% 78 90 85 82 81 40% 20% 3 34 27 16 17 8 13 19 0% 2 2 5 2 2 2 Teaching Awareness Research New work Country devt Collective Legislation practices str initiative and regulation Negative Influence (1 or 2) No Influence (3 to 5) Positive Influence (6 or 7) Perceived impact by different participant groups 6.6 There were statistically significant variations across different participant groups who reported activity-induced positive changes. · Participants from NGOs, the private sector, and the media reported higher levels of change than participants working in academic and government sectors (Figure 12). · Participants in entry-level positions gave high ratings to the program impact (Figure 13). However, this group is comparatively small and was not the key target audience for the PGP. Given that PRSP-related processes and events apply largely at the national level, this perception of impact may reflect different expectations for changes/impact for these two groups (entry-level and the remaining groups). · Consistent with participant perceptions of the relevance, effectiveness, and use value of the PGP, participants from low-income countries also gave significantly higher ratings to program impact than participants from middle- income countries (Mean = 5.74 vs. 5.28, t = 3.78, p = 0.000). In fact, 22 percent more low-income country participants gave an impact score above 6 (Figure 14). 31 · Furthermore, participants who attended activities that were categorized as action learning, delivered via face-to-face, and that did not involve partners, gave higher ratings to program impact than did their counterparts (Table 8). Figure 12: Activity impact as perceived by participants, by type of organization (mean score) N=248 Government (n=132) 5.41 Academic/research 5.59 (n=44) NGO/Private (n=72) 5.92 1 2 3 4 5 6 7 Negative to positive change Figure 13: Activity impact as perceived by participants, by position (mean score) N=249 Entry level 5.96 (n=15) Junior level 5.54 (n=35) Middle level 5.48 (n=79) Senior level 5.55 (n=77) Highest level 5.48 (n=28) 0 1 2 3 4 5 6 7 32 Figure 14: Distribution of rating score for perceived program impact, by country status Low income n=161 Middle income n=100 45 40 sesnops 38 35 30 25 re 20 of 15 16 % 10 5 0 1 >2<=3 >3<=4 >4<=5 >5<=6 >6<=7 Mean rating score 1-7 from low to high impact Table 8: Activity impact as perceived by participants, by course features (mean score) Course features Yes No t In-country event 5.63 5.51 0.91 Language of instruction same as participant's work language 5.53 5.65 -0.92 Action learning 5.76 5.48 2.10** Face-to-face 5.67 5.30 2.67** Conference 5.68 5.57 1.21 Partners involved in course design and implementation 5.32 5.71 -2.89*** Use of action plans during the course 5.67 5.50 1.32 Follow-up 5.66 5.56 0.73 ***p< = 0.01, **p< = 0.05, *p< = 0.10 Message from the focus groups 6.7 Because the survey questions are close-ended, participant surveys provided no substantive information about the impacts/changes noted. Using participant feedback from focus group discussions, however, we were able to glean some information about the nature of changes induced by PGP activities. The following examples show how poverty learning events had an impact at the individual, policy, organizational, and community and public levels. · Individual level - Integrated advanced knowledge, theories, and techniques in the financing field and in poverty assessment techniques; and 33 - Added new concepts, knowledge, and training materials in teaching. · Policy level - Influenced policy-making at strategic and sectoral levels and strategic and sectoral plans; - Incorporated pro-poor elements into tax policies such as taxation on lower incomes; - Developed a national policy on social protection; and - Improved sustainable water management in natural resource strategies. · Organizational level - Formed new coalitions and agreements between NGOs and government; - Worked with people from civil society and the private sector; - Reorganized the structure of a civil society network, with national, regional, and provincial units; and - Aligned an annual NGO work plan with the PRSP. · Community and public level - Developed community-driven approach to learning; - Popularized the PRSP process; - Produced radio programs on poverty reduction issues; and - Improved the quality of media reports on the PRS. BANK STAFF AND PARTNER VIEWS ON IMPACT 6.8 Interviews with Bank Operations staff, WBI TTLs, and local partners yielded mixed responses about the impact of the PGP. Bank staff views 6.9 Most Bank Operations staff interviewed in Washington, D.C., were not able to give detailed comments on the FY02­03 poverty program. When interviewed, most had different responsibilities than they did two years ago. Their comments about WBI and the PGP activities were general, rather than specific. For example, the efforts to get the WBI activities aligned with the CAS in Ghana were highly appreciated by the Operations staff, especially capacity-building activities and the economy. For Ghana, the RCET played a 34 very constructive role in the preparation phase of the CAS. The current WBI approach, which focuses on non-traditional sectors such as knowledge and capacity building, strengthening parliaments, and monitoring and evaluation, was highly appreciated. 6.10 In Ethiopia, World Bank staff13 perceived WBI's program impact as meaningful. Overall, the country is receptive to further connections with WBI products and stakeholders were willing to be involved. 6.11 By contrast, WBI program outcomes in Tajikistan are not evident to Operations staff. None could pinpoint a specific outcome of a WBI program in Tajikistan during FY02­03. The primary reason cited for lack of impact was that WBI activities tended to be supply-driven rather than demand-driven. Country office staff indicated that prior to FY04, WBI had not made any country needs assessments and was therefore not responsive to country needs. In addition, country staff indicated that there is no monitoring of the implementation process, therefore "the results do not materialize". Interviewees also noted that although there was "usually a lot of enthusiasm at the end of a course, nothing happens". The main reason cited for low impact in Tajikistan was the lack of follow-up. One interviewee from Operations commented that..."Irrespective of what feedback we have given, they (WBI) go out and do what they want to do". WBI TTLs views 6.12 WBI TTLs said it is very difficult to measure medium- or long-term outcomes at the institutional and policy levels. However, they have observed that Central and Eastern European participants from the development forum traveled to other countries after the forum for further discussions with their colleagues. In Macedonia the training on poverty measurement had a technical impact on their capacity to develop appropriate research. The Macedonia Poverty Forum provided an opportunity for discussions on achievements and progress, and allowed participants to exchange views with World Bank staff. Participants noted that they grew professionally and that development debates prepared countries very well for the PRSP forum. These events allowed interaction between government and civil society. The joint learning initiative with (United Nations Development Program (UNDP), and Department for International Development (DFID) increased the quality of the forum. Partners views 6.13 When asked about the overall impact of the PGP activities in FY02­03 on the capacity enhancement needs of Burkina Faso, one partner organization responded that impact was too difficult to measure due to the lack of provisions for follow-up support in partner contracts. One partner participated in, along with four economists from Burkina Faso, a course in Dakar on "Questions budgétaires et calcul du coût des DRSP et les liens entre commerce International et la Pauvreté", 2003. In the medium-term, partners mentioned that the activities themselves were the outcome, and that the development of the Capacity Enhancement Needs Assessment (CENA) concept in Burkina Faso was both 13. The Ethiopia country manager in FY04 was a former WBI Program Leader for the APP /PGP. 35 the result and outcome of a WBI discussion. Working with WBI is both challenging and stimulating. 6.14 More positive messages came from interviews with two WBI partners in Ethiopia -- the Civil Service College and the NGO Inter Africa Group. Both partners considered the impact of the WBI initiatives as certain, especially for governmental organizations because senior experts and even some Ministers and Vice-Ministers participated in the PGP events. Both partners were convinced that the courses on journalism had a real impact. Participants from government newspapers and from the private sector received accurate information on the PRSP process and on standards for professional journalism, expertise, and responsibilities. Several informed articles and reports on the PRSP appeared in these newspapers; one major newspaper published a special magazine with six articles on PRSP. SUMMARY 6.15 According to the survey, the impact of the PGP was greater in the areas of individual work and responsibility -- teaching, research, and raising public awareness. The observed impact was particularly pronounced in low-income countries. Interviews with some stakeholders were also evidence that the program brought some positive impact/changes at individual, policy, organizational, and community levels. Across various features of the PGP learning events, action learning and face-to-face delivery modes seem to be associated with a higher impact. The presence of a WBI focal point in Ethiopia was highly appreciated because such administrative arrangements ensured smooth implementation of WBI activities, including the PGP in the country. 36 7. FACTORS INFLUENCING PROGRAM OUTCOME 7.1 In the preceding chapters, we presented a descriptive analysis of the relevance, effectiveness, and impact of the WBI poverty program. As anticipated, the findings are encouraging. The PGP was relevant to most participants and effective in providing participants with useful and necessary content. The program also influenced use of the information by participants and was beneficial to both participants and client countries. Findings also suggest that the course characteristics of the PGP were largely well aligned with participant needs. Among the course characteristics studied, PGP influence on participant countries was particularly pronounced and consistent across all outcome measures (Figure 15). Figure 15: Relevance, effectiveness, use, and impact of the PGP, by country status (mean score) 7 )hgih 5.76 5.45 5.56 6 5.36 5.14 5.08 5.28 very 5 to 4.27 wol 4 (very 3 gn 2 Rati 1 Relevance Effectiveness Use Imapct Low-income country n=161 Middle-income country n=100 7.2 In this chapter, we extended our analysis to explore the factors that influence program outcomes. We seek answers to the following questions: What influences participant use of the learning content? What implications do these factors have on policy? Understanding the mechanisms through which these factors affect program outcomes will enable us to make more relevant recommendations to improve the program. 37 ANALYTIC STRATEGIES Conceptual model 7.3 The basic conceptual framework guiding our analysis is presented in Figure 16. The key outcome variables concerned in this evaluation are effectiveness, use, and impact. In this analysis, we are particularly interested in use because we believe that a causal link between the influencing factors (independent variables) and use is more explicit than with the final impact variable. The hypothesis represented by the framework is explained in Box 1. Box 1: Study Hypothesis Based on findings from the previous evaluations (Khattri & Quizon, 2002; Prom-Jackson et al. 2002; Quizon et al. 2004) and the current analysis results presented earlier, we hypothesize that the usefulness of the PGP is determined or influenced by three groups of factors -- participants, courses, and external environment. We assume that participant use of learning is influenced by individual characteristics, including gender, language capacity, and work environment. Participants from low-income and middle-income countries, and different geographic areas/regions may also have different use potential. We also believe that course characteristics influence the success of the program. Activity characteristics include location of the event, mode of delivery, duration, and action learning. Moreover, we consider the influence of participant perceptions of the relevance and effectiveness of the event. The inclusion of these two perception variables is based on behavioral theory, and previous research showing that both factors are strong predictors for use behavior (Ajzen & Fishbein 1980). Finally, we believe that use is influenced by participants' professional and country environments. Positive or negative environments will enhance or prevent the level of use. a Detailed information on how these factors were defined, measured, and used in the analysis is summarized in Annex 6B. a. Given the relatively high degree of correlation between the two types of facilitators and barriers (Coefficient of correlation = 0.42, p = 0.004), we simplified our analysis by combining the two types of facilitators and barriers by taking means. 38 Figure 16: Basic model of studying the influencing factors of the program outcome Block 1 Individual variables Gender External Familiarity with terminology* factors Organization County status/features Block 2 Course features Location Duration Knowledge Action learning Use Partners involvement Delivery mode Use of action plans Language of instruction Block 3 Course content Relevance Block 4 Effectiveness * As there is a significant correlation between position and familiarity with technical terminology, we only included the variable of familiarity with technical terminology. Analysis procedure 7.4 We performed the analysis in two steps.14 First, we examined three basic models in sequence: influence of trainees (participants), influence of training (course features), and combined influence of trainees and training. At this stage of the analysis, we intentionally focused only on these two sets of variables because we believe that most of these factors are objective measures and can be obtained before the learning event is offered. Because the nature of these variables is pre-determined, the results of the analysis would be meaningful for course designers, particularly to choose targets and delivery modes. 7.5 In the second step, we studied how participant perceptions, together with pre- determined variables, influenced the final outcome. As such we have extended the first stage analysis by including two perception variables: relevance and effectiveness. Specifically, we treated effectiveness as an intervening variable that we hypothesize may bridge predetermined variables and the final outcome. The purpose of the extended analysis is to increase our understanding of the mechanism and condition under which the program outcome occurs. Technically, in addition to a simple multiple regression study used to estimate the direct effect of "relevance" on use, we applied a path analysis to determine "effectiveness" (Duncan 1971). By borrowing the path analysis technique, we are attempting to investigate some possible mediating effects of perception variables. The detailed path analysis procedure is explained in Annex 8B. 14. We reported the results based on the un-weighted sample because a similar analysis using the weighted sample showed that the results were almost identical. 39 RESULTS Influence of the pre-determined variables (trainees and training factors) 7.6 Three multiple regression analyses were performed to test the effect of predetermined variables on the impact variable "use of content". These variables were introduced into the regression models by blocks as summarized in the box below. The advantage of introducing independent variables by block is that this process creates an opportunity for the evaluation to look not only at the relationship between dependent and independent variables, but also to explore possible interactions between independent variables. Model A: utilization=f (participant characteristics) Model B: utilization=f (course characteristics) Model C: utilization=f (participant characteristics + course characteristics) Annex 7 lists the coefficients of these regression analyses. In sum, the regression analyses performed for the impact variable use of content supports our primary study hypothesis about the influence of participant characteristics and course features. Taken in order of the relative importance of the significant effects, the coefficients in the full basic model (Model C ­ participants + course variables) show that: · Participants from low-income countries were nearly 100 percentage points more likely to use the course content than participants from middle-income countries; · Participants who attended courses that did not use a native/participant work language for course instruction were 71 percentage points more likely to use the course content than the courses using a participant's native language; · For every increase of one unit of familiarity with terminology of the event, participants were 22 percentage points more likely to use what they learned in the WBI course; · Compared to courses involving partners, participants who attended a course without partners involved were 60 percentage points more likely to use course content; · For every decrease of one unit of course duration, participants were 7 percentage points more likely to use what they learned from the WBI event. · Compared with the course that did not use/develop action plans, participants who attended the course involving the use of an action plan were 27 percentage points more likely to use course content; and · Participants who attended the courses delivered within their own countries were nearly 40 percentage points more likely to use the course content, compared with those who attended the courses that were offered off shore. 40 7.7 In addition to the influence of individual variables, the regressions implied some possible interactive effects among the independent variables, i.e., duration of the event event location and language of instruction are the result of accumulated or interacted effects of participant and course variables. This finding suggests a need to further investigate the predictive interaction of the variables in terms of how they affect the outcome variable. Influence of perception variables 7.8 The purpose of the first step analysis was to describe the direct influence of participant characteristics and course features on use of content. From the policy point of view, it seems possible to improve use by refining the course design. However, it is unlikely that the influence of individual characteristics will change even if improvements are made to course design. Thus, the analysis results have raised questions: · Do we need to worry about the differences across various demographic groups and types of events in terms of how they affect use? If the answer is "yes", then · Should the program redefine its targets, focusing only on those with high use potential? Or should the program consider leveraging the differences by improving program features and effectiveness? 7.9 To answer these questions, we turned our analysis to the perception variables -- relevance and effectiveness. Results 7.10 The regression result for combined effect of relevance with other pre-determined variables on use is shown in the last column (Model D) of the basic model table in Annex 7. Compared to the basic Model C, the presence of relevance in the model does not significantly alter greatly the influence of participant characteristics and course features. Rather, it shows its own strong and significant influence on the outcome; i.e., if participants perceived the course was designed to be relevant to them, they would be likely use the course content. 7.11 Detailed regression results of the path models "effectiveness and use" are presented in Annex 8. The significant paths established are displayed in Figure 17. The conclusions we draw from the second stage analysis are summarized as follows: · Participant perceptions of program effectiveness are associated with the participant characteristics and course features. Participants with high levels of proficiency of the event/course terminology and Participants from low-income countries are more likely to give a higher rating to the effectiveness of the event. Of the seven course feature variables, using an action plan and non- native language of instruction are positively associated with a higher effectiveness rating (Model E). 41 · However, the significant effects of all pre-determined variables on use disappear completely when course relevance is considered. This suggests that the influence of relevance on effectiveness is rather strong, standing well ahead of the influence of other factors that are important on their own (Model F). · Effectiveness as perceived by participants is also a strong predictor of use. However, it does not play a significant role in mediating the effects of pre- determined variables on use. That is to say, the effects of participant characteristics and course features on the final outcome, or use, were mainly direct and independent. On top of the influence of pre-determined and perception variables, a positive external environment enhances the utilization (Model G). SUMMARY 7.12 Overall, the results presented in this chapter have confirmed most differences observed in the earlier chapters about use across various participant groups and course features. Participant utilization behavior is positively influenced by their familiarity with course terminology, country status, and event location (in country vs. outside country). All these findings are consistent with all results derived from various sources and highlight the importance of the country focus approach. Moreover, the positive association between events of short duration and a higher level of use may support the finding that participants who attended conferences (often short in duration) were more likely to use course content than those who attended courses and seminars. 7.13 Conversely, longer-lasting events, partner involvement, and use of native/participant work language have a negative affect on use. Such results seem to be contradictory to some findings from other sources. For example, messages from most focus groups and interviews have emphasized the importance of using native languages to deliver learning events and build strong partnerships. This discrepancy may be due to the fact that some terms specified in CRS as discussed by other stakeholders (participants and Operations staff) are quite different. Participant perception of "partner involvement", for instance, may totally differ from what is categorized in CRS. 7.14 Nevertheless, the important message derived from these analyses is that both pre- determined variables and participant perceptions had direct and independent effects on outcome. The analyses of both the basic and extended models highlight the strong influence of pre-determined variables on the outcome. The perception variables indicate only their own direct and strong effects on use of content. The mediating role of effectiveness seems rather weak. These results imply that although participant perception variables could profitably be a focus of policy attention by course designers to maximize program impact, the strong influence of pre-determined factors cannot be forgotten and negated. As we pointed out earlier, it may be possible to modify some pre-determined course features, however, it is impossible to alter participant characteristics, unless we change the recruiting policy and implementation approach (from global to country focus). 42 Figure 17: Final model of factors influencing use Familiarity of terminology 0.12** External Factors Low-income country 0.71*** 0.14** In-country event 0.33* Knowledge Duration -0.07* Use Partner involvement -0.41* Language of instruction -0.44** 0.34*** 0.21*** 0.21** Effectiveness Relevance 0.42*** ***p< = 0.01, **p< = 0.05, *p< = 0.10 43 8. SUSTAINABILITY OF THE PGP 8.1 From the earlier chapters we have seen that the PGP has been considerably successful in achieving its stated objectives. In this chapter we explore possibilities to sustain the program. Using information collected from various stakeholders, we attempt to address the following sustainability questions: (1) Were the participants provided with a contact list of other participants and did they use the contact list to continue discussions and organize follow-up activities? (2) Was a follow-up to the learning activity implemented? (3) Does WBI have a comparative advantage in providing training on issues related to poverty reduction? CONTACT INFORMATION AND ACTIVITY FOLLOW-UP 8.2 The program appears to be making good use of the low-cost means of ensuring sustained presence by providing contact information to participants. Contact information for other participants was provided to 71 percent of participants. Of these, more than 25 percent of the participants never used the contact information, 54 percent used the contact information to organize discussions, and 13 percent used the information to organize follow-up activities. 8.3 On the other hand, evidence from participant questionnaires and qualitative feedback during focus groups and interviews show that there is a lack of follow-up by WBI after the event is over. According to the survey, three-quarters of the participants (73 percent) reported that there was no follow-up by WBI. Only 14 percent of the participants contacted WBI to follow-up on issues or questions related to content of the learning activity. 8.4 In focus group discussions and interviews, feedback indicated that lack of adequate follow-up was a major impediment to use of new knowledge and skills. Tajikistan participants indicated that there was no follow-up "in any shape or form" by WBI. One participant said "by creating action plans and discussing PRSP, we were actually under the impression that there would be follow-up on implementation -- but nothing!" More comments include, "It would be very ambitious to see any real changes happening without any monitoring or follow-up." "They spent a lot of money to send us to training, but no one followed up to see if this money was well-spent." 44 8.5 Similar opinions were also voiced during interviews with Operations staff and partners in Ethiopia, Indonesia, and Tajikistan. In Ethiopia, two partners who were interviewed mentioned that improved follow-up should be a priority. "A lot of courses are available; we have the list and marked those courses that could be interesting for Ethiopia. Ninety percent of our civil services requested courses on good governance, but the college received no immediate feedback from WBI and took contact with the International Development Law Organization (IDLO) of the International Labour Organization in Rome." o Another partner interviewed in Burkina Faso commented: "it is very difficult to measure medium-term outcomes since there is no follow-up support foreseen in the contract with the partners." 8.6 Similar experiences were shared by some Bank staff members in Ethiopia, who mentioned that feedback questions and remarks from participants and facilitators were sent to WBI Washington DC, but that no reply had ever been received. Follow-up must be taken seriously. 8.7 This lack of follow-up is not surprising to the PGP TTLs. "There is only so much follow-up that one can do from Washington, D.C., but there is also a shortage of complementary partners to take over this task", according to the PGP TTLs. 8.8 It is also apparent from the TTL interviews that despite difficulties, the program has been making efforts to ensure the sustainability of program impact through a set of new initiatives. The Poverty Action Community, multi-lingual Web sites, and video conference/internet follow-up courses are just a few examples. Because most of these initiatives started after FY02­03, some initiatives affecting impact may not have been captured by the current evaluation. WBI'S COMPARATIVE ADVANTAGE 8.9 We assessed WBI's comparative advantage in providing learning events related to poverty reduction by asking participants to compare the quality of the WBI activity they attended to that of activities/services provided by other institutions. 8.10 Only 26 percent of the respondents had participated in similar learning activities offered by non-WBI organizations in their country. Participants noted that the activities provided by other institutions were, on average, slightly less effective than the WBI learning activity they attended. The average rating comparing WBI activities to non-WBI activities was 4.7 on a scale of 1 to 7, with 1 indicating that the WBI activity was much less useful, 4 denoting that both were about the same, and 7 denoting that the WBI activity was much more useful. 8.11 In focus groups and interviews, participants indicated that depending on different countries, WBI faced competition from non-WBI sources in offering training related to the PGP. According to interviewers conducted in Tajikistan, WBI is not unique in 45 Tajikistan. Several agencies, such as UNDP, the German Technical Assistance Agency, the United States Agency for International Development (USAID), the Asian Development Bank (ADB), and the Soros Foundation, are all competitors. "While WBI is the leader in the training for government, for country-driven development, others are equally good". 8.12 Likewise, focus group discussants in Indonesia indicated that similar learning products are offered in universities, although WBI does bring in a useful international perspective. Similar events are also offered by the Japan International Cooperation Agency (JICA) and ADB in Jakarta. 8.13 On the other hand, the PGP TTLs felt that despite the many donor agencies residing in many of our client countries, the learning products provided by WBI are unique. Participants in low-income countries tended to value PGP activities highly because there are not many players in capacity enhancement in these countries. SUMMARY 8.14 Thus, while it appears that WBI has a comparative advantage in providing unique learning products related to poverty reduction in many client countries, there was not enough effort to ensure that initial activities pay off on a longer-term basis through provision of on-going support and follow-up. Due to the lack of local counterpart institutions (e.g., local universities or research institutes), follow-up is substantially difficult. It is hoped that by focusing on building specific institutions for analytical work on poverty within the country focus approach, this problem should be mitigated gradually. 46 9. CONCLUSIONS AND RECOMMENDATIONS 9.1 The WBI PGP was designed to support the Bank's mission of poverty reduction. By providing learning products to its client countries, the PGP objective is to contribute to reducing poverty in client countries by building capacity for both individuals and countries in preparation for and effective implementation of PRSP. This evaluation has examined the extent to which the objective was achieved by program activities in FY02 and FY03. CONCLUSIONS 9.2 The evaluation concluded that the PGP has achieved its stated objective of supporting client countries to fight poverty by building capacity for individuals in designing and implementation of PRSP. In particular, the program successfully addressed the needs of low-income countries and brought a positive impact. These conclusions are based on the following key messages extracted from the analysis: 9.3 The PGP was more significant for participants from low-income countries. · Participants from low-income countries rated the PGP as more relevant to their work and countries than those from middle-income countries (Mean 5.76 vs. 5.36). · Twice as many participants from low-income countries -- overall one-half -- reported high use of learning content compared to one-quarter of participants from middle-income countries. · Nearly 40 percent of participants from low-income countries compared to less than 20 percent of participants from middle-income countries reported that the PGP learning activity resulted in positive changes in their countries. 9.4 Participants rated the PGP as moderate for its overall relevance, effectiveness, usefulness, and impact. . · Participants rated the program as moderately relevant with a mean score of 5.60 on a 7-point scale. · Participants gave program effectiveness an average rating of 5.33 on a 7-point scale. Of the six effectiveness dimensions measured, the participants gave high ratings in the areas of providing knowledge and skills (Mean = 5.75), raising awareness of development issues important to their countries (Mean = 5.51), and helping participants understand their role as agents of change (Mean = 5.34). The activities were rated as relatively less effective in providing strategies and approaches to address the needs of the participant's organization. 47 · Participants reported moderate use of knowledge and skills gained from the PGP activities (Mean = 4.76). The use was strongest in teaching (Mean = 5.13) and raising public awareness (Mean = 5.05), followed by research applications (Mean = 4.73). · The program brought a moderate impact in the countries studied (Mean = 5.56). Greater impact was evident notably in the areas of individual work and responsibility -- teaching, research, and raising public awareness. However, a relatively large group (19 percent) reported that the activity had a negative impact on implementing country development strategies. 9.5 Within the PGP, activities with different design and delivery features received different ratings for program outcome. Although many of the quantitative and qualitative results were consistent, a few were not. · Activities rated higher in terms effectiveness and use were (a) held within the participants' countries; (b) used an action plan; (c) were relatively short in duration (less than seven days); and (d) perceived as relevant to participants and their countries. · All stakeholders interviewed recommended offering more activities in the participants' local language; however quantitative analysis shows that the correlation between holding activities in the participants' work language and perceived effectiveness was negative. Likewise, the analysis shows that partner involvement in PGP events did not yield a higher outcome rating, although stakeholders underscored the importance of working with partners. 9.6 Among the PGP participants, some were able to make comparatively more use of the learning content. · The higher users: (a) came from low-income countries; (b) were more familiar with technical terminology; and (c) perceived the events as more relevant to their specific needs. · The evaluation revealed that lower use may be attributable to barriers existing in participant work and country environments, such as instability, low financial and technological capacity, and the influence of cultural perceptions. 9.7 A systematic follow-up with past participants is lacking. · The survey shows that there was a lack of systematic follow-up with the FY02­ 03 activity participants. However, many initiatives and strategies have been designed and implemented to ensure sustainability of the program as part of the WBI country focus initiative since FY03. 48 LESSONS LEARNED AND RECOMMENDATIONS 9.8 According to the analysis and in conjunction with information collected from various stakeholders, the evaluation offered several lessons and recommendations. A country-focus approach is key to program success. · The higher rating from low-income country participants indicated program design was appropriate for the PRSP countries. The program needed to place more emphasis on strengthening organizational capacity within these countries, as well as ensure intensive and widespread training within those organizations. · While continuously focusing on low-income countries, the program might also consider revising or redesigning program content for participants from middle- income countries who might benefit from greater sensitization to poverty reduction and country-wide inequality issues. The PGP could improve its overall modest impact by enhancing its design and implementation in four ways: · Increasing the relevance of activities to the needs of participating countries; · Using action plans during learning events; · Delivering learning events in the participant countries; and · Selecting participants with a high level of proficiency with technical terminology. Long-term sustainability is a major challenge. · Long-term sustainability of PGP impact is a major challenge. The new WBI country focus approach may present a great opportunity for the program to address and meet the challenge by establishing in-country partnerships and designing long-term sustainability strategies at a relatively early stage. 9.9 Participants, Operations staff, donors, and local partners have made numerous suggestions for improving the PGP. They include: · Target all levels of government. Procedures in a work environment of organizations are very difficult to change. In order to apply certain concepts the existing system needs to change. There is a need to train people from different government agencies in different positions in order to foster alliances that can promote change. · Pay serious attention to systematic follow-up. Most participants agreed that in order to ensure sustainability and monitor program impact, a systematic follow- up is a must. 49 · Establish a country partnership to improve sustainability. The presence of a focal point/partner for WBI at the country level is very important to increase the efficiency of the program and guide present and future beneficiaries. This focal point can help identify needs, targets, and follow up which could improve overall distribution of information to local governments, partners, and donors. Creating and training a pool of local facilitators to disseminate knowledge in client countries is one suggestion. Such an approach may expand the training scale and limit costs. Local facilitators can easily blend local knowledge and experience with international concepts and input and guarantee the sustainability of PGP efforts. · Offer more events locally and in the local language. Both focus groups and partners strongly urged WBI to offer more courses locally and in the local language. · Use most convenient and inexpensive modes of delivery. Financial constraints are a predominant barrier for client countries trying to take advantage of WBI learning opportunities. It is suggested that the program consider increasing the use the most convenient and inexpensive modes, such as distance learning, in low-income countries. · Emphasize pre-training preparation. The program could create a number of pre- event activities that may ensure and enhance the quality and impact of the event, such as: - Sharing information with various stakeholders prior to the events so that they may better select and attract the most appropriate participants; - Selecting partners and facilitators on the basis of both technical and communication skills; - Helping participants to prepare (e.g., requesting that they share certain problems they face); and - Informing participants about events so that they can plan expenditures well in advance. 50 ANNEXES 51 52 ANNEX 1: LIST OF SAMPLED EVENTS/ACTIVITIES No Title FY Location Duration Participants (days) (n=424) 1 Advanced Impact Evaluation (C203) FY02 Korea 6 4 2 Basic Poverty Measurement and Diagnostics FY02 Thailand 10 31 3 Basic Impact Evaluation Course - Franco Africa FY02 Burkina Faso 5 9 4 Developing a Spatial Poverty Map in Africa: Principles and Practice FY02 Kenya 5 8 5 Intro to Poverty Measurement & Diagnostics Course (C101) FY02 Kenya 6 6 6 Introduction to Poverty Measurement and Diagnostics FY02 Russian 10 5 7 Introduction to Poverty Monitoring and Evaluation FY02 Tanzania 5 9 8 PRSP COURSE - Francophone Africa FY02 Burkina Faso 3 26 9 Poverty Reduction Strategies Forum for Bank/Fund Staff. Challenges Ahead and Operational Implications FY02 USA 1.5 6 10 Regional Forum ­ Hanoi FY02 Vietnam 3 17 11 Second Africa Forum on Poverty Reduction Strategies FY02 Senegal 4 43 12 Second Poverty Forum for Europe and Central Asia FY02 Hungary 3.5 18 13 Training of Trainers Course on Basic Poverty Analysis (C101) FY02 Thailand 10 8 14 Advanced Impact Evaluation FY03 Korea 6 15 15 Advanced Impact Evaluation Course FY03 Ethiopia 4 14 16 Attacking Poverty Course - APC for Civil Society FY03 Ethiopia 9 23 17 Attacking Poverty Course- Central Asia FY03 Kyrgyz 5 16 18 Basic Poverty Measurement Diagnostics Course FY03 India 10 24 19 Basic Poverty Measurement and Diagnostics- Distance learning. Anglophone Africa FY03 Tanzania 7 23 20 Basic Poverty Measurement and Diagnostics Course for Balkan Region FY03 Hungary 9 14 21 Basic Poverty Measurement & Diagnostics Course FY03 Indonesia 6 27 22 CIS-7 Third Poverty Forum FY03 Kazakhstan 3 1 23 Development Debates - Balkan countries: Bosnia Herzegovina- Albania- FR Yugoslavia- Macedonia FY03 USA 2.5 9 24 Poverty Reduction Strategies Forum for Albania- Bosnia Herzegovina and FR Yugoslavia FY03 Austria 4 25 25 Regional Seminar for Managers of Monitoring and Evaluation FY03 Burkina Faso 3 18 26 Regional Seminar for Asian Managers of Monitoring and Evaluation of Poverty Reduction Program FY03 Thailand 3 23 Training of Trainers - Basic Poverty Course ­ 27 FY03 Kenya 10 2 Anglophone Africa Total 27 424 53 ANNEX 2: PARTICIPANT SURVEY Instructions WBI had the pleasure to have you participate in the following learning activity: Title: _____________________________________________________________ Held from: ________________________ to________________________ In: _______________________________________________________________ Getting your opinion of the above-mentioned activity--now that you have had time to reflect on it--is very important to help WBI improve its programs. For this, we ask you to complete this questionnaire. The questionnaire has four sections and should take approximately 20 minutes to complete. · Section 1 asks about the relevance of the activity. · Section 2 asks about the usefulness of the activity. · Section 3 asks you to compare this activity with similar learning activities offered by other organizations. · Section 4 asks about the characteristics of the activity, its follow-up and your background. We need your honest feedback. Please keep in mind that your responses will be kept confidential, and will be used for the sole purpose of improving WBI programs. If you have any questions about the questionnaire please call or send a message to XXXX at XXXXX. Thank you for agreeing to complete this questionnaire! 54 I. Relevance of the Activity 1. Since the end of the activity, to what degree has the activity been relevant to your work? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 2. To what degree have the topics covered in the activity been relevant to your country's needs? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 3. Was the activity designed specifically for participants from your country? Yes No Don't know 4. Was the activity related to the country development goals listed below? a. Eradicate extreme poverty Yes No Don't know b. Achieve universal primary education Yes No Don't know c. Promote gender equality and empower women Yes No Don't know d. Reduce child mortality Yes No Don't know e. Improve maternal health Yes No Don't know f. Combat HIV/AIDS, malaria, and other diseases Yes No Don't know g. Ensure environmental sustainability Yes No Don't know h. Develop global partnerships for development Yes No Don't know i. Ensure water sanitation and supply Yes No Don't know j. Improve investment climate and finance Yes No Don't know k. Promote trade Yes No Don't know II. Usefulness of the Activity 5. Please rate the degree of effectiveness of the activity in each area noted below. (If the area was not an objective of the activity, please mark "not applicable.") Not effective Extremely Not Areas at all effective applicable 1 2 3 4 5 6 7 NA a. Raising your awareness and understanding of the development issues important to your country b. Providing you with knowledge or skills c. Helping you better understand your role as an agent of change in your country's development d. Helping you develop strategies or approaches to address the needs of your organization e. Helping you develop strategies or approaches to address the needs of your country f. Helping you develop contacts, develop partnerships and build coalitions in the field 55 6. How would you rate the change--brought by the activity--in the main topic or issue it addressed? Strong Strong negative positive Don't change No Change change Know 1 2 3 4 5 6 7 DK 7. How often have you used the knowledge and skills you acquired in the activity for the following purposes? (If you have not worked in the given area since this activity, please mark "Not applicable.") Not Purposes Very Not at all often applicable 1 2 3 4 5 6 7 NA a. Conducting research b. Teaching c. Raising public awareness in development issues d. Implementing new practices within your work organization e. Organizing collective initiatives f. Influencing legislation and regulation g. Implementing country development strategies 8. To what extent did the following factors help or hurt the process of using the knowledge/skills that you acquired at the activity? Neither Greatly helped Greatly Not Factors hurt nor hurt helped applicable 1 2 3 4 5 6 7 NA a. Your work environment (e.g., work procedures, colleagues, incentive system, funding, etc.) b. Your county's development environment (e.g., country policies, social groups, political groups, readiness for reform, etc.) 9. How has the activity influenced or led to changes in the following areas? (If the area is not relevant to the activity, please mark "Not applicable.") Negative No Positive Not Areas influence influence influence applicable 1 2 3 4 5 6 7 NA a. Research b. Teaching c. Public awareness in development issues d. New practices within your work organization e. Collective initiatives f. Legislation and regulation g. Country development strategies 10. Since the activity, have you discussed the issues raised in the activity, at work, with local partners, government officials, NGOs, or in the media? Never Thoroughly discussed discussed 1 2 3 4 5 6 7 56 III. Comparison of the WBI Activity with Similar Activities Offered by Other Organizations 11. Did you participate in any similar learning activities offered by other (NON-WBI) organizations in your country? (If no, please skip to question 14.) Yes No 12. If yes, please provide the name(s) of the organization(s): 1. _____________________________________________________________________________________________ 2. _____________________________________________________________________________________________ 13. How would you rate the usefulness of the WBI activity compared to NON-WBI activities? WBI much WBI much less useful About the same more useful No opinion 1 2 3 4 5 6 7 IV. Characteristics of the WBI Activity, its Follow-up and Your Background 14. How would you describe the type of the WBI learning activity that you attended? Video Sessions Mix of Video (Distance Class room and Face to Web-based Learning) (Face to Face) Face Conference Learning Study tour 1 2 3 4 5 6 15. How effective was this type of learning activity in helping you learn? Not effective Extremely at all effective No opinion 1 2 3 4 5 6 7 16. During the WBI activity, did you develop an action plan/strategy (e.g., work plans, strategy papers, or policy documents) to apply the knowledge and skills you learned? (If no, please mark "no" below, then skip to question 18.) Yes No 17. If yes, did you use part or all of the action plan in your work? Yes No 18. Were you provided with the contact information of other participants in the activity, such as e-mail addresses, telephone numbers or mailing addresses? (If no, please mark "no" below, then skip to question 20.) Yes No 19. If yes, how did you use it? Used it to Used it to continue organize joint Never used activity related follow-up Other uses it discussions activities (Please specify briefly) __________________________ 20. Was the language of instruction used during the activity the same language you use at work? Yes No 57 21. At the time of the activity, what was your level of proficiency in the language of instruction? Not proficient Highly at all proficient 1 2 3 4 5 6 7 22. At the time of the activity, what was your level of proficiency in the technical terminology used in the activity? Not proficient Highly at all proficient 1 2 3 4 5 6 7 23. After the activity, did WBI contact you for follow-up issues regarding the activity? Yes No 24 After the activity, did YOU contact WBI for follow-up issues or questions on the content of the activity? (If no, please skip to question #23) Yes No 25. If yes, please rate WBI's helpfulness in addressing your issues. WBI WBI responded I did not responded, and was have follow- WBI did not but was not extremely up requests respond helpful at all helpful for WBI 0 1 2 3 4 5 6 7 NA 26. Which of the following best describes the organization in which you have worked the longest since the activity? (Select one.) University/research institution National/central government Non-governmental organization (not-for-profit) Provincial/regional government Media Local/municipal government Private sector Other, specify: ____________________________ 27. Which of the following best describes the primary type of work you have done the longest since the activity? (Select one.) Research Teaching Policymaking/legislation Provision of services (e.g., financial, health, etc) Management/administration Other, specify: ____________________________ 28. How would you best describe the level of the position you have held the longest since the activity? Highest level (e.g., Minister, Deputy Minister, Top Government Official, Full Professor, President of an organization) Senior level (e.g., Department Head, Division Head, Associate Professor, Senior Researcher) Middle level (e.g., Program Manager, Project Leader, Assistant Professor, Technical Expert) Junior Level (e.g., Research associate, Ph.D. level graduate student, Technical Specialist) Entry level (e.g., Intern, assistant) Other, Please specify: __________________________________________________________________ 29. What is your gender? Male Female Thank you for your feedback. We appreciate very much your cooperation. 58 ANNEX 3: ACTIVITIES REPRESENTED BY SURVEY RESPONDENTS 3A: ACTIVITY INFORMATION Duration Participants No. TITLE FY Location (days) (n=261) 1 Basic Poverty Measurement and Diagnostics FY02 Indonesia 10 25 Russian 2 Introduction to Poverty Measurement and Diagnostics FY02 Fed. 10 4 3 Training of Trainers Course on Basic Poverty Analysis (C101) FY02 Thailand 10 4 4 Advanced Impact Evaluation (C203) FY02 Pakistan 6 4 5 Intro to Poverty Measurement & Diagnostics Course (C101) FY02 Kenya 6 3 Burkina 6 Basic Impact Evaluation Course - Franco Africa FY02 Faso 5 2 Developing a Spatial Poverty Map in Africa: Principles and 7 Practice FY02 Kenya 5 9 8 Introduction to Poverty Monitoring and Evaluation FY02 Tanzania 5 8 9 Not Available FY02 5 2 10 Second Africa Forum on Poverty Reduction Strategies FY02 Senegal 4 14 11 Second Poverty Forum for Europe and Central Asia FY02 Hungary 3.5 11 Burkina 12 PRSP COURSE - Francophone Africa FY02 Faso 3 7 13 Regional Forum - Hanoi FY02 Vietnam 3 4 14 Basic Poverty Measurement Diagnostics Course FY03 India 10 21 15 Attacking Poverty Course - APC for Civil Society FY03 Ethiopia 9 14 Basic Poverty Measurement and Diagnostics Course for Balkan 16 Region FY03 Hungary 9 14 Basic Poverty Measurement and Diagnostics- Distance Learning. 17 Anglophone FY03 Tanzania 7 22 18 Advanced Impact Evaluation FY03 Korea 6 11 19 Basic Poverty Measurement & Diagnostics Course FY03 Indonesia 6 14 Kyrgyz 20 Attacking Poverty Course - Central Asia FY03 Rep. 5 11 21 Advanced Impact Evaluation Course FY03 Ethiopia 4 17 Poverty Reduction Strategies Forum for Albania- Bosnia 22 Herzegovina and FR Yugoslavia FY03 Austria 4 18 Burkina 23 Regional Seminar for Managers of Monitoring and Evaluation FY03 Faso 3 2 Regional Seminar for Asian Managers of Monitoring and 24 Evaluation of Poverty Reduction FY03 Thailand 3 17 Development Debates - Balkan countries: Bosnia Herzegovina- United 25 Albania- FR Yugoslavia- Mac FY03 States 2.5 3 59 3B: LEVEL ONE SCORES BY ACTIVITY Activity Title FY Q1 Q2 Q3 Q4 Q5 Q6 1 Basic Impact Evaluation Course - Franco Africa FY02 4.64 4 4.5 4.11 4.11 4.5 2 Basic Poverty Measurement and Diagnostics FY02 4.19 4.12 4.29 4.19 4.38 4.43 3 Developing a Spatial Poverty Map in Africa: Principles and Practice FY02 4.38 4.38 4.33 3.96 4.38 4.48 4 Intro to Poverty Measurement & Diagnostics Course (C101) FY02 4.4 4.27 4.87 4.53 4.47 4.8 5 Introduction to Poverty Measurement and Diagnostics FY02 4.48 3.91 4.26 4.03 4.36 4.5 6 Introduction to Poverty Monitoring and Evaluation FY02 4.67 4.53 4.77 4.57 4.43 4.8 7 PRSP COURSE - Francophone Africa FY02 4.3 3.89 4.29 3.5 4 4.2 8 Training of Trainers Course on Basic Poverty Analysis (C101) FY02 4.74 3.57 4.27 4.39 4.17 4.48 WBI (295 activities) FY02 4.22 3.93 4.14 3.87 4.11 4.22 9 Advanced Impact Evaluation (C203) FY03 4.06 3.88 4.03 3.69 3.61 4.09 10 Attacking Poverty Course - APC for Civil Society FY03 4.46 3.57 4.32 3.74 4.43 4.46 11 Attacking Poverty Course- Central Asia FY03 4.69 4.81 4.78 4.56 4.58 4.86 12 Basic Poverty Measurement & Diagnostics Course FY03 3.92 4.12 3.96 3.36 3.52 3.96 13 Basic Poverty Measurement and Diagnostics Course for Balkan Region FY03 4.14 4.14 4.55 4.37 4.43 4.62 14 Basic Poverty Measurement and Diagnostics-DL Anglophone Africa FY03 4.26 4.07 4.35 4.18 4.38 4.42 15 Basic Poverty Measurement Diagnostics Course FY03 4.28 3.5 3.89 3.56 3.94 4.28 16 Regional Seminar for Asian Managers of Monitoring and Evaluation of Poverty Reduction FY03 4.49 3.82 4.22 3.98 4.11 4.29 17 Regional Seminar for Managers of M E FY03 4.59 4.19 4.56 4.31 4.41 4.73 18 Training of Trainers- Basic Poverty Course- Anglophone Africa FY03 4.73 3.95 4.68 4.09 4.18 4.5 WBI (341 activities) FY03 4.23 3.96 4.16 3.91 4.17 4.28 Q1 ­ Relevance Q2 - Gain new information Q3 ­ Usefulness of information Q4 ­ Focus on your learning needs Q5 ­ Match course objectives Q6 - Overall usefulness 60 ANNEX 4: FOCUS GROUP QUESTIONS 1. How were you selected to participate in the WBI activity? Explain. 2. What did you gain from this activity? Please detail. 3. What were the most useful aspects of the activity? What were the least useful aspects of the activity? Explain. 4. Can you offer specific examples of how you applied what you learned in the activity in your official duties? Probe 1: Have these led to any changes in your organization, or changes in policies and practices in your country? Probe 2: What were these changes? 5. What were some of the obstacles that you encountered in implementing the new knowledge and skills you acquired? Examples? 6. Were the knowledge and skills you acquired available from another source ­ either internationally or locally? What are the prominent non-WBI training institutions/agencies in your country? 7. How can we improve WBI activities in the future? 8. Lastly, would you be willing to pay for this course? And if yes, how much? 61 ANNEX 5: INSTRUMENTS FOR INDIVIDUAL INTERVIEWS A. SEMI-STRUCTURED INTERVIEWS WITH WBI TTLS I. Training Design and Relevance (1) How was the cooperation/partnership/course initiated? (2) What kind of needs assessment did you conduct for this activity? (Explain all the different processes for obtaining information on country needs.) (3) Are there any "special" features about the activity that distinguishes it from other WBI activities (e.g., partnerships, target audience, etc.)? (4) Were there any opportunities provided for course participants to continue their learning after the course ended (e.g., web-based discussion groups)? How about opportunities to maintain contacts with other participants? (5) How was this activity related to the program strategy at the time of your offering? To the overall "body of knowledge" covered by the program? To other products in your program (e.g., books, web-sites, and/or newsletters)? (6) Is WBI uniquely positioned to deliver this program? If so, why? II. Effectiveness and Results (1) Since the course, what were the most important outcomes (medium-term) that resulted from this learning event? Are there one or two anecdote(s) that display a result (direct or indirect)? Please describe. (2) In retrospect, what would you have done differently? (3) What are the most important barriers to use of the training knowledge, skills, or information by your participants? (4) How do you see your program (as it exists today) evolving? (5) Is there anything that we have left out and that you would like to add? 62 B: INTERVIEWS WITH COUNTERPARTS IN OPERATIONS (ACTIVITY LEVEL) Country: __________________________Interviewee: ________________________ Date of Interview: ________________Location of Interview:__________________ 1. Overall, to what degree do you think WBI learning events are aligned with the CAS? If you were to rate the degree of alignment on a scale of 1 to 10, (where 1 is no alignment and 10 is complete alignment), how would you rate it? 2. What was the extent and nature of your involvement in WBI activities during FY02 and 03? (e.g., development of course material and content, identifying participants, facilitating their attendance, providing instructors, providing funding for participants, technical/administrative assistance, etc.,) 3. Within the past two years, WBI's training events delivered in XXX (country/region) have been mostly in four areas: poverty, XXX, XXX and XXX. In this period, do you know of any outcomes (medium-term) that resulted from WBI learning events? What sort of evidence -- on the ground -- have you seen that shows results? Please describe. 4. What are the main barriers to applying new knowledge toward change in XXX? 5. In your opinion is WBI uniquely positioned to deliver learning activities for capacity building purposes in XXX (name of country)? Why? What is it that WBI brings -- or could bring -- to the country that others don't? 6. As the operational counterpart to WBI activities, do you think things should have been things done differently by WBI or yourself to better serve the learning needs of the country? Have you communicated any such "lessons learned" to the WBI task manager(s) in charge? 63 C: LIST OF INTERVIEWEES ORGANIZATIONS BELGIUM Belgian Directorate General for Development Cooperation Department Multilateral Cooperation, Belgium BURKINA FASO WB Operations staff Dutch Embassy, Ouagadougou. UNDP Assistant Resident Representative UNDP Office Ouagadougou CEFOC Centre de Formation Continue Groupe des Ecoles EIER-ETSHER CEFOC Office Ouagadougou IPD_AOS Institut Panafricain pour le Développement ­ Afrique de l'Ouest et Sahel IPD_AOS office Ouagadougou. CAPES Centre d'Analyse des Politiques Economiques et Sociales. CAPES Office Ouagadougou. Attaché for Development Cooperation Belgian Embassy Ouagadougou WASHINGTON DC Ghana country coordinator WB AFR Operations Kenya Country Economist Lead Economist WBI Regional Coordinator for Africa WBI Partnership Unit. WBI Poverty Team Leaders Attachés for Development Cooperation 64 Belgian Embassy, Washington DC ETHIOPIA WB Librarian World Bank Country Office Addis Abeba. World Bank Country Director, Ethiopia World Bank Office Addis Abeba World Bank Institute Focal point World Bank Office Addis Abeba Ethiopian Civil Service College Civil Service College, Addis Abeba GDLN Coordination Team GDLN Centre - Civil Service College Addis Abeba Inter Africa Group Partner for the APP course for civil society (media) Attachés for Development Cooperation Embassy of Belgium, Addis Abeba 65 ANNEX 6: VARIABLES DESCRIPTION A: FACTOR LOADING FOR RELEVANCE, EFFECTIVENESS, USE, AND IMPACT RELIABILITY STATISTICS (CORNBACH'S ALPHA) (N=261) Factors Loading Reliability Effectiveness Raise awareness 0.78 0.85 Provide knowledge or skills 0.70 Help understand role as an agent of 0.82 Help develop strategies addressing the organizational needs 0.79 Help develop strategies addressing the country needs 0.81 Help develop contacts and develop partnerships (network) 0.71 Utilization Conduct research 0.61 0.84 Teach 0.71 Raise public awareness about development issues 0.79 Implement new practices within work organization 0.73 Organize collective initiatives 0.80 Influence legislation and regulation 0.70 Implement country development strategies 0.70 Impact Conduct research 0.64 0.89 Teach 0.77 Raise public awareness about development issues 0.76 Implement new practices within work organization 0.75 Organize collective initiatives 0.77 Influence legislation and regulation 0.70 Implement country development strategies 0.72 66 B: DESCRIPTIVE ANALYSIS OF VARIABLES IN REGRESSION Variables Item Composition Mean Standard Deviation Dependent Variables Utilization* Aggregated score of 7 variables, 1-7, low to high 4.76 1.33 Independent Variables Relevance* Aggregated score of 2 variables, 1-7 low to high 5.61 1.09 Effectiveness* Aggregated score of 6 variables, 1-7 low to high 5.33 1.05 External environment* Aggregated score of 2 variables, 1-7 low to high 4.94 1.25 Event/course features Location 1=in country, 0=outside country Action learning 1=yes, 0=no Partner involvement 1=yes, 0=no Mode of instruction Face-to-face=1, 0=else Duration >1 day 6.24 2.59 Use of action plans 1=yes, 0=no Language of instruction 1=If Language of instruction is the same as used at work, 0=else Demographics Country status 1=low income, 0=middle income Type of Government=1, 0=else organization Familiarity with 1-7, not proficient to very proficient terminology 5.38 1.39 Gender 1=female, 0=male *Utilization = mean (q7a, q7b, q7c, q7d, q7e, q7f, q7g) q7a: Conducting research q7b: Teaching q7c: Raising public awareness in development issues q7d Implementing new practices within work organization q7e: Organizing collective initiatives q7f: Influencing legislation and regulation q7g: Implementing country development strategies Relevance = mean (q1, q2) q1: Relevant t to your work q2: Relevant to your country Effectiveness = mean (q5a, q5b, q5c, q5d, q5e, q5f) q5a:Raising awareness q5b Providing with knowledge or skills q5c Helping understand role as an agent of change country's development q5d Helping develop strategies address the organizational needs q5e Helping develop strategies address the country needs q5f Helping develop contacts and develop partnerships External environment = mean (q8a, q8b) q8a: Working environment q8b: Country environment 67 ANNEX 7: RESULTS OF MULTIPLE REGRESSION ANALYSES ­ BASIC MODELS Independent variables Dependent variable ­ Use Model A Model B Model C Model D Beta t Beta t Beta t Beta t Gender 0.13 0.81 0.13 0.82 0.13 0.94 tsn Familiarity with terminology 0.27 4.45*** 0.22 3.82*** 0.13 2.29** Government sector -0.32 -2.00* -0.16 -1.01 -0.15 1.03 Participa Low-Income country 0.61 3.58*** 1.01 5.54*** 0.79 4.59*** In-country event 0.07 0.30 0.39 1.73* 0.42 1.96** Duration -0.06 -1.60 -0.07 -2.00** -0.09 -2.62** ures Action learning -0.25 1.49 -0.24 -1.56 0.18 1.23 feat Partner involvement -0.77 -4.23*** -0.60 -3.52** -0.55 -3.42*** Face-to-face 0.34 1.86* 0.09 0.53 0.00 0.02 Course Use of action plans 0.29 1.78* 0.27 1.82* 0.10 0.72 Language of instruction -0.21 -1.19 -0.71 -4.05*** -0.53 -3.21*** Relevance 0.39 5.57*** R2=0.18 R2=0.15 R2=0.32 R2=0.40 Adj R2=0.16 Adj R2=0.12 Adj R2=0.28 Adj R2=0.36 N = 246 N = 252 N = 241 N = 241 ***p< = 0.01; **p< = 0.05, *p< = 0.10 Technical note · The regression for Model A shows that of four participant variables, three had significant effects on knowledge use -- the level of familiarity with the course terminology, sector representation and country status. All other things being equal, participants with higher terminology proficiency are more likely to use the course knowledge than those with low proficiency; participants working in private and research sectors are more likely to be the higher users; or participants from low-income countries tend to use more than participants from middle-income countries. However, the influence of all participant characteristics explains a very small variation of the model (R2=0.18). · Model B shows that three course feature variables (face-to-face, no partner involvement and use of action plans) appear to be important to influence the use level, however, the explanatory power of the model is again rather low (R2=0.15). 68 · Finally, when both sets of variables are introduced to the full estimation model (Model C) by blocks, the significant influences of participants and courses revealed in Models A and B remain mostly unchanged. Additionally, three course feature variables (location of the event, duration and language of instruction) became the third and fourth significant factors influencing the use level, i.e., participants attending the course that was delivered within their country or attending a short duration event or courses using a language of instruction that was different from participants' working language were likely to use more than those attending the courses delivered outside the country At the mean time, the explanatory power of the model has improved considerably (R2=0.32). 69 ANNEX 8: PATH ANALYSIS OF EFFECTS OF PERCEPTION VARIABLES ON USE A PATH ANALYSIS PROCEDURE According to our concept framework, we proposed that the course impact on individual use could be explained by four blocks of variables. The relationships between the variables were examined in a two-step sequence, using a multiple regression procedure. First, we placed the individual characteristics as pre-determined (exogenous) variables in the model, i.e., these variables existed before a given WBI training event occurred and most were not subject to WBI intervention. Equally, we treated the course features as ascribed variables in the model because most course features were also likely pre-determined by WBI before the learning event was offered. Second, we introduced two perception variables in the model. Despite the fact that relevance was a perception variable, we believe it indirectly measured the content of the event, which in most cases should be determined before the training event is delivered. As such, we positioned it as the same as the pre-determined variables. We then arranged perceived effectiveness between the pre-determined variables and the final outcome, or use. By doing so, we wanted to know if participant use behavior was predicted by their perception of course effectiveness. Meanwhile, we were interested in knowing if effectiveness, as an intervening variable, could play a role in mediating the effects of pre-determined variables on use of content. Finally, we added another variable to our model: participants' work and country conditions. However, we treated these characteristics as enabling and disabling factors that only enhance or diminish the influence of the preceding factors in the model on use, rather than as causal factors. We hypothesized that both these characteristics may have independently affected participants' use behavior. . 70 B: PATH ANALYSIS RESULTS - EFFECTS OF PARTICIPANTS AND COURSE FEATURES ON USE VIA PERCEPTION VARIABLES ­ RELEVANCE AND EFFECTIVENESS Independent variables Dependent variables Model E Model F Model G Effectiveness Effectiveness Use Beta t Beta t Beta t tsn Gender 0.05 0.39 0.06 0.52 0.15 1.08 Familiarity with terminology 0.15 2.87*** 0.05 1.01 0.12 2.28** Government sector -0.03 -0.22 -0.02 -0.02 -0.13 -0.97 Participa Low-Income country 0.33 2.07** 0.11 0.76 0.71 4.25*** In-country event 0.13 0.56 0.14 0.75 0.33 1.63* Duration -0.11 -0.53 -0.04 -1.21 -0.07 -2.14* ures Action learning -0.02 -0.64 -0.02 -0.16 0.13 0.96 feat Partner involvement -0.27 -1.79* -0.22 - 1.57 -0.41 -2.65* Face-to-face 0.06 0.41 -0.06 -0.24 0.01 0.07 Course Use of action plans 0.38 2.87*** 0.38 1.61 0.02 0.15 Language of instruction -0.37 -2.42** -0.19 -1.33 -0.44 -2.82** Relevance 0.42 7.70*** 0.21 2.97*** on Effectiveness 0.34 4.39*** Percepti External factors 0.14 2.29** R2=0.15 R2=0.29 R2=0.47 Adj R2=0.11 Adj R2=0.26 Adj R2=0.44 N = 241 N = 241 N = 241 ***p< = 0.01; **p< = 0.05, *p< = 0.10 71 REFERENCES Ajzen, I. & Fishbein, M. (1980). Understanding attitudes and predicting social behavior. Englewood Cliffs, NJ: Prentice-Hall. Duncan, O.D. (1971). In H.M. Blalock, Jr. (Ed), Causal Models in the Social Sciences. Chicago: Alidine-Atherton. Khattri, N. & Quizon, J (2002). Impact Evaluation of WBI Client Programs, FY00-01. No. EG03-63. The World Bank Institute. The World Bank, Washington, D.C. Prom-Jackson, S. et al. (2002). The Impact of FY01 Formal Training ­ Turning Knowledge and Reflection into Action in The World Bank. No. EG03-62. The World Bank Institute. The World Bank, Washington, D.C. Quizon, J. Chard, C. & Lockheed, M. (2004). The Effectiveness and Use in FY03 of WBI FY01-02 Activities: A Baseline Assessment in Five Countries. No. EG04-86. The World Bank Institute. The World Bank, Washington, D.C. 72