Country Focus Evaluation of WBI Client Activities in FY02-03, Guatemala: A Baseline Assessment Chaoying Liu Jonathan Richter Laurie Choi WBI Evaluation Studies No. EG05-93 The World Bank Institute The World Bank Washington, D.C. July 2004 ACKNOWLEDGEMENTS This evaluation report was prepared for WBI under the overall guidance of Marlaine E. Lockheed, Manager, World Bank Institute Evaluation Group (IEG). It was prepared by Chaoying Liu, Jonathan Richter, and Laurie Choi. Many thanks also go to Rosa Tock and Denis Nikitin who worked diligently to collect and analyze the data and help with the statistical work. The report benefited greatly from discussions with Eduardo Somensatto, the Country Manager, Guatemala and Philip Karp, WBI Regional Coordinator, Latin America and the Caribbean, who served as peer reviewers. Humberto S. Diaz contributed to document production. WBI Evaluation Studies are produced by the WBI Evaluation Group (IEG) to report evaluation results for staff, client, and joint learning events. An objective of the studies is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at: http://info.worldbank.org/etools/WBIEG/publications/index.cfm?pg=getPubs&category= Publications&Intro=yes&instructions=no&showDetails=no&ID= Vice President, World Bank Institute Ms. Frannie Léautier Manager, Institute Evaluation Group Ms. Marlaine E. Lockheed Task Team Leader Ms. Chaoying Liu ii ACRONYMS ACP Anti-Corruption Project AFR Africa Region CAS Country Assistance Strategy CPB Country Program Brief CRS Client Registration System CESI Community Empowerment and Social Inclusion DL Distance Learning EAP East Asia and Pacific Region ESW Economic and Sector Work ECA European and Central Asia Region F2F Face to Face FGD Focus Group Discussion FY02 Fiscal Year 2002 FY03 Fiscal Year 2003 FY04 Fiscal Year 2004 IEG Institute Evaluation Group GAC Governance and Anti-Corruption IADB Inter-American Development Bank LCR Latin American and Caribbean Region LICUS Low Income Country Under Stress MDGs Millennium Development Goals MNA Middle East and North Africa Region NGOs Non Governmental Organizations PRSP Poverty Reduction Strategy Paper PRSC Poverty Reduction Credit RCET Regional Capacity Enhancement Team SAR South Asia Region UN United Nations USAID US Agency for International Development WB World Bank WBI World Bank Institute WBIGG World Bank Institute Government Group iii iv TABLE OF CONTENTS ACKNOWLEDGEMENTS ................................................................................................ ii ACRONYMS..................................................................................................................... iii TABLE OF CONTENTS.....................................................................................................v EXECUTIVE SUMMARY ............................................................................................. viii 1. CONTEXT AND METHODOLOGY......................................................................... 1 Evaluation Objectives.................................................................................................. 1 Evaluation Questions and Methodology...................................................................... 2 2. DATA COLLECTION RESULTS.............................................................................. 4 Desktop Review........................................................................................................... 4 Participant Survey........................................................................................................ 4 Focus Group Discussions ............................................................................................ 6 Individual Interviews ................................................................................................... 7 Case Study................................................................................................................... 7 3. PROGRAM RELEVANCE AND IMPLEMENTATION .......................................... 8 Relevance of WBI Activities to the Country's Needs ................................................. 8 Stakeholders' Views of Relevance ............................................................................ 12 Activity Planning and Implementation...................................................................... 14 4. PROGRAM OUTCOME - EFFECTIVENESS AND IMPACT............................... 17 Program Effectiveness ............................................................................................... 18 Utilization of Learning .............................................................................................. 20 Program Impact ­ Changes Brought to Guatemala ................................................... 22 Factors Influencing the Program Outcome ................................................................ 24 How to Sustain the Built Capacity? ........................................................................... 27 The Comparative Advantage of WBI in Guatemala.................................................. 28 5. CONCLUSIONS AND LESSONS LEARNED........................................................ 30 Key Findings.............................................................................................................. 30 Lessons Learned ........................................................................................................ 32 v ANNEXES ........................................................................................................................ 33 Annex 1: List of WBI Learning Events for Guatemala FY02-FY03 (>1 day events) .......................................................................................................... 35 Annex 2: World Bank Institute (WBI) Country Focus Evaluation Questionnaire - Guatemala ........................................................................................ 36 Annex 3: List of WBI Learning Events Attended by Surveyed Participants............ 43 Annex 4: Questions of Focus Group Discussions ..................................................... 44 Annex 5: Individual Interveiws ................................................................................. 45 Annex 6A: Review of Alignment of WBI Activities with WB Country Assistance Strategies ................................................................................................. 47 Annex 6B: Mapping WBI Activities with WB Country Assistant Strategies FY02-03......................................................................................................................53 Annex 7: Variables Description..................................................................................60 vi vii EXECUTIVE SUMMARY By training and supporting key individuals in a country, WBI expects to contribute to the country's development and poverty reduction. This evaluation examines the extent to which this expectation has been achieved in Guatemala. Focusing on WBI activities carried out during the period of FY02 and FY03, this evaluation assesses both the process and the impact of WBI programs in Guatemala. The process evaluation addresses how WBI sets strategic directions, designs and implements learning activities for the country. The evaluation looks at the relevance of WBI activities in terms of addressing key issues of the country, the level of the alignment with the Bank's assistance strategy for Guatemala and the program implementation. The impact evaluation considers the outcome of a WBI intervention, i.e., the extent to which it has met its activity and program objectives. The impact issue is addressed by examining the following evaluation questions: · What is the effectiveness of WBI activities in Guatemala? · Did participants utilize what they learnt from the WBI activities? · What changes have WBI activities brought to individuals, organizations and the country? Additionally, the evaluation explores some of the conditions under which WBI could enhance the impact of learning programs on individuals and on the country, as well as bring a comparative advantage in Guatemala. Both qualitative and quantitative data collection methods were employed in order to benefit from new and available information. The final analysis was based on the following information : · A desktop review of alignment of WBI activities with the Bank's Country Assistance Strategy (CAS) for Guatemala; · A survey of 45 Guatemalan participants who attended at least one WBI learning activity during FY02 and FY03 (a longer than one-day event); · Two focus group discussions with selected participants; · Several individual interviews with the WB Operations staff members, WBI task managers, local partners and client representatives; and · A case study of the WBI Anti-Corruption Project (ACP) implemented in Guatemala. The primary conclusion from this evaluation is that WBI programs have brought some positive impact to Guatemala, but that the impact could be greater if WBI improves the viii design and implementation of the programs offered. This conclusion is based on ten findings. WBI activities in Guatemala were partially aligned with the Bank's Country Assistance Strategy. During the study period, there were a total of 33 Bank projects and 57 WBI events for Guatemala. More that 700 participants were trained by WBI. By mapping the topic areas and size of the WBI events with the four CAS priority areas, the WBI activities in Guatemala seem to be aligned partially with the Bank's assistance strategy, especially for the topics of Reducing Poverty and Improving Economic Management. WBI activities offered were mostly relevant to participants' work and to their country. Both survey participants considered that that most WBI activities delivered were relevant to their work and to their country (mean rating = 4.7 and 5.1 on a 7-point scale), but only moderately relevant to some country development goals. The Anti-Corruption Project was regarded as "highly relevant" by both the Operations team and the client. WBI activities in Guatemala were not designed strategically at the country level. By- and-large, the WBI activities in Guatemala were not designed strategically at the country level, reflecting the stage of the country's development. According to the Operations staff, WBI had little flexibility on the offerings in the country and most offerings were not completely tailored for the needs of the CAS and the choices were mainly supply driven. On the other hand, WBI activities were generally organized well at event and activity levels. The majority of WBI activities were implemented independently in the country, facilitated through local partners. Over 60 percent of activities were delivered through local partners. The Anti-Corruption Project in Guatemala is believed to be a very successful example of joint WBI and Operations implementation. WBI activities were effective in influencing individual knowledge and perceptions. The participants gave high ratings to the WBI activities for raising awareness, providing knowledge and skills, and helping understand the participant's role as agent (mean ratings=5.2, on a 7-point scale). The level of perceived effectiveness is positively influenced by the perceived relevance of the event to the country's needs. Participants reported moderate utilization of knowledge and skills gained from WBI activities. The mean rating of utilization was 4.5, on a 7-point scale. Utilization was strongest in teaching and research applications, followed by raising public awareness. Higher utilization of knowledge and skills was associated with having developed an action plan during the course, or having a positive country environment. WBI has brought some positive impact/changes. Participants believe that WBI activities have brought a moderately positive change to the country in the course topic areas (mean rating = 5.1, on a 7-point scale). The changes were greater at individual levels than at organizational or country level. The level of impact was influenced by the level of utilization. ix Policy advisory support is seen as an effective intervention strategy in helping the country build capacity. The case study of the Anti-Corruption Project has demonstrated that policy advisory services as a non-traditional learning activity can be very effective in enhancing capacity, especially at the country level, through building the country's ownership and providing technical assistance. WBI made little effort to ensure the sustainability of built capacity. The level of follow up and facilitation of continued learning for participants is low. WBI' s advantage in Guatemala is not apparent. According to participants and Operations staff, unless WBI is willing to be more flexible and to provide a range of products that are tailored according to the country's needs, it will not have comparative advantage over other providers in Guatemala. Drawing on the key findings, the evaluation offers several lessons that can be used by WBI to improve its programs: The first lesson is that WBI can provide important technical assistance or more direct knowledge products to the country. The content relevance to participants and to the country is the key to the success of programs offered in the country. This is true regardless of kinds of activities, types of delivery modes or location. This lesson has been already applied by WBI since Guatemala became a focus country and a new regional program coordinator has been assigned. A strategic plan for Guatemala at the country level has been developed for FY05 in consultation with the Operations team. The second lesson is that the influence of the WBI intervention cannot reach maximum impact if it focuses only on a relatively small number of individual participants. Within a given organization or institution there must be a critical mass of participants in order to substantially influence policy. WBI needs to place more attention to building institutional capacity in the country. The third lesson is that program impact can be enhanced strategically by: · designing and implementing programs jointly with Operations; and · promoting country- and particularly government­ownership of the activity. The fourth lesson is that program impact can be enhanced technically by: · using an action plan during the learning event; and · using the non-traditional learning approach of policy advice services. The final lesson is that program impact cannot be sustained in the country without the continuing involvement and effort of WBI after the learning event is over. x xi 1. CONTEXT AND METHODOLOGY 1.1 In collaboration with the regions, the World Bank Institute (WBI) has identified a group of focus and priority countries. These countries have been selected because of their size, regions and their status as Low Income Country Under Stress (LICUS) where knowledge and learning activities may dominate the Bank's engagement strategy, or by their status as Poverty Reduction Strategy Paper countries (PRSP), in which learning programs are particularly important. 1.2 Strategically, WBI is gradually moving from a thematic focus on training individuals in global events to a country focus, i.e., to enhancing in-country capacity by tailoring WBI programs to the specific needs of each country. Training and other learning products continue to be the main element in WBI's package of assistance, but they are increasingly conceived as part of a menu of services which includes diagnostics and advisory work, strengthening of learning and service delivery organizations, and consulting on capacity enhancement issues. This new country focus is aimed at increasing the efficacy and impact of the WBI intervention and ensuring that these results are sustainable. 1.3 In response to this strategic change, the World Bank Institute Evaluation Group (WBIEG) has conducted a country-focused evaluation in the 12 WBI focus countries. In FY03, the evaluation occurred in the following five countries: Brazil, Egypt, Russia, Sri Lanka, and Thailand. In FY04, the evaluation was carried out in seven countries: Burkina Faso, Guatemala, Indonesia, Kenya, Nigeria, Tajikistan, and Yemen. Table 1 summarizes the scope for these two evaluations. Table 1: WBI Country Focus Evaluation Region FY04 FY03 LAC Guatemala Brazil EAP Indonesia Thailand AFR Burkina Faso, Kenya, and Nigeria ---- MNA Yemen Egypt ECA Tajikistan Russia SAR ---- Sri Lanka EVALUATION OBJECTIVES 1.4 Building on findings and experience of the first five country focus evaluations, the FY04 evaluation encompasses the WBI learning activities implemented in FY02 and 1 FY03 in seven countries.1 Guatemala is one the countries included. The overall objectives of the evaluation are to: (1) understand the process used by WBI to design and to deliver learning activities in Guatemala; (2) assess the effectiveness and impact of WBI programs at individual, institutional and country levels; (3) establish a baseline for further monitoring and evaluation activities; and (4) form recommendations for the improvement of WBI programs. 1.5 Additionally, the evaluation explores the conditions under which WBI produces impact on individuals and on the country, as well as the comparative advantage of WBI in the country. EVALUATION QUESTIONS AND METHODOLOGY 1.6 The evaluation design is based on several assumptions. It acknowledges that achieving a country development goal is a result of collective effort, and that knowledge intervention is only part of the effort. It considers the staged change that determines how the country's capacity can be influenced by learning events. As such, the evaluation seeks to address both the process and impact issues of a WBI intervention within a common framework as shown in Table 2 below. Table 2: WBI Intervention Framework Process Impact Effective designing and Strategic planning delivering Capacity building Benefit gain Right content/topic Right people Individual Improved health, Alignment with country Right mode Awareness education and social priority Right time Knowledge economic indicators Alignment with Bank program Right location Skills Poverty reduction priority Quality Institution Research Teaching Dissemination Government/country Reform Strategy Policy Process questions 1.7 Process reflects how WBI sets strategic directions, designs and implements learning activities for a country. The process questions include: 1Zia (2004). Concept Note: The Impact of WBI Activities, FY02-03, on Participants from Seven Countries: Burkina Faso, Guatemala, Indonesia, Kenya, Nigeria, Tajikistan, and Yemen 2 · What was the relevance of WBI activities to Guatemala's needs? · How WBI activities were implemented for Guatemala? Impact questions 1.8 The evaluation considers the impact of the WBI intervention as the extent to which it has met its activity and program objectives. At the activity level, impact is concerned with building capacity of individuals, institutions and the government. At the program level, impact is concerned with the benefit that the country gains as a result of changes in individuals, institutions and the government. Realistically, the current evaluation places attention on the following impact questions: · What is the effectiveness of WBI activities in Guatemala? · Did participants utilize what they learnt from the WBI activities? · What changes have WBI activities brought to individuals, organizations and the country? Data collection and sources 1.9 Five main data collection methods were used, including qualitative and quantitative methods. First, we searched relevant program documents and data to gather a descriptive record of activities and outputs of each activity implemented by the Bank and by WBI in Guatemala. The findings of the review set the context for the evaluation. Additionally, the following data collection activities were employed: (1) A participant survey (2) Focus group discussions with selected participants (3) Interviews with Bank Operations staff, especially those located in the country office, interviews with WBI task managers, and interviews with local program partners or client representatives (4) A case study of a successful WBI learning program implemented in Guatemala Study population 1.10 The WBI Client Record System (CRS) shows that over 700 Guatemalans attended various WBI learning events during FY02 and FY03. Based on the study criteria, the evaluation identified all participants who attended learning activities that were longer than one day in duration, as impacts of shorter activities are likely to be negligible or non-existent. As a result, 163 individual participant records from 27 learning events were extracted from CRS. Forty percent of these participants (n=66) attended four WBI learning events delivered in Guatemala, and the remaining sixty percent (n=97) participated in 23 learning events outside the country. Annex 1 provides a list of WBI learning events that include Guatemalan participants during FY02 and FY03. 3 2. DATA COLLECTION RESULTS 2.1 Data collection was carried out between December 2003 and March 2004. This section provides a summary of the scope, process and results of each data collection activity. DESKTOP REVIEW 2.2 The evaluation conducted a desktop review and analysis of all Bank activities implemented in Guatemala during FY02 and FY03. This covered the following information: · 2002 Guatemala PRSP; · 1998 Guatemala Country Assistance Strategy (CAS) and 2002 CAS Progress Report; · WBI's Country Program Brief (CPB); · Information from the Business Warehouse documenting all Bank lending and no-lending activities carried out in Guatemala; and · All WBI activities in Guatemala and WBI activities that included Guatemalan participants in other locations. 2.3 The review and analysis centers on assessing the relevance of the WBI intervention to Guatemala's needs. PARTICIPANT SURVEY 2.4 A survey was carried out with Guatemalan participants who attended WBI learning activities longer than one-day in FY02 and FY03 and who also had valid contact information. Instrument 2.5 The survey instrument used follows the one applied across all eleven countries studied in FY03 and FY4. A copy of this survey instrument is attached at Annex 2. Specifically, the survey instrument seeks information from participants about: · their perceptions of WBI activities regarding relevance and effectiveness; · their use of knowledge and skills learnt from the WBI event; and · their perceived impact of knowledge utilization at individual, organizational and/or country levels. 4 Response rate and representation 2.6 Of 163 individual participants extracted from CRS, 132 had a contact information were targeted for the survey. The survey was carried out during December 2003 and February 2004 through a local consultant. A total of 48 participants were found to have either wrong contact information or could not be located during the survey period, making the final adjusted sample 84 participants. Forty-five participants completed the survey, making an adjusted response rate of 54 percent. These respondents represented 15 learning events, four offered in Guatemala and 11 outside the country. The list of learning events attended by the surveyed population, together with some features of the events, is summarized in Annex 3. 2.7 To understand the representation of the final sample, comparisons were made between survey respondents and the remaining population. Table 3 shows that, overall, there are no significant differences between the two groups with respect to available demographic information, the venue of learning events they attended, or the sectors they represented at the time they were trained. Table 3: Comparison of some characteristics of survey respondents with the non-surveyed (remaining) study population Variables Respondents Remaining population N=45 N=118 Location learning event In country 28 (62.2%) 67 (56.8%) Outside country 17 (37.8%) 51 (44.2%) Delivery mode F2F 16 (35.6%) 58 (40.2%) DL 29 (64.4%) 60 (50.8%) Sector representation Academic 15 (33.3.%) 29 (24.6%) NGOs/Private 12 (26.7%) 37 (31.4%) Government 18 (40.0%) 52 (41.1%) Characteristics of respondents 2.8 Table 4 presents the key characteristics of survey respondents and the WBI activities they attended during the study period. Nearly 40 percent of respondents represent academic society. The rest almost equally represent Non-Government Organizations (NGOs) and government sectors. Over half of the respondents have been working in teaching and research fields since completing the WBI learning events. Only two respondents stated that they work directly in making policy and legislation. More than 55 percent (n=25) are at senior and top levels of their organization, 27 percent are middle managers, and less than 10 percent are at junior and entry levels. There are somewhat more males among the respondents. 5 Table 4: Characteristics of respondents Variables Respondents N=45 (%) Sector representation* Academic 17 (37.8%) NGOs/Private 12 (26.6%) Government 13 (28.9%) Other 3 (6.7%) Type of work Teaching/research 25 (55.6%) Policy 2 (4.5%) Management 14 (31.1%) Other 4 (8.9%) Levels Junior 4 (8.9%) Middle 12 (26.6%) Senior 18 (40%) Highest 7 (15.6%) Other/missing 4 (8.9%) Gender Male 25 (55.6%) Female 20 (45.4%) *Some participants changed their job since completing the training, thus some figures are different from the figures exacted from CRS in Table 3. FOCUS GROUP DISCUSSIONS 2.9 The purpose of focus group discussions (FGDs) was to obtain more detailed information from participants, to elaborate their feedback on the effectiveness of the learning event, and to solicit their recommendations for improving WBI programs. 2.10 Two focus group discussions were conducted during March 2004 in the World Bank Country Offices in Guatemala City. The discussions were moderated by a local consultant, following a standard interview protocol, a copy of which is attached at Annex 4. 2.11 Of 27 individuals invited, 13 participated in the focus groups. The most common reason for declining to participate was that they were too busy with their current work. The characteristics of the focus group attendees and their training information are summarized in Table 5. Table 5: Characteristics of FGDs attendees Characteristic No. Gender Male 7 Female 6 Sector NGO/Private sector 4 Government 4 Academic 5 Learning programs CESI 4 Education 7 Economics 2 Method of learning F2F 2 Blended 4 DL 7 6 INDIVIDUAL INTERVIEWS 2.12 Unstructured interviews were conducted to address both the process and impact of WBI activities in Guatemala. The interviewees were: · Four WB Operations staff who had worked in Guatemala; · Three WBI task managers whose learning activities were physically delivered in Guatemala; · Two local partners who were involved in implementing WBI activities within the country; and · One client of the WBI Anti-Corruption Project (stakeholder). 2.13 These interviews took place in Washington and in the country office. Interviews lasted 30 to 60 minutes and all were conducted face-to-face. All interviews were guided by a set of questions whichare attached at Annex 5. CASE STUDY 2.14 A case study of the Anti-Corruption Project (ACP) is analyzed to (a) illustrate how WBI designed and delivered an intervention for Guatemala; and (b) explore the extent to which WBI has enhanced country capacity, through a non-traditional learning event. The case study largely relies on a desktop review of program briefs, press and government documents, and interview information with the three Operations staff, one WBI task manager and one client representative. 7 3. PROGRAM RELEVANCE AND IMPLEMENTATION 3.1 WBI activities in Guatemala were reviewed from two perspectives: the extent to which WBI designed the programs that were relevant to Guatemala, and how effectively WBI activities were implemented for the country during the study period. RELEVANCEOF WBI ACTIVITIES TO THE COUNTRY'S NEEDS 3.2 This section presents some key messages derived from the analysis of the degree to which WBI activities offered for Guatemala reflect the country's needs and were aligned with CAS. A detailed review and analysis results, including a mapping table, are provided in Annex 6. Critical issues identified in Guatemala PRSP 3.3 The 1996 Peace Accords ended 36 years of armed conflict within Guatemala. The fundamental cause of the conflict has been widely attributed to the historical pattern of exclusion, particularly with regards to indigenous groups. The Peace Accords placed poverty reduction at the forefront of the national agenda in order to build a more equitable and inclusive society. The 1998 Guatemala PRSP aims to reconcile the past by focusing the national agenda on three primary pillars: economic growth with equity, investment in human capital, and investment in physical capital. · Economic Growth with Equity. Although Guatemala has had surprisingly solid economic growth for a historically exclusive society, the growth rates have not been sufficient to benefit the poor. Guatemala has one of the highest poverty rates in Latin America. The Guatemala PRSP asserts that the country must achieve stronger economic growth in order to achieve the underlying goal of reducing poverty. · Investment in Human Capital. The second pillar of the PRSP focuses on building human capital while targeting the poorest citizens. Investments are concentrated in the areas of greatest need which includes health, education and food security. · Investment in Physical Capital. The third pillar invests in the physical infrastructure of Guatemala's poorest and most remote areas. The areas of priority include: water supply, basic sanitation, roads and energy. 3.4 Additionally, three areas of need are outlined in the PRSP as part of Guatemala's efforts to reduce poverty and build social inclusion: i.e., public sector modernization, governance, and decentralization and participation. Issues identified in the World Bank's Country Assistance Strategy (CAS) 3.5 Subsequent to the 1996 Peace Accords, the World Bank Group's assistance intensified during the implementation of the CAS strategy from 1998 to 2001. The 2002 8 CAS Progress Report builds on the foundation laid in the 1998 CAS, which shared the national PRSP goals of reducing poverty and building a more inclusive society. The CAS has four interlinked development priorities that closely mirror, though do not perfectly align with, the pillars of the PRSP: · Reducing poverty and promoting social inclusion; · Strengthening social cohesion and participatory decision processes; · Improving economic management to maintain macroeconomic stability and foster growth; and · Improving governance and public sector management. 3.6 The World Bank's products are designed to make progress in the priority areas, and are composed of lending and non-lending products. There were a total of 19 active lending products during FY02-03 that were valued at a total of $760.5 million. During the same period, there were nine non-lending products valued in the amount of $731,000. 3.7 The review of alignment of the Bank activities and PRSP suggests that the goals outlined by the CAS, along with Bank products, are clearly strongly aligned with the goals stated in the Guatemala PRSP (Annex 6). Alignment of WBI activities with the CAS 3.8 This section examines the degree to which WBI activities appear to align with the World Bank Group's products during the fiscal years of 2002 and 2003. During the study period, there were a total of 33 Bank projects, and 15 WBI in-country events. There were also 42 WBI regional and global events that included participants from Guatemala. More than 700 individuals were trained by WBI. 3.9 The types of WBI activities are mapped into two categories: those held within Guatemala, and those held out-of-country that included Guatemalan participants. The contents of WBI activities and Bank products are mapped under the four CAS priorities and within the major WBI classifications. For example, the Universatization of Education lending product is mapped under the "Reducing Poverty and Social Inclusion" CAS priority, and further mapped under the WBI "Education Program" classification. This helps broadly reveal which lending and WBI activities address specific CAS priorities while gaining additional insights by the area of focus. The detailed mapping result is tabled and included in Annex 6. Some key information is extracted in Table 6 and displayed in Figure 1. 9 Table 6: Topic areas and size of the Bank and WBI activities in FY02-03 WBI activities PRSP priority CAS priority Number and volume of WB* (Number of offerings (1998) (1998 and 2002 progress Report) products and participants) Priority areas Priority areas Lending ($) Non-lending ($)In-country Out-of- country Economic Growth with Equity Investment in human (1) Reducing Poverty and Promoting 10 5 6 21 capital Social Inclusion ($174.3 m) ($571,000) (159 par) (117 par) Investment in physical capital Decentralization and (2) Strengthen Participatory Decision 7 0 1 1 Participation Making ($138.1 m) ($0) (19 par) (14 par) (3) Improving Economic Economic Growth Management to Maintain 4 4 2 11 with Equity Macroeconomic Stability and Foster ($188.3) ($160,000) (32 part) (54 par) Economic Growth Economic Growth with Equity (4) Improve Governance and Public 3 0 6 9 Public Sector Sector Management ($577.6 m) ($0) 187 par) (194 par) Modernization Total 24 ($760.5m) 9 ($731,000) 15 (397) 42 (379) *There were only 19 active lending products during FY02-03, however five of them could not be disentangled between the CAS priorities of Reducing Poverty and Decentralization and Participation. The costs of those activities were split bet ween the two priority areas, and that is why there appears to be more lending products in the chart above. Figure 1: Alignment of WBI activities with CAS priorities 60% 50% 40% 30% 20% 10% 0% (1) Reducinging Poverty (2) Strengthening (3) Improving Economic (4) Improving Governance and Promoting Social Participatory Decision Management to and Public Sector Inclusion Making maintain... Mangement Bank funding share Bank activity number proportion WBI participant share WBI activity number proportion Reducing poverty and promoting social inclusion 3.10 Poverty reduction is the centerpiece of Guatemala's development agenda, and this development priority overall has the greatest number of WB products and WBI activities: 10 lending products, five non-lending products, six in-country WBI activities, and 21 out- of-country WBI activities with Guatemalan participants. Breaking this development priority down by WBI classification reveals that in the Education Programs classification, 10 the Universalization of Basic Education lending product is aligned with four WBI activities aimed at the education sector. In Poverty and Growth there are two active lending and four non-lending products that are aligned with WBI activities (both in and out of country). There are three WBI classifications in which there were Bank products: Environment and Natural Resources, Rural Poverty and Development, and Social Protection and Risk Management ­ yet no WBI activities within Guatemala. The same is true for Health, Nutrition and Population, where there is one non-lending product. There were no Bank products or in country WBI activities in the City and Urban Management classification. This makes sense since the focus for this priority is on rural communities. Similarly, in the area of Water Programs there were neither in-country WBI activities nor Bank products. Strengthening social cohesion and participatory decision processes 3.11 Both the CAS and PRSP stated that poverty reduction is the primary goal, which is strongly tied to the goal of building social inclusion. The WBI classification of Community Empowerment and Social Inclusion was the only classification that aligned with this development priority. There were seven lending products in this classification, yet only one WBI activity was held in Guatemala (FY2003). Furthermore, there were no Guatemalan participants involved in WBI activities held outside of the country in this area. This could be interpreted as a lack of demand, or more likely, a lack of opportunity for the poor to be involved in activities of this sort due to financial constraints. Improving economic management to maintain macroeconomic stability and foster growth 3.12 Under this CAS priority, there were a total of four lending products, four non- lending products, two in-country WBI activities, and nine out-of-country WBI activities with Guatemalan participants. Improving governance and public sector management 3.13 This fourth development priority had a total of three lending products, six in- country WBI activities, and six out-of country activities with Guatemalan participants. There were two WBI classifications which appear to be least aligned: Governance and the Public Sector. Governance had one lending product, four in-country WBI activities, and four out-of-country WBI activities with Guatemalan participants. The Public Sector had two lending products, two in-country WBI activities, and two out-of-country WBI activities. Summary 3.14 The level of alignment of WBI activities with CAS objectives and Bank lending/non-lending products is somewhat mixed. Using the criteria of WBI classification, of the fourteen categories, only half seem to be aligned relatively well between Bank lending/non-lending products and in-country WBI activities, i.e., Reducing Poverty and Improving Economic Management. The areas where the alignment was least are Improving Governance and Public Sector Management. 11 STAKEHOLDERS' VIEWS OF RELEVANCE 3.15 Both survey and focus groups participants were asked their opinions about the relevance of WBI activities to their individual work, to their country's needs and to country development goals. Relevance to participants' work 3.16 WBI activities were highly relevant to participant's work. As shown in Figure 2, using a 7-point scale (1=not relevant and 7=extremely relevant), a relatively large share of survey respondents (>40 percent) felt that the WBI activity they attended was highly relevant to their work (6 and 7). And another 37 percent respondents gave a rating of relevance at or just above the midpoint (4 and 5). The mean rating score of the perceived relevance is 4.7, which indicates that the WBI activity designed was somewhat relevant to the participants needs. Figure 2: Relevance to the work 7 6.7% 6 35.6% (Low-high) 5 20.0% 4 17.8% relevance 3 11.1% of 1 8.9% Level 0% 5% 10% 15% 20% 25% 30% 35% 40% % of respondents Relevance to the country's needs Views of participants 3.17 Survey participants felt that WBI activities were highly relevant to their country's needs. As seen in Figure 3, the majority of respondents rated the WBI activity as having high relevance to their country needs. The mean rating is 5.1 on a 7-point scale. 3.18 On the other hand, focus group participants indicated that some of the contents and models of WBI activities were not easily adaptable to a more complex and culturally diverse society like Guatemala, where there are more than twenty different ethnic groups. Some of the models exposed were seen as better suited to more homogenous societies. Thus, researchers participating in one of the focus groups insisted on the importance of doing more national research, and directing international financial resources to strengthen local capacities to conduct social research. 12 Figure 3: Relevance to the country 7 26.7% 6 20.0% 5 22.2% (Low-high) 4 11.1% 3 13.3% relevance of 2 2.2% 1 4.4% Level 0% 5% 10% 15% 20% 25% 30% 35% 40% % of respondents Views of Operations staff and partners 3.19 At the country level, Operations staff felt that WBI had little flexibility on the offerings, and most activities were not tailored for the needs of the CAS. 3.20 On the other hand, all partners/clients interviewed felt that most WBI activities delivered in Guatemala were relevant to the country's needs. According to data from the Global Distance Learning Network (GDLN) Center in Radefal Landivar University, of the 38 programs broadcasted during the period of May 2002 and January 2004, 14 or 37 percent were from WBI, the highest offering of any organization. This high acceptance rate, as commented by the director of GDLN, reflects that WBI events were considered to be relevant to the country's needs. However, in most cases the "relevance" referred to by partners is largely in narrow terms, i.e., not in reference to other national priority areas, rather it is concentrated on a particular topic field. The exception is the Anti-Corruption Project, which is highly regarded as "meeting the country's needs" by both the Operations team and the client country. As one client representative interviewed indicated at the interview: "Anti-corruption/transparency is one of the themes identified by the government to be addressed in the next five years2. The Bank's support is just right on the topic (to her, WBI is the Bank). Importantly, the Bank plays its role in a very smart way. Rather than standing in the front to tell what the government should do, the Bank has stayed behind the scene to facilitate the process. That certainly enhanced the ownership of the government and civil society trust of the government." 2Moving Toward Peace with Reconciliation and Development ­ Final report on the follow-up Matrix of Main Issues in Consulting Group of February 2002. Publication of Government of Guatemala, April 2003. 13 Relevance to the country development goals 3.21 Despite the fact that most survey respondents rated the WBI activities as having relatively high relevance to their work and to their country, WBI activities were rated as moderately relevant to country development goals. Table 7 shows between 30 and 40 percent survey participants felt that the WBI activities were not directly relevant. women According to these respondents, the WBI activities were moderately relevant to the following three country development goals: eradicate extreme poverty, achieve universal financeeducation primaryempowerdevelopmentprimaryandforclimateextreme education, and promote gender equality and empower women. andproverty Table 7: Relevance of WBI activities to the country development goals (N=45) (%) (%) equalityuniversalsustainabilityinvestmentsupplyEradicatepartnershipsdiseasesgenderandAchieve Yes No 44.0 33.0 33.0 40.0 globalotherImprove 24.0 38.0 PromoteenvironmentalandsanitationtradeDevelophealthmortalityEnsurewatermalaria, 22.0 42.0 20.0 44.0 16.0 51.0 childPromote 15.0 42.0 maternalEnsure 11.0 49.0 HIV/AIDS,Reduce 8.9 48.0 Improve 8.9 48.0 Combat 4.4 53.0 ACTIVITY PLANNING AND IMPLEMENTATION Activity planning for the country's needs 3.22 Although the review of alignment of WBI activities and CAS shows some degree of coherence, it may be largely interpreted as coincidence rather than a result of strategic planning. From the interviews with the Operations staff, there is no evidence showing that WBI activities for Guatemala (in-and out-country) in FY02 and FY03 were strategically planned at the country level. The Operations staff were not aware of a big picture, i.e., what WBI tried to achieve in Guatemala. Rather, they were simply aware of individual training courses held from time to time which they considered to be of "benefit most to individuals." Some staff commented: "There was little flexibility on the offerings by WBI, and they were not tailored for the needs of the CAS. There is a major effort recently by WBI to better coordinate, particularly since Guatemala became a focus country and they assigned a new program coordinator." "Six months ago, I was not aware about WBI activities, even no idea about WBI. Over the past 6 months, I became more aware about WBI, especially the Anti- Corruption Project." 14 3.23 On the other hand, the planning and organizing process seems to be carried out well at the activity and event levels. Most courses/events delivered within the country, either by face to face or through DL, were carefully planned and tailored in some degree to meet participants' needs. In the case of the Community Empowerment and Social Inclusion program (CESI), a formal needs assessment was conducted through a survey and focus group interviews. Despite all these efforts at the activity level, Operations staff still felt that WBI activities were largely supply driven. One country office staff pointed out: "Yet, the WBI activities were not "strategic" designed, I think you can say that the organization of the events was well designed, but their choices were not really demand-driven or requested. My sense is that they were mainly supply-driven, and just now we can see a major change and effort to ensure that the events are not only better aligned but also demand-driven." Implementation 3.24 It is observed that more than half of in-country courses and events of FY02 and FY03 were implemented in an independent fashion. Even though some courses were aligned well with ongoing WB project topics, there was no much interaction between the Operations team and WBI. For example, one Operations staff commented during the interview: "Many WBI programs are relevant to what we are doing in the country, such as community empowerment. However, we were not involved in decision-making about how the program could be implemented in the country, nor consulted about who should be given a priority for the training. From the participant list, we can see most of these WBI selected trainees were likely to gain benefit for themselves and had little influence on the community." 3.25 On the same line, one former participant commented during the focus group discussion: "It is important that before designing a course WBI takes into account the social, political and cultural reality of the countries in order to improve its success. Countries, even in the same region, are not homogenous but respond to different needs and specific topics." 3.26 Meanwhile, most WBI activities were implemented through a local partner. Of the 15 FY02 and FY03 activities covered in the survey, at least two third (n=10) used a local partner during the implementation. In most cases, local partners were individuals or consulting firms. The exception is the GDLN Center in Radefal Landivar University. The most common use of the local partner was as a facilitator, or as someone to make logistical arrangement and recruit participants. It has been shown that using a local partner have made the WBI implementation in country easier. On the other hand, this approach has weaken somewhat the interaction between the Operations and the WBI team. Many times the Bank country office was not fully aware of what these partners were doing. During the interview, two country Operations staff mentioned that, before 2002, they were often consulted when WBI offered a course, especially regarding 15 participant selection. Since GDLN has been involved, they have become less aware of WBI courses in a timely fashion. Sometime, they were copied emails just before the course commenced. Given a short notice, there was little time for them to review and to make suggestions. 3.27 During the study period, only one WBI activity has been integrated or joint implemented with the Operations team. In this category, WBI and Operations share the ownership of the activity. Mutual WBI and Operations involvement is seen not only in program/activity design but throughout the entire implementation process. The impact of the activity implemented through this model will be examined in the report through a case study of the Anti-Corruption Project in Guatemala. Box 1: WBI Anti-Corruption Project in Guatemala In 2000, Alfonso Portillo became the first democratically elected president since the conflict. In the post-war period, support and trust for the government was at an all time low. There were rampant allegations of corruption against both the public and private sector. It became increasingly apparent that the government needed to change its image to renew citizen support, strengthen investor confidence, improve human rights, and recover from economic hardships caused by the civil war. Due to increasing pressure from the public and international community, President Portillo tried to reestablish public faith in the democratic process and requested support from the World Bank to design an Anti -Corruption strategy for the country. In January 2002, following the request of the Guatemalan government, a team of individuals from Latin American and Caribbean Region (LAC) and WBI Government Group (WBIGG) met with the Guatemalan officials to discuss designing and implementing an Anti-Corruption strategy that could work in the context of Guatemala's post-war environment. Highlights: An important issue identified by the country Operations involvement starting from the design stage Promoting and building the country's ownership 16 4. PROGRAM OUTCOME - EFFECTIVENESS AND IMPACT 4.1 In the previous section, we present how WBI activities for Guatemala were designed, planned and implemented. In this section, we turn attention to evaluation results concerning the outcome of WBI activities in Guatemala. As discussed earlier in Chapter 1, the evaluation is concerned with specifically the impact on capacity building at individual, organizational and country levels. Understanding that changes in individual behaviors and organizations occur in stages, a set of outcome proxy measures is used to assess the extent to which the WBI intervention has enhanced capacity of individuals, organizations and the country, namely, effectiveness, utilization and impact (change). 4.2 The analyses presented here are based on information derived from the survey, focus groups and case study, as the evaluation did not obtain much impact information from other data sources. Box 2: WBI Anti-Corruption Project in Guatemala: Stage of Change In April 2002, Francesca Recanatini from WBIGG visited Guatemala to present to the government, civil society and international community the major components of the Anti-Corruption strategy, how it has been implemented in other countries, and the ways in which it can be adapted to fit Guatemala's particular circumstances. This seven-stage framework also outlines the roles that the World Bank and the country would play in various stages of the intervention. 17 PROGRAM EFFECTIVENESS 4.3 Through the participant survey and focus groups, we asked the participants to assess the effectiveness of the WBI activities in respect with the following dimensions: · raising awareness; · providing knowledge and skills; · helping understand their role as agent in development; · helping develop strategies to address the needs of organizations; · helping develop strategies to address the needs of the country; and · helping develop contacts and partnerships. 4.4 Figure 4 shows that although moderate to high mean effectiveness scores are obtained across all dimensions (mean=4.0-5.2, on a 7-point scale), the WBI activities appear most effective in influencing individuals. Approximate 50 percent of the respondents gave higher rating scores (6 and 7) to the aspects of raising awareness, providing knowledge and skills, and helping understand their role as agent in development ­ notably focusing on individual level dimensions (Figure 5). Respondents considered the WBI activities as less effective in helping participants develop strategies to address the needs for organizations and the country, or to develop contacts and partnerships. Figure 4: Effectiveness of the WBI activity (mean score) Contacts & partnerships 4.0 Strategies for country 4.9 Strategies for organization 4.9 Role as agent in development 5.2 Knowledge & skills 5.2 Awareness 5.2 1 2 3 4 5 6 7 Rating (very low to very high) 18 Figure 5: Effectiveness of the WBI activity Contacts & partnerships 33.3% Strategies for country 39.5% Strategies for organization 42.9% Knowledge & skills 47.7% Role as agent in development 51.2% Awareness 55.0% 0% 10% 20% 30% 40% 50% 60% % of rating 6 & 7 4.5 More detailed comments came from the focus group discussions. Participants reported that the WBI learning events helped them in four main ways: 4.6 First, the learning events helped them develop a new way of thinking and wider vision. The courses generated a more accurate knowledge about a specific topic through learning what is happening in other countries. The content, theoretical framework, and concepts offered a wider vision of the topics and allowed participants to at least discuss issues that were not yet on the political agenda of Guatemala. For example, national educational assessments were only a part of a broader reform in education currently going on in the country. Participants learned some new concepts such as the importance of adopting new ways of evaluating individuals and institutions, as well as students in classrooms. They also learned new ways of framing anti-poverty strategies. 4.7 Second, participants learnt advanced knowledge, theory and techniques. Participants found that the WBI courses represented advanced knowledge or theory and techniques in some fields, particularly in educational reform, quantitative analysis, and competitiveness ­ although some strategies and tools introduced in the learning events will not necessarily be applied with new governmental measures. Additionally, the course material provided was seen as very useful and up-to-date. 4.8 Third, the WBI activities helped participants establish networks with international and national counterparts. One participant noted that: "The WBI courses allow us to debate issues that are not necessarily in the national agenda and to start a conversation among professionals as a way to create a rich network of people interacting about a specific issue." 4.9 Finally, the learning activities gave participants a sense of achievement and reward. Most participants found that the courses have had an impact on their professional activities, although they reported that the kind of impact and change that they could make would depend on the political will of decision-making officials. 19 Box 3: WBI Anti-Corruption Project in Guatemala: Earaly Indicators of Impact At the first meeting in April 2002, WB met with the Guatemalan government and civil society to establish the foundation of partnership necessary to begin the Anti-Corruption Project (ACP). Though the discussion was productive, it was clear that the "fragmentation of the civil society and the unwillingness of some civil society groups to work jointly with the government" would create obstacles in forming the Steering Committee. In response, members of WBIGG met with civil society representatives to exchange information and ideas about how to encourage a more participatory A-C process. Both parties agreed to form a preliminary working group with the government and that both would choose the future members of the ACP Steering Committee. WBIGG also met with members from the international donor community to get their support. UN representatives, USAID, IDB, and the Japanese and Italian embassies welcomed WBI's role in helping the Guatemalan government. After various negotiations and discussions, the Guatemalan government signed a formal Memo of Understanding on April 26, 2002 outlining their strong commitment to working with the international community and the World Bank to fight corruption. Highlights · Built country's ownership · Civil society involvement ­ public awareness · Memo of Understanding - Government commitment UTILIZATIONOF LEARNING 4.10 Participants were asked if they were able to use the knowledge and skills that they learnt from the WBI activities in the following areas: · Research · Teaching · Public awareness in development issues · New practices within your work organization · Collective initiatives · Legislation and regulation · Country development strategies 4.11 Between 50 and 70 percent of respondents said that they had been able to make use of the knowledge and skills acquired from learning events. However, as shown in Figure 6, the reported utilization is at a moderate level. On average, only about a third of the respondents reported that they used the new knowledge or skills very often in their work (ratings of 6 or 7 on a 7-point scale, from low to high). 20 Figure 6: Utilization of learning Legislation & regulation 28.0% Research 30.6% Implement new practices 33.3% Country development strategies 34.5% Collective initiatives 36.4% Public awareness 39.4% Teaching 53.1% 0% 10% 20% 30% 40% 50% 60% % of rating 6 & 7 4.12 Moreover, considerable variation can be seen in utilization across the areas of knowledge and skills (see Figure 7). The most use of the WBI activities is in teaching, followed by raising public awareness -- areas reflecting individual work and responsibility. Similar to the perceived effectiveness, the level of utilization at organizational and at country levels (e.g. influencing legislation or developing country strategy) is rather limited (mean ratings=3.7 and 4.2). Figure 7: Utilization of learning (mean score) Legislation & regulation 3.7 Country development strategies 4.2 Collective initiatives 4.2 Implement new practices 4.5 Research 4.6 Public awareness 4.7 Teaching 5.0 1 2 3 4 5 6 7 Rating (very low to very high) 4.13 The message derived from the focus group discussion supports the above finding. Some participants have applied new perspectives learned on a daily basis, particularly in their professional activities. Participants also used knowledge and skills learnt to influence organizational or institutional change. Some concrete examples include: · Use of some of the methodologies and contents in distance learning programs. One participant tried to use WBI training programs methods in a virtual course related to indigenous rights; · Introduction of new knowledge and pedagogy in already existing higher education programs (Masters degrees in Social Sciences); 21 · Application of new evaluation paradigms in educational institutions; · Generation of institutional discussions about the importance of development models or national education reforms; and · Application of evaluation tools in the classroom. Box 4: WBI Anti-Corruption Project in Guatemala: Utilization Following the Memo of Understanding, a series of meetings of the government with civil society was facilitated by the Bank during May and June 2002. These meetings led to two National Workshops that were facilitated by the Bank. The final outcome of the first workshop in July 15, 2002 was the formation of a temporary committee called the "Comision Transitoria." This governing body consisted of members from government and civil society. The Comision Transitoria built upon the suggestions made in the first National Workshop and created a draft proposal to discuss at the second Workshop held on September 9, 2002. This proposal outlined the structure of the Steering Committee, which is: · Consejo Coordinador (Committee Coordinator): to coordinate various activities and work related to anti - corruption; · Technical Unit: to collect and analyze empirical data using Governance and Anti -Corruption (GAC) diagnostics; and · Foro (Forum): to discuss issues regarding the two other bodies' work In the following months, the Consejo Coordinator was responsible for choosing the members of the Technical Unit to perform three GAC diagnostics and an action plan. WBI held a number of videoconferences with the Consejo to guide them in these efforts. In May 2003, another workshop was held with the international anti-corruption community to discuss transparency and promotion of the Consejo's efforts in fighting corruption in Guatemala. As a result, by October 21, 2003, the Consejo was ready to launch phase two of the ACP model. Indicators of the use of concepts and tools · Use of a participatory approach · Establishment of a steering committee · Development of an action plan PROGRAM IMPACT ­ CHANGES BROUGHT TO GUATEMALA 4.14 Survey participants were asked to rate (from 1 to 7) the degree to which the WBI activities have led to either negative or positive changes in the general course topic area and in the following six specific areas: · Research · Teaching · Public awareness in development issues · New practices within your work organization · Collective initiatives · Legislation and regulation · Country development strategies 22 4.15 As seen in Figure 8, the WBI activities have produced moderate impact. Using a 7-point scale (1=strong negative; 4=neutral; 7=strong positive), over 60 percent of respondents rated changes brought by the WBI activity in the course/event topic areas as positive. Close to half of this group believe that the change is fairly strong. Few participants felt that the changes were negative. Finally, nearly nine percent of participants answered "did not know". The mean rating of the impact is 5.1. Figure 8: Perceived overal changes brought by WBI activities 7 4.4% 6 40.0% positive strong 5 20.0% to 4 17.8% negative 3 6.7% Strong 2 2.2% 0.0% 5.0% 10.0% 15.0% 20.0% 25.0% 30.0% 35.0% 40.0% 45.0% Percentage 4.16 More closely, Figures 9 and 10 show that the areas rated by most participants (60 percent) as having the most positive changes are research (mean=5.5) and teaching (mean=5.4), which are again concentrated at individual levels. Public awareness, which shows the third most positive change, may be seen at both individual and community levels. Figure 9: Perceived impact of the WBI activity Collective initiatives 41.4% Implement new practices 41.9% Country development strategies 42.9% Legislation & regulation 45.5% Public awareness 50.0% Research 57.1% Teaching 60.0% Overall impact 48.9% 0% 10% 20% 30% 40% 50% 60% 70% % of rating 6 & 7 23 Figure 10: Perceived impact of the WBI activity (mean score) Country development strategies 4.4 Legislation & regulation 4.8 Collective initiatives 4.8 Implement new practices 4.8 Public awareness 5.0 Teaching 5.4 Research 5.5 1 2 3 4 5 6 7 Rating (very low to very high) 4.17 Moreover, participants attending the focus group discussions stated that the WBI training courses met the demand of Guatemala's development needs, and were relevant to participants' professional development tasks. However, the instability or even non- existence of a national policy in the sectors where they work or where they could influence the most has made their contribution difficult. Government changes in 2004 have been crucial for most high-ranking officials, including those who attended the courses in previous years. Without policy changes, even if most employees were selected to be trained by their institutions, some governmental constraints would still exist. However, participants felt that the education and research institutions are more flexible to adopting or adapting some of the tools learned in their WBI learning activities. Box 5: WBI Anti-Corruption Project in Guatemala: Impact on institutions and legislation A list of some major institutional and legal changes since President Portillo's first meeting with the World Bank in January of 2002 includes*: · October 2002: Anti-Narcotics Information and Analysis Service replacing previous agency that was seen as embroiled in scandals. · December 2002: National Commission for Transparency and Against Corruption established · December 2002: Congressional approval of law that allows Attorney General's office to defend interests of the State domestically and abroad, which has implications for repatriation of assets. · December 2002: Law on Pre-Trial hearings extending "greater number of public officials the right to have cases heard by congress before tried in court." Congressional committees also allowed to decide to leave or lift immunity of accused in most cases. · December 2002: Law on Probity and Responsibilities containing rules and processes on administrative procedures against public officials and sanctions when found guilty. *Entirelist from Transparency International's Global Corruption Report 2004 FACTORS INFLUENCING THE PROGRAM OUTCOME 4.18 A common Ordinary Least Squares and Two-Stage Least Squares regression analyses were conducted to determine what factors influence participants' perception of impact. Three models are examined in sequence: influences on effectiveness, influences on utilization and influences on change (impact). Based on the findings from the previous evaluation study, three groups of influencing variables were of primary interest 24 for the evaluation: (i) the perceived relevance of the activity; (ii) course features; and (iii) participant characteristics.3 In addition, the study extracts some results of Level 1 evaluation to explore their usefulness for the impact evaluation.4 A description of all variables included in each category is summarized in Annex 7. Model 1: effectiveness=f(perceived course relevance)+.... Model 2: utilization=f(predicted perceived effectiveness)+... Model 3: change=f(predicted utilization)+... 4.19 Considering the small number of respondents, the dependent variables (effectiveness5, utilization and impact) were first correlated with relevant independent variables to eliminate the total number of independent variables introduced into each of the models. For example, four independent or predictor variables are found to be significantly correlated with effectiveness, the first dependent variable (see Table 8). Each dependent variable was then regressed onto those variables shown to be significantly correlated withit. Factors influencing the effectiveness 4.20 Table 8 shows the regression results on effectiveness with the four relevant variables. The results suggest that the perceived effectiveness cannot be explained by a participant's assessment of whether he/she acquired new information from the event at the time they attended the event (Level 1). Neither can it be explained by his/her proficiency in language of instruction. Rather, the perceived effectiveness is enhanced by the level of perceived relevance of the activity to the country's needs. Table 8: Model 1 - Regression results for perceived effectiveness Effectiveness Independent variables Coefficient t Level 1questions Acquired new information 0.112 0.19 Perceived relevance To your work 0.015 0.14 To your country needs 0.412 3.82** Demographics Proficient in language of instruction 0.017 0.11 Constant 2.997 0.99 R2 (Adj R2 ) 0.43 (0.37) N 39 ** Statistically significant at the 0.01 significance level. 3Khattri, et al. (2002). Impact Evaluation of WBI Client Programs, FY000-01. WBI Evaluation Studies, No. EG03-06. The World Bank Institute. 4A mean score for six mandatory Level 1 questions for each course evaluated assigned to each survey respondent who attended that course. (1.Relevance of this activity to your current work or functions; 2. Extent to which you have acquired, information that is new to you; 3. Usefulness for you of the information that you have acquired; 4.Focus of this activity on what you specifically needed to learn; 5. Extent to which the content of this activity, matched the announced objectives; and 6.Overall usefulness of this activity.) 5An aggregated measure of effectiveness was constructed using an un-weighted mean of the six individual measures of effectiveness. 25 Factors influencing the utilization 4.21 To simplify the analysis of predictors of utilization, in addition to constructing an overall aggregated measure of use, two sub-dimensions of use were constructed as well: use at individual level and use at organization/country level6. Table 9 shows the result of the regression of utilization on the predicted effectiveness, plus three independent variables that are correlated with this outcome (not the same three found for effectiveness). It shows that across all three aggregated use measures, the level of utilization is significantly influenced by the predicted overall effectiveness of the WBI event. Moreover, a positive country development environment predicts the utilization for two of these three dependent variables, as does the course feature variable, development of an action plan. The fourth predictor variable, working environment, is not found to significantly affect the level of utilization. Table 9: Model 2 ­ Regression results for the level of utilization Overall use Use at Individual level Use at organization Independent variables Coefficient t Coefficient t Coefficient t Effectiveness (predicted) Overall effectiveness 0.898 3.21** 1.002 3.69** 0.843 2.64* Course features Action plan development 1.179 2.20* 0.929 1.79 1.23 2.01* External Environment Working 0.044 0.24 0.142 0.82 0.129 0.06 Country 0.372 2.76** 0.444 3.4** 0.277 1.80 Constant -1.84 -1.22 -3.18 -2.18* -0.792 -0.46 R2 (Adj R2) 0.56 0.66 0.43 9 (0.50) (0.61) (0.33) N 29 29 26 ** Statistically significant at the 0.01 significance level. * Statistically significant at the 0.05 significance level. Factors influencing the impact/changes 4.22 Table 10 shows the results of the three separate regressions testing the influence of the four variables correlated with impact (change). Notably, the perceived impact of WBI intervention (measured in three dimensions ­ overall, individual and organization), is significantly associated with the same utilization dimensions, i.e., individual use predicts the change at individual levels. This result holds for all WBI events, regardless where the event was held. In addition, no course features seem to have strong influence on impact, nor do the demographic characteristics of participants. 6 Use at individual level includes conducting research, teaching and raising public awareness. Use at organization/country level includes implementing new practices within the organization, organizing collective initiatives, influencing legislation and regulation and implementing country development strategies. 26 Table 10: Model 3 ­ Regression results for the perceived impact/change Overall impact Change at individual Change at organization Independent variable Coefficient t Coefficient t Coefficient t Use (predicted) Overall use 0.671 3.55** Academic use 0.566 2.13* Practical use 0.675 4.627** Country Venue of the event 0.481 0.60 1.378 1.56 -0.206 2.06 Demographics NGO 0.227 0.38 -0.224 -0.34 0.877 1.51 National institute -0.897 -0.16 -1.136 -1.72 -0.382 -0.64 Constant 1.672 1.48 1.809 1.25 2.056 2.06 R2 (Adj R2 ) 0.51 0.31 0.62 (0.42) (0.18) (0.55) N 29 28 28 ** Statistically significant at the 0.01 significance level. *Statistically significant at the 0.05 significance level. 4.23 In sum, the regression analyses suggest that a perception of course effectiveness and developing an action plan, coupled with a positive country environment can enhance the utilization of knowledge and skill acquired from the learning event, which may then lead to broader impact. Box 6: WBI Anti-Corruption Project in Guatemala: Political influences on impact In December 2002, Oscar Berger Perdomo, leader of the conservative Grand National Alliance party, won the presidency and changed the pace of the Guatemalan A-C project. The election brought in a new President as well as a different set of political actors. In February 2004, new members were elected for the Comision Coordinador to continue the implementation of the GAC diagnostics and A-C project. The field work began in May 2004 with a draft report following in October 2004. In March 2004, newspaper Prensa Libre broadcasted a high level meeting of the WB with the new president and representatives from civil society in Guatemala. The objective of the meeting was to exchange the view of corruption issues with the new government, the nature and consequence of corruption, and to reinforce the government commitment to fighting corruption. Mass media and civil society groups continue to raise public knowledge on corruption issues. In recent articles of the Guatemalan newspaper Prensa Libre from March 11 and 12 of 2004, there are reports that the United States denied nine visas to former President Portillo and several ex -ministers who have been accused of illegal activities and are still under investigation. In the context of this increasing public awareness and desire for government accountability, President Berger and his administration appear willing to continue supporting initiatives against corruption and for transparency. At the May meeting, officials recognized that initiatives such as the System Integrated of Financial Administration, the System BancaSAT and entru Guatecompras are achievements that already produce positive fruits. Meanwhile, the newly appointed presidential agent for Transparency, Carols Vielman, emphasized that the government finds "important coincidences among its plans and the approaches of the World Bank". It confirmed that a diagnosis of corruption will be carried out. HOW TO SUSTAIN THE BUILT CAPACITY? 4.24 The evaluation has revealed that WBI activities have brought some benefits to the country, particularly to individuals. The next question is whether, and through what mechanism, WBI has ensured and enhanced the sustainability of the built apacity. To 27 answer this question, the evaluation seeks past participants' experiences with WBI follow-up activities and networking with other course participants. 4.25 The evaluation reveals that WBI took little initiative to ensure the sustainability of impact and that the level of facilitation of continued learning and exchange was low. 4.26 Only a few survey respondents (n=9) experienced a follow up by WBI. Of the six follow-up respondents who contacted WBI, four rated that WBI was very and extremely useful. 4.27 Over 50 percent (n=25) of the respondents said that they were provided with the contact information of other participants during the learning event. However, 15 of them never used it. Eight people showed initiative in using this contact information to continue activity-related discussion and two used it to organize follow-up activities. 4.28 Consistent results are derived from the focus group discussion and interviews with the WBI task managers. There was no WBI-initiated follow up after the event. Focus group attendees noted that, while the WBI courses create national and international networks between researchers and specialists working on the same issues, these connections are not utilized after the event. Participants strongly expected WBI to facilitate the networking process. 4.29 A lack of follow up with the participants after the activity may also explain why both the WBI task managers and partners interviewed can not make any comments about the impact of the event. THE COMPARATIVE ADVANTAGEOF WBI IN GUATEMALA 4.30 This section examines general perceptions of participants and Bank staff about WBI's role in Guatemala, in comparison with other similar organizations/institutions. 4.31 Of the participants surveyed and interviewed, only a few attended similar learning events organized by other organizations/institutions. According to these participants, WBI's comparative advantage in Guatemala is not apparent. The most prominent non- WBI institutions in Guatemala that offer similar courses are the Inter-American Development Bank (IADB) and the Swedish, Canadian, and Spanish cooperation agencies. In the case of IADB, one participant mentioned that the difference is that after trainees finish the course, IADB organizes monthly conferences about the topic covered and invites trainees to attend them. They also believe that there is more flexibility with other organizations/institutions in terms of being able to change some of the course contents. 4.32 Moreover, GDLN in Rafael Landivar University conducted an end course evaluation for 15 courses delivered through the center in 2003. Of the 15 courses evaluated, eight were organized by WBI. Figures 11, 12, and 13 present the comparison of evaluation results using four indictors: quality of content, quality of methodology, applicability of course to Guatemala, and overall satisfaction with the course. 28 Figure 11: Rating on quality Figure 12: Rating on methodology Quality of the content Quality of the methodology 100% 100% 90% 90% 80% 80% 70% 70% 60% 60% 50% 50% 40% 40% 30% 30% 20% 20% 10% 10% 0% 0% Good Excellent Combined Good Excellent Combined WBI (n=117) Non-WBI (n=184) WBI (n=114) Non-WBI (n=175) Figure 13: Rating on applicability and satisfaction Answer=Yes 102% 100% 98% 96% 94% 92% 90% 88% Applicability Satisfaction WBI (n=117) Non-WBI (n=182) *There are some missing value, thus the total is not 100% 4.33 Despite the fact that participants gave a high rating to both WBI activities and non-WBI activities, WBI receives more ratings of "excellent" for the quality of content and methodology than non-WBI organizers. Applicability is perhaps not a very good criterion, as, in principle, the course selected to broadcast by GDLN has to be relevant and applicable to the country. Ratings of applicability are very high for both sources of activities: all participants vote WBI activities applicable to the country, compared to 95 percent of participants for non-WBI activities. Finally, a similarly high number of participants showed satisfaction with the courses organized by WBI and non-WBI, with non-WBI only about 3 percentage points higher. 4.34 All Bank staff interviewed (Operations and WBI task managers) agreed that WBI is unique in Guatemala if it provides services that meet the country's needs. It is unique in (a) providing knowledge products as a means to aid programs; (b) accessing and connecting with international experts and new cutting-edge knowledge and technology; and (c) providing a high quality product. However WBI has to be more flexible and is willing to provide a range of products that are tailored according to the country's needs. 29 5. CONCLUSIONS AND LESSONS LEARNED 5.1 By training and supporting key individuals in a country, WBI's objective is to contribute to the country's development and poverty reduction. This evaluation examines the extent to which this objective has been achieved in Guatemala. Focusing on WBI activities carried out during the period of FY02 and FY03, the evaluation looks at both the process and the impact of WBI programs in Guatemala. 5.2 The primary conclusion from this evaluation is that WBI programs have brought some positive impact to Guatemala, but that the impact can be greater if WBI improves the design and implementation of the programs offered. This conclusion is based on the following key findings. KEY FINDINGS · WBI activities in Guatemala were partially aligned with the Bank's assistance strategy. During the study period, there were a total of 33 Bank projects and 57 WBI events for Guatemala. More that 700 participants were trained by WBI. By mapping the topic areas and size of the WBI events with the four CAS priority areas, WBI activities in Guatemala seem to be aligned partially with the Bank's assistance strategy, especially for the topics of Reducing Poverty and Improving Economic Management. The areas where the alignment was least are Improving Governance and Public Sector Management, and Strengthening Participatory Decision-Making. · WBI activities offered were highly relevant to participants' work and to their country. Both the participant survey and the focus groups indicated that WBI activities delivered were highly relevant to their work and to their country (mean ratings=4.7 and 5.1, on a 7­point scale), but only moderately relevant to some country development goals. Among all activities in Guatemala, the Anti-Corruption Project has been regarded as "highly relevant" to the country by both the Bank Operations staff and client country. · WBI activities in Guatemala were not designed strategically at the country level. Despite some of alignment of WBI activities with the Bank's assistance strategy, by-and-large, WBI activities in Guatemala were not designed strategically at the country level, reflecting the stage of the country's development. Operations staff claimed that WBI had little flexibility on the offerings, and most activities were not tailored for the needs of the CAS and the choices were mainly supply-driven. This is perhaps due to the fact that Guatemala became a focus country after FY03, and for most non-WBI focus countries, global and regional approaches were still predominant in FY02 and FY03. On the other hand, most WBI activities were organized well at the event and activity levels. Some assessments were carried out to tailor the content of the activity to meet participants' needs. 30 · The majority WBI activities were implemented independently in the country, facilitated through local individual partners. The evaluation reveals that most WBI activities were implemented independently in the country. More than 60 percent of activities were delivered through local partners. However, the role of the local partners was limited to facilitation, logistical arrangement and participant recruitment. The Anti-Corruption Project in Guatemala is believed to be a very successful example of joint WBI and Operations implementation. · WBI activities were very effective in influencing individual knowledge and perceptions. Of the six effectiveness dimensions measured, the participants gave higher ratings for raising awareness, providing knowledge and skills, and helping understand the participant's role as agent (mean rating = 5.2 on a 7- point scale). The perceived activity effectiveness is seen to be positively influenced by the perceived relevance of the event to the country's needs. · Participants reported moderate utilization of knowledge and skills gained from WBI activities. The majority of survey respondents used knowledge and skills gained from the WBI activity at a moderate level (mean rating = 4.5, on a 7-point scale). The utilization was strongest in teaching and research applications, followed by raising public awareness - all areas reflecting individual work and responsibility. Similar to the perceived effectiveness, the level of utilization at organizational and at country levels (e.g. influencing legislation or developing country strategy) is rather limited (mean ratings=3.7 and 4.2). · The level of utilization of knowledge and skills is associated with having developed an action plan during the course, or having a positive country environment. · WBI has brought some positive impact/changes. As reported by participants, WBI activities have brought a moderately positive change in the course topic areas (mean rating = 5.1 on a 7-point scale). However, the changes were seen greater at individual levels than at organizational and the country levels. The level of impact is found to be influenced by the level of utilization. · Policy advisory support is seen as an effective intervention strategy in helping the country build capacity. The case study of the Anti-Corruption Project has demonstrated that policy advisory services as a non-traditional learning activity can be very effective in enhancing capacity, especially at country levels, through building the country's ownership and providing technical assistance. The degree of civil society involvement and having established a Memo of Understanding may suggest that the impact of the WBI intervention has gone beyond individual levels. · WBI made little effort to ensure the sustainability of built capacity. The evaluation shows that WBI took very little initiative to ensure sustainability of capacity that was built. The level of follow up and facilitation of continued learning for participants is low. · WBI' s advantage in Guatemala is not apparent. The message from both participants and Operations staff interviews suggests that although WBI is 31 unique in providing cutting-edge and high quality knowledge products to the country, WBI has to be more flexible and should be willing to provide a range of products that are tailored according to the country's needs. Otherwise, WBI will not have apparent advantage in Guatemala. LESSONS LEARNED 5.3 Drawing on the findings, the evaluation offers several lessons that can be used by WBI to improve its programs: 5.4 The first lesson is that WBI can provide important technical assistance or more direct knowledge products to the country, thus brining positive impact to the country. The content relevance to participants and to the country is the key to the success of programs offered. This is true regardless of kinds of activities, types of delivery modes or location. WBI has done well in ensuring the relevance of learning events to individual participants in Guatemala, and it should do more to ensure the relevance of WBI activities to the country's needs. In this regard, CAS can be used as an instrument to identify needs and to set priorities for the country programs. This lesson has been already applied by WBI since Guatemala became a focus country and a new regional program coordinator has been assigned. A strategic plan for Guatemala at the country level has been developed for FY05 in consultation with the Operations team. 5.5 The second lesson is that the influence of the WBI intervention cannot reach maximum impact if it focuses only on a relatively small number of individual participants. Within a given organization or institution there must be a critical mass of participants in order to substantially influence policy. WBI needs to place more attention to building institutional capacity in the country. This requires that WBI identifies the institutions or organizations that are critical in influencing policy and ensures intensive and widespread training within those institutions or organizations. 5.6 The third lesson is that program impact can be enhanced strategically by: · Designing and implementing programs jointly with Operations. The experience of the Anti-Corruption Project clearly shows that the involvement of Operations is extremely helpful in moving the project forward. · Promoting country-and particularly government­ownership of the activity. Again, the experience of the Anti-Corruption Project shows how government ownership of the intervention activity can influence the progress and the impact of the intervention. 5.7 The fourth lesson is that program impact can be enhanced technically by: · using an action planning during the learning event; and · using the non-traditional learning approach of policy advice services. 5.8 The final lesson is that program impact cannot be sustained without the continuing involvement and effort of WBI. A strong voice heard from the focus group is that WBI needs to take the initiative to follow up and to facilitate continued learning and networking after the event is over. 32 ANNEXES 33 34 ANNEX 1: LIST OF WBI LEARNING EVENTS FOR GUATEMALA FY02-FY03 (>1 DAY EVENTS) Event Title FY Duration Participants In country events 1 Strategic Choices for Education Reform - Central America I FY02 15 24 2 Curso de Economía y Regulación de Servicios Públicos y Defensa de la FY02 9 21 Competencia 3 Governance and A -C Diagnostics- Guatemala FY02 2 2 4 CESI and Governance FY03 8 19 Out of country events 5 Clean Air Initiative for Latin American Cities: the Case of Lima-Callao- Peru FY02 3 2 6 Macroeconomic Management for Financial Stability and Poverty Reduction: Brazil FY02 9 2 7 Reproductive Health and Health Sector Reform Course UNFPA FY02 5 1 8 International Conference on Environmental Compliance and Enforcement (INECE) FY02 5 5 9 Maximizing the Benefits of the Clean Development Mechanism (Honduras)) FY02 4 2 10 Water Information in Latin America FY02 4 2 11 Regional Forum - LAC FY02 2 1 12 Poverty Impact Evaluation: Alternative Methods FY02 5 2 13 Curso de Gestion Urbana para Centroamerica FY02 10 8 14 Curso en Economía de la Regulación de Servicios Públicos y Defensa de la FY02 20 1 Competencia 15 Adapting to Change: Core Course on Population- Reproductive Health and Health FY03 10 1 Sector Reform 16 Market Creation For Biodiversity FY02 5 3 17 Judicial Reform: Improving Performance and Accountability in Latin America FY03 2 4 18 Micro-Macro Simulation Methods for Poverty Analysis: An Introduction to IMMPA FY03 5 2 19 Internet-based Macroeconomic Management: Analytical Foundations and Case FY03 9 1 Studies 20 Latin America: Fiscal Policy for Poverty Alleviation Conference FY03 4 4 21 National Strategy Study (NSS) Workshop on Capacity Building Needs under the FY03 3 1 Kyoto Protocol - Switzerland 22 Regional Workshop on Non-Bank Financial Institutions Development in Latin America FY03 3 5 23 Global E-Conference on CSR- Education and Technology: How Can Corporations FY03 3 1 Contribute to Bridging Educational and Digital Divides? 24 Introduction to Corporate Social Responsibility - 11 FY03 3 2 25 CESI: Latin America Social Development Capacity Building FY03 15 14 26 Regional Small and Medium size Enterprises Training with Citi FY03 3 4 27 Using National Educational Assessments Results FY03 5 29 35 ANNEX 2: WORLD BANK INSTITUTE (WBI) COUNTRY FOCUS EVALUATION QUESTIONNAIRE - GUATEMALA INSTRUCTIONS WBI had the pleasure to have you participate in the following learning activity: Title: _____________________________________________________________ Held from: ________________________ to: ________________________ In: _______________________________________________________________ Getting your opinion of the above-mentioned activity--now that you have had time to reflect on it--is very important to help WBI improve its programs. For this, we ask you to complete this questionnaire. The questionnaire has four sections and should take approximately 20 minutes to complete. · Section 1 asks about the relevance of the activity. · Section 2 asks about the usefulness of the activity. · Section 3 asks you to compare this activity with similar learning activities offered by other organizations. · Section 4 asks about the characteristics of the activity, its follow-up and your background. We need your honest feedback. Please keep in mind that your responses will be kept confidential, and will be used for the sole purpose of improving WBI programs. If you have any questions about the questionnaire please call or send a message to XXX or WBI_Survey@world.org. Thank you for taking time to complete this questionnaire! ID:_________________ 36 I. Relevance of the Activity The activity that you are asked to evaluate is mentioned on the first page of this questionnaire. 1. Since the end of the activity, to what degree has the activity been relevant to your work? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 s s s s s s s 2. To what degree have the topics covered in the activity been relevant to your country's needs? Not relevant Extremely at all relevant 1 2 3 4 5 6 7 s s s s s s s 4. Was the activity related to the country development goals listed below? a. Eradicate extreme poverty s Yes s No s Don'tknow b. Achieve universal primary education s Yes s No s Don'tknow c. Promote gender equality and empower women s Yes s No s Don'tknow d. Reduce child mortality s Yes s No s Don'tknow e. Improve maternal health s Yes s No s Don'tknow f. Combat HIV/AIDS, malaria, and other diseases s Yes s No s Don'tknow g. Ensure environmental sustainability s Yes s No s Don'tknow h. Develop global partnerships for development s Yes s No s Don'tknow i. Ensure water sanitation and supply s Yes s No s Don'tknow j. Improve investment climate and finance s Yes s No s Don'tknow k. Promote trade s Yes s No s Don'tknow 37 II. Usefulness of the Activity 5. Please rate the degree of effectiveness of the activity in each area noted below. (If the area was not an objective of the activity, please mark "not applicable.") Not Extremely Not Areas effective at all effective applicable 1 2 3 4 5 6 7 NA a. Raising your awareness and understanding of the development issues important to your country s s s s s s s s b. Providing you with knowledge or skills s s s s s s s s c. Helping you better understand your role as an agent of change in your country's development s s s s s s s s d. Helping you develop strategies or approaches to address the needs of your organization s s s s s s s s e. Helping you develop strategies or approaches to address the needs of your country s s s s s s s s f. Helping you develop contacts, develop partnerships and build coalitions in the field s s s s s s s s 6. How would you rate the change--brought by the activity--in the main topic or issue it addressed? Strong Strong negative No positive change change change Don't know 1 2 3 4 5 6 7 s s s s s s s s 7. How often have you used the knowledge and skills you acquired in the activity for the following purposes? (If you have not worked in the given area since this activity, please mark "Not applicable.") Not Very Not Purposes at all often applicable 1 2 3 4 5 6 7 a. Conducting research s s s s s s s s b. Teaching s s s s s s s s c. Raising public awareness in development issues s s s s s s s s d. Implementing new practices within your work organization s s s s s s s s 38 e. Organizing collective initiatives s s s s s s s s f. Influencing legislation and regulation s s s s s s s s g. Implementing country development strategies s s s s s s s s 8. To what extent did the following factors help or hurt the process of using the knowledge/skills that you acquired at the activity? Neither Factors Greatly helped Greatly Not hurt nor hurt helped applicable 1 2 3 4 5 6 7 NA a. Your work environment (e.g., work procedures, colleagues, incentive system, funding, etc.) s s s s s s s s b. Your country's development environment (e.g., country policies, social groups, political groups, readiness for reform, etc.) s s s s s s s s 9. How has the activity influenced or led to changes in the following areas? (If the area is not relevant to the activity, please mark "Not applicable.") Areas Negative No Positive Not influence influence influence applicable 1 2 3 4 5 6 7 NA a. Research s s s s s s s s b. Teaching s s s s s s s s c. Public awareness in development issues s s s s s s s s d. New practices within your work organization s s s s s s s s e. Collective initiatives s s s s s s s s f. Legislation and regulation s s s s s s s s g. Country development strategies s s s s s s s s 10. Since the activity, have you discussed the issues raised in the activity, at work, with local partners, government officials, NGOs, or in the media? Never Thoroughly discussed discussed 1 2 3 4 5 6 7 s s s s s s s 39 III. Comparison of the WBI Activity with Similar Activities Offered by Other Organizations 11. Did you participate in any similar learning activities offered by other (NON-WBI) organizations in your country? (If no, please skip to question 14.) s Yes s No 12. If yes, please provide the name(s) of the organization(s): 1. 3. 13. How would you rate the usefulness of the WBI activity compared to NON-WBI activities? WBI About WBI much less the much more No useful same useful opinion 1 2 3 4 5 6 7 s s s s s s s s IV. Characteristics of the WBI Activity, its Follow-up and Your Background 14. How would you describe the type of the WBI learning activity that you attended? Video Sessions (Distance Class room Mix of Video and Conference Web-based Study tour Learning) (Face to Face) Face to Face Learning 1 2 3 4 5 6 s s s s s s 15. How effective was this type of learning activity in helping you learn? Not effective Extremely No at all effective opinion 1 2 3 4 5 6 7 s s s s s s s s 16. During the WBI activity, did you develop an action plan/strategy (e.g., work plans, strategy papers, or policy documents) to apply the knowledge and skills you learned? (If no, please mark "no" below, then skip to question 18.) s Yes s No 17. If yes, did you use part or all of the action plan in your work? s Yes s No 18. Were you provided with the contact information of other participants in the activity, such as e- mail addresses, telephone numbers or mailing addresses? (If no, please mark "no" below, then skip to question 20.) s Yes s No 40 19. If yes, how did you use it? Used it to Used it to Never used continue organize joint Other uses it activity related follow-up (Please specify briefly) discussions activities s s s s _____________________________ 20. Was the language of instruction used during the activity the same language you use at work? s Yes s No 21. At the time of the activity, what was your level of proficiency in the language of instruction? Not proficient Highly at all proficient 1 2 3 4 5 6 7 s s s s s s s 22. At the time of the activity, what was your level of proficiency in the technical terminology used in the activity? Not proficient Highly at all proficient 1 2 3 4 5 6 7 s s s s s s s 23. After the activity, did WBI contact you for follow-up issues regarding the activity? s Yes s No 24 After the activity, did YOU contact WBI for follow-up issues or questions on the content of the activity? (If no, please skip to question #26) s Yes s No 25. If yes, please rate WBI's helpfulness in addressing your issues. WBI WBI I did not responded, responded have WBI did but was not and was follow-up not helpful extremely requests respond at all helpful for WBI 0 1 2 3 4 5 6 7 NA s s s s s s s s s 26. Which of the following best describes the organization in which you have worked the longest since the activity? (Select one.) s University/research institution s National/central government s Non-governmental organization (not-for-profit) s Provincial/regional government s Media s Local/municipal government s Private sector s Other, specify: ____________________________ 41 27. Which of the following best describes the primary type of work you have done the longest since the activity? (Select one.) s Research s Teaching s Policymaking/legislation s Provision of services (e.g., financial, health, etc) s Management/administration s Other, specify: ____________________________ 28. How would you best describe the level of the position you have held the longest since the activity? Highest level (e.g., Minister, Deputy Minister, Top Government Official, Full Professor, President of an s organization) s Senior level (e.g., Department Head, Division Head, Associate Professor, Senior Researcher) s Middle level (e.g., Program Manager, Project Leader, Assistant Professor, Technical Expert) s Junior Level (e.g., Research associate, Ph.D. level graduate student, Technical Specialist) s Entry level (e.g., Intern, assistant) s Other, Please specify: __________________________________________________________________ 29. What is your gender? s Male s Female Thank you for your feedback. We appreciate very much your cooperation. 42 ANNEX 3: LIST OF WBI LEARNING EVENTS ATTENDED BY SURVEYED PARTICIPANTS Action RequestedPartner Participants No Title Learning by involved Product Line n=45 1 CESI and Governance Yes External No Skill building 1 2 CESI: Latin America Social Development Capacity Yes Operation No Skill building 5 Building 3 Curso de Economía y Regulación de Servicios No External Yes Knowledge 7 Públicos y Defensa de la Competencia exchange 4 Curso de Gestion Urbana para Centroamerica No External Yes Skill building 1 5 Global E-Conference on CSR- Education and No External No Knowledge 1 Technology: How Can Corporations Cont exchange 6 International Conference on Environmental No External Yes Skill building 1 Compliance and Enforcement (INECE) 7 Internet-based Macroeconomic Management: Yes External No Skill building 1 Analytical Foundations and Case Studies 8 Introduction to Corporate Social Responsibility - 11 No External Yes Skill building 2 9 Macroeconomic Management for Financial Stability No External Yes Skill building 1 and Poverty Reduction: Brazil 10 Micro-Macro Simulation Methods for Poverty No External Yes Skill building 2 Analysis: An Introduction to IMMPA 11 National Strategy Study (NSS) Workshop on Yes External Yes Skill building 1 Capacity Building Needs under the Kyot 12 Regional Small and Medium size Enterprises Yes External Yes Skill building 1 Training with Citi 13 Regional Workshop on Non bank Financial No Operation Yes Policy Service 1 Institutions Development in Latin Americ 14 Strategic Choices for Education Reform - Central No Operation Yes Skill building 11 America I 15 Using National Educational Assessments Results Yes External No Skill building 9 43 ANNEX 4: QUESTIONS OF FOCUS GROUP DISCUSSIONS 1. How were you selected for participating in the WBI activity? Explain 2. What did you gain from this activity? Please detail 3. What were the most useful aspects of the activity? What were the least useful aspects of the activity? Explain. 4. Can you offer specific examples of how you applied in your official duties what you learned in the activity? · Probe 1: Have these led to any changes in your organization, or changes in policies and practices in your country? · Probe 2: What were these changes? 5. What were some of the obstacles that you encountered in implementing the new knowledge and skills you acquired? Examples? 6. Were the knowledge and skills you acquired available from another source ­ either internationally or locally? Who are the prominent non-WBI training institutions/agencies in your country? 7. How can we improve WBI activities in the future? 8. Lastly, would you be willing to pay for this course? And if yes, how much? 44 ANNEX 5: INDIVIDUAL INTERVEIWS 5A: Unstructured Interviews with the Operations staff 1. Overall, to what degree do you think WBI learning events are aligned with the CAS? If you were to rate the degree of alignment on a scale of 1 to 10, (where 1 is no alignment and 10 is complete alignment), how would you rate it? 2. What was the extent and nature of your involvement in WBI activities during FY01 and 02? (e.g., development of course material and content, identifying participants, facilitating their attendance, providing instructors, providing funding for participants, technical/administrative assistance, etc.,) 3. Within the past two years, WBI's training events in Guatemala has been in mostly in four areas: poverty, education, community development and a governance. In this period, do you know of any outcomes (medium-term) that resulted from WBI learning events? What sort of evidence - on the ground ­ have you seen that shows results? Please describe. 4. What are the main barriers to applying new knowledge toward change in Guatemala? 5. In your opinion is WBI uniquely positioned to deliver learning activities for capacity building purposes in Guatemala? Why? What is it the WBI brings ­ or could bring ­ to the country that other's don't? 6. As the operational counterpart to WBI activities, do you think there should have been things done differently by WBI or yourself to better serve the learning needs of the country? Have you communicated any such "lessons learned" with the WBI task manager(s) in charge? 7. Taking a more forward looking approach, what are some ways that WBI can assist the country in moving closer its development goals? 45 5B: UNSTRUCTURED INTERVIEWSWITH WBI TASK/PROGRAM MANAGER A. Training design and relevance 1. To what degree was the activity driven by and/or linked to Bank Operations or networks? (includes CASs, PRSP/Cs, and ESWs) 2. What kind of needs assessment did you conduct for this activity? (Explain all the different processes for obtaining information on country needs.) 3. Are there any "special" features about the activity that distinguishes it from other WBI activities (e.g. partnerships, target audience, etc.,)? 4. Were there any opportunities provided for course participants to continue their learning after the course ended (e.g. web-based discussion groups)? How about opportunities to maintain contacts with other participants? 5. How was this activity related to the program strategy at the time of your offering? To the overall "body of knowledge" covered by the program? To other products in your program (e.g., books, web-sites, and/or newsletters)? 6. Do you have partnerships with other training institutions? If so, are you disseminating and sustaining this program through them? How? 7. Is WBI uniquely positioned to deliver this program in the country (Guatemala)? If so, why? B. Effectiveness and results 8. Since the course, what were the most important outcomes (medium-term) that resulted from this learning event? Are there one or two anecdote(s) that display a result (direct or indirect)? Please describe. 9. In retrospect, what would you have done differently? 10. What are the most important barriers to use of the training knowledge, skills, or information by your participants? 11. How do you see your program (as it exists today) evolving? 46 ANNEX 6A: REVIEW OF ALIGNMENT OF WBI ACTIVITIES WITH WB COUNTRY ASSISTANCE STRATEGIES The following examines the critical issues in Guatemala, the World Bank Group's efforts to meet the development priorities, and then assesses the degree of alignment between WBI activities in FY02-03 and WB efforts. The first section describes the critical issues that were identified by the Guatemalan government in the 2002 Guatemala PRSP. The second section uses the 1998 Guatemala CAS and 2002 CAS Progress Report to describe the issues that the WB tried to address through its lending and non-lending products, and explains how those efforts aligned with the priorities of the PRSP. The final section uses the 2004 CBP and information from the Business Warehouse to describe and assess the level of alignment that WBI activities had with WB products. THE CRITICAL ISSUES IN GUATEMALA (PRSP) The 1996 Peace Accords ended 36 years of armed conflict within Guatemala. The fundamental cause of the conflict has been widely attributed to the historical pattern of exclusion, particularly with regards to indigenous groups. This prevented the majority of citizens from improving their economic standing, gaining access to social services, and playing a significant role in their own development. The Peace Accords therefore placed poverty reduction at the forefront of the national agenda in order to build a more equitable and inclusive society. The Guatemala PRSP aims to reconcile the past by focusing the national agenda on three primary pillars: economic growth with equity, investment in human capital, and investment in physical capital. Economic Growth with Equity The Guatemala PRSP asserts that the country must achieve stronger economic growth in order to achieve the underlying goal of reducing poverty. Although Guatemala has had surprisingly solid economic growth for a historically exclusive society, the growth rates have not been sufficient to benefit the poor. Guatemala has one of the highest poverty rates in Latin America, and continues to put the majority of the burden on rural communities with 81 percent the poor and 93 percent of those in extreme poverty living in the countryside. The strategies for improving economic growth include: maintaining macroeconomic stability, reforming the financial sector, reorienting public expenditures to the social sectors and infrastructure, and promoting agricultural and rural development. Investment in Human Capital The second pillar of the PRSP focuses on building human capital while targeting the poorest citizens. Investments are concentrated on the areas of greatest need which includes health, education, and food security. Guatemala has some of the worst health facilities in Latin America, the lowest life expectancy, and highest maternal and infant mortality. The national interventions aim to extend coverage of basic health services, improve access to the services, and strengthen health promoting and disease prevention services. In education, Guatemala ranks third from last for illiteracy. Investments are targeted at increasing net enrollment in pre- and primary schools, and reducing the 47 illiteracy rate. In regards to food security, the focus is again on the poor and excluded, who are highly dependant on agriculture. Women and indigenous populations in particular have limited opportunities and face discrimination in the labor market. Furthermore, land ownership in Guatemala is one of the most unequal in the world which constrains investment. Investment in Physical Capital The third pillar invests in the physical infrastructure of Guatemala's poorest and most remote areas. The areas of priority include: water supply, basic sanitation, roads, and energy. Progress has been evident in water supply and electricity, with 70 percent of households having piped water and electricity. Sanitation is mixed, with 90 percent having some kind of basic sanitation but fewer than half having access to sewage systems. The roads are especially deficient in the rural areas, which constrains access to markets and basic services. Additional areas of need As part of Guatemala's efforts to reduce poverty and build social inclusion, three additional areas of need are outlined in the PRSP yet do not fall neatly into the primary pillars: public sector modernization, governance, and decentralization and participation. Progress in modernizing the public sector includes instituting an integrated financial management system within all ministries to improve management, such as with tax collection and transparency. In the area of Governance, the PRSP includes actions to combat corruption and improve transparency, which are expected to supplement and enhance the influence of efforts at the public sector. Finally, decentralization and participation will be the means of expanding coverage to the rural populations. It will enable traditionally excluded groups to take ownership over program design and oversight, which simultaneously empowers them to drive development in their own communities and strengthen expenditure management. ISSUES THE WORLD BANK GROUP TRIED TO ADDRESS Subsequent to the 1996 Peace Accords, the World Bank Group's assistance intensified during the implementation of the CAS strategy from 1998 to 2001. The 2002 CAS Progress Report builds on the foundation laid in the 1998 CAS, which shared the national PRSP goals of reducing poverty and building a more inclusive society. The CAS has four interlinked development priorities that closely mirror, though do not perfectly align with, the pillars of the PRSP: · Reducing poverty and promoting social inclusion. · Strengthening social cohesion and participatory decision processes. · Improving economic management to maintain macroeconomic stability and foster growth. · Improving governance and public sector management. 48 The World Bank's products are designed to make progress in the priority areas, and are composed of lending and non-lending products. The lending products are long-term projects that can last many years and include large loans. The non-lending products are learning and research-based activities that involve less funding and are typically shorter in duration. There were a total number of 24 active lending products during FY02-03 that comprised a total of $760,500,000. During the same period, there were nine non- lending products in the amount of $731,000. Reducing poverty and promoting social inclusion This first development goal is broken down into four sub-priorities which cut across several of the PRSP priorities. First, the CAS seeks to induce economic growth and create income-generating opportunities for rural populations much in the same way the PRSP seeks economic growth with equity by promoting agricultural and rural development. Bank lending projects that aligned with this priority include the Land Administration and Land Fund Projects. Second, the CAS promotes more effective and far reaching social programs, while the PRSP seeks to reorient public expenditures to the social sectors and invest in human capital by focusing in food security, education, and health. Lending products that aligned with this priority include the Universalization of Basic Education Project and the Social Investment Fund Project. The third priority focused on ensuring effective delivery of social services targeted at the poorest, which aligns with the PRSP's goal of reorienting public expenditures to the social sector. Lending activities tied to this priority include the Reconstruction and Local Development Project and the Poverty Analysis, Policy Formulation and Monitoring Evaluation Project. There are also five non-lending products in this area. The fourth priority seeks to foster rapid improvements in infrastructure, and promote better functioning markets and rural financial systems. This aligns with the PRSP priority of investing in physical capital among poorest areas, and extending coverage of infrastructure in the sectors of water, sanitation, roads, and energy. Lending projects that align with this priority include the Rural and Main Roads Project, the Western Altiplano Natural Resources Management Project, Community Management of the Bio-Itza Reserve, and the Management and Protection of Languna del Tigre Project. Strengthening social cohesion and participatory decision processes This second development goal of the CAS focuses on improved targeting of government spending, aiding poorer communities, and empowering them to take greater control over their own development. This aligns with the decentralization and participation priority of the PRSP, which seeks to improve targeting, increase community ownership of program design, and strengthen participatory expenditure management. Bank lending products that aligned with these priorities include the Second Rural and Main Roads Project and the Strengthening the Ombudsman Office for Indigenous Women within the Guatemala Government Project. Other relevant lending products that also address that cut across additional development priorities include the Universalization of Basic Education Project, Community Management of the Bio-Itza Reserve Project, and the Land Administration and Land Fund Projects. 49 Improving economic management to maintain macroeconomic stability and foster growth This third development goal of the CAS seeks to maintain a stable macroeconomic environment. This aligns with the imbedded priorities of the PRSP pillar of Economic Growth with Equity. The first imbedded priority is maintaining macroeconomic stability. Lending products that aligned with this priority include the Competitiveness Project and Private Participation in Infrastructure Project. The second imbedded priority is instituting reforms to strengthen the financial sector. The lending products that align with this include the Financial Sector Adjustment Loan and the Financial Sector Technical Assistance Loan. There were four non-lending products that supported this development goal. Improving governance and public sector management This fourth overarching development goal of the CAS focuses on reforms that combat corruption and misuse of public funds. It seeks to concentrate these reforms in the areas of the financial sector, tax administration, public sector procurement, and the judicial sector. The PRSP pillar of Economic Growth with Equity and the imbedded priority of reforming and strengthening the financial sector is in alignment with the lending product Integrated Financial Management III. The PRSP's pillar of Public Sector Modernization has two priorities that align with relevant Bank lending products. The first is combating corruption and improving transparency, which is aligned with the Judicial Reform Project. The second priority is modernizing institutions in the public sector by integrating financial management systems and strengthening tax collection mechanisms. This is in alignment with the Bank lending product entitled Tax Administration Technical Assistance Loan Project. WBI ACTIVITY ALIGNMENT WITHTHE BANK'S INTERVENTIONS The goals outlined by the CAS along with Bank products clearly have strong alignment with the goals stated in the Guatemala PRSP. This section examines the degree to which WBI activities appear to align with the World Bank Group's products during the fiscal years of 2002 and 2003. WBI activities are mapped into two categories: those held within Guatemala, and those held out-of-country that included Guatemalan participants. The WBI activities and Bank products are mapped under the four CAS priorities and within the major WBI classifications. So, for example, the Universalization of Education lending product is mapped under the "Reducing Poverty and Social Inclusion" CAS priority, and further mapped under the WBI "Education Program" classification. This helps broadly reveal which lending and WBI activities address specific CAS priorities while gaining additional insights in the area of focus. Reducing poverty and promoting social inclusion Since poverty reduction is the centerpiece of Guatemala's development agenda, this development priority overall has the greatest number of WB products and WBI activities: 10 lending products, five non-lending products, six in-country WBI activities, and 21 out- of-country WBI activities with Guatemalan participants. Breaking this development 50 priority down by WBI classification reveals that there were no Bank products or WBI activities in the City and Urban Management classification. This makes sense since the focus for this priority is on the rural communities. There appears to still be a level of demand in this area, however, as there were two WBI activities held out-of-country that included Guatemalan participants. Similarly, in the area of Water Programs there were neither in-country WBI activities or Bank products. The same is true for Health, Nutrition and Population excepting for one non-lending product. As with the Water Program, Guatemalan participants were involved in out-of-country WBI activities. What is different about the lack of activities and products in these classifications is that they are more in line with the priorities stated in both the CAS and the PRSP, which seek to build rural access to clean water and health services. In the WBI classifications of Education Programs and Poverty and Growth classifications, there were both WBI activities held within Guatemala as well as Bank lending products. In the Education Programs classification, the Universalization of Basic Education lending product is aligned with four WBI activities aimed at the education sector. There were no Guatemalan participants involved with Education activities held outside of Guatemala. In Poverty and Growth there are two active lending and four non- lending products that are aligned with two WBI activities. There were also six WBI activities held outside of the country that included Guatemalan participants. There are three WBI classifications in which there were Bank products yet no WBI activities within Guatemala: Environment and Natural Resources, Rural Poverty and Development, and Social Protection and Risk Management. The Environment and Rural Poverty and Development classifications each had three lending products. Furthermore, there appears to have been a significant level of demand for Environment, as there were nine out-of-country WBI activities with Guatemalan participants. The Social Protection classification had only one lending product and there were no out-of-country WBI activities that were attended by Guatemalans. Strengthening social cohesion and participatory decision processes Both the CAS and PRSP stated poverty reduction to be the primary goal, which is strongly tied to the goal of building social inclusion. The WBI classification of Community Empowerment and Social Inclusion was the only classification that aligned with this development priority. There were seven lending products in this classification, yet only one WBI activity was held in Guatemala (FY2003). Furthermore, there were no Guatemalan participants involved in WBI activities held outside of the country in this area. This could be interpreted as a lack of demand, or more likely, a lack of opportunity for the poor to be involved in activities of this sort due to financial constraints. Improving economic management to maintain macroeconomic stability and foster growth Under this CAS priority, there were a total of four lending products, four non-lending products, two in-country WBI activities, and nine out-of-country WBI activities with Guatemalan participants. There were four WBI classifications under this development priority: Financial Sector Policy Outreach, Private Sector Development, Public-Private 51 Partnerships in Infrastructure, and Trade. The Financial Sector and Private Sector classifications each had lending products, no WBI in-country activities, and no Guatemalan participants attending WBI activities held out-of-country. More specifically, the Financial Sector had two lending products, three non-lending products, and one external WBI activity with Guatemalan participants (2002). The Private Sector has one lending product, one non-lending product, and seven out-of-country WBI activities. The Public-Private Partnership had one lending product and one WBI activity in Guatemala. In the area of Trade, there were no lending activities, one in-country WBI activity, and one out-of-country WBI activity with Guatemalan participants. Improving governance and public sector management This fourth development priority had a total of three lending products, no non-lending products, six in-country WBI activities, and six out-of country activities with Guatemalan participants. There were two WBI classifications which appear to be least aligned: Governance and the Public Sector. Governance had one lending product, four in-country WBI activities, and four out-of-country WBI activities with Guatemalan participants. The Public Sector had two lending products, two in-country WBI activities, and two out- of-country WBI activities. CONCLUDING REMARKS The level of alignment between WBI activities in fiscal years 2002 and 2003 with the Bank's lending products appears to be somewhat mixed. In regards to the four overarching CAS development priorities, there seems to be good alignment. The first development priority of Reducing Poverty had a total of ten lending products, five non- lending products, six in-country WBI activities, and twenty one out-of-country WBI activities. The second development priority of Strengthening Participation marked the least alignment of the four with seven lending products, no non-lending products, one in- country WBI activity, and no out-of-country WBI activities. The third development priority of Improving Economic Management and Growth had four lending products, four non-lending products, two in-country WBI activities, and nine out-of-country WBI activities. The fourth development priority had three lending products, no non-lending products, six in-country WBI activities, and six out-of country activities. Beyond the four overarching development priorities of the CAS, the extent to wich WBI activities are aligned with Bank lending products becomes more questionable to the degree of precision. Using the criteria of WBI classification, of the fourteen categories, only half were aligned between Bank lending/non-lending products and in-country WBI activities. That is, if a WBI classification did or did not have a lending/non-lending product, seven times out of fourteen WBI activities followed suit. Therefore there were also seven classifications in which there either was a WBI activity but no lending/non- lending product, or there was a lending/non-lending product and no in-country WBI activity to complement it. In sum, the WBI activities in 2002-2003 seem generally aligned in support of CAS objectives and Bank lending/non-lending products, however the extent of precision remains questionable. 52 ANNEX 6B: MAPPING WBI ACTIVITIES WITH WB COUNTRY ASSISTANT STRATEGIES FY02-03 WB Products* WBI Activities Non-lending Products (9 Activities in Guatemala Total: Activities with Guatemalan CAS Priorities Lending Products (24 Products @ 15 Activities, 397 participants outside Guatemala PRSP National Priorities (1-4) Products @ $760.5 million) $731,000) Participants) (Total: 42 Activities, 379 Participants) Economic growth with equity -Reorient public expenditures to the social sectors and infrastructure. -Promote agricultural and rural development. Investment in human capital (1) Reducing Poverty and Promoting Social Inclusion -Concentrate investments in the - Induce economic growth and income generating opportunities with a focus on rural areas . areas of food security, education, - Promote more effective and far reaching social programs. and health. - Ensure effective delivery of social services targeted at the poorest. - Foster rapid improvements in infrastructure, and promote better functioning markets and rural financial systems. Investment in physical capital -Focus on poorest and most remote areas of the country. -Extend coverage of basic infrastructure in regards to water supply, basic sanitation, roads, and energy. City and Urban Curso en Gestisn del Suelo Urbano Management (Panama, FY03, 2) Curso en Reduccisn del Riesgo Urbano (Panama, FY03, 2) Education Universalization Of Basic Strategic Choices for Program Education Project (Fy01- Education Reform - Central Active, $62.2) America I (FY02, 24) Eight Reasons to Invest in Early Childhood Education (FY03, 41) Using National Educational Assessments Results (FY03, 29) "Global Dialogues on Early Childhood Development Education" (FY03, 20) (Annex 6B continues on next page.) 53 (Annex 6B continued.) WB Products* WBI Activities Non-lending Products (9 Activities in Guatemala Total: Activities with Guatemalan CAS Priorities Lending Products (24 Products @ 15 Activities, 397 participants outside Guatemala PRSP National Priorities (1-4) Products @ $760.5 million) $731,000) Participants) (Total: 42 Activities, 379 Participants) Environment and Community Management of Market Creation For Biodiversity Natural the Bio-Itza Reserve (Peten) (Mexico, FY02, 3) Resources (GEF-MSP) (FY03-Active) Western Altiplano Natural Maximizing the Benefits of the Clean Resources Management Development Mechanism (Honduras, Project (Fy03-Active, $32.8) FY02, 2) Management and Protection International Conference on of Laguna del Tigre Project Environmental Compliance and National Park (GEF-MSP) Enforcement (INECE) (Costa Rica, (FY00-Active) FY02, 5) Clean Air Initiative for Latin American Cities: the Case of Lima-Callao (Peru, FY02, 2) UNITAR BRIEFING ON FINANCING SUSTAINABLE DEVELOPMENT (USA, FY03, 1) National Strategy Study (NSS) Workshop on Capacity Building Needs under the Kyoto Protocol (Switzerland, FY03, 1) Global Dialogue on Sustainable Development: Corporate Social Responsibility (USA, FY03, 21) Fiscal Policy and the Environment Seminar (Chile, FY03, 3) IWRM - Training of Trainers with UNESCO-CapNet (Switzerland, FY03, 1) Health Impacts of Health, Nutrition Reproductive Health and Health Traditional Fuel and Population Sector Reform Course UNFPA (Italy, Use (FY02, FY02, 1) $97,000) (Annex 6B continues on next page.) 54 (Annex 6B continued.) WB Products* WBI Activities Non-lending Products (9 Activities in Guatemala Total: Activities with Guatemalan CAS Priorities Lending Products (24 Products @ 15 Activities, 397 participants outside Guatemala PRSP National Priorities (1-4) Products @ $760.5 million) $731,000) Participants) (Total: 42 Activities, 379 Participants) Adapting to Change: Core Course on Population, Reproductive Health and Health Sector Reform (Italy, FY03, 1) Health Sector Reform and Sustainable Financing-Flagship Core Course (USA, FY03, 60) Poverty and Reconstruction and Local Guatemala CAS Attacking Poverty Course for Poverty Impact Evaluation: Growth Development Project (FY99- (2002, $95,000) Civil Society in Latin America Alternative Methods (Peru, FY02, 2) Active, $30.0) (FY02, 21) Poverty Analysis, Policy Poverty Study Attacking Poverty Course - Regional Forum - LAC (Bolivia, Formulation and Monitoring (2002, $249,000) GDLN LAC (FY03, 24) FY02, 1) and Evaluation (FY01-Active) Country Poverty Reduction Strategies Forum Assistance for Bank/Fund Staff. Challenges Evaluation (FY02, Ahead and Operational Implications $70,000) (USA, FY02, 2) Country Macroeconomic Management for Accountability Financial Stability and Poverty Assessment Reduction (Brazil, FY02, 2) (FY02, $60,000) Water Information in Latin America (Chile, FY02, 2) Internet-based Macroeconomic Management: Analytical Foundations and Case Studies (USA, FY03, 1) Rural Poverty Land Administration Project and Develop- (FY99-Active, $31.0) ment Land Fund Project (FY99- Active, $23.0) Rural and Main Roads Project (FY98-Active, $66.7) (Annex 6B continues on next page.) 55 (Annex 6B continued.) WB Products* WBI Activities Non-lending Products (9 Activities in Guatemala Total: Activities with Guatemalan CAS Priorities Lending Products (24 Products @ 15 Activities, 397 participants outside Guatemala PRSP National Priorities (1-4) Products @ $760.5 million) $731,000) Participants) (Total: 42 Activities, 379 Participants) Micro-Macro Simulation Methods for Water Program Poverty Analysis: An Introduction to IMMPA (Brazil, FY03, 2) Social Protection Social Investment Fund and Risk Project (FY98-Closed FY04, Management $20.0) Decentralization and Participation Improve targeting and reducing expenditure bias towards urban areas. (2) Strengthen Participatory Decision Making Allow communities to take Better targeting of government spending. ownership in the design of their Aiding poorer communities, and empowering them to take a greater control of their own development. programs. Strengthen oversight of expenditure management. Community Second Rural And Main CESI: Latin America Social Empower-ment CESI and Governance (FY03, Roads Project (Fy03-Active, Development Capacity Building and Social $46.7) 19) (Worldwide, FY03, 14) Inclusion Strengthening the Ombudsman Office for Indigenous Women within the Guatemala Government (FY03-Active) Universalization Of Basic Education Project (Fy01- Active) Community Management of the Bio-Itza Reserve (Peten) (GEF-MSP) (FY03-Active) Land Administration Project (FY99-Active) Land Fund Project (FY99- Active) (Annex 6B continues on next page.) 56 (Annex 6B continued.) WB Products* WBI Activities Non-lending Products (9 Activities in Guatemala Total: Activities with Guatemalan CAS Priorities Lending Products (24 Products @ 15 Activities, 397 participants outside Guatemala PRSP National Priorities (1-4) Products @ $760.5 million) $731,000) Participants) (Total: 42 Activities, 379 Participants) Rural and Main Roads Project (FY98-Active) Economic Growth with Equit (3) Improving Economic Management to Maintain Macroeconomic Stability and Foster Economic Growth -Maintain macroeconomic - Maintain a stable macroeconomic environment. stability - Create favorable economic conditions in order to attract foreign and local investment. -Reform and strengthen the - Design targeted programs for increasing participation of the rural poor in economic growth. financial sector. Financial Sector Financial Sector Adjustment Financial Sector Risk Management Workshop for Policy Outreach Loan (FY02-Active, $150.0) Assessment Regulators (USA, FY02, 2) (FY02) Financial Sector Technical Post Financial Regional Workshop on Nonbank Assistance Loan (FY02- Sector Financial Institutions Development in Present. $5.0) Assessment Latin America (Chile, FY03, 5) (FY02) Country Economic Memorandum (FY03, $126,000) Investment Private Sector Competitiveness Project Curso en Economia de la Regulacion Climate Development (FY01-Active, $20.3) de Servicios Publicos y Defensa de Assessment la Competencia (Agentina, FY02, 1) (FY03, $34,000) Curso de Gestion Urbana para Centroamerica (El Salvador, FY02, 8) Training Seminar on the Role of Associations in the Promotion of the use of Technology in Training for Local Governments in Latin America (USA, FY02, 3) Metropolitan Governance Study (USA, FY02, 25) (Annex 6B continues on next page.) 57 (Annex 6B continued.) WB Products* WBI Activities Non-lending Products (9 Activities in Guatemala Total: Activities with Guatemalan CAS Priorities Lending Products (24 Products @ 15 Activities, 397 participants outside Guatemala PRSP National Priorities (1-4) Products @ $760.5 million) $731,000) Participants) (Total: 42 Activities, 379 Participants) Global E-Conference on CSR, Education and Technology: How Can Corporations Contribute to Bridging Educational and Digital Divides? (Worldwide, FY03, 1) Anti-Corruption and Accountability in Government (USA, FY03, 2) Regional Small and Medium size Enterprises Training (USA, FY03, 4) Introduction to Corporate Social Responsibility - 11 (Worldwide, FY03, 2) Public Private Private Participation in Curso de Economia y Partnerships in Infrastructure Technical Regulacion de Servicios Infra-structure Assistance (FY97-Closed Publicos y Defensa de la FY04, $13.0) Competencia (FY02, 21) FTAA and Regional WTO Course for Trade Policy Trade Integration in Central America Officials (Switzerland, FY02, 1) (FY02, 11) Economic Growth with Equity Reform and strengthen the financial sector. Public Sector Modernization (4) Improve Governance and Public Sector Management -Combat corruption and improve -Reforms to combat corruption and misuse of public funds. transparency -Reforms in the areas of the financial sector, tax administration, public sector procurement, and the judicial sector. -Modernize institutions in the public sector by integrating financial management systems and strengthening tax collection mechanisms. Judicial Reform Project Governance and A -C ICAC Global Initiatives For Governance (FY99-Active, $33.0) Diagnostics, Guatemala Controlling Corruption (Korea, FY03, (FY02, 29) 1) (Annex 6B continues on next page.) 58 (Annex 6B continued.) WB Products* WBI Activities Non-lending Products (9 Activities in Guatemala Total: Activities with Guatemalan CAS Priorities Lending Products (24 Products @ 15 Activities, 397 participants outside Guatemala PRSP National Priorities (1-4) Products @ $760.5 million) $731,000) Participants) (Total: 42 Activities, 379 Participants) Judicial Reform: Improving Judicial Reform: Improving Performance and Performance and Accountability in Accountability in Latin Latin America (Spain, FY03, 4) America (FY02, 35) Governance and A -C Latin American Economics Diagnostics, Guatemala Journalism: "Los Temas Sociales en (FY03, 5) Tiempos de Crisis" (Colombia, FY03, 19) Latin America Open and Parliaments, Governance and Transparent Municipal Anticorruption in LAC (Brazil, FY03, Governance Program (FY03, 3) 66) Integrated Financial Fiscal Policy for Poverty An Evaluation and Planning Public Sector Management Iii - Technical Alleviation: Guatemala (FY02, Workshop - Action Learning (Spain, Assistance Project (Fy02- 31) FY02, 2) Active, $15.7) Tax Administration Technical Intergovernmental Fiscal Municipal Finance Distance Learning Assistance Loan Project Relations and Local Financial Program for Latin America (Mexico, (FY98-Active, $28.2) Management in Latin America FY02, 5) (FY03, 21) Latin America: Fiscal Policy for Poverty Alleviation Conference (USA, FY03, 4) Public Finance, Decentralization, and Local Financial Management (Mexico, FY03, 150) Public Finance, Decentralization, and Local Financial Management (Chile, FY03, 6) *There were only 19 active lending products during FY02-03, however five of them could not be disentangled between the CAS priorities of Reducing Poverty and Decentralization and Participation. The costs of those activities were split between the two priority areas, and that is why there appears to be more lending products in the chart above. 59 ANNEX 7: VARIABLES DESCRIPTION Dependent Variables Items Scoring Effectiveness Raising awareness 1=not effective Providing with knowledge or skills 7=extreme effective Helping understand role as an agent of development Helping develop strategies address the organizational needs Helping develop strategies address the country needs Helping develop contacts and develop partnerships Utilization and Conducting research 1=not effective Impact/Change Teaching 7=extreme effective Raising public awareness in development issues Implementing new practices within work organization Organizing collective initiatives Influencing legislation and regulation Implementing country development strategies Exogenous/Independent Variables Item Scoring Level 1 measures Relevance of this activity to current work or functions 1= not effective Acquired information that is new 5=very effective Usefulness of the information acquired Focus of this activity on what you specifically needed Content of this activity matched the announced objectives Overall usefulness of this activity Relevance Relevanceto work 1=not relevant Relevance to country needs 7=extreme relevant External environment At working 1=very hurt At country 7=very helpful Event/course features Mode of instruction Dummy: DL, F2F, Mix, Conference, Web, Study tour 1=yes, 0=else Develop action plan Dummy 1=yes, 0=else Duration Continuous >1 Venue Guatemala 1=yes, 0=else Demographics Type of organization Dummy: University, NGOs, Media, Private, National government, 1=yes, 0=else Regional government, Local government Type of job Dummy: Research, Teaching, Policymaking/legislation, Provision 1=yes, 0=else of s ervices, Management/administration Position 1=entry level 5=highest Language proficiency 1= not proficient 7=very proficient Gender 1=female, 0=male 60