31294 The Impact of WBI Activities, FY01-02, on Participants from Brazil: A Baseline Assessment William Eckert Graça Sousa Victoria Gunnarsson WBI Evaluation Studies EG04-82 The World Bank Institute The World Bank Washington, D.C. January, 2004 i ACKNOWLEDGMENT This evaluation report was prepared for WBI under the overall guidance of Marlaine E. Lockheed, Manager, World Bank Institute Evaluation Group (IEG). It was prepared by William Eckert, Graça Sousa and Victoria Gunnarsson. The report benefited greatly from discussions with Tim Campbell and Cristina Ling Chard, who served as peer reviewers. Tanya Loftus and Humberto S. Diaz contributed to document production. The report was discussed by the WBI Regional Capacity Ehancement Team (RCET) at a meeting chaired by Philip E. Karp on January 13, 2004. WBI Evaluation Studies are produced by the WBI Evaluation Group (IEG) to report evaluation results for staff, client, and joint learning events. An objective of the studies is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. WBI Evaluation Studies are available on line at: http://www.worldbank.org/wbi/evaluation/puball.htm. Vice President, World Bank Institute Ms. Frannie Léautier Manager, Institute Evaluation Group Ms. Marlaine E. Lockheed Task Team Leader Mr. William Eckert ii ACRONYMS AFR Africa Region BN Bank of Northeast CAS Country Assistance Strategy CMU Country Management Unit CPB Country Program Brief CRS Client Registration System EAP East Asia and the Pacific ECA Eastern and Central Asia ECLAC Economic Commission for Latin America and the Caribbean FY01 Fiscal Year 2001 FY02 Fiscal Year 2002 IETS Institute for Studies on Labor and Society IEG World Bank Institute Evaluation Group IPEA Institute of Applied Economic Research IPIECA Industrial Petroleum Industry Environmental Conservation Association LACEA Latin American and the Caribbean Economic Association LCR Latin American and the Caribbean Region M&E Monitoring and Evaluation MDGs Millennium Development Goals MNA Middle East and North Africa NGOs Non-Governmental Organizations NIP Network for Inequality and Poverty Research OAS Organization of American States OECD Organization for Economic Cooperation and Development OLS Ordinary Least Squares RCET Regional Capacity Enhancement Team SAR South Asia Region WB World Bank WBI World Bank Institute 2SLS Two-Stage Least Squares iii iv TABLE OF CONTENTS ACKNOWLEDGEMENT................................................................................................ ii TABLE OF CONTENTS ................................................................................................. v EXECUTIVE SUMMARY............................................................................................ vii 1. INTRODUCTION..................................................................................................... 1 Background ................................................................................................................... 1 Evaluation Framework: Objectives and Questions ....................................................... 2 Hypothesis for Exogenous Factors ............................................................................... 5 2. METHODOLOGY ................................................................................................... 7 Participant Population and Survey Sample ................................................................... 7 Analysis....................................................................................................................... 11 Study Limitations ........................................................................................................ 11 3. MAJOR FINDINGS ............................................................................................... 13 Relevance of WBI Learning Activities....................................................................... 13 Sustainability of WBI Learning Activities ................................................................. 15 Effectiveness of WBI Learning Activities.................................................................. 16 Impact of WBI Learning Activities ............................................................................ 20 Facilitators and Barriers in the Utilization of Learning.............................................. 35 Alignment of WBI Activities with the Bank Groups' Country Assistance Strategy (CAS) for Brazil ........................................................................................... 35 4. RESULTS AND CONCLUSIONS ........................................................................ 39 REFERENCES................................................................................................................ 41 ANNEXES ANNEX 1 Quantitative Analysis................................................................................... 45 ANNEX 2 ...................................................................................................................... 53 A: WBI Participant Questionnaire .............................................................................. 53 v B: IETS Presentation................................................................................................... 62 C: IETS Report............................................................................................................ 63 D: Interview And Focus Group Questions For Country Focused Impact Evaluation............................................................................................................. 66 E: Focus Group Results - Rio De Janeiro ................................................................... 67 F: Focus Group Results-Fortaleza .............................................................................. 71 G: CAS Summary Of Development Priorities, Wbi Activities And Brazilian Participants Distribution. ...................................................................................... 75 vi EXECUTIVE SUMMARY The World Bank Institute (WBI) has reorganized its strategy for organizing and delivering its products to one which emphasizes impact at the country level. Twelve countries were selected as focus countries to begin implementing this new strategy, which included Brazil. In order to begin evaluating this new strategy in Brazil and the other focus countries, the WBI Evaluation Group (IEG) designed an approach that includes an assessment of previous training events for Brazilians in order to establish a baseline for a later, prospective evaluation. This study attempts to answer questions on the nature and causes of WBI's training relevance, effectiveness, utilization and influence for Brazilian participants during the fiscal years FY01 ­ FY02. Data to answer these questions were collected from a survey of 300 participants and in-depth, focus group interviews held in Rio de Janeiro and Fortaleza. The adjusted response rate for the survey was 52 percent. Descriptive statistics and a set of two-stage regression models were used to analyze the data and answer the research questions. Data from the focus groups was content analyzed and used in tandem with the survey data to help answer the research questions. Following is a summary of the findings: (a) Responding to questions about the relevance of the training for Brazilians, a majority of participants (55 percent) reported that the activity they attended was designed specifically for Brazil, while an even greater number (74 percent) said the activity was "Extremely relevant" to the specific needs of Brazil; (b) Course participants rated the effectiveness of their WBI training high. The main determinants of these ratings appear to be factors related to how closely the training focused on Brazil's needs and conditions; (c) The high rating of effectiveness held across groups, although some group differences were found. Women and government workers tend to rate the training as more effective than their counterparts; (d) A larger number of participants who were asked to rate training utilization believed that one or more of the eight dimensions of utilization was not applicable to their situation. This result may have confounded the overall rating of course utilization, which was lower that the other dimensions, and the model, which found only a weak relationship with the factors identified as potential determinants; (e) Influence of the training was rated highly by participants. Again, women and government workers were more positive about the course's influence. Perceived vii effectiveness of the course and country level facilitators were the main determinants of influence; (f) Most organizational and country level factors presented to the participants were seen as facilitators rather than barriers to incorporating their new knowledge and skills into their workplace; (g) Generally, the WBI courses offered during the study period were aligned with CAS priority areas. However, the categories used in the CAS were very broad and may have affected this conclusion. This report concludes that WBI may be in a relatively strong position to have a significant impact in Brazil through its new country focused strategy. Results show a generally high level of satisfaction with past training events among Brazilian participants across the domains of relevance, effectiveness, utilization and influence. Further, we see that important features for enhancing effectiveness are directly related to how "Brazil- focused" the event was perceived by participants but that other design features such as action planning and numbers of people from the same unit trained do not affect effectiveness or impact. The general environment for accepting change in Brazil also emerges as a key determinant of overall influence of these past training events. These suggest that a future effort, which designs training events more focused on country needs, will likely succeed in increasing its impact. viii 1. INTRODUCTION BACKGROUND 1.1 The World Bank Institute (WBI) supports the World Bank's overall agenda by providing learning programs and other related services to assist and enhance development. These activities are focused in the strategic development areas of environment and sustainable development, poverty reduction & economic management, finance and private sector development and human development. The magnitude of this effort is substantial. In the past year alone, 2002, WBI delivered more than 500 activities to clients in more than 150 countries. 1.2 WBI's primary strategy for organizing and delivering its products has been around key thematic areas such as health and population, education, governance and macroeconomic policy. Within these and other key infra-structural areas, learning-based services have been delivered within regions and countries, but not with a direct focus on country level development. This strategy changed in 2001. In order to achieve greater impact at the country level, WBI redirected its activities to be more closely integrated with other Bank-wide operations in order to increase capacity within individual countries. While the thematic sectors remain important, greater emphasis is now given to country needs and an integrated program of learning-based services. WBI will now align its programs more directly with country needs and broader country assistance strategies, in order to reduce poverty and progress more directly towards the Millennium Development Goals (MDGs). 1.3 As part of this new approach, WBI created a regional capacity enhancement pillar, the Regional Capacity Enhancement Team (RCET), to work with country teams and units of the WBI thematic pillars, in priority countries, to develop more country focused enhancement programs. WBI identified 45 countries overall, 12 of which were selected as focus countries for more intensive, multi-sectoral capacity enhancement activities over the next three years. These countries include all six World Bank Regions and were selected because of their priority needs for learning and knowledge programs. Table 1 shows the targeted countries by Region and their income status. Included in this initial group of 12 is Brazil. 1 Table 1: Targeted WBI Priority Countries Lower-middle Region Low income income LCR Guatemala Brazil EAP Laos Thailand AFR Nigeria Burkina Faso MNA Yemen Egypt ECA Tajikistan (representing 4 other central Asian countries) Russia SAR Afghanistan Sri Lanka 1.4 The greatest problem facing Brazil is the existence of inequalities in wealth, and subsequent disparities in health and welfare. While classified as one of the "low to middle income countries," Brazil is one of the world's more inequitable societies, with the wealthiest one-percent of the population receiving the same share of total income as the poorest fifty-percent. This disparity is even more concentrated regionally, within the country. The Northeast region consists of nine federal states: Alagoas, Bahia, Ceará, Maranhão, Paraíba, Pernambuco, Piauí, Rio Grande do Norte and Sergipe. Within this region, the poverty rate is nearly double the country's average. While rapid economic development has occurred in other regions of the country, enhancing their institutional and human capital infrastructure, the situation and prospects in the Northeast are less certain. Even though some progress has been made in recent years in some sectors and in some states, the area, as a whole, remains a dual economy, with the greatest poverty concentrated in the rural area. 1.5 The WBI Brazil country team is working to develop a comprehensive strategy for the overall country, which acknowledges the special conditions in the Northeast. As it develops the final Country Brief, the team notes that the program will be more fully aligned with that of the operational CMU's and the new CAS, also under development. This new country-focused direction of WBI poses a challenge for the World Bank Institute Evaluation Group (IEG). IEG has responded to this need by developing a framework for the evaluation of these country-focused activities (Quizon, 2002). This approach calls for a comprehensive evaluation of the sectoral-based activities within each country while also assessing their systematic effects at the country level. Part of this approach will be the establishment of an "impact" baseline, and its updating over time, by which these activities can be both monitored and evaluated. This current study is an attempt to establish such a baseline for Brazil. EVALUATION FRAMEWORK: OBJECTIVES AND QUESTIONS 1.6 The overall objectives of this study are to: (a) assess the effects of prior training activities delivered to Brazilian participants; and, (b) establish a baseline for utilization in a subsequent prospective evaluation of the WBI country-focused program. To meet these objectives, this evaluation study attempted to answer the following questions: 2 (a) What was the perceived relevance of WBI learning activities among Brazilian participants? (b) What was the perceived effectiveness of WBI learning activities among Brazilian participants and what factors are related to perceived effectiveness? (c) What was the perceived impact of WBI learning activities among Brazilian participants on: · Utilization and what factors are related to perceived impact on utilization? · Influence and what factors are related to perceived impact on influence? (d) What were the perceived barriers to and facilitators of the impact of WBI learning activities among Brazilian participants? (e) How aligned were WBI learning activities with the Brazil Country Assistance Strategy (CAS) for the study period? 1.7 Specifics of how WBI's training program impacted Brazilian participants are summarized in Figure 1. This figure represents a conceptual model of training effects that shows the major components of the process and their interrelationships. According to this model, training impact is composed of two primary domains, perceptions of utilization and influence. Each domain is represented by a number of more specific dimensions, which both define and help measure the overall quality of that domain. For example, utilization consists of eight distinct dimensions. 1.8 Perceived effectiveness of the training has a direct effect on these impact domains.1 This, too, is represented by a number of specific dimensions. Also part of the model is a set of exogenous factors that include gender, government employment, type of work, course characteristics and barriers to and facilitators of utilization of course materials/skills. The model specifies that these factors may influence impact, either directly or indirectly, through their direct effect upon the perceived effectiveness of the courses. Effectiveness, too, is composed of several features, including raising awareness, updating or refining skills and providing new knowledge or skills. A final element of the model is the presence and function of perceived course relevance. Relevance is expected to have a direct influence upon effectiveness, but exercises no direct effects upon the two impact components, utilization and influence. Relevance consists of participants' views on how well the training events were designed especially for the participants and an overall rating of training relevance to meet Brazil's needs. 1In theory, influence is largely determined by the frequency of utilization of knowledge and skills acquired in the course. Hence, we expected much of the effect of perceived effectiveness and the exogenous variables to be affecting influence indirectly through utilization. However, a test of this assumption using an Ordinary Least Square (OLS) model showed that effectiveness has an independent effect upon influence, independent of utilization. Because of this result, we constructed the model in Figure 1 to reflect the condition that both utilization and influence are independent dimensions of impact. 3 Figure 1: Model of the Effects of WBI Learning Activities in Brazil EXOGENOUS FACTORS · Gender Demographics · Government · Position · Research/teaching Course Characteristics · Length · Action plan · Facilitators organization Work Environment · Facilitators country · Colleagues attending EFFECTIVENESS · Raising awareness/understanding · Updating/refining knowledge and skills RELEVANCE (Instruments) · Providing new knowledge and skills · Designed for Brazilians · Helping in networking · Addressing Brazil specific needs · Providing strategies/approaches to Brazil's needs · Providing strategies/approaches to participants' org. needs UTILIZATION INFLUENCE · Research · Research publications · Develop strategies · Legislation · Implement strategies · Teaching material · Change legislation IMPACT · Consensus building · Teaching · Initiatives · Raising awareness · Work practices · Community initiatives · New practices 4 HYPOTHESIS FOR EXOGENOUS FACTORS 1.9 We hypothesize that women perceive effects of training more positively than men, as shown in previous evaluation studies (Khattri et al., 2002)2. Hence, we assume that women will rate effectiveness, utilization and influence higher than men. Moreover, government participants may find more use for courses in policy related issues than people in non-policy related field. We therefore expect government participants to rate the training more positively than non-government participants. 1.10 Secondly, we expect longer training activities to increase the ratings of the activity in all fields because the participant is exposed to the course material longer. Also, creating an action plan during the training should increase the usage of the new knowledge and skills because it lays out the particular areas where the new material could be used in the participant's work. Furthermore, facilitators are factors that help implement new ideas into the participant's work and should aid the usage and influence of the acquired skill. Also, attending courses together with colleagues is assumed to increase the usage and influence of the course material because it affects a larger part of the work group and creates a broader understanding of the new ideas within the participant's unit. 1.11 Finally, training that is perceived as targeted towards the participants and specifically towards Brazil's needs are assumed to be more effective, to stimulate greater use of the acquired knowledge and to influence the participant's work more positively than training that is designed in a general manner. 1.12 In addition, we expect that people in different position levels (senior management versus junior professionals for instance) may have different degrees of experience and may therefore be able to take in more or less of the training material depending on the appropriateness of the difficulty level of the specific course. We have no predetermined hypothesis on the direction of the effect however. Likewise, we assume that researchers and teachers may be affected differently from people who are not from university or research fields. Again, we do not have any present hypothesis of the direction of the effect of being a teacher or researcher on the ratings of the training activities. 1.13 In order to answer some of the key questions about relevance, effectiveness and impact, a statistical model was constructed from survey data collected from Brazilian participants in WBI courses during FY01-02. The following section describes the complete methodology used in this evaluation study. 2In this study, bivariate comparisons of participants' ratings of effectiveness and impact revealed that women rate some dimensions of effectiveness and use higher than men. 5 6 2. METHODOLOGY 2.1 This study used three sources of information to answer the evaluation questions presented in the previous section. One was a survey of Brazilian participants in WBI- sponsored courses during the period FY01-02. The second was a set of in-depth focus group interviews with selected course participants in Rio de Janeiro and Fortaleza. The third, was a review of Brazil Country Assistance Strategy (CAS) document. Quantitative and qualitative methods were used to analyze these data. Both the collection and analytical methods are presented below. PARTICIPANT POPULATION ANDSURVEY SAMPLE Participant Survey 2.2 The survey was carried out by the Institute for Studies on Labor and Society (IETS ­ Instituto de Estudos do Trabalho e Sociedade), located in Rio de Janeiro. A list of all Brazilian participants in WBI training activities for the period FY01-02 was obtained from the World Bank's Client Registration System (CRS), yielding a population of 300 eligible subjects from 31 activities. It was decided that all 300 would be included in the survey. IETS was instructed to attempt to administer the survey primarily by e- mail. When e-mail delivery and/or administration was not possible, the survey was to be conducted by telephone. The instrument for the survey was developed by the IEG evaluation team (see Annex 2A) and translated into Portuguese under their direction. A Portuguese language version of the instrument was made available to IETS for administration by e-mail using a web-based ColdFusion questionnaire with an access database. This system allowed responses to be sent directly to IEG. Telephone interviews by IETS staff also used the same system. 2.3 The survey was conducted during the period May 5 to July 4, 2003. From the initial list of 300, two individuals were listed twice because of being enrolled in two separate WBI learning events during the study period, reducing the number to 298. Out of this group, 67 could not be reached because of incomplete or invalid addresses, or other reasons. (For example, one of the respondents was deceased). The final, accessible population was 231 participants. A total of 119 completed questionnaires was received for an adjusted response rate of 52 percent (51.5). A description of IETS and their full report on the survey execution and results are provided in Annex 2B and 2C. Respondent Population 2.4 In addition to the 119 respondents' rating of the WBI activity we also extracted information on various characteristics of the respondent population (demographics), course and work environment. These characteristics are listed in Figure 1 as Exogenous Factors. Figure 2 displays the distribution of the respondents by gender. Out of the 113 7 people that participated in training and answered the question on gender 66 (58 percent) were men and 47 (42 percent) were women. When analyzing the total population3, we actually find a quite similar gender distribution, with 62 percent of the participants being male and 38 percent female. Figure 2: Distribution of Respondents by Gender (N=113) Women 42% Men 58% 2.5 In addition, 89 (79 percent) people out of 113 respondents to this question came from the federal, regional or local government while eight people (7 percent) represented universities or research institutions and seven people (6 percent) were employed by private businesses. There were no participants from donor agencies. The distribution of respondents by type of organization is displayed in Figure 3. If we consider the total population there are no significant differences. Out of the 298 participants, 232 (78 percent) were government officials, seven (2 percent) represented ministers and eighteen (6 percent) belong to the private sector. Academics, NGOs and Research Institutes were represented by 26 people (8 percent) and only one person represented bilateral aid donors. The rest of the participants belong to international organizations or other. Figure 3: Distribution of Respondents by Type of Organization (N=113) 7% 1% Gov. Federal 6% Gov. Regional 31% 7% Gov. Local University/Research Private NGO 27% 21% Other 3 When analyzing gender we consider a total population of 298 participants, counting once the two participants that had two registers for attending different activities. 8 2.6 Figure 4 shows the frequency distribution of the 112 people who answered the question about primary type or work. Almost half of the sample (49 percent) were management or administrative staff, eleven people (10 percent) were researchers, ten people (9 percent) were primarily teaching and eight people (7 percent) reported that they mainly provided services. Only a few participants combined research and teaching. Figure 4: Distribution of Respondents by Type of Work (N=112) 13% Mgmt./administration Research 4% 7% Teaching 49% Policy making/ 8% Legislation Provide services 9% Research/Teaching 10% Other 2.7 The age of the respondents ranged between 24 and 71 years old with a mean of 41 years. Forty-one percent of the 114 respondents were between 30 and 39 years old and 30 percent between 40 and 49 years old. Furthermore, the proficiency of the language and terminology used in the training was high. Seventy-nine percent reported that they were very knowledgeable or highly proficient in the language of instruction while less than three percent reported that they were not proficient at all. Eighty-three percent of the respondents reported that they were very knowledgeable or highly proficient in the terminology of instruction while as little as one percent believed that they were not at all proficient in the terminology used during the activity attended. Delivered Programs 2.8 According to CRS data, during the period FY01-02, 31 different WBI training activities were delivered in Brazil, under 15 different programs. Table 2 shows the 15 programs, the number of activities within the programs and the number of Brazilian participants per program, for the total and the respondent population. 9 Table 2: Programs Delivered in Brazil, Number of Activities and Participants Number of Number of Participants Percentage of Participants PROGRAM Activities with from Brazil from Brazil (FY01-02) Participants per Program per Program from Brazil Total Respondent Total Respondent Population Population Population Population Clean Air Initiative 1 31 15 10.33 12.61 Decentralization and Governance: Intergovernmental Relations & Local 1 44 24 14.67 20.17 Financial Management Education Program 3 42 32 14.00 26.89 Education Reform Core Learning Program 1 1 1 0.33 0.84 Finance and Banking 4 108 21 36.00 17.65 Flagship Program on Health Sector Reform and Sustainable Financing 2 5 2 1.67 1.68 Infrastructure: Finance and Regulation 1 2 0 0.67 0.00 Knowledge for Development 1 7 3 2.33 2.52 Macroeconomics and International Finance 1 8 3 2.67 2.52 Market Solutions for Development 2 3 0 1.00 0.00 Public Finance Decentralization and 1 9 5 3.00 4.20 Local Financial Management Rural Development and Natural Resource Management 1 2 2 0.67 1.68 Social Protection Core Program 2 5 3 1.67 2.52 Sustainable Development: Environmental Management 5 18 3 6.00 2.52 Sustainable Development: Natural 5 15 5 5.00 4.20 Resources Management Total 31 300 119 100 100 2.9 Of the WBI programs delivered in Brazil during this period, Finance and Banking enrolled the largest number of participants, with 36 percent of participants attending activities under this program. The course "Domestic Debt Market Workshop" was attended by 102 Brazilian participants. The activities under the Decentralization and Governance program and the Education Program had around 14 percent of the total number of Brazilian participants each, followed by the Clean Air Initiative, with ten percent. All the other courses had less than 20 participants from Brazil. 2.10 Participants in two programs did not respond to the survey: Infrastructure: Finance and Regulation, and Market Solutions for Development. These two programs, however, included only five participants from Brazil, making the issue of bias due to under-representation moot. A more serious issue of being over-represented can be found with the Education Program. Almost 27 percent (26.89 percent) of respondents attended these activities, compared to only 14 percent in the general population. There was a similar pattern for the Decentralization and Governance program. On the other hand, Finance and Banking were under-represented among respondents. Except for the above mentioned three programs, the total population was overall well represented by the respondents in the distribution across programs. 10 Focus Group Interviews 2.11 Two in-depth focus group interview sessions were held with Brazilian participants. Sites selected for these interviews were Rio de Janeiro and Fortaleza. Rio de Janeiro was selected because the greatest number of participants resided in this area; Fortaleza because it was important to include a site in the Northeast Region. Each focus group interview followed a standard protocol (see Annex 2D). The focus groups were led by IEG staff, with assistance from IETS in Rio de Janeiro. Each session lasted three hours. A total of ten participants took part in the focus group interviews, five in Rio de Janeiro and five in Fortaleza. Brazil CAS Review 2.12 A review of the CAS for Brazil, covering the period 2000-2002, was conducted in order to assess the alignment of WBI learning activities with the objectives of World Bank assistance as stated in that document. WBI courses offered in Brazil in FY01-02 were organized and matched against the stated priority areas presented in the CAS. Results from this matching exercise are discussed in section 3 and presented in Annex 2G. ANALYSIS 2.13 Data collected from the participant survey, gathered using an electronic questionnaire, were exported into SPSS for Windows for analysis. After the data were cleaned and coded, two basic types of analysis were conducted: (a) Descriptive statistics in the form of frequencies and percentages were calculated and used to summarize results in the key areas of relevance, effectiveness, utilization and influence. These same indicators were used to describe the responding population; (b) A set of statistical models were constructed using a two-stage least squares method (2SLS). These models were used to answer questions about factors which influence outcomes in the key areas of inquiry, effectiveness, utilization and influence. A full description of these models and the method of construction is found in Annex 1A. 2.14 Data from the focus group interviews were content analyzed. This analysis used key categories from the interview protocol as a guide to the analysis. Key points of the respondents were summarized for integration with the survey data results. A summary of these results by key category and participants list is reported in Annexes 2E and 2F. STUDY LIMITATIONS 2.15 Several features of this study place limitations on its findings and must be considered when interpreting the final results. First, participant self-reported impression of training may not be the most objective or valid indicator of that training's results. This 11 may be even more immediate given the time lag between the training and this survey.4 But the indicators used have been used in previous studies (Khattri et al., 2002) and are consistent with the overall logic of the expected training effects. Minimally, this information provides valuable insight into participants' perceptions of the extent to which the training was effective. 2.16 A second issue is the response rate. Fifty-two percent (52 percent) of those participants with valid and active addresses completed the survey, from the adjusted population of 231. While this is a reasonable response rate, it still leaves a sizeable number of respondents unaccounted for. This may be the basis for bias in the final results. Few pre-survey indicators were available to assess the differences between the adjusted population and respondents. Two of those that were available, gender and type of organization, showed little difference between these groups. However, there was a difference in the courses represented by the two groups (see section 2.10). It also should be noted that nearly two-thirds of the respondents came from specific sectors, particularly finance, education and intergovernmental relations. This may also be a source of bias in the participants' responses. Care should be exercised when attempting to generalize to the full population of participants from these data given these limitations. 2.17 Next, in the assessments of the various training events dimensions and effects, it should be recognized that these events were not designed to be "country-focused" in the same way as those in the future strategy. While some of these events may have been Brazil "specific," they were more likely organized around and anchored in the earlier WBI strategy which emphasized sectoral themes over country-focused effects. This "program" dimension to the training was not considered or used when questioning participants or course deliverers/planners about the effectiveness of the training. This feature should be considered when interpreting the effects of the training assessed in this report. 2.18 Moreover, training events are assessed individually and not as parts of larger programs. When taking individual events out of their context much of the impact and effectiveness may be reduced since the true impacts of the series of activities may only come about from the program as a whole including all parts of the intervention. 2.19 Finally, the reliability of the list of WBI training participants for the study period should be considered. These data, as noted, were taken from the CRS data base list of participants. This list represents those participants actually entered by Task Managers or other course personnel. Its completeness is questionable. While the CRS lists 1,826 Brazilian participants for the two years FY01 and FY02, anecdotal information from interviews with Task Managers and others working in Brazil suggests that this number may be significantly higher. In this case, the survey data would be even less representative of the total population of participants. 4Most of the Focus Groups participants, both in Rio and Fortaleza, argued that the activity had been held a long time ago (in some cases more than two years) and therefore it was difficult for them to specify and give details on the attended activity. 12 3. MAJOR FINDINGS 3.1 The main findings of this study are outlined in this section. We discuss relevance, effectiveness, the two impact measures, facilitators and barriers and alignment with the Brazil CAS. RELEVANCEOF WBI LEARNING ACTIVITIES 3.2 Survey respondents were asked to rate the relevance of the course in three different areas: (a) Whether or not the activity attended was designed specifically for participants from Brazil; (b) To what degree the topics covered in the activity were specific to the needs of Brazil; and (c) If the activity was related to the WB corporate priorities and to the ten Millennium Development Goals (MDGs), and, if so, the extent to which the activity addressed these goals. Perceived relevance for Brazil and WB corporate priorities and MDGs was high. 3.3 More than half of respondents reported that the activity they attended was customized for participants from Brazil (Figure 5). However, because the activities were actually not country focused in design, the level of perceived customized design can be interpreted as relatively high. There is no direct information on whether participants perceived that attending a course they believed was designed specifically for participants from Brazil increased their level of satisfaction with that course. In section 3.16 we employ regression analysis to evaluate the impact of perceived relevance of specific design for Brazilian participants and level of relevance of the activity to Brazil's specific needs on the effectiveness of the WBI activity. Figure 5: Was the Activity You Attended Designed Specifically for Participants from Your Country? (n=115) No 45% Yes 55% 13 3.4 Focus Groups results confirm that participants recognize that the course content was not designed specifically for Brazil. Especially, one of the courses included an African audience as well, which made participants complain about the irrelevance of some presentations. 3.5 In contrast to the question on customized design, respondents rated the level of relevance of the activity to the specific needs of Brazil very highly. The mean rating of the relevance to Brazil's needs was 6.25 on a scale from 1-7 where 1 is "Extremely irrelevant" and 7 is "Extremely relevant". The distribution of relevance to Brazil's need is presented in Figure 6. Eighty-seven of the 118 respondents (74 percent) rated the WBI activity's relevance to Brazil's needs as extremely relevant while only ten respondents (9 percent) believed that the relevance to the specific needs of Brazil was average or below. Again, none of the WBI activities were country focused. The level of perceived relevance of the activity to the specific needs of Brazil was therefore very high. Figure 6: Distribution of Relevance to Brazil's Needs by Level (n=118) 80 74 70 60 50 40 30 respondents 20 18 %10 1 2 6 0 Extremely 3 4 5 Extremely irrelevant relevant (1-2) (6-7) 3.6 Focus Groups findings support the high ratings for relevance to Brazil's needs in the survey. When asked if the attended course benefited their country, most of the Focus Groups participants could not mention concrete direct benefits, but rated the activity program as highly relevant in that it addressed specific issues of concern to Brazil, such as decentralization and governance, or controlling air pollution. The third relevance measure estimated the relevance of the WBI activity to the WB corporate priorities such as improved trade and investment climate and to the ten World Bank MDGs. Table 3 shows the respondents' rating of the degree to which the activity they attended was related to each of these priorities and goals. 14 Table 3: Respondents Ratings of courses with respect to the WB corporate priorities and to the Millennium Development Goals (MDGs) Percentage of WB corporate priorities and MDGs respondents who believed the activity Mean ratinga was related (n=118) Eradicate extreme poverty 26 4.44 (n=59) Achieve universal primary education 37 5.28 (n=68) Promote gender equality and empower women 8 4.24 (n=41) Reduce child mortality 19 4.61 (n=51) Improve maternal health 12 4.15 (n=46) Combat HIV/AIDS, malaria, and other diseases 7 4.18 (n=40) Ensure environmental sustainability 26 5.12 (n=57) Develop global partnerships for development 42 5.42 (n=74) Ensure water sanitation and supplyb 11 4.32 (n=44) Improve investment climate and financeb 43 5.32 (n=77) Promote tradeb 18 4.26 (n=50) aNot all respondents rated the relevanceof the activities to the MDGs. Thus, the mean ratings in column 3 are based on a lower number of respondents. bEnsuring water sanitation and supply, improving investment climate and finance and trade promotion are not MDGs, however, they are WB corporate priorities. 3.7 The question relating to the WB corporate priorities and MDGs was divided into two parts. In part one respondents were asked to mark each priority and goal to which the activity was related. One-hundred and eighteen people answered this question. The highest rated priorities and goals were "Achieve universal primary education" (37 percent of respondents believed the activity was related), "Develop global partnerships for development" (42 percent) and "Improve investment climate and finance" (43 percent). These were also the priorities and goals for which respondents reported the highest degree to which the activity was related to the specific goal (Table 3, column 3). Not all respondents answered this question. Thus the mean ratings in column 3 are based on a lower number of respondents. These ratings accurately reflect the topics of the activities offered and the number of participants per program reported in Table 2. The topics of the activities that had the largest number of participants (and respondent participants) correspond to the WB corporate priorities and MDGs to which respondents related the course content. Almost 28 percent of the respondents attended training on education, 20 percent attended training on decentralization and governance and 17 percent attended training on finance and banking. SUSTAINABILITYOF WBI LEARNING ACTIVITIES 3.8 The questionnaire respondents were also asked whether or not they were aware of any follow-up activities to the WBI learning events. Follow-up activities were defined such as for example workshops, meetings or E-mail discussion groups. Of the 112 respondents to this question only 19 (17 percent ) were aware of such activities taking place immediately following the activity. Moreover, of these, only 46 people answered the following question of whether or not they participated in the follow-up activities, and, only 7 people (15 percent of the 46 respondents) said that they did participate. We can conclude that the level of sustainability of learning events for Brazilians was very low. However, based on the small number of respondents, we cannot draw any conclusions of 15 whether or not follow-up activities increase perceived effectiveness, utilization or influence of the course material. EFFECTIVENESS OFWBI LEARNING ACTIVITIES 3.9 The effectiveness of the WBI learning activities as defined in the questionnaire are measured by six different dimensions: (a) Raising participants' awareness/understanding of development issues; (b) Updating/refining participants' knowledge and skills; (c) Providing participants with new knowledge/skills; (d) Helping in networking among people with similar professional interests, and; (e) Providing strategies/approaches to address the development needs of Brazil; (f) Providing strategies/approaches to address the needs of participants' organizations. 3.10 Participants could choose to rate effectiveness of the activity on a 7-point scale with 7 being "Extremely effective" and 1 being "Not effective at all".5 Overall perceived effectiveness of the WBI learning activities was high. 3.11 Figure 7 shows the distribution of the overall perceived effectiveness ratings in Brazil. On the average, participants rated the course as effective with a mean rating of 5.69. Out of the 117 respondents, 66 (56 percent) participants rated the activity as extremely/very effective, 30 (26 percent) rated the activity as effective and no one rated the training activity as ineffective. The total percentage with ratings in the effective (above average) range was an impressive 82 percent. Figure 7: Distribution of Overall Effectiveness by Level 60 56 50 40 26 30 respondents 20 % 11 10 7 0 0 Ineffective 3 4 5 Extremely (1-2) effective (6-7) 5 Factor analysis showed that all six dimensions of effectiveness could be treated as one aggregate effectiveness variable created by computing the average of the six dimensions. 16 3.12 Analysis of the distribution of rating across the six effectiveness dimensions shows that WBI activities are the most effective when it comes to understanding country specific development issues, refining and providing knowledge and skills and stimulating interest in subject of the activity. Of the 118 respondents 79-81 (69-71 percent) rated these as extremely/very effective while only 1-3 people rated these dimensions as ineffective. The dimensions of the activity that were rated the least effective were providing country and organization specific development strategies with only 53-57 percent of the respondents rating these as extremely/very effective. In addition, five and nine respondents respectively thought that the WBI activity was ineffective in these two dimensions. Figure 8 depicts the distribution of effectiveness by level and by the six effectiveness dimensions. Figure 8: Distribution of Overall Effectiveness by Dimension and Level 80 69 69 68 71 70 1 or 2 57 60 53 6 or 7 50 40 respondents 30 %20 8 10 3 1 0 2 4 0 Raising Refining Providing Networking Country Org. awareness skills new skills strategies strategies 3.13 In fact, participants from both Focus Groups stressed the usefulness of the course in raising awareness of the presented issues while providing participants with new techniques and management strategies. One of the direct effects mentioned several times was the increasing interest in searching information for further activities on the same themes. Very small demographic differences in perceived effectiveness. 3.14 Figures 9 and 10 show the distribution of effectiveness by gender and by type of organization. Men tended to rate the overall effectiveness of the training lower than women, consistent with our initial hypothesis. Sixty-eight percent (68 percent) of women (32 out of the 47 female respondents) compared to only 46 percent of the men (30 out of the 65 male respondents) thought that the course was extremely/very effective. Furthermore, participants from the central, regional or local government rated the course effectiveness higher than non-government employees.6 Of the 88 representatives from the government, 54 (61 percent) thought that the course was overall extremely/very effective while only 12 (41 percent) of the 29 non-government participants agreed. On the contrary, only seven percent of the government participants compared to 24 percent of 6Non-government workers came from universities/research institutions, the private sector, NGOs or other organizations. 17 non-government participants rated the overall effectiveness as average. Seven percent of the participants from both groups rated the course as somewhat ineffective. Figure 9: Distribution of Overall Effectiveness by Gender and Level 80 70 68 Men 60 Women 50 46 40 29 respondents 30 17 21 %20 8 10 6 0 0 4 0 Ineffective 3 4 5 Extremely (1-2) effective (6-7) Figure 10: Distribution of Overall Effectiveness by Organizatrion and Level 70 61 Government 60 50 Other 41 40 28 30 24 25 respondents 20 % 7 7 10 0 0 0 0 Ineffective 3 4 5 Extremely (1-2) effective (6-7) Note: "Other" indicates all other non-governmental organizations, see Figure 3. 3.15 Figure 11 displays the frequency distribution of perceived overall effectiveness of the WBI activity by work organization and level of rating. Almost no one rated the activity as ineffective. Almost 70 percent of both activity groups thought that the course was extremely/very effective. For the middle category, six out of 24 (25 percent) researchers and teachers rated the course as average while six out of 93 (6 percent) representatives from other areas than research/teaching believed that the course effectiveness was average. Sixty-four percent of the non-researchers/teachers were involved in management and administrative work, hence, the distribution for management and administrative staff is most likely similar to that of people who were not primarily conducting research and/or teaching. For the respondents who thought the course was effective, only two (8 percent) were researchers/teachers while 22 (24 percent) had other work responsibilities (i.e. mainly management or administration). 18 Figure 11: Distribution of Overall Effectiveness by Work Type and Level 80 68 70 67 Research/ Teaching 60 50 Other 40 25 respondents 30 22 24 %20 6 8 10 0 0 0 0 Ineffective 3 4 5 Extremely (1-2) effective (6-7) Perceived relevance to Brazil is the most important determinant of effectiveness. 3.16 In order to determine what areas affect perceived training effectiveness, regression analysis was employed. Methods and results of the regression analysis are discussed in depth in Annex 1A. A variable description is also presented in Annex 1A, Table A1. Table 4 presents the results from the ordinary least squares (OLS) regression to predict overall effectiveness using a group of demographics, course characteristics and work environment characteristics. 3.17 The most important determinants of overall effectiveness were how specifically the course was designed for (a) participants from Brazil, and (b) for Brazil's special development needs. Designing the course specifically for participants from Brazil (ß = 0.666, Std. Err. = 0.244) and relating the material to address the specific needs of Brazil (ß = 0.250, Std. Err. = 0.107) significantly increased course effectiveness consistent with our initial hypothesis. In other words, compared to participants who did not perceive the activity attended to be designed uniquely for Brazilian participants, believing that the activity especially targets Brazilians raised perceived effectiveness by a third of a unit (say from a rating of 5 to a rating of 5.7). This corresponds to an increase in relevance of around 12 percent at the mean. Increasing the relevance to Brazil's needs by one unit increased effectiveness by one-quarter unit or four percent such that, in order to move from, say, the midpoint of the scale, 4, to a 5, relevance has to increase by four units. Other explanatory variables exhibit varying signs but lack significance. Beta coefficients were calculated in order to compare the effects of explanatory variables with different ranges. Of these two significant explanatory variables, the beta coefficients suggest that course customization and designed for Brazilians had a relatively greater impact on effectiveness within the model. The explanatory power of this model was impressive, with the two significant variables explaining nearly 60 percent (57.6 percent) of the variance in effectiveness. In contrast to our initial hypotheses, government employees did not rate course effectiveness significantly higher than non-government employees. Furthermore, training length, developing an action plan and working in a an environment with helpful facilitators did not raise perceived effectiveness. 19 Table 4: OLS Regression Results of Overall effectiveness Variable Coefficient Std. Err. t P>|t| Beta Coeff.a Demographics Female 0.057 0.235 0.24 0.808 0.024 Government 0.465 0.288 1.61 0.111 0.156 Position 0.008 0.091 0.09 0.926 0.008 Research/teaching -0.124 0.255 -0.48 0.630 -0.043 Course Characteristics Length 0.019 0.016 1.22 0.225 0.113 Design Brazilians 0.666** 0.244 2.73 0.008 0.279 Relevance Brazil 0.250** 0.107 2.34 0.022 0.211 Action plan 0.379 0.254 1.49 0.140 0.158 Work Environment Facilitators organization 0.160 0.094 1.71 0.092 0.209 Facilitators country 0.005 0.090 0.05 0.960 0.006 Colleagues attending -0.057 0.092 -0.62 0.538 -0.054 Constant 2.159** 0.746 2.89 0.005 R2 0.576 N 82 aThe beta coefficients indicate the number of standard deviations the effectiveness will change due to a one standard deviation increase in the respective explanatory variable. ** indicates significance at the 5 percent significance level. IMPACT OF WBI LEARNING ACTIVITIES 3.17 The impact of WBI activities are divided into two domains: (a) Utilization of the new knowledge and skills gained as a result of the learning activity; and (b) The nature of influence and changes brought about by the activity. 3.18 Survey respondents were asked to think about the WBI activity they attended and to evaluate how often they make use of the knowledge and skills they acquired in eight areas. The areas which utilization measures are the following: (a) Conducting research; (b) Developing country development strategies; (c) Implementing country development strategies; (d) Changing or influencing legislation or regulations; (e) Teaching; (f) Raising others' awareness in development issues; (g) Organizing collective community initiatives, and; (h) Implementing new practices within a participant's organization. 20 Overall utilization was rated as relatively modest. 3.18 Participants could choose to rate utilization of the activity on a 7-point scale with 7 being "Use very often" and 1 being use "Use not at all". On the average participants rated utilization low at 4.84. 3.19 Figure 12 shows the distribution of overall7 utilization ratings in Brazil. Only 43 out of 109 respondents (39 percent) rated overall utilization as "Use very often" while as many as 30 people (28 percent) rated overall utilization as low or "Use not at all". Figure 12: Distribution of Utilization by Level 40 39 30 22 20 16 12 respondents 11 %10 0 Use not at 3 4 5 Use very all (1-2) often (6-7) 3.20 Additionally, Figure 13 shows reported use of knowledge and skills by dimension and level. Again, scores are much lower than for effectiveness. Raising others awareness in development issues and implementing new practices within the work organization were the two areas where respondents most frequently made use of what they learned during the activity. Sixty-two and 54 percent of the respondents reported that they very often made use of the new material in the two areas respectively. On the other hand, less than 40 percent of the respondents made often use of the new material when conducting research, developing and implementing development strategies, changing legislation or regulations or when teaching. Figure 13: Distribution of Utilization by Dimension and Level 70 62 1 or 2 60 54 6 or 7 50 43 39 41 40 38 36 36 34 27 30 26 24 26 respondents 19 20 % 9 13 10 0 Research Develop Implement Change Teaching Raising Comm. New strategies strategies legislation awareness initiatives practices 7Due to very few overlapping complete observations on the various utilization measures, factor analysis could not provide any guidance on how to group the utilization measures in sub-groups. Instead we analyze the sum of all eight measures of utilization and on all eight sub-questions separately. Annex 1A discusses how the overall measure was constructed. 21 3.21 Curiously, all application examples given in both focus groups were on the lower rated dimensions, like developing and implementing strategies or changing legislation. For instance, in the Northeast there was no tradition of regulating voluntary transfers from the state to municipalities. A former participant of the fiscal relations course contributed to the implementation of a rule to normalize these transfers, which has already being applied. 3.22 Similar to the effectiveness ratings, women made more frequent use of the activity material than men. The distribution of overall utilization of the knowledge and skills learned in the course by gender and levels is shown in Figure 14. Twenty-three of the 45 female respondents8 (51 percent) rated overall utilization "Use very often" while only 19 out of the 61 male respondents (31 percent) made very often use of the new material. Twenty- one men (34 percent) compared to 8 women (17 percent) reported that they used the new material rarely or not at all. On the other hand, 18 percent of men (11 respondents) versus only four percent of women (2 respondents) did not use the course material at all. Figure 14: Distribution of Overall Utilization by Gender and Level 60 51 Men 50 Women 40 31 30 25 18 20 respondents 20 16 13 10 11 %10 4 0 Use not at 3 4 5 Use very all (1-2) often (6-7) 3.23 On the organizational level, government representatives made much more use of the material taught in the course than non-government workers (Figure 15) consistent with our initial assumptions. Forty-three of the 84 government representative respondents (51 percent) compared to only eight of the 25 non-government participants (32 percent) reported that they used the course material very often. Fourteen percent of the government workers used the material taught in the activity rarely or not at all. In contrast, as many as 40 percent of the non-government participants reported low use or "Use not at all". 8 When discussing the two impact measures the term "respondents" applies to all those who answered the various questions on utilization and influence but only to those who did not answer "Not applicable". The issue of applicability is discussed in section 3.24 and 3.37. The individuals who chose to answer "Not applicable" to any question are thus not included in our measure of total number of respondents for that question. 22 Figure 15: Distribution of Utilization by Work Organization and Level 60 51 50 Government 40 36 Other 32 30 24 24 16 16 respondents 20 10 12 12 %10 0 Use not at 3 4 5 Use very all (1-2) often (6-7) 3.24 There is also a distinction in how individuals, who mainly conduct research and/or teaching made use of the material taught in the activities, compared with participants from other fields, as shown in Figure 16. Eleven of the 24 researcher/teacher respondents (46 percent) reported that they use the knowledge and skills obtained by the course sometime or very often. In contrast, 63 of the 85 participants (75 percent) with professions other than research and teaching (mainly management and administration) reported that they made some or very often use of the material. Only 19 percent of the respondents with other work concentrations than research and teaching reported that they made average or rare use of the material from the course while as many as 50 percent of the researchers/teachers used the material only rarely or at average. Figure 16: Distribution of Utilization by Work Type and Level 60 51 Research/ 50 42 Teaching 40 Other 33 29 30 24 21 respondents 20 % 8 11 13 10 7 0 Use not at 3 4 5 Use very all (1-2) often (6-7) 3.25 Furthermore, respondents were asked to indicate whether the various dimensions of the two impact measures were applicable to them in their work. All those who answered 1-7 on the utilization dimensions are said to define the specific dimension as applicable to their work. In addition, a large number of individuals indicated that one or more dimensions of utilization were not applicable to them. Table 6 shows that for all dimensions except raising others' awareness in development questions and implementing new work practices within the organization, around 33-38 percent of the total number of survey respondents (118) indicated that the dimension was not applicable to them. Eleven percent of the total number of survey respondents indicated that raising others' awareness in development questions was not applicable to them. Eighteen percent reported that implementing new work practices in the organization was not applicable to them. 23 Table 6: Not Applicable Utilization Dimensions, Number and Percentage of Respondents (n=118) Utilization Dimensions Not applicable (n) Not applicable (%) Research 45 38 Develop strategies 41 35 Implement strategies 41 35 Change legislation 42 36 Teaching 44 37 Raising awareness 13 11 Comm. initiatives 39 33 New practices 21 18 Perceived effectiveness and position are the most important determinants of utilization. 3.26 We employed a regression analysis to explain what determines the dimensions of impact of the skills learned in the training. In the analysis we assume that the same factors explaining the variation in effectiveness also explain the variation in impact, but that effectiveness is a determinant of utilization as well. The determinants of utilization from the two-stage least squares, 2SLS9 regression analysis are summarized in Table 7.10 Table 7: 2SLS Regression Results of Overall Utilizationa Variable Coefficient Std. Err. t P>|t| Beta Coeff.b Overall effectiveness c 5.940* 3.519 1.69 0.096 0.513 Demographics Female -1.357 3.152 -0.43 0.668 -0.049 Government 0.277 4.363 0.06 0.950 0.008 Position 2.417* 1.288 1.88 0.065 0.193 Research/teaching 1.147 3.427 0.33 0.739 0.035 Course Characteristics Length -0.039 0.218 -0.18 0.860 -0.020 Action plan 1.001 4.120 0.24 0.809 0.036 Work Environment Facilitators organization 2.301 1.529 1.50 0.137 0.254 Facilitators country 0.157 1.135 0.14 0.890 0.018 Colleagues attending -1.121 1.237 -0.91 0.368 -0.090 Constant -22.422* 12.599 -1.78 0.080 R2 0.464 N 78 aThe first stage is the OLS regression on overall effectiveness presented in Table 4. Overallutilization is measured on a scale from 4 to 56, as a sum of all applicable areas of utilization. b The beta coefficients indicate the number of standard deviations the utilization will change due to a one standard deviation increase in the respective explanatory variable. cPredicted by the OLS regression in Table 4. * Indicates significance at the 10 percent significance level. 9Two-stage least squares (2SLS) estimator is obtained by least squares regression of Z on ? and X. The name stems from the two regressions in the procedure: Stage 1, Obtain the ordinary least squares prediction, ?, from regressing Y on X and Q where Q are variables that affect Y but that do not have any direct impact on Z. Stage 2, Regress Z on ? and X. 10 Because our data did not include any variables that we are certain influence effectiveness but neither of the impact measures, we rely on design of the activity especially for Brazilians and to address Brazil's needs as instruments of effectiveness based on statistical significance in the effectiveness regression. 24 3.27 When viewing the results of the regression analysis, the overall weakness of the model and its explanatory variables should be considered. Position and Overall effectiveness were weakly associated with utilization, showing significance only at the 0.05 level, the generally accepted social science standard. This may reflect the relatively large number of respondents who responded that the various aspects of utilization were "Not applicable," as reported in Table 6. The results indicate that the variation in overall utilization can mainly (46.4 percent) be explained by perceived effectiveness and the position level of the individual. Overall effectiveness increases utilization of the knowledge and skills obtained in the training (ß = 5.940, Std. Err. = 3.519). The higher the position the higher the perceived utilization of the WBI activity (ß = 2.417, Std. Err. = 1.288). When interpreting the size of the coefficients one has to keep in mind that the overall utilization is the sum of all applicable utilization measures for each individual (see Annex 1A for an extensive discussion on how and why such a measure was developed). Hence, a one unit increase in perceived overall effectiveness means an almost six unit increase in the sum of overall utilization which ranges from 4-56. This corresponds to an increase in utilization of about 21 percent at the mean. Moving up one position (say from middle management to senior management) increases overall utilization of the new knowledge and skills by only nine percent at the mean. No other explanatory variables have any significant impacts on overall utilization of the training material. Analysis of the beta coefficients further confirms the findings. The most important variables in explaining utilization are perceived effectiveness, position and organizational facilitators although organizational facilitators is not statistically significant in explaining the variation in utilization. As perceived effectiveness increases by one standard deviation, utilization increases by 0.513 standard deviations. People who perceive effectiveness one standard deviation higher than the norm rate utilization 14 percent higher, other things equal. 3.28 In contrast to our initial hypotheses, women did not rate utilization significantly higher than did men and government employees did not rate utilization significantly higher than did non-government employees. Furthermore, training length, developing an action plan and working in a an environment with helpful facilitators did not raise usage of knowledge and skills acquired in the training activity. 3.29 The effect of the main determinants of the eight 2SLS regressions on the separate dimensions of utilization is summarized in Table 8. Positive signs indicate positive and significant effects, negative signs indicate negative and significant effect, blank spaces indicate insignificant effects. Due to the small sample size for the analyses for the separate sub-questions of utilization the number of explanatory variables was reduced. The most important determinants of utilization are perceived overall effectiveness, position and the number of colleagues that attended the training activity. High overall perceived effectiveness increased utilization in most areas and the higher the position the individual occupied the higher the utilization of the training material in implementing country development strategies, changing legislation and regulations and organizing collective community initiatives. Surprisingly, the greater the number of colleagues attending the same activity the lower the utilization in research, teaching and raising others' awareness in development issues. However, the coefficients on colleagues attending are only significant at the ten percent level. Moreover, women made more use 25 (than did men) of the knowledge and skills learned from the activity in organizing collective community initiatives. Individuals from the government are better at using the new knowledge in implementing development strategies and changing legislation and regulations. Country level facilitators are also important positive contributors to making use of the training material in changing legislation and regulations. 3.30 Focus group participants mentioned the Bank of Northeast (BN) and the Brazilian Universities as main WBI activity facilitators, whose roles were crucial namely in disseminating information and material from the course. Suggestions were made to keep and increase partnerships with these and other facilitators in Brazil, in order to reach broader audiences. 26 Table 8: Direction of Effects of Explanatory Variables from 2SLS on the Different Dimensions of Utilizationa Explanatory Overall Impact variables effective- Govern- Research/ Facilita-tors Facilita-tors Collea-gues variables ness Female ment Position teaching Length Action plan org. country attending R2 N Overall utilization + + 0.464 78 Research + Excludedb Excluded Excluded Excluded - 0.314 60 Develop strategies Excluded Excluded Excluded Excluded 0.451 64 Implement strategies ++ ++ Excluded Excluded Excluded Excluded 0.401 64 Change legislation + + ++ Excluded Excluded Excluded Excluded ++ 0.479 64 Teaching Excluded Excluded Excluded Excluded - 0.233 60 Raising awareness ++ Excluded Excluded Excluded Excluded - 0.418 84 Community initiatives + + Excluded Excluded Excluded Excluded 0.449 65 New practices ++ Excluded Excluded Excluded Excluded 0.441 79 a++ indicates positive and significant effects at the 5 percent significance level, + indicates positive and significant effects at the 10 percent significance level, - - indicates negative and significant effects at the 5 percent significance level,- indicates negative and significant effects at the 10 percent significance level, blank spaces indicate insignificant effects at the 10 percent significance level. bDue to the small sample size for the analyses of the sub-questions, the number of explanatory variables are reduced. 27 3.31 One explanation of the negative effect (although only weakly significant) on research and teaching in Table 8 could be that these activities are most often performed by individuals rather than in group form. Hence, attending activities with a large number of colleagues may distract the individual's research and/or teaching methods and actually reduce usage in these areas. When considering the negative effect of the variable "Number of colleagues attending" for utilization, Focus Groups results do not support these findings. This counter-intuitive result could potentially be caused by the small sample size and can most likely be ignored. In fact, one of the main barriers for the utilization of the acquired knowledge noted were the difficulties of coordination that result from the low number of people within the same organization attending the course. 3.32 Influence and change due to the WBI learning activities is measured in the following six areas: (a) Research methodology or publications; (b) Legislation or regulations; (c) Teaching materials for the course; (d) Consensus building for change; (e) Community-based initiatives, and; (f) Work practices in participants' organization. 3.33 Participants could choose to rate influence of the activity on a 7-point scale with 1 being "Negative influence" and 7 being "Positive influence". The average utilization rating was relatively high, 5.65. Influence of WBI learning activities was high. 3.34 Figure 17 shows the distribution of reported overall influence from the WBI activity. An area is said to have been positively influenced by the course material if the average influence rating of that area was 5 or above. Compared to the reported utilization of course material the influence of the activity was high. Sixty-one of the 101 respondents (60 percent) reported that the activity had positive influence while only eight people (8 percent) thought that the activity had negative or somewhat negative influence on the average of the six dimensions. On average the training had somewhat positive or positive influence on every dimension. Among the dimensions, the activity had the highest influence on changing work practices in the organization with a mean rating of 5.808 and the lowest influence in research methodology or publications with a mean rating of 5.305. Seventy-one percent reported that the training had positive influence on work practices in their organization while only three percent believed that the training negatively influenced their work practices. Fifty-six percent of the respondents valued the Influence of the WBI training as a positive influence on research methodology or publications while almost seven percent thought that the training had a negative influence 28 on research or publications. Figure 18 shows the distribution of influence by areas of influence and levels. Figure 17: Distribution of Influence by Level 70 60 60 50 40 30 respondents 20 20 % 12 10 6 2 0 Negative 3 4 5 Positive influence influence (1-2) (6-7) 3.35 Interestingly, participants perceived influence as high even though they reported that they only made modest use of the knowledge and skills acquired in the training. This may seem like contradicting results but may very well be the case if we consider the demographic characteristics of the respondents. Seventy-nine percent of all respondents came from different areas of government and nearly half of the respondents were associated with administration and management while only eight percent were involved in policy making and legislation. Given this distribution, it is possible that only a few people are in a position to influence work practices, especially related to changing legislation or regulations, but that the usage of the knowledge and skills in such areas may be viewed and documented as perceived influence by a larger number of participants. Moreover, one has to be careful when interpreting the participants perception of influence. Influence at the institutional level may only be truthfully reflected by the organizations impacted and not by the individual participants. Figure 18: Distribution of Influence by Dimension and Level 80 71 1 or 2 70 65 58 61 61 6 or 7 60 56 50 40 respondents 30 % 20 7 10 3 5 5 5 3 0 Research Legislation Teaching Consensus Initiatives Work publications material building practices 29 3.36 Figures 19-21 show the distribution of perceived influence by gender, work organization, type of primary work responsibility and level. Women rated the influence slightly higher than men. Thirty-five of the 43 (81 percent) female respondents believed that the influence from the WBI activity was positive while only 33 of the 57 male respondents (58 percent) judged the influence as positive. On the lowest levels, low and "Negative influence" men were slightly higher represented than women. Figure 19: Distribution of Influence by Gender and Levels 90 81 80 Men 70 Women 60 58 50 40 respondents 30 21 % 20 12 12 10 4 5 0 2 5 0 Negative 3 4 5 Positive influence influence (1-2) (6-7) 3.37 Similar to the overall utilization of knowledge and skills, government workers rate the activity influence higher than non-government (mainly management and administrative) respondents. Fifty-eight of the 85 government respondents (73 percent) believed that the overall influence from the activity attended was positive. On the other hand, 11 of the 21 non-government respondents agreed (Figure 20). Moreover, slightly more non-government respondents reported that they believed that the training activity had a negative or somewhat negative overall influence. The influence rating was relatively even across work types as shown in Figure 21. Figure 20: Distribution of Influence by Work Organization and Level 80 73 70 Government 60 52 Other 50 40 respondents 30 24 18 % 20 14 10 5 1 4 5 5 0 Negative 3 4 5 Positive influence influence (1-2) (6-7) 30 Figure 21: Distribution of Influence by Type or Work and Level 80 75 67 Research/ 70 Teaching 60 50 Other 40 30 respondents 15 17 %20 9 10 5 0 2 0 1 0 Negative 3 4 5 Positive influence influence (1-2) (6-7) 3.38 Respondents were also asked to indicate whether the various dimensions of influence were applicable to them in their work. All those who answered 1-7 on the dimensions are said to define the specific dimension as applicable to their work. In addition, a large number of individuals indicated that one or more dimensions of influence were not applicable to them. Table 9 indicated that influence on research methodology and publications and influence on community-based initiatives had the highest non-applicability response of around 48-50 percent. Influence on consensus building for change had the smallest number of total survey respondents, 30 out of 118, who said that these dimensions were not applicable to them. Table 9: Not Applicable Influence Dimensions, Number and Percentage of Respondents (n=118) Influence Dimensions Not applicable (n) Not applicable (%) Research publications 59 50 Legislation 47 40 Teaching material 39 33 Consensus building 30 25 Initiatives 57 48 Work practices 40 34 Perceived effectiveness and country level facilitators are the most important determinants of influence. 3.39 We use regression analysis to evaluate the most important determinants of influence and to understand what aspects are important for increasing the WBI's influence on the participants' work. The results of the 2SLS regression is presented in Table 10. In the analysis we assume that the same factors which explain the variation in effectiveness also explain the variation in impact, but also that effectiveness is a determinant of impact. The first stage of the analysis is the OLS regression in Table 4 and we use the same instruments for effectiveness as we used in the utilization analysis. 31 Table 10: 2SLS Regression Results of Overall Influencea Variable Coefficient Std. Err. t P>|t| Beta Coeff.b Overall effectivenessc 5.083** 1.562 3.25 0.002 0.887 Demographics Female -0.029 1.582 -0.02 0.985 -0.002 Government 0.211 2.236 0.09 0.925 0.012 Position 0.065 0.617 0.11 0.916 0.011 Research/teaching 1.002 1.687 0.59 0.555 0.060 Course Characteristics Length -0.044 0.104 -0.42 0.672 -0.046 Action plan -1.168 1.811 -0.64 0.521 -0.085 Work Environment Facilitators organization 0.024 0.749 0.03 0.975 0.005 Facilitators country 1.200** 0.565 2.12 0.038 0.256 Colleagues attending -0.311 0.613 -0.51 0.614 -0.066 Constant -27.430** 6.245 -4.39 0.000 R2 0.476 N 77 a The first stage is the OLS regression on overall effectiveness presented in Table 4. Overall influence is measured on a scale from 17 to 18, as a sum of all applicable areas of influence after recoding "Negative influence" as ­3 and "Positive influence" as +3. bThe beta coefficients indicate the number of standard deviations the influence will change due to a one standard deviation increase in the respective explanatory variable. cPredicted by the OLS regression in Table 4. ** indicates significance at the 5 percent significance level. 3.40 The only significant determinants of overall influence from this analysis are course effectiveness (ß = 5.083, Std. Err. = 1.562) and country level facilitators (ß = 1.200, Std. Err. = 0.565). Both of these variables had positive and significant impacts on the influence of the training as hypothesized earlier. Similar to the analysis of utilization, overall influence is measured as the sum of rating of all applicable areas of influence. influence thus ranges form ­17 to 18.11 Hence, increasing overall effectiveness by one unit increases overall influence by five units or 21 percent at the mean. Having one unit more helpful country level facilitators (say moving from the midpoint of the scale, 4, to a 5) increased overall influence at the mean by only five percent, a very small effect. There is no significant difference in the level of influence between male and female participants. Similarly, according to the beta coefficients effectiveness and country level facilitators are the most important determinants of influence. As perceived effectiveness increases by one standard deviation perceived influence increases by 0.887 standard deviations. Individuals who perceive effectiveness one standard deviation higher than the norm rated influence 17 percent higher, other things equal. 3.41 Again, differently from our initial hypotheses, women did not rate perceived influence significantly higher than did men and government employees did not rate influence significantly higher than did non-government employees. Moreover, training length and developing an action plan did not raise perceived influence. 11Before summing influence rating were converted to a scale from ­3 to +3. See Annex 1A for a more extensive discussion. 32 3.42 The main determinants, as predicted by the 2SLS regressions, of the six dimensions of perceived influence are presented in Table 11. The higher the perceived effectiveness of the activity the more positive the learning activity influence except when it comes to influencing legislation or regulations. In addition, country level facilitators increased perceived influence in areas such as research methodology and publications, legislation or regulations and teaching materials. However, the coefficients on country level facilitators are only significant at the ten percent level. 33 Table 11: Direction of Effects of Explanatory Variables from 2SLS on the Different Dimensions of Influencea Explanatory Impact variables Overall variables effective- Govern- Research/ Facilita-tors Facilita-tors Collea-gues ness Female ment Position teaching Length Action plan org. country attending R2 N Overall influence ++ ++ 0.476 77 Research publications ++ - Excludedb Excluded Excluded Excluded + 0.261 49 Legislation Excluded Excluded Excluded Excluded + 0.296 61 Teaching material ++ Excluded Excluded Excluded Excluded + 0.468 64 Consensus building ++ Excluded Excluded Excluded Excluded 0.280 74 Initiatives ++ Excluded Excluded Excluded Excluded 0.567 49 Work practices ++ Excluded Excluded Excluded Excluded 0.306 64 a ++ indicates positive and significant effects at the 5 percent significance level, + indicates posit ive and significant effects at the 10 percent significance level, - - indicates negative and significant effects at the 5 percent significance level, - indicates negative and significant effects at the 10 percent significance level, blank spaces indicat e insignificant effects at the 10 percent significance level. bDue to the small sample size for the analyses of the sub-questions, the number of explanatory variables are reduced. 34 FACILITATORS AND BARRIERS IN THE UTILIZATION OF LEARNING Environmental facilitators help the implementation of knowledge and skills acquired from the WBI learning activities. 3.43 Figure 22 shows that two organizational factors and one country factor are the most important facilitators of implementing new ideas. A factor is considered a facilitator if participants rated the level of helpfulness of that factor as 5 or higher out of 7. Among the organizational level facilitators, helpfulness of organizational practices and procedures and helpfulness of colleagues are the strongest facilitators with mean ratings of 5.37 and 5.14. Resources and funding are considered as the least helpful factors with a neutral mean rating of 4.51, neither helpful nor unhelpful. Among the country level facilitators, country policies are the most helpful in implementing new ideas with a mean rating of 5.72 while political groups are the least helpful country level facilitators with a mean rating of 5.22. At the average, country level facilitators were rated as more helpful than organizational level factors. The average country level facilitator rating was helpful (5.43) while organizational facilitators were rated as neither helpful not unhelpful (4.88). If a factor's mean rating is 3 or below it is considered a barrier to implementation of new ideas. None of the factors were rated as barriers. Figure 22: Mean ratings of the Most Important Facilitators/Barriers of Implementing Ideas 7 5.72 6 5.37 5.14 5.22 5 4.51 rating 4 3 Mean2 1 Org. Org. Collegues Country Political practices resources policies groups and and funding procedures ALIGNMENTOF WBI ACTIVITIES WITH THE BANK GROUPS' COUNTRY ASSISTANCE STRATEGY (CAS) FOR BRAZIL WBI learning activities for FY01-02 appear to be aligned with the CAS for Brazil. 3.44 The CAS for Brazil covers the period 2000-2002. Its primary objective for Bank assistance is poverty reduction. The five priority areas identified in the Bank's strategy for Brazil are: (a) Poverty and inequality, with a need to focus on policies to increase income earning opportunities, through the enhancement of human, and physical resources of the poor, improving education, health, land reform, and urban, and rural 35 services and to provide the social safety nets needed for protection against economic shocks. (b) Fiscal adjustment and administrative reform, at all levels of Government, targeting social security and tax reforms. (c) Renewed growth of per capita income, through declining interest rates, and continued financial reforms. Private sector development improvement is also addressed, through appropriate regulatory framework. (d) Effectiveness of public institutions, particularly to strengthen sub-national institutional capacity, and continuation of the judiciary reform. (e) Environmental management, targeting protection of natural resources and ecosystems, ensuring compatibility and controlling pollution especially in urban areas. 3.45 The CAS development priorities are represented in detail in Annex 2G, with a priority rating of high (H), moderate (M) or low (L). To assess the alignment of WBI activities with the CAS, the WBI courses offered in Brazil in FY01-02 were organized under the different priority areas. Using this very general classification for the development priorities, it was possible to place all of the delivered courses under these priority areas. Some of the courses overlap and can be considered under more than one priority area. Because this classification of courses are quite broad, the results, while exhaustive, may not be sufficiently rigorous in order to judge how well the WBI courses aligned with the CAS priorities. Care should be used when interpreting these alignment results. 3.46 From the match of WBI courses with the CAS development priority areas, we can conclude the following: (a) From the total of 31 WBI courses delivered, the greatest single number was in the Environmentally & Socially Sustainable Development area (12 courses; 39 percent). Eleven of the 12 courses delivered in this area were on environmental issues. These courses accounted for 22 percent of the total number of participants, 66 out of a total of 300. (b) When considering the number of Brazilian participants, the greatest concentration was in the priority area of Finance. This was mainly due to the "Developing Domestic Debt Markets Workshop," which had over 100 Brazilian participants attending, accounting for more than one third of the total number. Another course, "Intergovernmental Fiscal Relations and Local Financial Management Course for Brazil and Lusophone Africa," also had relatively a high number of Brazilian participants (44; 15 percent). Because of these two courses, the more general priority area of Finance, Private Sector and Infrastructure, although having only 19 percent of the total number of deliveries, accounted for more than half of the total number of WBI Brazilian participants. 36 (c) There were four courses under the Education priority area, all on Education Reform. One of these courses, "Strategic Choices for Education Reform ­ Brazil", accounted for 12 percent (37) of the total number of participants. (d) No specific WBI courses on Social Development (high priority) nor Gender issues or Energy and Mining (moderate priorities) were delivered during this period. 3.47 As reported above, WBI's strategy for selecting and delivering its courses was based on thematic program areas and was not country focused in the same way as the new WBI strategy. However, when viewing the general development priority areas identified in the CAS, it appears that there is a general alignment with WBI learning activities. WBI offered courses under almost all of the high and medium priority areas outlined in the CAS. In addition, all WBI courses were covered by the CAS priority areas and only two courses were delivered in areas that were not high priority CAS areas. 3.48 Under the 5 priority areas, the CAS specifically mentions WBI's role in increasing the effectiveness of public institutions. According to the document, WBI should focus on public sector management, including management of smaller municipalities. During FY01-02 only two WBI courses were delivered to Brazilian participants on this issue. The course "Intergovernmental Fiscal Relations and Local Financial Management Course", with 44 Brazilian participants (15 percent of the total number of participants) and "City Strategies to Reduce Urban Poverty through Local Economic Development" with nine participants from Brazil, representing three percent of the total number of learning activities for that period. This one area suggests a non- alignment between the CAS and the WBI learning events. 3.49 In addition to the priority areas and according to the CAS Matrix12, WBI is specifically mentioned to assist in several development strategies: (a) Privatization of rail and ports (regulation WBI Focus); (b) System planning including environment (regulation WBI Focus); (c) Complete privatization and improve regulatory framework and competition (regulation WBI Focus); (d) Strengthening of local governments, particularly at municipal level (WBI decentralized public management, WBI city management); (e) Improve governance transparency and accountability (WBI Focus: transparency & accountability); (f) Modernize environmental instruments and institutions (WBI Focus); (g) Integrate environment concerns in other sectors (WBI Focus); and, 12The CAS matrix is under Annex B9 in the CAS for Brazil that covers the period 2000-2002. 37 (h) Prioritize environmental pollution problems, develop & implement action plans (WBI Clear Air Initiative). 38 4. RESULTS AND CONCLUSIONS 4.1 In sum, respondents rate the effectiveness of the WBI training high. This pattern holds across groups, although there are some differences. For example, women tend to rate the training higher than men, and government workers rate the activity higher than non-government participants. The main determinants of effectiveness appear to be whether or not the training was perceived to be designed especially for Brazilian participants and the perceived degree of relevance of the training to Brazil's unique needs. The higher the relevance to Brazil, the higher the perceived effectiveness. This is an important finding for the current plan of making WBI training even more country focused and relevant. It suggests that this strategy is likely to have the greater impact anticipated. 4.2 The respondents rating of the utilization of knowledge and skills obtained for attending the course were more moderate, although they still are generally positive. Again, there were some differences noted among groups. Women reported making slightly more use of the material than men, and government workers utilized the material more than non-government participants. Utilization is especially low among researchers and teachers. The regression analysis of utilization identifies only weak determinants of utilization, none being significant at the standard five percent level. High perceived effectiveness of the course and occupying a high position are the strongest variables, both having a positive effect upon utilization. These two variables were significant at the ten percent level. 4.3 Influence is rated relatively high. Among the groups observed, more women than men believe that the WBI activities have a positive influence, and government representatives rate influence as more positive than respondents from non-government agencies. The variables that explain most of the variation in perceived influence are perceived effectiveness and country level facilitators. Since the main determinants of effectiveness were the training's relevance to Brazil, this suggests that there may be an indirect influence on influence. 4.4 According to the participants perception, the influence of the learning events is higher than the reported use of the knowledge and skills obtained in the event. This appears inconsistent. However, as WBI has shifted its focus from increasing the level of knowledge and skills of the individual training participant to deepening the impact of the intervention on institutions and organizations, influence of the training event may be more accurately reflected by the involved institutions than in the eyes of the participants. 4.5 The analysis also finds that organizational and country level environmental factors affecting effectiveness, utilization and influence tend to facilitate implementing 39 ideas. Respondents perceive these factors as helpful rather than unhelpful in incorporating their new knowledge and skills into their work practices. 4.6 In general participants in the Focus Groups were satisfied with the learning activities attended, mentioning several gains in terms of increasing awareness, learning new techniques and concepts. One of the most noted useful aspects was the possibility of networking and experience sharing among participants from different areas and organizations. They continuously stressed WBI's role as a disseminator of Bank's knowledge in several areas and countries, and its role as a facilitator to gather people to exchange experiences. Institutional weight and credibility of WBI were considered to be a major tool in order to lead to social mobilization and overcome cultural barriers existing in Brazil. Nevertheless participants stressed the need to create dynamics and synergies with the courses, including more interactive actions between participants and speakers and more pedagogical material to support these actions. 4.7 Overall, these results appear encouraging for the newly enacted country focused strategy. It was noted early in this report that the training events observed were not planned as coordinated and specific to Brazil's in the same way that the new strategy does. Still, the training was viewed as highly positive in all of the areas, with the exception of utilization. It should also be noted that the participants suggest that there is a generally positive climate for making effective use of this training by the absences of major barriers. We also see in the statistical models evidence that "Brazil specific" features have a positive effect upon the outcome, both directly and indirectly. However, it should not be interpreted to mean that the new country-focused training will automatically be more effective. This will depend, of course, on many other factors as the program unfolds. 4.8 While there is a general alignment of WBI training events with CAS priority areas, this may be the result of a broad designation of those areas by the CAS. The new CAS being developed is expected to align effectively with the WBI Country Program Brief (CPB), which presents WBI's plan for Brazil over the next five years. It may be helpful when viewing this alignment to consider the earlier situation and assure that the level of precision and detail exists to provide a meaningful and valid alignment. 40 REFERENCES Khattri, Nidhi, et al., 2002. Impact Evaluation of WBI Client Programs, FY00-01. WBI Evaluation Studies. Washington, D.C.: The World Bank Institute, The World Bank. Quizon, Jaime B., 2002. Measuring, Monitoring and Managing for Development Results. A concept note prepared for the World Bank Institute. Washington, D.C.: The World Bank Institute, The World Bank. 41 42 ANNEXES 43 44 ANNEX 1: QUANTITATIVE ANALYSIS Rationale for Regression Analysis Regression analysis was used to identify what aspects determine the effectiveness of the WBI training and to what extent the effectiveness translates into impacting participants in their daily work and contribution to the development of Brazil. Two separate groups of regression analyses were performed, one to explain perceived course effectiveness and the other to explain what determines the two dimensions of impact of the skills learned in the course. In the analysis, we assume that the same factors which explain the variation in effectiveness also explain the variation in impact, but also that effectiveness is a determinant of impact. Conceptually, course effectiveness and impact (utilization of the new knowledge and skills gained in the course and the nature of the influences and changes brought about by the activity) on the participants' work are discussed in section 3. In the first analysis, we regress perceived effectiveness on participant characteristics, WBI course characteristics and participants' work environment characteristics. This analysis helps us understand what determines the level of effectiveness of the training. The second analysis attempts to determine what factors explain the impact of the course on the participants' work habits and on the influence that the course has had on the development work in Brazil. More specifically, the regression analysis explains the direction and magnitude of the effect of perceived effectiveness on the degree to which the participants employed their new knowledge in their work. This analysis goes beyond the basic descriptive statistics and evaluates, controlling for factors that jointly determine effectiveness and impact, whether the participants perception of effectiveness is a significant determinant of utilization and influence of the knowledge from the course. Effectiveness, Utilization and Influence Except for the measures of course effectiveness and utilization and influence of the new knowledge from the training, the variables used in the analysis are self-explanatory. A variable description and summary statistics is presented in Table A1. For effectiveness, utilization and influence, the survey included a series of questions, presented in section 3. To establish a generic effectiveness measure of the course the responses to the six sub- questions were averaged.13 A simple average is used because of the small variation in the mean and standard deviation of the six sub-questions (see Table A1). 13In addition to the seven answer alternatives to determine the level of effectiveness of the course the respondent could choose "Don't know" which has been recoded as average effectiveness (4). There were no missing observations or "Not applicable" options for the effectiveness questions. 45 For the utilization and influence variables the means and standard deviations were more diverse. Factor analysis14 was applied to the two variable groups to analyze how many joint factors should be used to explain knowledge utilization and knowledge influence. Due to the very diverse components of both variables and to the large number of observations rated as "Not applicable", no clear factors could be defined. 15 Therefore, the analysis was performed on overall measures of utilization and influence, and on each one of their dimensions. Because the influence measure ranges from "Negative influence" to "Positive influence", before summing the applicable sub-questions the answer alternatives were first recoded to range from ­3 indicating negative influence to +3 indicating positive influence with 0 as the average measure. In order to construct two overall measures of utilization and influence the dimensions that were applicable to each individual were summed for each individual. Summing the applicable dimensions (instead of taking an average) accounts for two dimensions of impact; intensity of impact and breath of impact. If an individual rated several of the dimensions as applicable and rates all these at a high level this individual is said to have a higher impact rating than a person that only finds one or a few dimensions applicable. On the other hand, by accounting for the breath, low impact rating on several dimensions is rated similar to high impact in only one or a few dimensions. Grouping and summing variables that do not have high correlations, neither statistically nor theoretically, does not truthfully reflect the overall utilization or influence of the new knowledge since some areas are more general to all participants, (i.e. impact on work practices in the organization) or more specialized than others (i.e. impact on implementing country development strategies). It should be kept in mind when interpreting the results of the analysis on the two overall impact indicators, that they are sums of all the applicable areas of the sub-questions and not weighted by the importance of the various utilization and influence dimensions. Regression Methods and ReSults: Effectiveness The results from the ordinary least squares (OLS) regression analysis of effectiveness are presented in Table 3.1. The main determinants of overall effectiveness seem to be whether or not the activity was designed specifically for Brazilians and level of relevance of the activity to Brazil's specific needs. Uniquely designed activities raise perceived effectiveness of the training (ß = 0.666, Std. Err. = 0.244). Courses that have a high relevance to Brazil's development needs also increase perceived effectiveness (ß = 0.250, Std. Err. = 0.107). The beta coefficients indicate the number of standard deviations effectives will change from a one standard deviation increase in the respective explanatory variable. The larger the beta coefficient (in absolute value) the larger the 14 Factor analysis is a statistical approach to data reduction concerned by finding a small number of common factors that linearly reconstruct the original variables. Factor analysis helps reduce the number of variables in the analysis by describing linear combinations of the variables that contain most of the variation. For this analysis orthogonal varimax rotation is performed. 15 Because less than 10 percent of the observations were missing for both impact variables, missing information and questions answered as "Not applicable" are treated as "Not applicable". We assume that the reason why a person did not answer a question was because the subject matter had no application for that person. There were no "Don't know" options for the impact questions. 46 effect of that coefficient on effectiveness. The variables that most significantly explain the variation in effectiveness are also those with the largest beta coefficients: whether or not participants perceived that the activity was designed specifically for Brazilian participants and the perceived degree of relevance to Brazil's needs. As a rough indication of the magnitude of these effects, note that roughly 68 percent of the distribution of effectiveness will lie within one standard deviation of the mean, so that people who perceive relevance to Brazil's needs one standard deviation higher than the norm rate effectiveness 6 percent higher, other things equal. Regression Methods and Results: Impact A.8 Because we assume that impact can be explained by effectiveness as well as the same exogenous variables that also determine effectiveness (see Table A1), effectiveness is modeled as endogenous in the impact regression. A two-stage least squares regression (2SLS) is used to determine effectiveness and impact jointly (utilization and influence) and allows for possible endogeneity. We relied on the two statistically significant explanatory variables from the effectiveness equation to instrument for perceived effectiveness.16 Selected results of the OLS regression analyses of utilization and influence are presented in Tables 3.4, 3.5, 3.7 and 3.8 and A2-A5 and are discussed in section 3. Analysis of the beta coefficients confirms the regression findings. Furthermore, all results were robust to changes in specification (adding and removing explanatory variables). 16Theoretically we could not find any variable in our data that we believed influenced course effectiveness but did not utilization or influence to serve as suitable instruments for effectiveness. Instead we chose the two statistically significant variables from the effectiveness equation as instruments based on statistical significance in explaining effectiveness. We assume that Design Brazilians and Relevance Brazil effect effectiveness but do not effect impact and are uncorrelated with the error term. 47 Table A1: Variable Description and Summary Statistics Variable Description N Mean Std. Dev. Min Max Dependent Variables Utilization (7=use very often, 1=use not at all) Overall utilization (1-7) Average of all dimensions of utilization of knowledge/skills from course 109 4.841 1.695 1 7 Overall utilization (4-56)* Sum of all dimensions of utilization of knowledge/skills from course 109 28.12 13.57 4 56 Research Conducting research 73 4.479 2.069 1 7 Develop strategies Developing country development strategies 77 4.377 2.165 1 7 Implement strategies Implementing country development strategies 77 4.390 2.135 1 7 Change legislation Changing or influencing legislation or regulations 76 4.474 2.163 1 7 Teaching Teaching 74 4.554 2.312 1 7 Raising awareness Raising others' awareness in development issues 105 5.495 1.727 1 7 Community initiatives Organizing collective community initiatives 79 3.873 2.451 1 7 New practices Implementing new practices within work organization 97 5.206 1.898 1 7 Influence (7=positive influence, 1=negative influence) Overall influence (1-7) Average of all dimensions ofinfluences/changes from course 101 5.650 1.308 1.17 7 Overall influence (-17-18)* Sum of all dimensions of influences/changes from course 101 6.931 6.404 -17 18 Research publications Research methodology or publications 59 5.305 1.744 1 7 Legislation Legislation or regulations 71 5.577 1.461 1 7 Teaching material Teaching materials for courses 79 5.544 1.575 1 7 Consensus building Consensus building for change 88 5.750 1.456 1 7 Initiatives Community-based initiatives 61 5.541 1.566 1 7 Work practices Work practices in your organization 78 5.808 1.469 1 7 Effectiveness (7=very effective, 1=not effective at all) Overall effectiveness Sum of all dimensions of effectiveness of the course 117 5.688 1.158 2.67 7 Understanding issues Understanding country specific development issues 118 5.839 1.352 2 7 Refining skills Refining existing knowledge/skills 118 5.831 1.303 2 7 Providing skills Providing new knowledge/skills 117 5.838 1.293 3 7 Interest Helping get people interested in subject of activity 118 5.941 1.404 1 7 Country strategies Providing country specific development strategies 118 5.475 1.512 1 7 Organization strategies Providing org. specific development strategies 118 5.246 1.704 1 7 (Table A1 continues on next page.) 48 (Table A1: continued.) Variable Description N Mean Std. Dev. Min Max Explanatory Variables Demographics Female Dummy if the individual is female 113 0.416 0.495 0 1 Age Age of the individual 114 41.090 9.509 24 71 Government Dummy if the individual's org. is central/regional/local government 120 0.742 0.440 0 1 Position Position of the individual (6=top mgmt., 1=junior/other) 99 3.162 1.210 1 5 Research/teaching Dummy if the individual primarily conducts research or teaches 120 0.200 0.402 0 1 Term proficiency Level of proficiency of terminology of training (7=high, 1=low) 112 5.991 1.159 1 7 Course Characteristics Length Number of days the activity lasted 102 9.314 6.888 1 44 Follow-up Dummy if there were follow up activities after course 112 0.170 0.377 0 1 Design Brazilians Dummy if activity was designed specifically for Brazilian participants 115 0.548 0.500 0 1 Relevance Brazil Degree of relevancefor Brazil's specific needs (7=high, 1=low) 118 6.254 1.134 2 7 Extreme poverty Dummy if activity is related to eradicate extreme poverty 118 0.263 0.442 0 1 Education Dummy if activity is related to universal primary education 118 0.373 0.486 0 1 Gender Dummy if activity is related to gender equality/women's empowerment 118 0.076 0.267 0 1 Child mortality Dummy if activity is related to reduce child mortality 118 0.186 0.391 0 1 Maternal health Dummy if activity is related to improve maternal health 118 0.119 0.325 0 1 Diseases Dummy if activity is related to combating HIV/malaria/diseases 118 0.068 0.252 0 1 Environment Dummy if activity is related to ensure environmental sustainability 118 0.263 0.442 0 1 Partnerships Dummy if activity is related to global partnerships for development 118 0.424 0.496 0 1 Water Dummy if activity is related to ensure water sanitation and supply 118 0.110 0.314 0 1 Investment Dummy if activity is related to improve investment climate/finance 118 0.432 0.497 0 1 Trade Dummy if activity is related to promote trade 118 0.178 0.384 0 1 Action Plan Dummy if an action plan was developed in the activity 114 0.553 0.499 0 1 Work Environment Facilitators organization Average helpfulness of org. factors in using new knowledge and skills 108 4.880 1.545 1 7 Practices/procedures Helpfulness of org. practices/procedures in using knowledge and skills 113 5.372 1.733 1 7 49 (Table A1: continued.) Variable Description N Mean Std. Dev. Min Max Resources/funding Helpfulness of org. resources/funding in using knowledge and skills 112 4.509 1.977 1 7 Incentives Helpfulness of org. incentive system in using knowledge and skills 113 4.885 1.931 1 7 Supervisor Helpfulness of supervisor in using new knowledge and skills 110 4.691 1.811 1 7 Colleagues Helpfulness of colleagues in using new knowledge and skills 113 5.142 1.757 1 7 Facilitators country Average helpfulness of country factors in implementing ideas 112 5.426 1.478 1 7 Country policies Helpfulness of country policies in implementing ideas 112 5.723 1.514 1 7 Social groups Helpfulness of social groups in implementing ideas 113 5.336 1.751 1 7 Political groups Helpfulness of political groups in implementing ideas 113 5.221 1.816 1 7 Country "readiness" Helpfulness of country "readiness" to reform in implementing ideas 114 5.412 1.723 1 7 Colleagues attending Index of number of colleagues attending activity (5=ten or more, 1=none) 113 2.611 1.129 1 5 * Variables marked with a star are those used as dependent variable in the regression analyses on overall utilization and overall influence. 50 Table A2: 2SLS Regression Analysis Outcome. Dependent Variable = Utilization in Implementing Country Development Strategies Variable Coefficient Std. Err. t P>|t| Beta Coeff.a Overall effectiveness 0.166 0.447 0.37 0.711 0.084 Demographics Female 0.592 0.514 1.15 0.254 0.137 Government 1.992** 0.917 2.17 0.034 0.326 Position 0.539** 0.221 2.44 0.018 0.280 Work environment Facilitators country 0.243 0.167 1.45 0.152 0.176 Colleagues attending -0.057 0.200 -0.28 0.778 -0.031 Constant -1.555 2.020 -0.77 0.445 R2 0.401 N 64 aThe beta coefficients indicate the number of standard deviations theutilization will change due to a one standard deviation increase in the respective explanatory variable. ** Statistically significant at the 5 percent significance level. Table A3: 2SLS Regression Analysis Outcome. Dependent Variable = Utilization in Changing Legislation or Regulations Variable Coefficient Std. Err. t P>|t| Beta Coeff.a Overall effectiveness 0.693* 0.387 1.79 0.079 0.364 Demographics Female 0.131 0.486 0.27 0.788 0.029 Government 1.383* 0.758 1.83 0.073 0.242 Position 0.546** 0.218 2.50 0.015 0.272 Work environment Facilitators country 0.369** 0.141 2.61 0.012 0.276 Colleagues attending -0.088 0.196 -0.45 0.656 -0.046 Constant -4.253* 2.124 -2.00 0.050 R2 0.479 N 64 aThe beta coefficients indicate the numberof standard deviations the utilization will change due to a one standard deviation increase in the respective explanatory variable. * Statistically significant at the 10 percent significance level. ** Statistically significant at the 5 percent significance level. 51 Table A4: 2SLS Regression Analysis Outcome. Dependent Variable = Influencing Legislation or Regulations Variable Coefficient Std. Err. t P>|t| Beta Coeff.a Overall effectiveness 0.488 0.340 1.43 0.157 0.365 Demographics Female 0.051 0.426 0.12 0.905 0.017 Government 0.550 0.646 0.85 0.398 0.121 Position 0.011 0.178 0.06 0.952 0.008 Work environment Facilitators country 0.262* 0.132 1.99 0.052 0.257 Colleagues attending 0.031 0.172 0.18 0.859 0.024 Constant 0.683 1.927 0.35 0.724 R2 0.296 N 61 a The beta coefficients indicate the number of standard deviations the influence will change due to a one standard deviation increase in the respective explanatory variable. * Statistically significant at the 10 percent significance level. Table A5: 2SLS Regression Analysis Outcome. Dependent Variable = Influencing Consensus Building for Change Variable Coefficient Std. Err. t P>|t| Beta Coeff.a Overall effectiveness 1.139** 0.313 3.64 0.001 0.838 Demographics Female 0.019 0.402 0.05 0.963 0.006 Government 0.582 0.647 0.90 0.372 0.119 Position -0.070 0.146 -0.48 0.633 -0.055 Work environment Facilitators country 0.126 0.120 1.05 0.299 0.121 Colleagues attending 0.059 0.153 0.39 0.701 0.044 Constant -2.093 1.532 -1.37 0.176 R2 0.280 N 74 aThe beta coefficients indicate the number of standard deviations the influence will change due to a one standard deviation increase in the respective explanatory variable. ** Statistically significant at the 5 percent significance level. 52 ANNEX 2 A: WBI PARTICIPANT QUESTIONNAIRE World Bank Institute (WBI) Country Focus Evaluation Questionnaire Instructions for E-mail based Responses Thank you for agreeing to complete this questionnaire. The purpose of this questionnaire is to obtain base-line data on the performance of WBI learning activities in your country. We are going to ask you to think about the WBI learning activity you attended between the summer of 2000 and the summer of 2002. During this time period, if you participated in more than one WBI activity we would like you to answer the following questions with the most recent activity in mind. Before you start, please indicate (to the best of your recollection) the name of the most recent WBI activity that you attended ________________________________________________________________ Held from __________ to ___________ (date), in _____________________ (location) The questionnaire has four sections and should take between 20 to 30 minutes to complete. · Section 1 asks your opinion about the usefulness of the learning activity · Section 2 asks about the relevance of the learning activity to the specific needs of your country · Section 3 asks you to compare WBI learning activities with similar ones offered by other organizations · Section 4 asks about the characteristics of the learning event and your background. We would like to emphasize that your views are critical to us in our goal of continually improving WBI's learning activities. We need your honest feedback in order to get a clear picture of the effectiveness of the activity you attended. Please keep in mind that your responses will never be associated with you individually and that your participation in this survey will be kept confidential. If you have any questions about the questionnaire please send a message by e-mail to gmoreiradesousa@worldbank.org, by phone 1-202-473-3630 or by fax 1-202-522-1655. IN COMPLETING THIS QUESTIONNAIRE, PLEASE MARK YOUR ANSWERS WITH AN "X" OR PROVIDE A WRITTEN RESPONSE IN THE APPROPRIATE BOXES. 53 World Bank Institute (WBI) Evaluation Participant Questionnaire Section I. Usefulness of the WBI Learning Activity 1. Thinking about the WBI learning activity you attended, would you say that the activity has been effective or ineffective in the areas noted below. Not Extremel Don' Areas effective at 2 3 4 5 6 y t all Effective Kno w a. Raising your awareness and understanding of the development issues important to your country b. Updating or refining the knowledge or skills you already had c. Providing you with new knowledge or skills d. Helping you get to know people interested in the subject matter of the learning activity e. Providing you with strategies or approaches to address the development needs of your country f. Providing you with strategies or approaches to address the needs of your organization 2. Thinking about the WBI learning activity you attended, what was the main theme or area it addressed? 3. Did the WBI learning activity you attended lead to any overall changes in that Yes No Don specific area? (If you answer "No," or "Don't Know," skip to question #5.) 't kno w 4. How would you rate the nature of the overall change in that specific area? Strong No Strong Do negative change positive Not change change Know 1 2 3 4 5 6 7 8 54 5. Thinking about the WBI activity you attended, how often have you used the knowledge and skills you acquired in the following areas? Please mark "Not Applicable" if you do not work in the given area. Areas Not at all 2 3 4 5 6 Very Often N/A a. Conducting research b. Developing country development strategies c. Implementing country development strategies. d. Changing or influencing legislation or regulations d. Teaching e. Raising others' awareness in development issues f. Organizing collective community initiatives g. Implementing new practices within your work organization 6. How helpful or unhelpful are the following factors in actually using the new knowledge or skills that you acquired from the WBI learning activity? Not Neither Factors helpful 2 3 helpful 5 6 Extremely Don't at all nor Helpful Know unhelpful a. Practices and procedures in your work organization b. Resources and funding availability in your work organization c. Incentive system in your work organization d. Your supervisor e. Your colleagues 7. How helpful or unhelpful are the following factors in implementing the ideas covered in the learning activity? Not Neither Items helpful 2 3 helpful 5 6 Extremely Don't at all nor Helpful Know unhelpful a. Your country's policies b. Social groups in your country c. Political groups in your country d. Your country's general mood of "readiness" for reform and innovation 55 8. How has the WBI activity that you attended, influenced or led to changes in the following areas: (Please mark N/A if the particular area is not relevant to the activity.) Areas Negative Positive Influence 2 3 4 5 6 Influence N/A a. Research methodology or publications b. Legislation or regulations c. Teaching materials for courses d. Consensus building for change e. Community-based initiatives f. Work practices in your organization g. Other (Please specify) ____________________ 9. Have the issues raised in WBI learning activities been discussed at work, with local partners, government officials or NGOs? Never Discussed Don't discussed at all thoroughly Know 1 2 3 4 5 6 7 10. Thinking back to the time immediately following the learning activity you Yes No attended, were you aware of any follow-up activities (e.g., meetings, workshops, or E-mail discussion groups) arranged by participants? (If you answer "No," skip to question #15) 11. Did you participate in any of these follow-up activities? Yes No 12. In how many of these follow-up activities did you participate? 13. Are these activities still taking place? Yes No 14. If not, why? II. Relevance of WBI Learning Activities to the Needs of Your Country Recall, you are being asked to think about the activity you attended between the summer of 2000 and the summer of 2002. 15. Was the activity you attended designed specifically for participants from your Yes No Don't country? know 56 16. Thinking about the WBI learning activity, to what degree were the topics covered in the activity relevant to your country's specific needs? Extremely Extremely Don't Irrelevant Relevant Know 1 2 3 4 5 6 7 8 17. The table below lists the key goals for a country's development. Please review the list and indicate whether the WBI activity in which you participated was related to any of these goals by marking an "X" next to the relevant goal. Next, for each goal you marked as relevant please rate the extent to which the activity addressed the key issues in achieving that goal(s) in your country. (First review the development goals under column "A." Next, under Column "B," indicate with an "X" the relevant goal(s), then rate the extent to which the activity addressed key issues in reaching that goal(s) under Column "C.") A B C Mark an "x" If you marked "X," please rate the extent to which the here if WBI activity addressed key issues in achieving that goal. Activity was related to each goal Goals for Development Mark with an "X' Low 2 3 4 5 6 High N/A a. Eradicate Extreme Poverty b. Achieve universal primary education c. Promote gender equality and empower women d. Reduce child mortality e. Improve maternal health f. Combat HIV/AIDS, malaria, and other diseases g. Ensure environmental sustainability h. Develop global partnerships for development i. Ensure water sanitation and supply j. Improve investment climate and finance k. Promote trade 57 III. Comparison of WBI Learning Activities with Similar Activities Offered by Other Organizations 18. Thinking about the WBI learning activity you attended, are you aware of any learning activities with similar content offered by organizations other than WBI, in your country? Yes No (If you answer "No," skip to question #21.): 19. If yes, please provide the name(s) of the organization(s): 1. 2. 3. 20. In comparing the activities offered by WBI and non-WBI, would you say that, overall, the non-WBI learning activities were more effective or less effective than WBI training? Much About Much less Do Not more effective the effective Know Same 1 2 3 4 5 6 7 8 IV. Characteristics of the Learning Event and Your Background Now, we would like to ask you to think about some of the things that took place during the activity you attended and to recall some of the logistics. 21. How many of your colleagues, or others with whom you work closely (both within or outside of your organization), attended the same activity? None 1 to 2 3 to 4 5 to 10 Over 10 22. During the learning activity, did you develop an action plan/strategy (e.g., work plans, strategy papers, policy documents, assessment of country needs, assessment of sectoral needs) to apply the new ideas you learned? (If you answer Yes No "No," please skip to question #25) 23. If yes, did you work as a team with your colleagues to produce the action plan? Yes No 24. Did you actually use parts or all of the action plan in your work? Yes No 25. To your knowledge was the learning activity a part of a series of related Yes No 58 activities? 26. If yes, did you participate in other parts of the series? Yes No 27. Were you provided with the contact information of other participants in the Yes No activity you attended such as email addresses, telephone numbers or mailing addresses? 28. If yes, please indicate whether the contact information was: Not useful Extremely Do Not at all useful Know 1 2 3 4 5 6 7 8 29. After the WBI learning activity, were there any follow-up contacts made by the Yes No organizers? 30. If yes, please indicate the nature of the follow-up: Face-to-Face Web-based Other (please specify) meetings/activities discussions E-mail discussion __________________ 31. After the learning activity, did you receive any newsletters related to the Yes No learning activity? 32. If yes, please indicate whether it was: Not useful at Extremely Do Not all useful Know 1 2 3 4 5 6 7 8 33. In retrospect, what aspect(s) of the learning activity did you find most useful for your work, organization, or country? 59 34. In retrospect, what aspect(s) of the learning activity did you find least useful for your work, organization, or country? 35. What was the primary language used during the learning activity? ___________________ Not (For each item below, select one answer on the scale, where Highly 1=Not proficient at all and 7=Highly proficiency.) proficient 2 3 4 5 6 Proficient at all 36. At the time of the learning activity, what was your level of proficiency in the language of instruction of the learning activity? 37. At the time of the learning activity, what was your level of proficiency in the technical terminology utilized in the learning activity 38. How long was the learning activity? Please indicate number of days 39. What proportion of the learning activity did you attend? A quarter Less than More All or or less half Half than half almost all 40. At the time of the learning activity, which of the following best characterizes the organization in which you worked? (Select one.) University /Research Institution Donor Agency Non-Governmental Organization (NGO) (not-for-profit) National/Central Government Private Sector (for profit) Provincial/Regional Government Other ____________________________________ Local Government 41. At the time of the learning activity, which of the following best describes the primary type of work you did? (Select one.) Research Management/Adminis tration Teaching Policymaking / Legislative Research / Teaching Provide Services (e.g. financial, health, etc.) Other ____________________________________ 60 42. At the time of the learning activity, how would you best characterize your position in your organization? Top management (e.g. Minister, Deputy Minister, Top government official) Senior management/administration (e.g. Department Head, Division Head) Middle management/administration (e.g. Program Manager, Project Leader) Senior professional/technical/research staff Junior professional/technical/research staff Other (Please specify) ____________________________________ 43. Please indicate your gender Male Female 44. In which year were you born? (Please fill in the year in the boxes provided.) 1 9 Thank you for your feedback. We greatly appreciate your cooperation. 61 B: IETS PRESENTATION The Instituto de Estudos de Trabalho e Sociedade (IETS - Institute of Studies of Labor and Society) is an organization of civil society located in Rio de Janeiro, Brazil. Its main purpose is to gather professionals from different public and private institutions and from multiple disciplines to produce and publish knowledge related to the social area. IETS researchers belong to the staff of the best reknown academic institutions of Rio de Janeiro, among which are the Federal University of Rio de Janeiro (UFRJ), the Catholic University of Rio de Janeiro (PUC-Rio), the Getúlio Vargas Foundation (FGV-RJ) and the Institute of Applied Economic Research (IPEA). Studies carried out by IETS concentrate, mainly, in two dimensions: monitoring peoples´ life conditions and studying the determinants of collective and individual well being. It is also IETS's purpose to consolidate itself as a social actor capable of contributing to the design, the implementation and evaluation of public policies, either local or national, which aim at making peoples´ quality of life better, in an efficient and sustained way. The Institute also plans on becoming a center of qualification of professionals for the different levels of government and civil society. All these initiatives and activities, however, would lose sense if they remained closed to themselves. Therefore, it is one of the main functions in IETS to promote a wide and plural debate among different segments in society interested in the reflection of the multiple angles of social issues. The basic tools of these activities are a continuous and diverse schedule of seminars and the definition of an editorial line that can make public, in a consistent way, the researches carried out in this field. The attention to distinct media channels represents one of the key elements of interaction with society. For this reason, IETS maintains a news agency, so that this interaction can happen in a dynamic, regular and consistent way. Still, IETS directs effort in building and consolidating exchange nets with universities, public and private institutes, government and non-government organizations, both national and international, directly interested in the diagnosis and fight against poverty and inequality. IETS is a member of the NIP ­ Network for Inequality and Poverty Research, an initiative of the Latin American and Caribbean Economic Association (LACEA), the Inter-American Development Bank and the World Bank. Bringing the poor to light and forcing them to the moral conscience of the Nation are fundamental tasks to overcome inequality and build a project of development anchored in social justice. This is a challenge that IETS is willing to face. 62 C: IETS REPORT IMPACT EVALUATIONOF WBI TRAINING ACTIVITIESHELD IN BRAZIL Final Report The World Bank Institute (WBI) through its evaluation group (IEG) is conducting an impact evaluation of WBI's training programs delivered in Brazil between July 2000 and June 2002. The main objectives of this evaluation consist of the following: (a) To determine the training's impact on participants and on the different institutions; and (b) To use this information as a baseline for future and further impact evaluations on the country in focus. The evaluation is divided in two stages: the first consists of the recruitment of two focus groups each with a small number of participants, and the second consists of the application of questionnaires to the 298 participants of different Brazilian states.17 The Instituto de Estudos do Trabalho e Sociedade (IETS ­ Institute of Studies of Labor and Society) was hired by WBI to help in the recruitment of focus groups and survey administration. This report aims to describe to WBI the main procedures used during the work and the methodology applied. Activities Description: The research began on May 5th, 2003 and the first activities were related to the recruitment and formation of the focus groups. Focus Groups Formation The focus groups were held, at IEG's request, in Rio de Janeiro and Fortaleza. Rio de Janeiro was the State in Brazil with the greatest concentration of course beneficiaries. From the Northeastern States, Ceará had the highest number of participants. The number of participants in each focus group (between 5 and 8), the time, the place, the schedules18 and the requirement that the participants should be able to speak English were defined by IEG. For the focus group formed in Rio de Janeiro, several attempts to contact 72 participants were made (Rio de Janeiro city and surrounding cities). The main difficulties found were: (a) Wrong e-mail addresses, and/or telephone numbers: either non-existent or outdated; (b) The schedule of the meeting not being available; 17To be exact, in the database there are 300 participants, but there are two duplicate registers, from participants that attended two different activities and therefore will answer according to the more recent one. 18 The Focus group in Rio de Janeiro was conducted on May, 12 between 12:00 pm and 3:00 pm, in Pestana Hotel in Copacabana. In Fortaleza, the meeting was held on May 14 between 12:00 pm and 3:00 pm in Luzeiros Hotel. 63 (c) Unreachable (ex: holidays, on leave, traveling, not answering the phone, always busy, etc.)19 As a result of these multiple attempts, seven participants confirmed their presence for the meeting in Rio de Janeiro, but two of them did not show up on the focus group day. Focus group recruitment for Fortaleza was a little more difficult because the number of participants was smaller. Even when trying to contact people in cities near Fortaleza the total number of participants was less than 20. The obstacles were different in this case. The number of wrong or outdated phone numbers and/or emails was not significant and the people contacted were more receptive. Meanwhile, the meeting had to be conducted in Portuguese because most of the people could not speak English. In the end of this process, there were five people confirmed and they all showed up. Questionnaire Administration Immediately after the focus group interview in Fortaleza the process of administrating the questionnaires started, the procedures of which are described herewith. The first step was to send an email to all the participants whose information was registered in the database supplied by IEG. The email was personalized and contained a brief explanation of the survey, the title and the date of the activity each participant attended and a code for their identification. As 98 participants had no e-mail, they were not part of this first attempt. From the 200 participants to whom emails were sent, 62 electronic addresses were inaccessible and, therefore, did not receive the message. From this point the work was divided in two groups: (a) Contacting by phone all the participants that did not receive the e-mail (160) and; (b) Waiting a few days and starting to contact by phone the people that did receive the e- mail (148), but did not complete the questionnaire20. In the case of group 1 all the phone numbers that were in the database were used and, as a final attempt, tried to locate the institution where the participant used to work by the time of the course to obtain information about him/her. This group proved the most demanding in terms of work, because many of them did not have a phone number (for example: among the 98 that did not have email, 63 percent did not declare any phone number, which led to research on the institution involved or via the Internet 21 . When the participants were found, their e-mail addresses were requested in order to send the message, taking advantage of the opportunity to explain the importance of the survey. Among the participants that received the e-mail (group 2), the response rate was much lower than expected and after several days the process of gathering responses began. Several other 19 It is interesting to notice that two people did not accept because they could not speak english and one did not remember having participated in the training activity. 20IEG has informed periodically about the number of participants that replied to the questionnaire, according to their database. 21Internet browsers were used to track information through the participants' name. This method proved not to be very efficient but resulted in some contacts. 64 problems were identified, such as: non-existent or incorrect phone numbers, many people no longer worked in the same place and no information was held on them. Some were on holiday, on leave or traveling, others were never located and never returned the phone calls. Another problem was related to the participants who, after phone conversations confirmed that they were going to fill out the questionnaire but, until now, failed to do so. The problems encountered with the second group were repeated among the participants of group 1 in the sense that there was the need to find electronic addresses so that the emails could be sent. Simultaneously to this process, an email was sent from IEG to the participants, which had a direct effect on the response rate. The survey was extended until July 4th, with the intention of having the maximum number of answers possible. Until now, 115 questionnaires have been registered and 64 participants informed IEG by phone that they would answer the questionnaire but have yet to do so. Of the 300 participants, 81 were excluded from the survey because it was impossible to track them or because they refused to fill out the questionnaire. The remaining 40 participants did not give a final response for several reasons. All the cases are analyzed one by one in an annex provided by IETS. 65 D: INTERVIEW AND FOCUS GROUP QUESTIONS FOR COUNTRY FOCUSED IMPACT EVALUATION 1. What did you gain from the activity? 2. Was this activity part of a larger program? 3. What were the most useful aspects of the activity? What were the least useful aspects of the activity? Probe 1: For what reasons? Please describe in detail. 4. Can you offer specific examples of how you applied what you acquired in the activity? Probe 1: Have these led to any changes in your organization, or changes in policies and practices in your country? Probe 2: What were these changes? 5. In your view, how has the activity benefited your country? Probe 1: In what way? Please provide examples. 6. When you used the information that you acquired from the activity, what were some of the obstacles that you encountered? Probe 1: Please explain and give specific examples. Probe 2: In what ways were these obstacles? 7. Since attending the activity, have you attended other courses addressing the same topics? Probe 1: What were they? Probe 2: How useful were they? 8. Were the knowledge, skills or new ways of thinking that you obtained in the activity available from another source (e.g., institution, self-study materials, Internet site, etc.)? Probe 1: If yes, what were these sources and what were the names of the activities? Probe 2: How useful were they? 9. How can we improve similar WBI activities in the future? Probe 1: Please give specific examples. Probe 2: In particular, what types of courses would you like to see offered? 10. Is there any other assistance that WBI could provide in future activities to meet your specific professional needs? Probe 1: If so, what would it be? 66 E: FOCUS GROUP RESULTS - RIODEJANEIRO The first focus group interview was held in Rio de Janeiro on Monday, May 12, 2003 with five former participants of WBI training events, further listed in this Annex. These participants worked at the Brazilian Bank, Municipal Agencies and in Petrobras, the largest corporation in Brazil. Overall, two courses on finance and banking were represented - Developing Domestic Debt Markets Workshop22 and Intergovernmental Fiscal Relations. The other course represented was the Clean Air Initiative Urban Air Quality Management course. A summary of the discussions with the Brazilian participants, based on the interview questions (Annex 2D) is presented below. Participant Comments Gains and Usefulness of the Attended Activity · Increased awareness of the issues of the course as well as greater technical support. For instance, the Finance and Banking course approach was mentioned to allow financial analysts and other participant technicians to begin conceptualizing existing possibilities of debt markets including those they could take for their organization in order to attract investments. · Participants' heterogeneity (different government levels), experience sharing and the establishment of personal contacts with people in the same field were considered to be of high importance. · Working group discussions and participants' presentations, allowing knowledge sharing to occur as well as discussion forums, including the possibility for participants to launch themes for discussion were useful interactions. · Summary document distribution by the end of the activity deemed useful. Observations · Lack of time for group work preparation as well as short duration of the course in order to approach all the issues on the agenda. · Content of the course not tailored specifically for Brazil. · High number of participants attending the course made it very difficult to directly access the presenters for deeper discussions. Integration in Broader Programs · In the majority of cases, participants were not certain if the course was integrated in a broader program but they did mention various other courses that might have been related. Furthermore, they stressed the importance of having the information from the attended course to raise awareness and interest to participate in further activities. 22According to CRS data for FY01-02, more than 1/3 of the total number of participants in Brazil attended the Developing Domestic Debt Market Workshop, within the Finance and Banking program. 67 · In some cases the activity was part of a series of training courses in several areas that the government proposes to their staff. Application Examples · The majority of the participants referred to "changes in terms of perception" but did not refer to concrete or specific practical application examples of the knowledge and skills obtained in the course. · One example was mentioned on the Clean Air Initiative course. The participant stated to put into practice the knowledge acquired a few months after the course, when a large oil spill occurred, affecting several municipalities around Rio de Janeiro. A methodology to quantify the environmental damage was established based upon some of parameters discussed in the course. As a result, the responsible company has submitted a fee and the revenues reverted to a municipality park budget, but is still in its implementation phase. Benefits for Brazil · Overall, it was difficult for participants to mention direct benefits for their country, they could only appoint trends while referring to long-term processes. Courses on specific issues such as controlling air pollution, finance and banking or decentralization and governance were generally considered useful for the country as a whole because they are related to Brazil. · Nevertheless, WBI training activities contributed indirectly to the decision-making processes while allowing technical decisions to be taken more firmly, at the national level. Obstacles · Government change and political instability were stated as the main obstacles to the implementation of new practices. · Difficulties of coordination inside the government were mentioned because not everybody attended the activity and therefore not everybody was aware of the importance of the issues to the same degree. · People were skeptic when considering information searches at the WB. They thought considering it to be a long and painful process, which suggests the need to clarify basic WB search routines in order to ease the process, facilitate access and establish contacts. Other Attended Courses · Seminars promoted by OECD and WB. · Academic courses (masters degrees, etc). · IPIECA and Petrobras course. · Banking and finance courses, ex: BBVA (international bank group in Europe and Latin America). 68 Other Available Sources · Available bibliography is not as specific as the course approach. · World Bank and partners websites published the course information. · World Bank publications. Suggestions and Recommendations · Explanation of information search through the web during the training activities, in order to facilitate the common user information search process. · Organized literature and database on the website. Provision of a database with case studies, establishing contacts and lists with bank consultants' contacts in order to exchange ideas on specific situations. · Discussion forums on the course topics. · Delivery of activities tailored to Brazilian needs and/or coordinated with other Latin American countries that have similar problems. Courses focused on Brazil and regions within the country, namely generic training with teachers developing projects on these areas. · Creation of an analysis routine and discussions of practical cases, including more case studies. Greater focus on methodology development, including general models with mechanisms for capturing resources and valorization criteria, for instance, on environment. · Specific discussions with small groups of participants (working groups), in order to facilitate closer interaction with the speakers. · Bigger offer of initiatives on financial markets. · Environmental courses with a component on sanitation were considered a fundamental area of concern in Latin America, providing and gathering solutions using useful and low cost technologies. · Inclusion of a section on the training activities about bank's structure and organization, including the areas where the bank intervenes. 69 70 F: FOCUS GROUP RESULTS-FORTALEZA A second focus group interview was conducted in Fortaleza, part of the Brazil Northeast Region, on May 14. This region is generally considered the most impoverished region of the country and is targeted by WBI's country-focused program in Brazil. The five participants of this focus group were mainly government officials, representing several municipalities in the region, one of them was the former president of the Bank of Northeast, the leading financial institution in the Northeast Region. Overall two WBI learning activities were represented: Intergovernmental Fiscal Relations and Local Financial Management Course for Brazil and Lusophone Africa, and City Strategies to Reduce Urban Poverty Through Local Economic Development. One of the participants in the group interview also had a role as a presenter in the course he attended. When comparing with the previous focus group, these participants were more focused on local issues, especially municipality concerns. A summary on their comments on the courses, following the interview questions structure (Annex 2D), is presented below. Participants' Comments Gains and Usefulness of the Attended Activity · Expansion of perspectives mainly through the presentation of concrete examples an methodologies in execution out of Brazil. In terms of local planning and development strategies, the fact that the methodologies presented at the course were feasible and applicable in different realities (small and large municipalities) was considered to be very useful. · New techniques, planning strategies and new cultural concepts on how to deal with local development. · Learning how to interact, plan and articulate the development of a small city, taking into account the problems of a big metropolis, including social mobilization strategies. · For the Intergovernmental Fiscal Relations Course, the course helped to evaluate the role and importance of resource transfers among the several government spheres for sub-national entities. · High quality of the delivered material during the course, for instance, a CD distributed among the participants with all the presentations. Observations · Some participants complained they had no access to the presentations after the course, not even via the internet. 71 · Even though the pedagogical material was deemed to be useful, including slides and other presentation techniques, the instruments used to do the exercises were criticized. Having some background experience included and introducing them when applied to case studies could have been helpful facilitating the use of the material/instruments. Integration in Broader Programs · In some cases, the course was integrated in a human resources qualification policy, promoted by the government and not attached to any other WBI program. · The City Strategies course, which was a teleconference course, was integrated in a program developed with several national and sub-national organisms in India to study development, with the goal of transmitting the results later on to municipalities and States in Brazil. For this, one of the facilitators was the Bank of Northeast, which role was underlined by participants. Application Examples · Several examples of application were given, namely the implementation of a local sustainable development program in Fortaleza municipality by the local government. This direct program includes recuperation measures to reurbanize the center and create, and establish, an incentive system to bring back commercial activities and sources to the city, promoting growth using concepts and knowledge from the course. · The participant that was also a presenter gave two voluntary lectures in municipalities using the knowledge gained on the course. · One participant who attended the fiscal relations course contributed to the implementation of a rule on voluntary resource transfers from the state to the municipalities. Benefits for Brazil · As in the Rio de Janeiro focus group, participants did not appoint direct benefits from the course to the country. Participants considered that in the current situation the results are less important than the process, because there are still many conditions to aggregate and many people to mobilize, to make the process inclusive. This should carry more weight than the results, being stated that "while the knowledge is concentrated, the results will not show up". Obstacles · Cultural obstacles on the instruments' application, of political and technical perspectives. · Rejection reaction when installing new management cultures to some local realities. This is due to a significant and already existing gap among the local bureaucracies 72 when considering new techniques and managing different alternatives for development, which are seen negatively as evaluation. Other Attended Courses · Courses promoted by the WB and the BN on program and project evaluation and poverty combat. · Course promoted by the OAS on social entrepreneurism · ECLAC course in Chile with focus a on regional development · Specialization course on public finance promoted by the Ministry and the region's stock exchange. Other Available Sources · The material presented on the course, which in generally was very satisfactory, was available but participants referred to a need for additional bibliographical sources, such as references for books corresponding to the themes of the course, websites, newsletters, etc. Suggestions and Recommendations · The main recommendation refers to pedagogical practices, to overcome some cultural barriers and the gap mentioned previously, considering WBI's defining role as a multilateral organism in establishing such mechanisms in a convincing and stimulating way. · Conduct database maintenance with the presented experiences, in order to create a synergetic system among the speakers and participants. · Scientific production stimulation as a WBI role, stimulating people to write about their experiences, suggestions made taking into account mainly WB experience in several countries and relevant areas. · Focus on courses presenting solutions to combat corruption was considered one of the biggest problems in Brazil and with a direct correlation with poverty, promoting transparent actions that would change the Brazilian accounting courts. In turn, stimulating a more effective action of internal control that does not practically exist at the municipality level. · Presentation of a WBI program to manage ranking and rewarding systems, listing the best municipalities according to the results of their resource application and evaluating them. This proactive action would also have reflection in broader acceptance of the programs. 73 · Courses combining development and planning strategies with fiscal issues, finance and public management. In effect gathering these two themes - public and social control instruments and the results on the quality of life. The interactions between these two themes are not directly seen and therefore the suggestion to WBI using the knowledge to show them. · Course aggregating examples for Municipalities, with greater public diversity, in order to reach the different bureaucratic realities which would gain support for new policies and practices. · To face the dilemma of decentralization with control, WBI should open space to involve the qualification process and training. Thus responding to the need for social and policy control instruments, not to lose efficiency and scale economics. · Broader offer of courses in M&E and impact measurement for the different spheres of government, which would lead to practical changes. The need for practical cases of evaluation was mentioned as well. · Training managers with the vision of participatory management and democratization of management so as to include the beneficiaries of the programs in the process. · WBI should stress the qualifications of responsible managers for public expenses, from the viewpoint of spending efficiency, because expenditure when compared with income/revenues has a secondary role. · WBI could gather people of all government spheres, trying to give them the knowledge and management culture of development, not only centered in the natural or state power; this may benefit the country as a whole. · Policy taking into consideration the formation of study groups of local realities, which can serve as a reference to other countries and forming practical networks (where people interact) through the internet, newsletters, etc. · Delivery of courses in Brazil approaching cases that apply to the country and preferencialy to the regions within Brazil, gathering specialists in different areas but applying the knowledge to the same reality. A suggestion was made to conduct a "case study city" showing the generated results which pedagogically would have a large impact in Brazil. Participants stressed the need to use successful cases in a reality which people are familiar with, in order to reach greater support and acceptance of the new practices. · More partnerships with local institutions in order to reach municipalities and broader areas. 74 G: CAS SUMMARY OFDEVELOPMENT PRIORITIES, WBI ACTIVITIES AND BRAZILIAN PARTICIPANTS DISTRIBUTION. CAS Summary of Development Priorities23 WBI Learning Activity Title (FY01-02) No. Part. Brazil % Part. Brazil Poverty Reduction & Economic Management - Poverty reduction (H) City Strategies to Reduce Urban Poverty through Local Economic Development 9 3.00% Macroeconomic Management for Financial Stability and Poverty Reduction Course in 8 2.67% Brasilia - Economic policy (H) Using Knowledge for Development: A Policy forum for Latin America 7 2.33% - Public sector (H) Curso de Economia de la Regulación de los Servicios Públicos y Defesa de la 1 0.33% Competencia Curso de Economia de la Regulación de Servicios Públicos y Defesa de la 2 0.67% Competencia24 - Gender (M) ............................................................................................ Subtotal 5 delivered courses (16 percent of total deliveries) 27 9.00% Human Development Department - Education (H) Strategic Choices for Education Reform Global Core Course 4 1.33% Strategic Choices for Education Reform ­ Central America 1 0.33% Strategic Choices for Education Reform ­ Brazil 37 12.33% Strategic Choices for Education Reform 1 0.33% - Health, nutrition & population (H) Health Sector Reform Capacity Building in Latin America and the Caribbean ­ Second 4 1.33% Regional Course 23Indicating Bank priority: H­ high, M ­ moderate and L-low 24There were 2 courses with the title "Curso de Economia de la Regulación de los Servicios Públicos y Defesa de la Competencia", both held in Buenos Aires, Argentina. The first course, with one Brazilian participant, was during FY01 (3-28 September, 2001) and the second during FY02 (4February-1 March, 2002). 75 CAS Summary of Development Priorities23 WBI Learning Activity Title (FY01-02) No. Part. Brazil % Part. Brazil Health Sector Reform Course via Distance Learning 1 0.33% - Social protection (H) Pension Reform Executive Workshop 4 1.33% Safety Nets Core Course 1 0.33% Subtotal 8 delivered courses (26 percent of total deliveries) 53 17.67% Finance, Private Sector & Infrastructure - Financial Sector (H) Developing Domestic Debt Markets Workshop 102 34.00% Intergovernmental Fiscal Relations and Local Financial Management Course for Brazil 44 14.67% and Lusophone Africa Non-Bank Financial Sector in the Context of Globalization 3 1.00% Policy Challenges for the Financial Sector in the Context of Globalization 1 0.33% - Private Sector (H) The Legal and Regulatory Environment for Credit Reporting Systems ­ Sao Paulo Brazil 2 0.67% - Energy & mining (M) ............................................................................................ - Infrastructure (H) Frontiers in Infrastructure Finance 2 0.67% Subtotal 6 delivered courses (19 percent of total deliveries) 154 51.33% 76 CAS Summary of Development Priorities23 WBI Learning Activity Title (FY01-02) No. Part. Brazil % Part. Brazil Environmentally & Socially Sustainable Development - Rural development (M) Making Biodiversity Conservation Pay 2 0.67% - Environment (H) Clear Air Initiative Urban Air Quality Management 31 10.33% Clear Air Initiative for Latin American cities: the Case of Lima-Callao Peru 7 2.33% Conferencia Internacional Aplicación y Cumplimento de la Normativa Ambiental en 6 2.00% América Latina Environmental Economics & Development Policy 1 0.33% Forest Law Enforcement and Governance 2 0.67% International Conference on Environmental Compliance and Enforcement (INECE) 2 0.67% International Waters: Training Course for Bank staff 1 0.33% Market Creation For Biodiversity 2 0.67% Orientation Course for Controlling Air Pollution 2 0.67% Water Information in Latin America 2 0.67% Water Forum 2002 ............................................................................................ 8 2.67% - Social development (H) Subtotal 12 delivered courses (39 percent of total deliveries) 66 22.00% 77