40575 WBI Evaluation Briefs I I RFPflRTlhlT: nhl P I IFhIT Ahln STAFF I FARhllhlC Ahln PAPAPlTV nF\IFI flPhAFhIT PRnCRAhAC Nidhi Khattri. Acting Manager Institute Evaluation Group November 2004 What Improves the Utility of WBI Activities? Evidence from Five Countries The Challenge Pillar to ensure that thematic programs respond to country As the capacity-developmentarm of the World Bank, WBI has priorities and work toward producing measurable results at engaged in global and regional knowledge sharing for nearly the country level. fifty years. Much of the expertise within the organization has been arranged along sectoraland thematic lines. Sectoraland UnderlyingAssumptions thematic programs delivered globally or regionally, however, The assumptions underlying the shift to a country focus have not always addressed conntry-specific issues, given that include the following: capacity constraints differ widely from country to country The potential for capacity development at the individual, according to their stage of development. institutional, and policy levels will increase as WBI programs are adapted to address local needs. Thetraditionalemphasison thematicknowledgesharinghas also A country focus approach will mitigate the difficulty of constrained WBI from fnlly aligning its capacity-development producing evidence of cost-effective capacity development activitieswith Bank operations. In addition, this approach has stemmingfrom WBI thematic programs that involve many not provided snfficient focus on local development priorities countriesin various stagesof development. in order to systematically complement the activities of key regional partners. Evaluation Methods and Instruments World Bank Institute Intervention The evaluation summarizedin this BrieP is a meta-analysis of data from five country studies,' supplemented with original In Fi02, WBI embarked on a new, country-focused data collected from the WBI staff involved in delivering WBI capacity-development strategy This strategy helped WBI learning activities for participants in those countries. The begin to align its knowledge agenda more closely with Bank objectiveof the evaluationwas twofold: operations.' As part of this effort,WBI created aCountryPillar that is responsible for working with regional and country (a) to develop baseline estimates of the effectiveness and teams and with WBI thematic teams to develop and deliver utilityof WBI learningactivities,against which to measure sustained programs of capacity development in target the effectiveness and utility of the country-focused countries. The Country Pillar follows a two-pronged approach in the future, and approach to carryout its mandate. First,the Pillarworks with (b) to address a key research question: What elements and operationalcountryteams to develop a strategic approach to charactmj.tics of WBI learning activities result in capacity development in priority countries and to strengthen greater effectivenessanddesired capacig development the knowledge and learning inputs of the Bank country impacts at the individzla1, institutional, and programs. Second, it works with units of WBI's Thematic policy/cozntly levels? I .This occurs through the alignment of WE1 learning activities with thc Bank's Country Assistance Srratrgics, Poverty Reduction Strategy Papers. Community Driven Development approaches, and Comprehensive Development Frameworks. 2. Quizon, J.. C. Ling Chard, and M. Lockheed. "Thr Effectivmess and Use in FY03 ojWBI FYOI-02 Activities: A Baseline Assessment in Fivr Countries," WBI Evaluation Studies No. EG04-86, World Bank Institute, May ZW4. 1. These reports are authored hy IEG staff and consultants: Khattri. K., P. Bachrach, T. Jiang, "The impact ojWBIAdivities, FYOI-02, on Pnnicipunfs from Sri Lanka: A Bosrline Assessmenr."WBI Evaluation Sedies No, E004-73. World Bank Institute. October 2003. Zia. H.. M. Al-Savvid. S. Tawila. and V. Ounnarsson . . ., ."Thelmnoct of,WBI . Client Acriviri, FYOI-02, on Pnrricipnrztsfmm Egypt: A Bnselirze Assessment," WE1 Evaluation Studies No. EGO4-78, World Bank Institute, October 2003. Bardini, M., V. Gunnurason. E. Maniieuu. and Y. Narozhnava. "Thelmovct of WBI Client Aciiviiirr. FYOI-02. on Poriicioann fmm Russia: A Bnsrline Assesrmenr." WBI Evaluation Studies W O R L D B A N K I N S T I T U T E Pmmoting knowledge and learning for a bener worM Figure I. CompositeDependentVariables:Activity Effectivenessand Use items (see Figure 1). In both cases, six categories of independent variables were examined related to Dependentvariables activities: objectives, design, delivery Enhanced understandingof - development issuer features, follow-up, participant Increased knowledge and skills characteristics, and other. The Activity Strategiesfor addressing country'sr analysisused a two-stage instrumental development needs Effectivene regression model, where the first Independent . f Appmachesforaddressing Variables lat~on'rneedr stage explained activity effectiveness (categories) uction to others interested b and the second stage used the Activity objective of activi I estimates of activity effectiveness and Activity design activity features to predict participant Activity delivery features use of the learning activity. Activity follow-up Participant Participants and characteristics -3 Other + Overall Use Response Rates The design and implementation of the 2 Operational Use #.=&,.,%, +developingcountrystrategies Use L participant surveys in each of the five +takingcommuniryinitiatives countries were very similar and +changingregulations involved either the full population or a -t implementing country strategies random sampleof theFY01-02 learning. activityalumniwith contact information in WBI's Client RecordingSystem (CRS) The original studies evaluated the Data from the participant and TTL database (see Table l).iOf the 1,268 medium-termeffectsof FY01-02activities surveys were combined and analyzed participants identilled by name in the in fiveWBIfocus countries:Brazil,Egypt, jointly to explore how WBI learning CRS, 1,006had contactinformation. Of Russia, Sri Lanka,andThailand. activity objectives, designs, delivery those 1,006,793participants from 131 features and related follow-up activities activities responded. While the overall Data in the original five studies came influence participants' ratings of responserate was 79 percent, response from participant questionnaires effectiveness and use. The analysis rates by country varied from 52 to 91 (standardized, translated for and defined both "activity effectiveness" and percent, with most non-responses pretested in each country), in-country "activityuse" as composite dependent due to inaccurate or outdated CRS focus groups with WBI activity variables, each composed of several contactinformation. participants,and interviews with World Bank country operations staff. These Table I.Samplingof Participants,by Country datawere collectedin EY03, nine to 31 months after the activitvhad been held. Total Participants Toal Participants New survey data were "llected from Country Total Pardcipants* Identified by Name SampledWith Response Rate the task team leaders (TTLs) of the (number sampled) Contact Information evaluated activities, using a Brazil 1,826 300 23 1 52% questionnaire with closed-ended ~ g y p t 882 248 203 84% questions related to the objectives, :ryznka 3.046 1628 (300 sampled) 217 91% 258 184 169 80% design, and delivery of the learning 1.536 768 (236 sampled) 186 86% activity, including later follow-up Tom1 7.548 I.2ML rampied 1,006 79% activitieswith participants: *These totals exclude one-day and World Links activities 4. To minimizeburden. TTLs wen asked to pmvide responsesfor no momthan two learningactivities that they hadcooducted: WBI activitieswere therefon selected for this study based on which ones had the highestnumberof respondentsfrom the participantsurveys. 5. Possiblecontactinformalionincludedemail address, homeloftice telephone number, far number, or homeloffice address. Page 2 WBI EvaluationBriefrNavember 2004 W O R L D B A N K I N S T I T U T E Promotingknowledgeandlearningfora b e m world The formal survey of TTLs was associated with activities tailored to Evaluation Results: Use administeredvia email to 42 TTLs of 62 meet regional needs. 5. WBI activities were not often WBI learning activities. Questionnaires 3. Tbcee activity delivery features used by participants,particularlyfor were completedby 36task managers of contributedto greatereffectiveness. operational purposes--defined as 52 learning activities, for a response Activities that involved action plans developing country strategies, taking rate of 83 percent. All programs were were 11percentmore effective than community initiatives, changing represented in the sample, and TTL those that were not. regulations or implementing country data were matched with 405 strategies. Figure 2 illustrates the respondents to the participant survey Those learning activities that were average frequency of use in operational (52 percent of the full respondent part of a series were seven percent compared with academic functions. population and 40 percent of the more effective than one-time Mean ratings of academic use were activitypopulation). activities. slightly above the mid-point, and mean Activities in which teams of ten or ratings for operational use were at or Evaluation Results: more colleagnes attended were six slightly below the mid-point. The Effectiveness percent more effective than those in reported frequency of use varied by which individuals attended with 1. Participants' ratings of activity country, and fewer than half of the fewer than ten colleagues. effectiveness were positive across participantsin allcountriesbut SriLanka 4. all five countries, particularly for Unmeasured country-level reported that they frequently used their factorsaccountedfor alarge share individual benefits. Respondents WBI-acqniredknowledge and skills. of the variance in participants' rated WBI activities above the mid-point 6. The academicoroperationaluse ratingsof effectiveness. Exogenous rating' of "4" in all five areas of of the activitywas strongly related effectiveness: offering networking country-specific attributes affected to reports of its effectiveness. The oppormnities (5.17), raising awareness participant ratings of effectiveness of participants' perception of the of development needs (5.10), WBI activities. ParticipantsfromBrazil, effectivenessof a WBI activitywas a key increasingknowledgeand skills (5.08), Egypt and Sri Lanka perceived activities determinant in whether the knowledge providing strategies for development as ten percent more effective than did and skillsacquiredfrom theWBI activity needs (4.8), and giving approachesfor participants from Thailand and Russia. were used. addressing the needs of participant's organization (4.6). These ratings Figure 2. Reported Use ofWBl-Acquired Knowledgeand Skills varied widely across countries with (I=not at all; 7=very often) participants in Brazil, Sri Lanka, and Egypt rating activities more highly than participants in Thailand and Russia. 2. '%o design features were Academic Use Operational Use important for activity effectiveness:designingthe activity with a partner and tailoring it to country needs. Activities with these design features were 30 percent more effective than activities lacking both features. Interestingly, ratings of effectiveness were significantly positively associated with activities tailored specificallyfor the needs of the country, but significantly negatively 6. Ratiogs were horn I(noteffective at all) lo 7 (ertrernely effective), wiUl4 at Ulemid-point WBI Evaluation BriefrNavember 2004 Page 3 W O R L D B A N K I N S T I T U T E Promotingknowledgeand learningfor a better worM 7. The participant's position was significant in Delivering activities as part of a series of offerings and predictingoverall activityuse and operational use but following up with participants will increase the degree to not academic use. Participants in higher level positions which the content of the WBI activity is used locally after appeared to apply more of what they learned from the WBI the learningactivity activity. Focus group discussions revealed that middle-level For programs to follow up with their participants, it is WBI alumni were less likely to implement new ideas without essentialthat archival data with more accurate participant the agreementof their superiors,whilehigherlevel alumnidid contact information be available in the CRS. Systems and not face this constraint. procedures need to he in place to enable continuous 8. lbo delivery features had significant impact on updating of this information. activity use. Acknowledgement Using English' as the language of instruction increased the The evaluation summarized in this brief was conducted with reported academic and overall use of the learning event the hancial support of the World Bank Institute. hut not its operational use. Participants were also likely to use more of what they acquiredfrom thelearningeventwhen the activitywaspart of a series. Tbis was true for overalluse and operational use, but not for academic useperhaps because courses do not need to be part of a series to have a simcant The World Bank Institute WBI) helps people, institutions,and impact on scholarlyactivities. countriesto diagnose problemsthat keep communities poosto make infomed choices to sobe those problems, and to share 9. Electronic follow-up was the most effective means what they learn with othen.Thmugh tradrtional and distance for raising the use of a learning activity. Contacting leamingmethods,WBIand hs partnen in manycountriesdeliver knowledge-based options to policy makers, technical experts, participants via electronic means significantly increased business and community leaden,and civil society stakeholdem participants' ratings of both overall use and academic use. fostering analytical and networking skills to help them make sound decisions, design effective socioeconomic policies and 10. Activity objectives did not influence participants' programs,and unleashthe productive potewtialoftheir societies use of WBI learning events. Activity objectives as dehed The Institute Evaluation Gmup (IEG) work; with WBl's byWBITTLswere not sipdicant determinantsof participants' LeadershipTeamand with theWorld Bank's Learning Boardto actualuse of what they acquiredfrom the learningevent. This desin, implement, and report evaluation results for M,cliemt, observation may indicate that WBI learning events would and joint learning events IEG also offers distance learning and face-to-face training in pmgram evaluation. WE1 Evaluation benefit from more realistic activity objectives dehed in more Briefsreport on thse evaluxtjonrerub. specific, concrete terms. Contacts Implications and Conclusions WBI Evaluation Briek joy Behrens,EvaluationAnalyst This study establishes some baseline measures of the 202 4585053 or ~tehrens@voridbanko~ effectiveness and utility of WBI learning activities. These benchmarks provide useful anchors for future country- The Effectivenessand Use in FY03 ofWBl FYOI-02 focusedimpact evaluations. In addition,thisanalysisprovides Activities:A BaselineAssessment in Five Countries lalme Qulzon.Senlor EvaluauonOmcer practical guidance for improvingthe overall effectivenessand 202 458 19I 6 or~qu~zon@worldbankorg utility of WBI activities: Visit our website for more infonnm'onon the EvaluationGmup Theeffectivenessof WBI activitiescan be improvedthrough ofWBl and electronic copies of all EvaluationBriefs: the inclusion of partnerships, country-focused activity hm!/,~.w~@k&~18iwbiiev~!~,@/ designs, action plans and exercises,as well as by targeting activitiesto teams of participants. 7. l'his included English only and English with simultaneous translation to the local language Page4 WBI EvaluationBriefrNavernber 2004