Evaluation of the Quality and Impact of Programs Facilitated by the Global Development Learning Network (GDLN) Helene Jennings Macro International Dawn Roberts Macro International WBI Evaluation Studies No. EG04-88 The World Bank Institute The World Bank Washington, D.C. June 2004 1 FORWARD This independent evaluation report was prepared for the World Bank Institute (WBI) under my overall direction as Manager of the WBI Evaluation Group (IEG), and at the request of WBI Vice President Frannie Léautier. The firm Macro International was awarded the contract for this study following an internationally competitive selection process and in accordance with the World Bank's Consultant Guidelines and Administrative Manual Statement 15.00. An expert panel was held in April 2003 to develop the study's Terms of Reference (TOR). Joining me on this panel were Juan Moreno (Senior Education Specialist, Human Development Network), Eva Baker (Director, UCLA Center for the Study of Evaluation), Christopher Curran (Professor, Dublin City University and retired Director of the National Distance Education Centre in Ireland), Peter Dirr (Consultant and retired President of Public Service Telecommunications Corp.), Ryan Watkins (Assistant Professor, George Washington University) and Maurya West Meiers (IEG Task Manager for this project). Monika Weber-Fahr (Manager, Global Development Learning Network Services) and Michael Foley (Lead Distance Learning Specialist, GDLN Services) participated in sessions of the expert panel, presenting information on GDLN. Jean-Louis Ginnsz (Procurement Advisor/Consultant, Operational Core Services Department) advised us on procurement matters and served on the evaluation committee to select the firm along with Maurya West Meiers, Juan Moreno, Peter Dirr and me. In June 2004 Eva Baker, Christopher Curran and Peter Dirr independently reviewed this study for IEG. The Macro International team presented key findings to the GDLN Management team on May 18, 2004, and to the WBI Extended Institute Leadership Team (EILT) meeting chaired by Frannie Léautier on June 16, 2004. I would like to thank all of the persons who participated in this project, particularly the evaluation team from Macro International, led by Helene Jennings and Dawn Roberts, for their professionalism, flexibility and commitment to this project. WBI Evaluation Studies are produced by IEG to report evaluation results for staff, client, and joint learning events. An objective of the studies is to get the findings out quickly, even if the presentations are less than fully polished. The papers carry the names of the authors and should be cited accordingly. The findings, interpretations, and conclusions expressed in this paper are entirely those of the authors and do not necessarily represent the view of the World Bank Group. These studies are available online at: http://www.worldbank.org/wbi/evaluation/index.html Marlaine Lockheed Manager, Institute Evaluation Group ii ACKNOWLEDGEMENTS This report was prepared for the World Bank Institute (WBI) under a contract for an independent evaluation under the overall direction of Marlaine Lockheed, Manager of the Evaluation Group. A team of evaluators and specialists from Macro International carried out the study. The evaluation team benefited immensely from ongoing communication and support from Maurya West Meiers, Evaluation Analyst. The primary evaluators for this assignment were Helene Jennings and Dawn Roberts, who developed the instruments, conducted the interviews at headquarters and in the field, planned and carried out the site visits, and authored the report. They were ably supported by a number of Macro staff. Kate Goddard competently implemented and monitored the web-based survey, with assistance from Sean Stevens, Taryn Dalton, and Timothy Prand. Bryan Rhodes was especially helpful in cleaning and processing the survey data. Anthony Rodell was invaluable in the formatting of the graphics and the overall look of the final report. Other Macro staff who pitched in as needed were Dianne Fragueiro, Ronaldo Iachan, and John Costenbader. The evaluators gratefully acknowledge the assistance provided at the four site visit locations: Brazil, Senegal, Tanzania, and Vietnam. The DLC managers from those locations again demonstrated their talents and thoughtfulness in arranging for interviews with a wide array of stakeholders during the intensive 5-day visits. Fernando Félix de Oliveira, Mor Seck, Charles Senkondo, and Andrew Scyner graciously answered hours of questions and put their staffs and centers at our disposal. Their insights were especially valued. Interpreters and translators also assisted us in our work, and we appreciate their efforts--especially those of Le Thu Hang who assisted with a number of interviews in Vietnam. Obviously a critical part of the study was the feedback from former participants and partners in the sites visited. The many individuals interviewed were forthcoming with their experiences and suggestions. It was a pleasure to meet them and hear about their activities on behalf of development in their countries. The evaluation also benefited from the many staff members of the World Bank and the World Bank Institute who contributed in innumerable ways to the understanding of GDLN and the collection of qualitative and quantitative data. We appreciate the input from the large number of respondents to the web-based survey and from the many staff members who made time in their schedules to be interviewed at length and/or to answer questions and supply data. We thank other DLC managers who provided additional information from their vantage points through videoconference and telephone. Monika Weber-Fahr, Manager, GDLN Services, encouraged staff and managers to assist with the study and shared her views with the evaluators. We especially want to express iii our gratitude to Michael Foley, who spent considerable time providing his historical perspective and his vision of knowledge sharing and distance learning to the researchers. Finally, we appreciate the support of Frannie Léautier, WBI Vice President to this evaluation and the time she took to discuss her interest in this effort. Helene Jennings, Macro International Dawn Roberts, Macro International Vice President, World Bank Institute Ms. Frannie Léautier Manager, Institute Evaluation Group Ms. Marlaine E. Lockheed Task Team Leader Ms. Maurya West Meiers iv ACRONYMS AFR................Africa Region AMS...............Activity Management System CAS................Country Assistance Strategy CRS................Client Registration System DLC................Distance Learning Center EAP................East Asia & Pacific Region ECA................Europe & Central Asia Region EILT...............World Bank Extended Institute Leadership Team FY01 ..............Fiscal Year 2001 FY02 ..............Fiscal Year 2002 FY03 ..............Fiscal Year 2003 FY04 ..............Fiscal Year 2004 FHA................Federal Highway Administration GDLN ............Global Development Learning Network IEG.................World Bank Institute Evaluation Group ISG.................Information Solutions Group LAC................Latin America & Caribbean Region M&E ..............Monitoring and Evaluation MDG ..............Millennium Development Goal MNA ..............Middle East & North Africa Region MOU ..............Memorandum of Understanding NGO...............Nongovernmental Organization NOC...............Network Operations Center OED ...............World Bank Operations Evaluation Department OPCS..............Operations Policy and Country Services PRSP ..............Poverty Reduction Strategy Paper SAP ................The World Bank's Cost Accounting System SAR................South Asia Region SARS..............Severe Acute Respiratory Syndrome SAS ................Statistical Analysis System TOR................Terms of Reference UNDP.............United Nations Development Programme WBI................World Bank Institute WBIEN ..........World Bank Institute Environment & Socially Sustainable Development Division WBIFP ...........World Bank Institute Finance & Private Sector Development Division WBIHD..........World Bank Institute Human Development Division WBIKP...........World Bank Institute Knowledge and Learning Partnerships WBIPR...........World Bank Institute Poverty Reduction & Economic Management Division WBIRC ..........World Bank Institute Regional Capacity Enhancement v vi TABLE OF CONTENTS EXECUTIVE SUMMARY...............................................................................................XI Introduction and Methodology ................................................................................... xi Perspectives within the World Bank..........................................................................xii Organization and Operation of DLCs: The Managers' Perspective.........................xiii Delivery of GDLN Programming: The Participants' Perspective............................ xiv Findings and Recommendations................................................................................ xv 1. INTRODUCTION............................................................................................................ 1 Background on GDLN................................................................................................. 1 Research Questions...................................................................................................... 5 Organization of the Report .......................................................................................... 5 2. METHODOLOGY .......................................................................................................... 7 3. PERSEPECTIVES WITHIN THE WORLD BANK .......................................................... 11 Understanding of GDLN ........................................................................................... 12 Experience Using GDLN........................................................................................... 19 Perspectives on Content............................................................................................. 27 Impact for Development............................................................................................ 31 Sustaining the Network through Partnerships ........................................................... 36 Critical Factors for GDLN Centers............................................................................ 41 4. ORGANIZATION AND OPERATION OF DLCS: THE MANAGERS' PERSPECTIVE ....... 45 The Nature of Differences ......................................................................................... 45 Common Viewpoints................................................................................................. 48 Conditions for Impact................................................................................................ 53 Accomplishments ...................................................................................................... 54 Key Issues for Enhancing Impact.............................................................................. 55 5. DELIVERY OF GDLN PROGRAMMING: THE PARTICIPANTS' PERSPECTIVE............... 67 What is Known about Participants ............................................................................ 67 Feedback on GDLN Activities .................................................................................. 69 Results of the GDLN Experience .............................................................................. 73 Key Issues for Enhancing Impact.............................................................................. 78 6. FINDINGS AND RECOMMENDATIONS........................................................................ 81 Overall Conclusion.................................................................................................... 81 Supports for GDLN Centers...................................................................................... 81 vii Issues Related to Content........................................................................................... 87 Quality Assurance...................................................................................................... 89 Monitoring and Evaluation........................................................................................ 90 Promoting the Network.............................................................................................. 92 Addressing the Research Questions........................................................................... 94 viii LIST OF FIGURES Figure 1: Importance of Achieving Various Objectives Through GDLN Events....... 16 Figure 2: Clarity about Whom to Contact with GDLN-related Questions.................. 18 Figure 3: Number of Activities Conducted by GDLN Users During FY04................ 20 Figure 4: Level of Ease or Difficulty for GDLN-related Tasks.................................. 22 Figure 5: Compelling Reasons to Use GDLN............................................................. 24 Figure 6: Reasons to NOT Use GDLN in the Future.................................................. 26 Figure 7: Where Respondents Assign Responsibility for Quality Assurance............. 29 Figure 8: Themes in AMS, Compared to World Bank Themes.................................. 33 Figure 9: Alignment of GDLN Programs or Activities............................................... 34 Figure 10: Factors for Development Impact ................................................................. 35 Figure 11: Perceived Incentives for Organizations to Become Partners....................... 39 Figure 12: Perceived Barriers for Organizations to Become Partners .......................... 40 Figure 13: Ten Most Important Conditions for Distance Learning Centers ................. 42 Figure 14: Factors in Effective and Efficient DLC Operation ...................................... 50 Figure 15: Most Important Factors for Development Impact ....................................... 54 Figure 16: How DLCs Contributed to Development Dialogue in FY04....................... 55 Figure 17: Challenges to Strengthening Development Dialogue in FY04.................... 56 Figure 18: Motivation for Enrollment in Course/Activity ............................................ 69 Figure 19: Valuable Components in Course/Activity ................................................... 70 Figure 20: Participants' Assessment of Quality and Relevancy ................................... 73 Figure 21: Aspects of Learning Event Where Most Effective for Professional Development................................................................................................ 74 Figure 22: Work Accomplished as a Result of DLC Learning Event........................... 75 Figure 23: Assessment of Capacity Enhancement by Participants ............................... 76 LIST OF TABLES Table 1: Number of GDLN Learning and Knowledge Exchange Offerings, by Year................................................................................................................ 2 Table 2: Numbers of Respondents Who Were Approached and Established Partnerships, by Type of Organization ........................................................ 38 Table 3: Utilization Rates--FY03 ............................................................................. 48 Table 4: Clarity in Knowing Whom to Contact......................................................... 59 Table 5: Ease of Use of GDLN Tools........................................................................ 60 Table 6: Sources of Revenue for DLC Managers...................................................... 64 Table 7: Agreement with Statements About Future DLC Attendance ...................... 72 LIST OF ANNEXES Annex A: Methodology........................................................................101 Annex B: Web-based Survey..................................................................111 Annex C: Protocols..............................................................................131 ix Annex D: Participatory Survey...............................................................141 Annex E: Measuring Thematic and Sectoral Alignment...................................147 Annex F: Geographic Distribution of GDLN Centers......................................149 x EXECUTIVE SUMMARY INTRODUCTION AND METHODOLOGY 1. The World Bank Institute (WBI) awarded a contract to Macro International to undertake an evaluation of the quality and impact of programs facilitated by the Global Development Learning Network (GDLN). This study was conducted from October 2003 to June 2004. Background on GDLN 2. GDLN is an international network for learning and knowledge sharing that has grown dramatically since its launch in June 2000: in June 2004, there were 68 GDLN Centers. The network is a partnership of public, private, and non-governmental organizations and is dedicated to poverty reduction and sustainable development by facilitating knowledge and capacity-building programs. 3. The activities provided through GDLN include courses and seminars, global dialogues, and web-based courses. These activities encompass a range of formats, with some occurring as part of a series while others are one-time events. For most, two-way videoconferencing has served as a key tool for enabling multi-site and instant exchange among participants. Although the GDLN management is housed largely within WBI in the office of GDLN Services, the network is a broad development initiative that extends beyond WBI offerings: of the 673 GDLN learning and knowledge exchange offerings during FY04, 207 were provided by WBI and the rest (466) were provided by either other Bank units or external partners. Study Design 4. In accordance with the Terms of Reference, this study relied on both qualitative and quantitative data to address four research questions: · What is the effectiveness and impact of the programs offered through GDLN? · What are the conditions leading to greater distance learning center (DLC) effectiveness? · What is the congruence between content and use of GDLN and the World Bank goals and Millennium Development goals? · What are the capacity enhancement effects of GDLN? 5. Field work for this study was conducted during the second half of Fiscal Year 2004. Major data collection activities included a review of GDLN-related documents and xi systems data, semi-structured interviews with 32 World Bank employees at headquarters and in the field, a web-based survey of 221 World Bank personnel and 43 distance learning center managers, site visits to four distance learning centers, and a videoconference focus group with 8 DLC managers and coordinators from centers based at universities. Site visit activities included touring the facilities; observing GDLN activities; reviewing center data regarding utilization, activities, and participants; and interviewing key stakeholders such as the center manager and staff, participants, and local partners. Finally, 58 of the participants who were interviewed also completed a closed-ended survey. 6. The study looked at three perspectives--stakeholders at the World Bank, DLC managers, and participants--and provided findings and recommendations in concert with the research questions. PERSPECTIVES WITHIN THE WORLD BANK 7. Perceptions about GDLN at the World Bank greatly inform the research questions since these various stakeholders represent not only those who manage and provide support services for the network but also those who foster partnerships and those who are active or potential users of the network. This evaluation therefore explored experiences and attitudes at the Bank related to six areas: the understanding of GDLN, experience using the network, perspectives on content, impact for development, partnerships, and critical factors for GDLN Centers. 8. Understanding of GDLN: Survey and interview data revealed that there was no clear shared vision of what GDLN is or could become among Bank staff. Respondents expressed a range of opinions about what the purpose of the network was and also reflected uncertainty about how the network was governed and who had responsibility for specific roles. Among those who had experience using GDLN, WBI content providers were particularly likely to express confusion about whom to contact to schedule a GDLN event, to resolve technical difficulties, or to develop partnerships. 9. Experience Using GDLN: Active users expressed a high level of satisfaction with the network: of those survey respondents who used GDLN during FY04, nearly all (89%) rated their experience as "good" or better. GDLN users at the Bank also reported that the tasks required to design and implement a network event could be completed with relative ease and that there were compelling reasons to use the network more in the future. Those incentives cited by a majority of respondents included that GDLN enables experts in other locations to contribute to the event and that using the network reduces travel costs. In general, respondents were less likely to identify reasons for not using the network in the future, and no specific disincentive for network use was identified by a majority of respondents. 10. Perspectives on Content: Bank staff familiar with GDLN generally recognized the network to be a vehicle for delivering content rather than a developer of curricula. At the same time, many noted that having sufficient high quality content to meet local demand is a critical network issue. Content providers at the Bank who use GDLN lack xii clear incentives to effectively adapt content to address local needs through distance learning, and a majority of respondents agreed that content delivered through GDLN tends to be supply-driven. Although GDLN stakeholders consistently identified quality as an important determinant for effectiveness and impact, no consensus existed as to who should have ultimate responsibility for quality assurance. 11. Development Impact: Bank staff often referred to GDLN as a public good with its intended use being as a vehicle for development. Some expressed concern that the mandate for centers to be financially self-sustaining undermined the focus on development. Others believed that there was an opportunity to tap into the development business through partnerships with donor agencies and others in the development community. To date, GDLN management has not provided central guidance regarding how curricula should promote development impact or whether programming should be aligned with World Bank goals or standard development measures across regions. 12. Sustaining the Network through Partnerships: World Bank staff believed that partnerships were critical for the future of GDLN, but the roles for establishing these are not clearly defined. Much of the confusion surrounding partnerships has resulted from the variety of types of partners and the diversity of GDLN centers and their needs. Those involved in partnership development identified clear incentives for organizations to collaborate with GDLN, and clarity about roles and how to pursue and develop partnerships was seen as critically important for enhancing network capacity. 13. Critical Factors for GDLN Centers: Although many factors were identified by stakeholders as important for a distance learning center to be effective, the most critical of these is the leadership of the DLC manager. Other conditions noted to be among the most important include capabilities for marketing GDLN events and having a consistent quality assurance system for content. ORGANIZATION AND OPERATION OF DLCS: THE MANAGERS' PERSPECTIVE 14. Distance learning centers (DLCs) in the network adhere to certain conventions and expectations, but they also represent great diversity depending on the country context, date of inauguration, conditions of establishment and direction provided by the DLC manager and staff. Site visits to four centers and survey responses from 43 DLC managers provided important insights into how centers are organized and how they operate. 15. The Nature of Differences: GDLN centers operate within seemingly unique environments, and several local factors influence quality and impact. Key factors that vary widely among centers include the center location or affiliation, the business model, the mix of activities offered and technologies used, the links with the local development community, and the marketing style of the manager. 16. Common Viewpoints: Despite the diversity among centers, many similarities exist related to the role GDLN plays in the country, the challenges faced by managers, and the future directions envisioned. DLC managers themselves and all other GDLN xiii stakeholders agreed that managers are key for center operations. Other commonalities across centers included the importance of effective local marketing efforts, the high administrative burden of setting up GDLN courses or dialogues, and the interest in broader dissemination through subregional networks. 17. Conditions for Impact: DLC managers shared similar views with GDLN stakeholders at the Bank about what conditions are needed for the network to most effectively contribute to the development agenda. The factors viewed to be most critical included the overall quality of the activity, the relevance of the specific thematic area, having the appropriate experts and facilitator, and involving the right participants. 18. Accomplishments: DLC managers reported how their centers had contributed to their countries' development dialogues during this fiscal year. Among the most common contributions cited were that centers had formed new partnerships in the development community and had increased their ability to develop local content. 19. Key Issues for Enhancing Impact: Responses from managers also illuminated what challenges currently limit the impact of GDLN. These impediments related to getting appropriate content to meet local demand, ensuring the overall quality of activities, relying on technology, receiving sufficient support from GDLN Services, successfully developing partnerships, integrating monitoring and evaluation practices, and achieving financial sustainability. DELIVERY OF GDLN PROGRAMMING: THE PARTICIPANTS' PERSPECTIVE 20. Information from past participants in GDLN activities was collected through interviews and surveys during site visits. The participants contacted constitute a convenience sample and cannot be said to be representative of the group as a whole. Their experiences and views do serve to illuminate key considerations for designing and implementing DLC events to better enhance capacity and strengthen impact. 21. Participant characteristics: Limited statistics are kept by DLCs about the background of their participants. Among those in the convenience sample interviewed and surveyed for this study, about two-thirds had participated in multiple DLC events and most (88%) had participated in their most recent event with other members of their organization or agency. These participants tended to be highly educated--with more than half having a master's, professional, or doctoral degree--and their primary motivation for attending the event was to learn new techniques and strategies. 22. Feedback on GDLN Activities: Participants identified the most valuable components of learning activities to be the contributions of other participants and the use of expert presenters. They also reported a high level of satisfaction and assigned high ratings for the quality and relevance of the events they had attended. 23. Results of the GDLN Experience: Activities delivered through GDLN were credited most often with updating knowledge or providing new information. Participants also commonly noted that the events provided contact with experts, enhanced skills, provided new strategies, and facilitated work with colleagues. When asked how they xiv used their GDLN experience, more than half reported that they transmitted new knowledge or skills to others in their organization. They also usually indicated that the GDLN activity had enhanced their own professional development, and to a lesser extent, the capacity of their organizations and even the development of their countries. In some cases, participants provided tangible examples to illustrate the impact of the GDLN program on their work. 24. Key Issues for Enhancing Impact: The translation from learning gains to changes in public policy or institutional capacity was only apparent in a few participant experiences, but those interviewed provided clear suggestions for how to improve effectiveness and increase impact. Courses and activities need to be strengthened--in length, depth, and format. The right participants need to be targeted. Centers need to increase their visibility. Participants need to continue the dialogues that began at the centers through communities of practice. The content offered needs to be relevant to address local demands, and where appropriate, needs to be presented in the local language by high-quality interpreters. Finally, evaluation practices need to be integrated into center operations to allow for continuous improvement. FINDINGS AND RECOMMENDATIONS 25. The GDLN is still in a developmental phase while the concept evolves through business and strategic planning processes. There is not a clear shared vision at the Bank for what the network is or could become. To date, the necessary steps have not been taken to transform GDLN into the knowledge-sharing vehicle of choice for the development community, but GDLN and World Bank management could play a critical central role to help this happen. Findings and recommendations are provided in the body of the report for refining network supports to centers, addressing issues related to content, assuring quality in content and operations, integrating evaluation practices, and actively promoting the network both within the Bank and to external partners in the development community. This study's main conclusions for the four research questions are below: 26. Effectiveness and Impact: Programs facilitated by GDLN have contributed to development in a limited number of cases, but this contribution has been constrained by uncertainties about roles and the governance of the network itself. A clear network philosophy needs to be established regarding effectiveness and impact as GDLN continues to build partnerships, add centers and subregional networks, and facilitate a growing number of meetings or dialogues with impromptu content. Practices regarding appropriate content, effective pedagogy, and the selection of participants directly influence the effectiveness of programs. Based on these findings, concrete recommendations for enhancing impact are included in the report. 27. Conditions for Greater DLC Effectiveness and Efficiency: The critical ingredient for a center's success is the leadership of the DLC manager. Given that each manager must determine the optimal mix of content, skills and knowledge exchange activities that will support a financially viable center, successful managers should have more opportunities to share their strategies with others in the network. Virtually all managers reported looking to GDLN operations in Washington for procedures, systems, xv and tools--and most did not feel supported by these services. Although many DLCs were struggling with the requirement for financial sustainability, managers saw this as a function of needing to generate revenue rather than of excessive costs for content or connectivity. Overall, a variety of conditions influence how well a GDLN center operates: some of these are under the control of the DLC manager and others are a product of the start-up terms, quality of the national infrastructure, economic and political circumstances, and the nature of and access to the development community in a particular country. 28. Alignment with World Bank Goals and Millennium Development Goals: WBI and GDLN management must clarify whether aligning GDLN content with World Bank goals or MDGs across all regions is part of the overall strategy of the network. While GDLN stakeholders agree that the network should be used for development, there is currently no central guidance that activities or results should focus on specific goals or measures. This absence of any push to align activities with standard international development objectives is arguably part of a network philosophy of decentralization, where centers themselves are likely in the best position to assess how to most effectively contribute to the development dialogues of their countries. However, this philosophy may be in conflict with efforts to transform GDLN into the knowledge-sharing vehicle of choice in the development community. 29. Impact on Capacity Enhancement: The research conducted for this study did not reveal any instances where programs delivered via GDLN resulted in capacity enhancement at the national level. However, the nature of knowledge transfer and knowledge sharing occurring through GDLN has a strong potential for achieving capacity building for institutions in that a team from an organization may attend together, providing a critical mass. Real outcomes related to capacity can only be achieved for GDLN through a more deliberate network strategy to link activities to goals, identify and target the right participants, and support a continuous process of capacity enhancement rather than an array of isolated events. In addition, a successful strategy would include an increased focus on country capacity needs and a coordinated effort with country teams and other Bank specialists on targeted problems. xvi 1. INTRODUCTION 1.1 The World Bank Institute (WBI) conducted a competition among consulting firms and awarded a contract in October 2003 to undertake an evaluation of the quality and impact of programs facilitated by the Global Development Learning Network (GDLN). GDLN is an international network for learning and knowledge sharing that was formally launched by the World Bank in June 2000, and has grown dramatically since that time. The designated contractor, Macro International of Calverton, Maryland conducted that evaluation from October 2003 to June 2004. This volume constitutes the comprehensive reporting of that study. BACKGROUND ON GDLN 1.2 The mission of the World Bank Institute is to "create learning opportunities for countries, Bank staff, and clients, and people committed to poverty reduction and sustainable development." WBI supports the learning and knowledge agenda by providing learning programs and policy advice in the areas of environmental and national resources; economic policy for poverty reduction; governance, regulation, and finance; and human development. WBI uses a variety of formats to deliver learning programs. Since its formal establishment, GDLN has been an important vehicle for delivering structured learning events as well as workshops and meetings. GDLN was an outgrowth of the World Bank and WBI's decision to meet future learning challenges using communications technology; promoting information networks; and involving partnerships with donor agencies, private foundations, and international and local organizations. The GDLN partnership is dedicated to poverty reduction and sustainable development by providing knowledge and capacity development programs. They provide the content, technology and financial and other support to operate this network. In June 2004, there were 68 centers in the network. 1.3 Since its inception, the Network has conceptualized the learning activities delivered into four different categories: 1) courses, which usually take place over time-- with a number of sessions spread over one or multiple weeks, 2) seminars or conferences, which are shorter in duration and less structured and formal than courses although they also have multiple sessions, 3) global dialogues, which are short videoconference sessions focusing on specific issues, knowledge exchange or analyses on a wide range of policy issues or development topics, and 4) web-based programs which use the Internet for delivery of synchronous or asynchronous learning. 1.4 Several other key distinctions in the nature of distance learning through these auspices are that some activities are one-time events (e.g., global dialogues) and others are offered as a series; some are focused on one country, while others are multi-site, either regional or international. In addition, some countries are able to use GDLN centers to connect their own networks to GDLN events; for example, in China, GDLN Distance 1 Learning Centers are able to connect to a larger distance learning network of China's universities. Beyond these four types of learning opportunities, GDLN also tracks knowledge exchange offerings, such as meetings and workshops. 1.5 GDLN reports that 673 activities took place through GDLN in FY04: almost 60 percent were formal learning events and courses and the rest were meetings and workshops via videoconferencing. Table 1 depicts the distribution of these different events by type and year, where the information is available. Table 1: Number of GDLN Learning and Knowledge Exchange Offerings, by Year1 Year Courses Seminars/ Global Web-based Knowledge Conferences Dialogues Courses Exchange Total FY01 82 13 173 5 N/A 273 FY02 72 70 85 2 122 351 FY03 70 98 135 16 260 579 FY04 100 44 249 1 279 673 1.6 The data for FY01 were reported in a World Bank Operations Evaluation Department (OED) study on the first two years of GDLN operation2 and did not break out data in terms of knowledge exchanges--i.e., meetings and workshops. (They may be included in one of the other categories; the breakdown in types also may not have been defined in the same way for the first year.) Regardless of how activities delivered through GDLN are categorized, the totals indicate that the number of activities has increased in each year of operation. (Of course, the number of centers has increased from 20 in FY01 to 68 at the end of FY04.) Looking at the last three years is instructive in terms of the patterns of increase. The numbers of courses increased considerably in the most recent year, but the number of seminars and conferences decreased. The number of global dialogues grew dramatically--almost doubling--from the previous year. Web- based courses had virtually no activity. The knowledge exchange offerings were fairly flat in FY04 over the previous year, but in FY03 they had more than doubled over the prior fiscal year. 1.7 The initiators of this range of activities can be WBI, other World Bank units, or external partners. The general pattern shown by the last three years' data is that WBI tends to use GDLN for learning activities, the other Bank units tend to use it for knowledge exchange, and the external partners take advantage of the network for both purposes. For FY04, WBI produced 41 percent of the learning offerings (up from 37% the previous year--generally due to the large increase in global dialogues), and external partners provided 42 percent of the learning offerings (with an especially high percentage of the seminars). On the other hand, the Bank took advantage of the network for 1The statistics for FY02-FY04 are supplied by GDLN services and based on their use of the more reliable videoconferencing scheduling system (RendezVous) and adjusted numbers of actual unstructured workshops and meetings. 2Alex Romiszowski (2003). "The Global Development Learning Network: A Review of the First Two Years' Operation." OED Background Paper. Washington, D.C. 2 meetings (40% of the knowledge exchange activities were by other Bank units), as did the external partners, which initiated 43 percent of the knowledge exchange events. 1.8 GDLN is an international system, made up of a diverse set of distance learning centers that have the capacity to connect to Headquarters in Washington, D.C. (the telecommunications hub) and to other centers around the globe (Chapter 4 describes the centers and their variations in more detail). When GDLN began, the World Bank had a comparative advantage through its high-capacity communication network that linked Bank field offices. 1.9 The current network is designed for transmission of video, voice, and data through a variety of technologies including satellite/VSAT (which provides a direct, reliable connection to the network--and is the means of connection from Africa), ISDN (which provides a dial-up connection--and is commonly used in Latin America and Eastern Europe), fiber optic (used to connect from British Council offices, Paris, and the Brasilia Bank office), and Internet Protocol, which provides more efficient use of bandwidth. The network is in the process of converting to IP where feasible. Importantly, there are different cost implications to users for each of the different technologies employed in the GDLN system. This contributes to difficulties in knowing and negotiating costs for content providers and DLCs, as highlighted in other parts of this report. 1.10 Advances in technology permit the inter-connection of independent centers worldwide, but the greater potential contribution of GDLN is in the development of curriculum that delivers new, current and cutting-edge knowledge and employs pedagogy from adult learning theory. Good instructional design takes into account what is known about preferred training approaches for adults: where courses or events are based on established learning needs with clear statements of outcomes, activity-based, experiential, participatory, interactive, and have facilitated learner support. 1.11 All these considerations frequently result in a blended approach for delivering knowledge services through synchronous videoconferencing and asynchronous materials and interactions using the Internet and print media. In many of the learning offerings there is an emphasis on group learning, often a focus on developing an action plan or local strategy paper, a high level of interactivity with reporting from the different participating countries, local facilitation to make the application more relevant to each audience, and follow-up supports. 1.12 Many benefits of this distance learning approach have been enumerated-- working adults are able to participate in courses or dialogues and network actively with like-minded professionals while staying on-site at their jobs and applying what they learn immediately. For some it is a new way of learning and a new way of doing business.3 During this phase, while the GDLN is still relatively new but developing and growing 3The value proposition of comparing the costs of sending professionals out of the country for face-to-face training with experts is also often cited. 3 rapidly, there is not yet a clear understanding of the range of variables that influence whether GDLN activities achieve their intended objectives. 1.13 Ideally, programs facilitated by the GDLN result in learning gains for their participants who then change their behavior and ultimately, over time, have an effect on development indicators in their countries. In reality, the efficiency and effectiveness of Distance Learning Centers (DLC) in delivering education services is dependent on an array of factors that vary widely between countries and regions. It is well known that DLC managers and World Bank country office staff deal with sharp variations in technology infrastructure depending on their locations. Differing business models tied to the funding of the centers' original establishment and differing cost recovery requirements significantly influence operations. 1.14 Content needs also differ: those in Russia might emphasize small and medium enterprise development while those in East Asia focus on information related to Severe Acute Respiratory Syndrome (SARS). Regions and some individual countries have gone through a needs assessment process to establish priorities of content development or acquisition. Common cross-cutting programs focus on attacking poverty, social protection, and good governance. Other substantive areas frequently emphasized are education, health (especially HIV/AIDS), environment, investment and finance. 1.15 The Bank's costs for the Network run about $10 million per year.4 In its four years of operation GDLN has moved from a focus on start-up to what is considered a process of consolidation. GDLN is looking at a transition towards credibility and sustainability. Concerns about governance, disparities among Distance Learning Centers (DLCs), low videoconferencing utilization rates, and future technology changes have led to a number of initiatives, presumably including this evaluation that is being directed by the WBI Evaluation Group. During the course of the evaluation study, the system and its governance have been evolving: several changes have been made in the organizational management, major revisions were set forth in the FY04 business plan, a collaborative process developed a strategic plan for FY05-FY07, new supports to centers such as a toolkit were announced, new reporting systems were designed, and improvements were underway for existing administrative systems. 1.16 A significant change developed through the strategic planning exercises by GDLN is the proposed evolution of the network from a "videoconference center" to a "development resource center." Recognizing that it is increasingly difficult--if not impossible--to centrally deliver and broker enough training courses, the GDLN unit is looking to broaden the constituency for operating, supporting, and using the network to 4 The GDLN FY04 Business Plan puts forth $10 million as the cost for GDLN teams in the Regions and WBI in FY03 (p. 22), and notes that the World Bank's reporting systems do not systematically capture information on expenditures related to GDLN. This amount covers services and support to the operation and growth of the Network and individual GDLN Centers, and investments to set up GDLN Centers in Bank offices. This amount does not take into account the fees that are paid by World Bank users (content providers and others) for use of the system. The plan also states that WBI's GDLN budget has remained flat since FY02 at $4.1 million. 4 others in the development community. GDLN management proposes to change the Bank's role in the network to a convener and advisor. RESEARCH QUESTIONS 1.17 The Terms of Reference (TOR) that guided this empirical study set forth four objectives that have been converted into research questions. · What is the effectiveness and impact of the programs offered through GDLN? · What are the conditions leading to greater DLC effectiveness? · What is the congruence between content and use of GDLN and the World Bank goals and Millennium Development goals? · What are the capacity enhancement effects of GDLN? A more detailed presentation of the research objectives put forth in the TOR may be found in Annex A. ORGANIZATION OF THE REPORT 1.18 This comprehensive report on the evaluation on the quality and impact of GDLN programs is organized in six chapters. In addition to the first two chapters (Introduction and Methodology), the next three discuss the various perspectives of the Network: that of the World Bank (Chapter 3), the managers on the organization and operation of DLCs (Chapter 4), and the participants on delivery of GDLN programming (Chapter 5). The final chapter presents findings and recommendations that respond to the research questions. 5 6 2. METHODOLOGY 2.1 This independent evaluation of the quality and impact of programs facilitated by GDLN relied on both qualitative and quantitative data to address the research questions presented in the previous chapter. A brief overview of the study methodology is presented here, with a more detailed description provided in Annex A. 2.2 To gain the perspectives of numerous types of stakeholders and systematically collect data on a range of critical factors, the evaluators relied on a mixed method approach. Comments and reflections were gathered through semi-structured interviews, and closed-ended quantifiable responses were compiled through two separate surveys. Field work for this study was conducted during the second half of Fiscal Year 2004, and major data collection activities included the following: · Review of GDLN Documents and Systems Data: Existing publications and program data provided a contextual and historical understanding of GDLN. This document and data review continued throughout the study, especially given the development of critical program elements such as a new strategic plan and revised business policies. · Interviews with Bank Staff: Semi-structured interviews were conducted with 22 World Bank employees at headquarters and 10 more from the field who had a variety of roles related to GDLN. Interviews relied on an open-ended protocol but also allowed for stakeholders to raise and explore issues they believed were critical. Most of these sessions lasted 45 minutes to one hour, with telephone follow-up in some cases to ask for clarification or learn about programmatic changes during the study. The interview protocol for these sessions is included in Annex C. · Web-based Survey of World Bank Personnel and DLC Managers: This survey was fielded in two waves. The first included a sample of mainly GDLN users and the second targeted a sample of those accredited to manage trust funds with the explicit intent of gaining more respondents outside of WBI, particularly those in operations at the task manager level. Overall, 264 individuals responded to the survey, representing 60% from the first wave and 17% from the second. Survey respondents represented many roles, including members of GDLN Services, content providers, GDLN regional teams, WBI instructional design staff, WBI managers, DLC managers, and others. In addition, survey questions spanned a broad range of 7 variables related to quality and impact, including center operations, content development, contributions of GDLN related to country development needs, marketing, and technology among others. A copy of the questionnaire with response frequencies is included in Annex B. A further discussion of the survey fielding and questionnaire design is included in Annex A. A brief overview of roles within WBI and the World Bank related to GDLN is presented in the introduction of the next chapter. · Site Visits to GDLN Centers: Site visits were conducted to distance learning centers in Brazil, Senegal, Tanzania, and Vietnam. The rationale for site selection is included as part of the complete methodology in Annex A. Site visit activities included interviewing DLC managers, DLC staff, local partners, participants of prior center events, key employers, and other stakeholders. Many of the 108 individuals interviewed had multiple roles. An interpreter was relied upon as needed to communicate in the appropriate local language. In some sites, focus groups were held so that participants could actively exchange ideas with each other. In most cases, participants who were interviewed also completed a closed-ended survey questionnaire which is described below. Although the GDLN center at Chulalongkorn University in Bangkok, Thailand, was not available for the proposed site visit during the period of study, a one-hour telephone call was conducted with the DLC coordinator using key questions from the DLC manager protocol. · Survey of Participants: Participants who had attended prior GDLN events were interviewed and then asked to complete a questionnaire if appropriate. The participant survey was translated into the local language and included a series of items to identify an individual's satisfaction with a GDLN event or series and how he or she used the experience in the weeks, months, or years following it. Completed questionnaires were submitted by 58 past participants. · Videoconference Focus Group: In order to further explore issues of DLCs located at universities, a 2-hour focus group was held via videoconference technology with a total of 8 DLC managers and coordinators from 7 different countries in the Latin America and Caribbean region. An abbreviated DLC manager protocol was used, highlighting key questions about the influence of the university setting on administration, operations, resources, marketing, and sustainability. The session was conducted in Spanish. 2.3 Qualitative data from the interviews and from the open-ended survey items were coded to identify dominant themes related to key areas. The quantitative data from the two surveys were entered into the Statistical Analysis System (SAS) for coding and analysis. Responses were analyzed for different types of respondents, with major 8 subgroups including GDLN Services team members, WBI content providers, GDLN regional teams in World Bank Operations, task managers in Operations who are not members of the GDLN regional teams, and distance learning center managers. Respondents were also classified based on whether they had conducted GDLN-related work for specific regions. Unless explicitly noted otherwise, differences in subgroup responses have only been included in this report where they were statistically significant at the p<.01 level and where there were sufficient numbers of respondents in each category to support valid tests. 2.4 Where survey items are referred to in the following chapters, the question numbers for the items cited are included as footnotes. In addition, the complete response frequencies and the number of respondents are listed for each item in the survey questionnaires, located in the Annexes. 9 10 3. PERSPECTIVES WITHIN THE WORLD BANK 3.1 An important starting point for looking at issues of quality and impact of GDLN is the World Bank staff. GDLN stakeholders at the World Bank headquarters or country offices are not limited to those who manage and provide GDLN support services directly. They also include those who foster partnerships; those who are active or potential users of the network in the course of lending, educating, and supporting development projects; and those who provide technical or financial support. Perceptions about the network from these various stakeholder groups greatly inform the research questions since these various roles do at times directly influence the impact and effectiveness of programs delivered through GDLN. 3.2 GDLN Services is a division within the World Bank Institute (WBI), headed by a Manager, who reports directly to the Vice President of WBI. The unit includes sections on marketing and business development, activity services, advisory services, and studio services among others.5 The GDLN Manager also heads the GDLN Management Team which was designed to coordinate the work of the different GDLN teams outside of WBI who are critical to the operation of the network--the World Bank's regional departments and the Bank's Information Solutions Group (ISG)--with GDLN Services in WBI. Other major GDLN stakeholders include task managers or unit managers either in WBI or in World Bank Operations who have provided content using GDLN or who are a potential source of content. The responsibilities of still others include developing partnerships involving GDLN among other Bank initiatives--for example, WBI staff in Knowledge and Learning Partnership or those in Regional Capacity Enhancement. 3.3 Perspectives about GDLN within the Bank were explored during this study to provide a contextual understanding of how the Global Development Learning Network (GDLN) has evolved and to what degree the network is now viewed as an effective vehicle for development among Bank staff. To this end, the evaluation included in-depth semi-structured interviews with a range of Bank staff combined with a web-based survey to systematically collect and quantify responses. Respondents in these data collection efforts included not only the range of GDLN stakeholders described above but also the managers of distance learning centers (DLC) in the network. Although most of these 5 The reorganization of GDLN Services and related units was in progress during the course of this evaluation. In February 2004, the decision was made to move the instructional design team into GDLN Services. This is an important component in continuing to promote GDLN use within the Bank and WBI. 11 managers are not Bank employees, they tend to interact frequently with GDLN Services, WBI content providers, and others at headquarters, and they provide further insights into how GDLN is used and perceived by the World Bank.6 UNDERSTANDING OF GDLN 3.4 There is currently no shared vision among Bank staff about what GDLN is or could become. The effect is that full support and utilization within the Bank is weakened. The GDLN has not been integrated into Bank practices where it might intelligently be applied. This overall finding was derived from interview and survey data that explored perceptions among Bank staff as to what the intended mission of GDLN is, what a typical GDLN event includes, and what the critical roles are for effective network operation. Overall Purpose 3.5 Initial interviews at headquarters included a question specifically asking what the purpose of GDLN was and what type of activity qualified as a "GDLN event." In general, those with more extensive experience using or marketing the network were more likely to view it as a comprehensive vehicle for learning, where videoconferencing was one technique integrated into a broader blended approach. At the other end of the spectrum, some personnel asserted that GDLN simply represented a technology tool available for conducting Bank business and that the network should not have any explicit mission of its own. The diversity of responses among Bank staff is best reflected in the following quotes representing commonly heard themes: · "GDLN is a knowledge-sharing tool, a broker of content for development. At four years, the network has not yet reached its cruise speed, but there is strong potential." · "GDLN evolved out of a need for an established booking and scheduling system. Distance learning started to take effect and the World Bank needed its own network." · "The primary purpose is to facilitate the exchange of knowledge and expertise among development practitioners (change agents) in a way that is cost-effective and learning-effective." · "GDLN is a technical tool, just refers to the infrastructure... GDLN should be understood as a means to provide technical support to content providers." 6 In-depth interviews were conducted with 22 individuals at headquarters and 10 additional World Bank personnel from the field. In total, 264 individuals responded to the web-based survey, representing World Bank staff and Distance Learning Center managers. A more detailed description of the study methodology and respondent characteristics is included in Chapter 2: Study Methodology and in Annex A: Detailed Methodology. 12 · "The network is a delivery modality to reach more people, get the knowledge out there. Ideally, GDLN centers in countries would strengthen country capacity, serve as training of trainers model." 3.6 As with views on the purpose of the network, characterizations of a "GDLN event" embraced a similar range--with some assigning this status to any activity having a videoconference booking and others being much more specific. For example, one respondent from the field emphasized that GDLN events include more than two parties and that a simple bilateral meeting conducted through videoconference would not qualify. Some respondents who had extensive roles related to GDLN described a progression of the network over time, where GDLN had evolved from being essentially a training channel to disseminate information from WBI to more of a vehicle for knowledge- sharing. These individuals reflected that GDLN events had formerly been predominantly courses delivered using videoconference and were now increasingly development dialogues or meetings where participants could exchange knowledge and build capacity. 3.7 To support further analysis of differing views among Bank staff, the survey included the following set of three statements related to the understanding of GDLN and respondents were asked to indicate their level of agreement for each. (a) "The GDLN is an ideal delivery modality to disseminate development knowledge to more people."7 (b) "The GDLN is primarily a tool to deliver learning activities through videoconferencing."8 (c) "The GDLN offers a comprehensive system to deliver distance learning through a blended approach involving a combination of distance learning, face-to-face, e-learning or other pedagogical methods."9 3.8 The first statement focused on whether GDLN was an effective vehicle for transferring knowledge out from the Bank or facilitating an exchange of information, and most respondents either strongly agreed (34%) or agreed (53%) that "the GDLN is an ideal delivery modality to disseminate development knowledge to more people." Some significant variations in responses occurred depending on the individual's role at the World Bank and the regions for which he or she had conducted GDLN-related work:10 · Respondents were significantly more likely to agree or strongly agree with this statement if they were a member of GDLN services or a DLC manager. 7Q9A: Responses were based on a four point scale--Strongly agree (34%), agree (53%), disagree (12%), and strongly disagree (1%). 8Item Q9B: Strongly agree (21%), agree (54%), disagree (23%), and strongly disagree (2%) 9Item Q9C: Strongly agree (33%), agree (52%), disagree (15%), and strongly disagree (0%) 10Differences are only noted in this report if they are statistically significant at the p<.01 level. 13 · Those in World Bank Operations who were not members of the GDLN regional teams were less likely to strongly agree with this item than those situated within the World Bank Institute (WBI). · Those respondents who had conducted GDLN-related work for the East Asia and Pacific region or the Latin America and Caribbean region (or both) were more likely to agree or strongly agree.11 3.9 The second statement--"the GDLN is primarily a tool to deliver learning activities through videoconferencing"--was intended to reflect a relatively narrow definition of the network. Three-quarters of respondents either strongly agreed (21%) or agreed (54%) with this item, and there were no significant differences in responses based on role, affiliation within the Bank, regional affiliation, or any other subgroup. 3.10 The third item in this series reflected more of an all-encompassing definition of the network, stating that "the GDLN offers a comprehensive system to deliver distance learning through a blended approach involving a combination of distance learning, face- to-face, e-learning or other pedagogical methods." Most respondents either strongly agreed (33%) or agreed (52%) with this description, and an analysis of subgroup responses revealed only one significant difference related to roles, regions, or affiliations within the Bank: · Those respondents who identified themselves as content providers (including task team leaders and program team members) were less likely to be in agreement with this statement--regardless of whether they were situated in World Bank Operations or in WBI. 3.11 Overall, a majority of respondents agreed or strongly agreed with all three of these statements. Those who agreed or strongly agreed that the GDLN "is an ideal delivery modality" were significantly more likely to also agree that "the GDLN offers a comprehensive system"--but many of these same respondents also agreed that "GDLN is primarily a tool." In general, respondents saw no inherent conflicts between these different statements, and 62 percent indicated agreement with both concepts of GDLN: one as a tool and one as a comprehensive system to deliver distance learning. Priorities for Using the Network 3.12 Because World Bank staff expressed a range of opinions in interviews about the appropriate uses for GDLN, the survey listed a series of six priorities for GDLN use and asked respondents to rank them. These items are shown below with the percentage of respondents who ranked them among the top three priorities:12 11Many respondents are linked to multiple regions since they were asked to indicate all regions where they had conducted GDLN-related work. Responses are therefore dichotomized for the analysis by region, with all those linked to a region being compared to all those who are not. 12 For a complete listing of how frequently each item was ranked as the top priority (1), see the survey questionnaire with associated frequencies in Annex B. The list of items presented above appear in the questionnaire as follows: 1--Q11C; 2--Q11B; 3--Q11D; 4--Q11F; 5--Q11A; and 6--Q11E. 14 1. Providing unique learning environment to key policymakers and other leaders (72%) 2. Disseminating knowledge to as many people as possible (69%) 3. Bringing training to people who do not have other options (56%) 4. Focusing intensively in a specific content area to address a major national problem (44%) 5. Maintaining high utilization of distance learning facility to successfully recover costs (35%) 6. Introducing modern technology to countries (19%) 3.13 For this item, respondents also had the option to select the "other" category and specify this use of GDLN in their ranking. Although 14 percent of respondents did identify another category to include in their list of priorities, no newly specified use appeared more than three times. The most common listings here included creating communities of practice, developing a network of specialists (either in a region or in a field), and supporting World Bank projects. 3.14 One interesting finding from this ranking question was the placement of "bringing training to people who do not have other options." Although this was selected as the top priority by relatively few respondents (12%), this item was the third most likely to appear among the top three priorities (56%). This was also the only item where a subgroup analysis revealed significant differences in responses: those in WBI were more likely to assign this item top priority than respondents in World Bank Operations or DLC managers. Reach seems to be primarily a concern of WBI. 3.15 To further explore the priorities of the network, the survey asked those respondents who had ever provided content or helped design a learning event hosted by GDLN to indicate how important it was for their GDLN events to achieve certain objectives.13 This question was designed to further enhance understanding about priorities in GDLN use, and mean responses by item are presented in Figure 1.14 3.16 Overall, respondents assigned the greatest importance for having an event "update or refine knowledge or skills." More than half (56%) of respondents indicated that this was "very important" and an additional third (34%) rated it as "important." Four items were considered only marginally less important than updating knowledge, and these included raising awareness of specific development issues, providing strategies to address the country's development needs, providing new knowledge, and facilitating network among people working on the subject matter. The two items that were rated as slightly less important in this list were still widely considered to be important. For "provide 13A set of questions (Q12A-C) were included to identify which respondents had played a role in providing content for GDLN at some point. This set of respondents is therefore larger than just those who identified their primary role as being "content provider" (Q1). 14Q16A-G: Responses are based on a four point scale--Very important, important, somewhat important, and not important. Respondents had the option of selecting "NA" if the objective was not relevant for their type of GDLN events. The means discussed here are for those respondents who did not select "NA." 15 strategies to address the problems faced by participants' organization," more than a third (37%) selected "very important" and an additional 43 percent indicated that this was an important objective. Similarly, nearly a third of all respondents (32%) noted that it was "very important" for an event to "facilitate the implementation of a reform program," and an additional third (33%) rated this as "important." Figure 1: Importance of Achieving Various Objectives Through GDLN Events (means) Update or refine knowledge or skills 3.5 Raise awareness of specific development issues or 3.4 trends Provide strategies or approaches to address the development needs of the 3.4 country Provide new knowledge or skills 3.4 Facilitate networking among people working on the subject matter of the 3.4 learning event Provide strategies or approaches to address the problems/issues faced by 3.2 participants' organization Facilitate the implementation of a reform program 3.2 1.0 2.0 3.0 4.0 Not Very Important Important 3.17 Subgroup analyses were conducted for responses to each of these items to determine whether importance ratings appeared to be influenced by an individual's role vis-à-vis GDLN, regional experience, or organizational affiliation within WBI or World Bank Operations. Only one characteristic appeared to have an effect: those who had conducted GDLN-related work for the South Asia region were significantly more likely to indicate that it was "very important" for an activity to raise awareness of specific development issues or trends. 16 Roles 3.18 The lack of consensus about the purpose of GDLN or the definition of a GDLN event was part of a broader uncertainty about how the network was governed and who had responsibility for specific roles. Several individuals interviewed expressed confusion as to how they might plan and implement a GDLN event or whom they should contact to receive additional information in a particular area. Others noted that they were not clear what their own roles were or what others perceived these roles to be. For example, a WBI task manager and a member of a GDLN regional team both observed that they felt substantial pressure to facilitate local partnerships with GDLN centers and development organizations even though this was not recognized to be part of their job descriptions 3.19 The survey explored the understanding of various roles related to GDLN, first asking respondents about their own responsibilities. Two-thirds indicated that they had a clear understanding of their role(s) related to GDLN.15 Among the others, 19 percent reported that their roles were not clear and 15 percent indicated that there was a conflict between their understanding of their roles and the expectations of others. 3.20 As a follow-up to this question, respondents were asked what proportion of time they spent working on GDLN-related activities and whether this time allotment was sufficient for them to effectively fulfill their responsibilities. Three-quarters of all respondents reported that they "usually" (45%) or "always" (30%) had enough time allotted to effectively complete GDLN-related activities, and these responses were not influenced by the proportion of time that an individual generally worked on GDLN- related matters.16 3.21 To examine the understanding of roles related to GDLN, the survey included a set of questions to ascertain whether respondents knew whom they should contact for GDLN-related questions and how well they understood their own responsibilities. Among those respondents who had used GDLN previously and felt the question was applicable to them, most (84%) reported that "it is always clear" (46%) or "usually clear" (38%) whom to contact to schedule a GDLN event.17 About three-quarters of respondents also indicated that it was at least usually clear whom to contact regarding technical difficulties (77%) or to make requests for content (73%). Results for the full series of items on whom to contact with GDLN-related questions are shown in Figure 2. 15Q7: Response options include yes; no; there is a conflict between my understanding and the expectations of others. 16Q8B: Responses were based on a five-point scale--Always, usually, sometimes, rarely, and never. 17Q6A-G: Responses were based on a three-point scale--Yes, it is always clear; yes, it is usually clear; or no, it is not clear. Respondents also had the option to select NA. The frequencies presented in this section reflect a subset of the total, namely those individuals who have used GDLN at least "some" before and have not selected "NA" as their response. 17 Figure 2: Clarity about Whom to Contact with GDLN-related Questions Event scheduling 45.5 37.6 83.1 Technical difficulties 36.4 40.6 77.0 Requests for content 27.2 46.1 73.3 Connectivity costs 28.0 43.0 71.0 Partnership development 18.4 39.7 58.1 Proposals for content 20.9 39.5 60.4 Content fees 21.5 34.3 55.8 0 10 20 30 40 50 60 70 80 90 100 Percentage of Respondents Always Clear Usually Clear 3.22 This set of respondents reported the lowest level of clarity for whom to contact regarding fees for content--only slightly more than half (56%) indicated that this was "always clear" (22%) or "usually clear" (34%). Respondents also were less clear about whom to contact regarding partnership development with just 58 percent indicating that this was "always clear" (18%) or "usually clear" (40%). 3.23 Cross-tabular analyses were conducted to identify differences in response patterns by role, placement within the World Bank or WBI, and regional affiliations. Subgroups were limited specifically to only those who had used GDLN in the past so that responses were based on actual experience rather than those of others in the Bank who might have little knowledge about GDLN in general. Not surprisingly, this analysis revealed that those in GDLN Services were generally more aware about whom to contact with questions, but this was not true in all instances. Several significant differences in responses across subgroups were identified: · Those respondents in World Bank Operations who were not members of GDLN regional teams were significantly more likely to respond with "it is not clear" for whom to contact related to event scheduling, technical difficulties, partnership development, content fees, and connectivity costs. However, responses from this subgroup were not significantly different for requesting or proposing content. 18 · Members of the GDLN Services Team and DLC managers were more likely to indicate that "it is always clear" whom to contact for event scheduling and technical difficulties. For questions related to partnership development, those in GDLN services were more likely to select "it is always clear" and DLC managers were more likely to select "it is usually clear" than other respondents. · Content providers were more likely to indicate that "it is not clear" whom to contact related to event scheduling, technical difficulties, and partnership development. These findings were true for both those content providers in WBI and those in World Bank Operations. Among those WBI, those in the Environment and Socially Sustainable Development Division (WBIEN) were more likely than other divisions to report that it was not clear whom to contact for partnership development.18 · WBI unit managers were more likely to indicate that "it is not clear" for whom to contact related to partnership development. · Those respondents who had conducted GDLN-related work for North America were more likely to report that "it is always clear" whom to contact for event scheduling and technical difficulties. Those who had conducted work for the Latin America and Caribbean region were also more likely to select "it is always clear" whom to contact regarding technical difficulties. 3.24 In some cases, how frequently respondents had used GDLN during this fiscal year appeared to have an influence on whether they knew whom to contact. Those respondents who had used GDLN more frequently were less likely to report that it was not clear whom to contact for content proposals, technical difficulties, and connectivity costs.19 However, the frequency of use appeared to make no difference for knowing whom to contact for event scheduling, requests for content, partnership development, and content fees. EXPERIENCE USING GDLN 3.25 Those at the Bank who work most actively with GDLN are positive about the network and find that the advantages outweigh the challenges. Several interview and survey questions focused on users' experiences with GDLN to explore why and how often they opted to use the network and what types of incentives or challenges they must consider in planning future use. 18Many divisions did not have a sufficient number of respondents to support this subgroup analysis. The two other divisions that also had sufficient numbers for this item were WBIFP and WBIHD, but no significant differences were found for these. 19These differences were statistically significant at the p<0.05 level. No differences were found to be statistically significant at the p<0.01 level, the standard used throughout this report. 19 Level of Use and Satisfaction 3.26 Of those survey respondents who had ever provided content or helped to design an activity or learning event hosted by GDLN, about one-quarter (24%) had provided more than 12 individual sessions through GDLN during FY04.20 An additional third used the network at least quarterly, with 15 percent having provided 7-12 sessions, and 17 percent having provided 4-6 sessions. Nearly a third (31%) had provided only 1-3 sessions during FY04 and 13 percent of the respondents had not provided any sessions during this time period. These categories in the frequency of use for this fiscal year are presented in Figure 3. Figure 3: Number of Activities Conducted by GDLN Users During FY04 0 13% More than 12 24% 1-3 31% 7-12 15% 4-6 17% 3.27 An analysis of response patterns by subgroup did not reveal any differences in the frequency of use based on regional affiliation or role; however, those respondents in Operations who were not members of the GDLN regional teams were less likely to have used the network 7 times or more during FY04. 3.28 Those who have used the network tend to report a high level of satisfaction. Survey respondents were asked to rate their overall experience with GDLN, and nearly all (89%) rated their experience with the network to be at least "good."21 More than half of those who had used GDLN reported that their overall experience was "excellent" (14%) or "very good" (45%). An additional 30 percent reported their experience as "good" and 10 percent noted that it was "fair." Those who had provided more sessions hosted by GDLN during FY04 were more likely to report a higher level of overall satisfaction. 20Q13 21Q14: Responses were based on a five-point scale--Excellent, very good, good, fair, and poor. 20 Ease of Use 3.29 GDLN users interviewed at World Bank headquarters noted various challenges in using the network even as they espoused the benefits of delivering events through GDLN. WBI staff in particular expressed concern about the administrative and monitoring systems involved. These include the Activity Management System (AMS) designed for GDLN, the Learning Catalog (a World Bank/WBI initiative), and the Client Registration System (CRS) used by WBI. Some WBI staff must use all three monitoring systems and find that there are redundancies in work that result from a lack of coordination.22 3.30 To provide a better understanding of how difficult or easy it is to use GDLN, the survey included a list of common elements necessary for setting up an event and asked respondents to rate the level of difficulty or ease for each. Results for this list are presented in Figure 4.23 22DLC staff and others outside of WBI may consult the Learning Catalog, but they enter data only into AMS. They do not have access to CRS. 23Q15A-I: Responses are based on a four-point scale--Very difficult, difficult, easy, and very easy. Respondents could also select "NA" if the item did not apply to them. The means and frequencies presented here are for those respondents who did not select "NA." 21 Figure 4: Level of Ease or Difficulty for GDLN-related Tasks (means) Reserving appropriate facilitites 2.8 Recruiting participants 2.7 Recruiting local facilitator 2.6 Entering data into the Learning Catalog 2.5 Entering data into AMS 2.5 Adapting content to local needs 2.5 Converting face-to-face content for distance learning 2.5 Entering data into CRS 2.4 Negotiating costs for learning events 2.3 1.0 2.0 3.0 4.0 Very Very Difficult Easy 3.31 The GDLN-related task that was reported to be the easiest was reserving appropriate facilities or equipment, with nearly three-quarters (71%) of respondents indicating this was "very easy" (15%) or "easy" (56%). Recruiting participants was also considered to be relatively easy, with 11 percent reporting this to be "very easy" and an additional 53 percent labeling it as "easy." 3.32 On average, the task considered to be most difficult was negotiating costs for learning events, with 14 percent indicating that this was "very difficult" and an additional 40 percent reporting it to be "difficult." Respondents also found entering data into the Client Registration System (CRS) to be relatively challenging, with approximately half (52%) finding this to be "very difficult" (19%) or "difficult" (33%). 3.33 An analysis of subgroups for this series of items did reveal some differences in response patterns. Notable findings include the following: 22 · Those who had conducted work for the Latin America and Caribbean region were also more likely to indicate that recruiting participants was "very easy" or "not applicable." · Those respondents who had conducted GDLN-related work in the South Asia region were more likely to indicate that both recruiting participants and adapting content to local needs were either "very easy" or "not applicable." · Respondents who had done GDLN-related work for the East Asia and Pacific region were more likely to indicate that entering data into the Client Registration System (CRS) was "very difficult" and they were less likely to report that this activity was "not applicable." · Respondents who worked in World Bank Operations but were not members of regional GDLN teams were more likely to select "NA" for recruiting participants, reserving appropriate facilities, negotiating costs for learning events, and entering data into the Activity Management System (AMS). · DLC Managers were likely to rate recruiting participants as being more difficult and entering data into AMS as being less difficult. Incentives to Use GDLN 3.34 Many content providers who had used GDLN were strong believers in the network and had clear motivations for using this delivery modality despite difficulties with some set-up tasks. As one WBI manager noted in an interview, "GDLN allows participants to not travel, to do a learning event and then apply what they learned on the job, then do another learning event and share the findings. They can really have action- based learning." Other users observed during interviews that GDLN allows expertise to be brought in easily from different regions and also forces discipline, since participants must wait for the opportunity to speak. 3.35 The survey provided a list of possible incentives for using GDLN and asked those who had provided content through the network to indicate which factors provided compelling reasons for them to use GDLN in the future.24 As demonstrated by Figure 5, respondents were most likely to indicate that GDLN enables experts in other locations to contribute (66%) or reduces travel costs (58%). Respondents were least likely to use GDLN simply because of demand from the target audience to use this mode (5%) or to support active learning approaches (17%). Very few respondents opted to use the "other" category for this question and no response specified here occurred more than once. 24Q17: Those considered to be "content providers" here include anyone who has provided content or helped design an activity hosted by GDLN, any one who has supervised another person in this content provider role, or anyone who has provided instructional design to the individual directly providing content. Respondents had the option of checking all incentives that applied to them and specifying an "other" category if appropriate. 23 Figure 5: Compelling Reasons to Use GDLN 100 90 80 70 66.4 60 57.5 spondents Re 50 of 39.0 40 37.7 34.9 Percentage 30 19.1 20 17.1 10 4.8 0 Enable experts Reduces travel Builds Enables senior Allows Provides Supports Target in other costs communities of professionals participants to effective link to active learning audience locations to practice to share learn on the in-country approaches prefers this contribute knowledge job network delivery 3.36 The reasons to use GDLN in the future generally did not vary significantly by group except in two instances: · Respondents in WBI who identified their primary role to be "content provider" were more likely to select "reduced travel costs" as a reason to use the network in the future. · Those respondents who had conducted GDLN-related work for the Latin America and Caribbean, North America, or South Asia regions were less likely to select "enables senior professionals to share knowledge" as a compelling factor for using GDLN. Challenges 3.37 While describing motivating factors for using GDLN, users who were interviewed also recounted frustrating experiences in using the network and explained why they might opt not to use this delivery modality again. Some content providers in WBI noted that the existing performance measures such as the number of participants reached and the cost per participant are quantitative measures that do not support the more qualitative concept of using GDLN to reach specific decision makers for maximum development impact. Other themes heard during interviews included that GDLN centers were located in capital cities and not appropriate for programs directed at communities or schools and that it was often difficult to market GDLN events and recruit sufficient participants. 3.38 Some users also noted that many existing networks were superior. One content provider reflected that existing networks or universities often have subject matter experts 24 available to support distance learning and that it is easier to channel programs through this existing infrastructure. That individual further observed that most educational institutions become known for a specific field and that it is not realistic for GDLN to promote development and build capacity in all subject areas. Instead, the network should focus on a specialty if it is indeed to be more than just a tool. 3.39 After several such discussions during interviews, a list of possible challenges was developed and included within the survey. Respondents were prompted to select up to three of the options listed that they viewed as reasons to not use GDLN in the future.25 Results from this question are shown in Figure 6. Challenges selected most commonly reflected what had been heard in the interviews: the event was not marketed sufficiently (39%) and the distance learning centers were not near the target audience (34%). In addition, more than a third of the survey respondents indicated that a key challenge was that technical difficulties had occurred (38%). Two other reasons selected by at least a quarter of the respondents were excessive costs26 and having content that was not appropriate for distance learning.27 Respondents were least likely to indicate that they were dissuaded from further use because of difficulty reserving facilities (14%) or because AMS is difficult to use (16%). 25Q18: Respondents could select up to three reasons. 26Costs include essential delivery costs of connection fees of $285 for set-up of a session plus connectivity fees by the minute for ISDN (if that is the technology used), room rentals (currently suggested at $200 per hour for VC room and $150 per hour for Internet/multimedia room) plus content development and delivery costs (instructors, facilitators, translation and interpretation services, and the like.). 27Based on a survey, it is impossible to assess whether, in fact, their content is not appropriate for distance learning or the individual was unfamiliar with the means of adapting the content or reluctant to try. 25 Figure 6: Reasons to NOT Use GDLN in the Future 100 90 80 ts 70 en ond 60 sp Re 50 of e 39.0 tag 40 37.7 34.3 en 29.5 30 Perc 24.7 20 16.4 13.7 10 0 Event not Technical Distance Excessive Content not AMS is difficult Difficult to marketed Difficulties learning cost per appropriate for to use reserve sufficiently centers not participant distance facilities near target learning audience 3.40 Although respondents overall did not indicate that difficulties reserving facilities might dissuade them from future network use, WBI content providers were significantly more likely to note this challenge: in fact, fully one-third (34%) selected this item. The only other significant variation by subgroup was that respondents who had conducted GDLN-related work in the ECA region were less likely to select "event was not marketed sufficiently." Other respondent characteristics such as how frequently they had used GDLN appeared to have no influence on what challenges were selected. 3.41 An open-ended item was included in the survey asking respondents to indicate what changes could be made related to GDLN that would encourage them to use the network more frequently. Out of the 52 responses received to this question, the most common themes related to lowering costs (11), developing content adapted for local use (9), improving the support systems provided through GDLN Services (6), and increasing the capacity of distance learning centers to market events (5). Sample quotes for each of these general themes are presented below: · Decreasing costs - "Subsidize connection costs for extremely low- income/marginalized audiences (but those with reach and multiplier effect)" - "Reduce or eliminate connectivity and facility usage fees" · Developing local content 26 - "Touch base with DLCs to find out the learning needs of each country and try to find content suppliers that meet the needs to improve on the relevance of learning materials" - "Better and more demand-driven content" · Improving systems provided through GDLN Services - "Drop the AMS. IT'S NOT WORKING. Improve services provided by GDLN Services." - Provide better and coordinated services from GDLN Team--often we are not sure what are the services we can expect: just connections?" · Increasing the capacity and expectations of distance learning centers to market events - "Make staff of GDLN Centers more proactive in the marketing and recruiting process. Often the content provider ends up doing his/her job in addition to theirs... extremely frustrating and reduces incentives to use DL format for courses because of the amount of extra energy required" - "Stronger capacity of DLC and local partners in organizing the distance learning event" PERSPECTIVES ON CONTENT 3.42 GDLN stakeholders at the World Bank are concerned about the availability of high-quality content to meet local demand. Issues related to content development surfaced repeatedly in interviews with World Bank personnel and covered a broad spectrum from initial development through marketing and delivery. Meeting Local Demand 3.43 Several of the individuals who were interviewed reflected on the challenge of effectively adapting content for local use in a cost-effective way. One instructional designer explained the rationale behind having off-the-shelf content in WBI given the labor intensive process of developing courses and the resulting desire and need to recycle them. This emphasis on recycling content with little effort may create a supply-driven system. In contrast, a content provider asserted that the supply-driven WBI is gone, and that the new dominant philosophy is to not do the off-the-shelf approach but instead to tap into local sources for content expertise, connecting with local institutions where possible. He also described the evolution of WBI from being a course provider to becoming a "key global player in capacity-building" with an emphasis on being involved in policy dialogues and institutional development. 3.44 This topic was further explored in the survey where respondents who had at least heard of GDLN were asked to indicate their level of agreement to a statement. Nearly two-thirds (62%) of those who responded strongly agreed (17%) or agreed (45%) that 27 "content delivered through the GDLN tends to be supply-driven and does not necessarily match local needs." 28 A cross-tabular analysis revealed no differences in responses based on an individual's role related to GDLN, regional affiliation, or frequency of using GDLN during FY04. WBI content providers also did not respond any differently than other groups. Quality Assurance 3.45 The quality of events delivered through GDLN relies on a range of factors, from the functioning of technology and the appropriateness of the facilities to the various aspects related to the content itself. Stakeholders at the Bank frequently emphasized the need for a quality assurance system, so that content delivered through GDLN adhered to consistent standards. Some noted that this was of particular importance given the pressure to increase the utilization rates of GDLN centers and funnel as much content as possible into the pipeline. However, despite the frequent discussion of quality assurance, no consensus existed about who should have responsibility for upholding consistent standards. 3.46 A few of those interviewed noted that content should be reviewed by an instructional designer before being delivered through GDLN but that this only happened if the content provider specifically requested it, usually either through the appropriate regional team or through the Activity Management System (AMS). An instructional designer countered this viewpoint by asserting that DLCs themselves should be responsible for quality assurance since content providers at headquarters can not necessarily assess the relevance or appropriateness for the local level. 3.47 Others reflected that there is no clarity on what the appropriate role for a GDLN Center is and that adapting content and assessing its value requires a much different capacity than just running a business and hosting events. In addition, one content provider argued that quality is best determined by the consumer, so the real measure is the return rate of participants. This diversity of perspectives about quality assurance was not surprising given the wide range of providers and processes through which content is delivered using GDLN. 3.48 The survey specifically asked respondents to indicate who should have responsibility for ensuring that GDLN content is of high quality and the results from this item are shown in Figure 7.29 Responses to this item underscored the lack of consensus, with less than a third (30%) indicating the content provider, the most frequently selected choice. Other relatively common responses included having quality be determined by the consumer or having the responsibility belong to GDLN Services (both 14%). Respondents were least likely to assign this responsibility to the instructional designer (7%). 28Q9F: Responses were based on a four-point scale--Strongly agree, agree, disagree, and strongly disagree. 29Q10 28 Figure 7: Where Respondents Assign Responsibility for Quality Assurance 100 90 80 70 60 pondents Res of 50 tagen 40 rceeP 29.6 30 20 17.1 14.4 13.9 10.7 10 7.4 6.9 0 C ontent D eterm ined G D LN D LC S taff G D LN Instructional O ther P rovider by C onsum er S ervices R egional D esigner Team s 3.49 This question also provided the option for respondents to specify an "other" category and 17 percent selected this category. Responses specified here covered a wide range, often assigning responsibility to two or more of the categories listed. The most common answer called for forming "an academic board" or a "quality control group" that would be comprised of representatives from most or all of the other categories. Some indicated such a group should be global while others described it as functioning on a regional level. 3.50 An analysis of subgroup responses for this item revealed no difference in the opinions of stakeholders in WBI versus those in World Bank Operations. However, several other respondent characteristics related to role or region did appear to have an influence: · Respondents in GDLN Services were more likely to assign this responsibility to themselves (the GDLN Services staff) or to indicate that quality should be determined by consumer demand (in terms of repeat business). 29 · DLC managers were less likely to assign this responsibility to the content provider and more likely to associate it with GDLN services. · Content providers in general were more likely to indicate that quality assurance should be determined by the consumer. · Those who indicated that they had directly provided content using GDLN were more likely to assign the responsibility for quality assurance to the content provider or to an instructional designer. They were less likely to assign this responsibility to the GDLN services team, the GDLN regional staff, or the distance learning center staff. · Among content providers in general (who may or may not have used GDLN), those in the WBI Finance and Private Sector Development Division (WBIFP) or in the WBI Poverty Reduction and Economic Management Division (WBIPR) were more likely to have assigned responsibility for quality assurance to the content provider.30 · Those who had conducted GDLN-related work in the Middle East and North Africa or North America regions were less likely to assign responsibility for quality assurance to the content provider or the GDLN regional staff and more likely to assign this responsibility to GDLN services team based in WBI. · Those who had conducted GDLN-related work in North America were also more likely to select the "other" category. Negotiating Terms for Use 3.51 The other major factors related to the delivery of content that surfaced during the interviews tended to revolve around cost agreements negotiated among World Bank content providers, GDLN Centers, and other external partners.31 In particular, some stakeholders expressed concern about the lack of clear systems to specify costs and support financial agreements. 3.52 One member of GDLN Services explained that the nature of activity agreements varies by region since some centers are very poor and participants are unable to pay whereas others can market their own activities and charge a fee. It is therefore not uncommon for the same activity to be delivered to five sites with a different content fee for each. This stakeholder reflected that this is not problematic since "prices never differ that radically, and centers never know what each other is paying." A content provider complained that this lack of transparency in paying GDLN Centers makes his role very 30Some divisions did not have sufficient numbers of respondents to support a valid analysis. Other divisions with enough respondents but no significant differences in responses included WBIEN and WBIHD. 31For a discussion of issues related specifically to external partners, please see "Sustaining the Network through Partnerships," pp. 32-37. 30 frustrating. He also noted that there appear to be different methods for paying a GDLN Center depending on the status of the center and whether the activity will be paid for directly through GDLN or through another part of the Bank. There is no clear formula for this process, but when Centers do not receive promised payments then the content providers may experience difficulties negotiating terms in the future. 3.53 Other stakeholders observed that some difficulties had occurred because activity agreements were not binding enough. One program officer described the frustration of recruiting partners to develop content only to have centers back out of the payment agreement when they were unable to recruit enough participants. A WBI manager reflected that "the experience has been disastrous with regard to marketing GDLN." If content providers have to contract trainers and pay costs up front for programming, then a GDLN Center should not be able to cancel an event at the last minute because of the lack of participants or other factors. This dynamic made some task managers wary of delivering events through GDLN. 3.54 Several individuals interviewed acknowledged these problems with negotiation and support systems and stated that GDLN was still in a developmental stage. Indeed, during the period of this evaluation, efforts were underway to revise the Memorandum of Understanding signed by GDLN Centers and improve the Activity Management System, among others.32 The GDLN Strategic Plan also noted current efforts to reassess the fee structure. IMPACT FOR DEVELOPMENT 3.55 Some GDLN stakeholders noted an inherent conflict in the network's joint focus on development impact and sustainability while others saw a rich opportunity for partnerships in the development business. World Bank staff often referred to GDLN as a public good with its intended use being as a vehicle for development. Some expressed concern that the network's potential was undermined by the pressure on centers to become financially sustainable. Given the network's emphasis on development impact, one of the four research questions for this study focused on whether content delivered through GDLN is aligned with World Bank goals or Millennium Development Goals (MDGs).33 Content Alignment 3.56 GDLN activities encompass a broad range of content: some is developed within WBI or by the World Bank while other programming is designed by external partners. Some events are planned in advance while others--particularly meetings--may rely on content of a more impromptu nature. At this point, GDLN does not promote a core 32The final draft of the GDLN Network Agreement (revised MOU) was released in April 2004. 33The MDGs are as follows: 1) Eradicate extreme poverty and hunger; 2) Achieve universal primary education; 3) Promote gender equality and empower women; 4) Reduce child mortality; 5) Improve maternal health; 6) Combat HIV/AIDS, malaria, and other diseases; 7) Ensure environmental sustainability; and 8) Develop a global partnership for development. 31 curriculum but instead, through AMS, provides a marketplace in which to match content proposals with requests. 3.57 Although GDLN is touted as a network for development, no central guidance is provided about what goals or standards programs should adhere to for optimal development impact. In some cases, strategic content development has been assessed at the regional level to establish priorities. The GDLN team for the Europe and Central Asia region, for example, conducted a review of the Country Assistance Strategies (CAS) and Poverty Strategy Reduction Papers (PRSP) available for all countries in the region to prioritize programming demand. This analysis was predicated on the belief that "the cornerstone for ultimate success of the network is the availability of well-designed content in local languages, adapted for the audience concerned, while a media mix should be used that is compatible with existing technical infrastructure."34 In some countries, there has been a deliberate effort to align GDLN content with the CAS, which in turn may be based on a PRSP. 3.58 Internationally recognized development objectives such as those represented by the World Bank themes or the Millennium Development Goals (MDGs) do not currently provide a framework for GDLN content. According to the Bank's Operations Policy and Country Services (OPCS) office, the World Bank themes represent the goals and objectives of Bank activities and are also used to capture Bank support to the MDGs.35 While some AMS themes such as "Rural Development" are consistent with those of the World Bank, others such as "communications" are not. In fact, it appears that categories such as this one more closely resemble sectors, focusing on the part of the economy which receives support. This categorization in AMS is further complicated by the lack of definitions, so that those activities included under "communications" may include a formal course on communication strategies and an impromptu meeting to negotiate GDLN use among other offerings. 3.59 Given these the considerations above, it appears that the themes used to categorize and organize content in AMS do not correspond to standard World Bank themes or sectors, as demonstrated by Figure 8. 34Assessment of Priorities and Strategic Content Development in Europe and Central Asia, June 2002, p. 4. 35See Measuring Thematic and Sectoral Alignment of Bank Activities in Annex E. 32 Figure 8: Themes in AMS, Compared to World Bank Themes Standard Themes in AMS Themes Corresponding to World Bank Goals 1. Communication--? 2. Education Economic Management 3. Environment Public Sector Governance 4. Finance 5. Gender Rule of Law 6. Governance Financial and Private Sector Development 7. Health/Nutrition Trade and Integration 8. Human Resources--? 9. Infrastructure--? Social Protection and Risk Management 10. Local Development--? Social development, Gender, and Inclusion 11. Politics--? Human Development 12. Poverty 13. Private Sector Development Urban Development 14. Rural Development Rural Development 15. Social Development Environment and Natural Resources 16. Social Protection Management 17. Telecommunication and IT--? 18. Transport--? 19. Urban Development = Term explicitly listed as part of World Bank Theme (The detailed listing of World Bank Themes is presented in Annex E) 3.60 This exercise of comparing themes revealed that seven themes from AMS did not match the explanation of Bank themes and three Bank themes did not appear to have a corresponding AMS category. 3.61 To further explore whether GDLN content is aligned with specific development indicators, the survey asked respondents to reflect on how well programs were aligned with various objectives. World Bank staff and DLC managers were able to indicate if they did not know enough about programming to answer or if they were unfamiliar with 33 specific measures such as the MDGs.36 Responses to this series are presented in Figure 9. Figure 9: Alignment of GDLN Programs or Activities 100 92.5 89.8 90 17.2 80.6 81.3 80 92 s 70 37.3 36 60 39.4 Respondent of 50 55.2 age 40 39.2 30 Percent 29.7 33.5 20 10 20.1 13.3 11.5 11.8 0 World Bank Goals Millenium Development CAS PRSP Goals Always Usually Sometimes 3.62 Respondents most frequently reported that content was aligned with World Bank goals, with three-quarters (75%) indicating that this was "always" (20%) or "usually" (55%) the case. An additional 17 percent reported that this alignment occurred "sometimes." Respondents were most likely to be aware of the World Bank goals and only a few (3 %) indicated that they were not familiar with these. Content was also often aligned with MDGs with more than half (52%) reporting that this occurred "always" (13%) or "usually"(39%), and an additional 37 percent responding with "sometimes." Five percent of respondents for this item noted that they were not familiar with the MDGs. 3.63 The survey also explicitly explored the perceived conflict in asking GDLN Centers to align content for development impact while becoming financially sustainable. More than two-thirds of respondents (70%) either agreed (44%) or strongly agreed (26%) that "it is not realistic to expect GDLN Centers to become fully sustainable while focusing on the development agenda."37 Response patterns for this statement were consistent across all subgroups. 36Q22A-D: Responses were based on a five point scale--Always, usually, sometimes, rarely, and never. Other possible responses included, "I don't know enough to answer this question" (regarding programming in general) or "I am not familiar with these" (regarding the goals). The frequencies presented for this series do not include those who did not know enough about programming in general to answer. They do however include responses from those who were not familiar with the respective goals since this is presumably an indication that aligning with those goals was not a priority for that respondent in working with the programming. 37Q9I: Responses were based on a four-point scale--Strongly agree, agree, disagree, and strongly disagree. 34 Critical Factors 3.64 Content providers described various components of program delivery that influenced impact and these ranged from finding the right facilitator and tapping into subregional networks to including the right level participants and involving partners. The survey listed the various factors voiced during interviews and respondents were asked to rate their importance for influencing whether a GDLN activity was effective in contributing to the development dialogue in a country.38 As shown in Figure 10, the factor deemed to be most important was the overall quality of the activity, including both its content and pedagogy. Nearly all (94%) rated this as "very important" (60%) or "important" (34%). Respondents also identified the relevance of the thematic area to the country's need and having the appropriate experts and facilitator as important factors. Figure 10: Factors for Development Impact (means) Overall quality of activity (content and pedagogy) 3.6 Relevance of the specific thematic area to country need 3.5 Having the appropriate experts and facilitator 3.4 Commitment of high level officials to follow through 3.3 Involvement of partners who focus on development issues 3.3 Participant level/occupation 3.3 Location of distance learning center in capital city 2.7 Link with CAS or PRSP 2.9 Connectivity with in-country network 2.9 1.0 2.0 3.0 4.0 Not Very Important Important 3.65 The factor identified to be the least important in the list was the location of the distance learning center in the capital city although this was still an important factor-- with slightly more than half (57%) rating this as "very important" (19%) or "important" (38%). Aligning content with the CAS or PRSP and connectivity with in-country networks were also seen as less important elements in effectively contributing to the development dialogue. 38Q23A-I: Responses were based on a four-point scale--Very important, important, somewhat important, and not important. Respondents could also select "don't know." The means presented here reflect all respondents except those who answered "NA." DLC manager responses are therefore also included here as part of the larger group, but their views are specifically presented in Chapter 4. 35 3.66 Cross-tabular analyses for this item revealed no differences in responses based on role or placement in WBI versus World Bank Operations. In addition, only one difference was noted based on regional affiliation. Those who had conducted GDLN- related work for the Africa region tended to assign a higher importance rating to the overall quality of the activity. 3.67 A number of Bank staff interviewed also pointed out that promoting communities of practice was important for increasing the impact of programs delivered through GDLN. Network users were therefore asked to indicate in the survey how frequently they convened communities of practice, and almost two-thirds (61%) reported that they did this at least "occasionally" which was defined as "several times a year."39 SUSTAINING THE NETWORK THROUGH PARTNERSHIPS 3.68 World Bank staff believe that partnerships are critical for the future of GDLN, but the roles for establishing these are not clearly defined. Discussions about the effectiveness and impact of programs facilitated by GDLN often focus on the importance of partnerships and how to develop more of them to expand and sustain the network. Responsibility for Building Partnerships 3.69 Definitions of partnerships among World Bank staff span a broad spectrum. Partners provide funding to support distance learning center staff or to fund participants attending a specific event. They finance the development of content. The distance learning centers themselves are partners in the network. Partners design GDLN events or provide content for programs. They contribute experts and facilitators for dialogues or learning activities. 3.70 Very few survey respondents identified themselves as partnership development specialists, but about one-third (32%) of those who had least heard of GDLN reported having been involved in developing co-financing or content partnerships for the network.40 Those who had conducted GDLN-related work for the Latin America and Caribbean region were more likely to have been involved in developing these partnerships, but no other variables appeared to have an influence. 3.71 During the interviews, stakeholders often emphasized the need for more partnerships of various types, but there was no consensus about who should have responsibility for this. One stakeholder who explicitly focused on partnership development observed that it was often not clear whom to contact or how to proceed when an organization had been successfully courted and was interested in working collaboratively with GDLN. This observation corroborated findings from the survey discussed earlier, where 42 percent of those who had used GDLN were not clear whom to contact regarding partnership development.41Some asserted that those at the World Bank 39Q19: Responses are based on a five-point scale: Very frequently (at least twice a month), Frequently (monthly), Occasionally (several times a year), Seldom (quarterly or less), and never. 40Q25 41Q6E: Results are discussed on p. 15. 36 needed to leverage their connections to facilitate building partnerships for GDLN Centers; others saw this as a local responsibility for centers to shoulder themselves. 3.72 To further examine this issue, the survey asked respondents to indicate their level of agreement to two statements related to the responsibility for partnerships. About three-quarters (76%) of respondents either agreed (49%) or strongly agreed (27%) that "the World Bank plays a critical role in making connections between distance learning centers and local offices of development agencies." Nearly all respondents also either agreed (44%) or strongly agreed (53%) that "distance learning centers need to be proactive in forming local partnerships with development agencies."42 The high level of agreement with both of these statements seemed to indicate that partnerships must be pursued from both angles. Respondent characteristics did not appear to influence answers to these items except that those respondents who had been involved in developing co-financing or training partnerships for the GDLN were more likely to strongly agree that centers need to be proactive in forming local partnerships. 3.73 Bank staff explained that much of the confusion surrounding partnership development resulted from the variety of types of partners and the diversity of GDLN centers and their needs. For example, one WBI content provider might be in a position to effectively approach a donor organization or a university whereas a GDLN Center might be in a stronger position to contact the central or regional government. 3.74 The survey respondents who had been involved in developing partnerships were asked to indicate what types of organizations they had approached to forge connections and how often partnerships had resulted from these efforts. They were also asked to note how many times such organizations had initiated this collaboration by contacting them instead. No concrete definition of what constitutes a partnership was presented with this question, but the responses illustrate trends in terms of what types of organizations are most likely to approach or be approached by GDLN stakeholders and how successful stakeholders perceive themselves to be in their partnership endeavors. Results for this series are presented in Table 2 and include only those respondents who worked at Bank headquarters or field offices.43 42Q9D-E: Responses are based on a four-point scale--Strongly agree, agree, disagree, and strongly disagree. 43Q27A-C: These numbers do not include responses from DLC managers. For a discussion of partnerships and DLC managers, see Chapter 4: Organization and Operation of DLCs. 37 Table 2: Numbers of Respondents Who Were Approached and Established Partnerships, by Type of Organization Number who N=54 Number who were Number who successfully (includes only approached by approached World Bank staff) organization organization established partnerships Happened Happened 5 Happened Happened Happened Happened at least or more at least 5 or more at least 5 or more once times once times once times Teaching or research institution 37 20 34 16 34 8 (public or private) NGO 30 12 23 8 24 4 Private Sector (for profit) 23 10 18 11 17 4 Donor Agency 23 6 28 10 17 2 National or central government 25 3 21 7 20 4 Regional or provincial 13 4 12 4 12 2 government Local government 10 2 9 2 6 0 3.75 In interviews, many GDLN stakeholders at the Bank emphasized a trend towards reaching out to the donor community with the thought that these agencies would adopt GDLN as a training and knowledge sharing vehicle. Survey responses regarding partnership development showed that stakeholders were most likely to have approached and established partnerships with teaching or research institutions, non-governmental organizations, and private sector companies. They were least likely to have connected with regional or local governments. Incentives for Partners 3.76 Survey respondents who had been involved in developing partnerships were also asked to indicate what they perceived to be the incentives for organizations to enter into partnerships with GDLN and results for this question are shown in Figure 12.44 More than three-quarters (76%) reported that the link to others working in development was an incentive--a finding that appeared to support the concept of GDLN being promoted a the vehicle of choice for the development community. In addition, more than two-thirds of the respondents (69%) noted the ability to implement development activity on a larger scale. Access to World Bank branding was also a perceived incentive with nearly two- thirds (65%) selecting this option. 44Q28 38 Figure 11: Perceived Incentives for Organizations to Become Partners 100 90 80 76.1 69.0 nts 70 64.8 pondes 60 57.8 Re 50.7 50 of ge tane 40 rc 30 Pe 20 10 0 Link to others A bility to A ccess to W orld G reater visibility / O rganizations w orking in im plem ent B ank branding exposure com plem ent each developm ent developm ent other in subject activity on larger area scale 3.77 A variety of reflections on the role of World Bank branding in marketing GDLN activities were heard during interviews, with some stakeholders even proposing that the Bank may jeopardize the future ability of the network to operate independently by not embracing it enough during the developmental phase. One individual in Operations recounted an instance where the European Union had considered partnering with a local GDLN Center but then had become suspicious after learning that the Center was never used by the local World Bank office. Most respondents (90%) either agreed (65%) or strongly agreed (25%) that "having the World Bank brand allows distance learning centers to market activities more successfully." At the same time, about three-quarters (74%) of respondents disagreed (60%) or strongly disagreed (14%) that "the GDLN would attract more external partners if it was not so closely affiliated with the World Bank."45 Data from both the survey and the interviews revealed a general perception that the World Bank does or could play a powerful role in marketing GDLN activities and centers. Barriers 3.78 In addition to incentives, stakeholders were asked to identify the challenges in recruiting partners. One individual from the field noted that AMS was frustrating to use and that he believed it acted as a disincentive for partnerships. Another content provider noted that even when a potential partner approached him, he was not sure whom to contact and the process became convoluted. When asked to indicate issues that affect an organization's willingness to enter into partnerships with GDLN, survey respondents 45Q9G-H: Responses are based on a four-point scale--Strongly agree, agree, disagree, and strongly disagree. 39 most frequently selected the lack of organizations' experience with such partnerships (71%) and the lack of clear communication channels (52%).46 Both of these obstacles seemed to clearly link back to the need for more procedural information from GDLN Services and the need for more clearly defined roles. Frequencies of responses for all of these issues or perceived barriers are shown in Figure 12. Figure 12: Perceived Barriers for Organizations to Become Partners 100 90 80 71.8 70 60 52.1 Respondents 50 of 40 33.8 32.4 Percentage 30 21.1 20 10 0 Lack of Lack of clear B urdensom e to D ifficulties in O rganization's experience w ith com m unication enter data into negotiating M O U visibility is such partnerships channels A M S overshadow ed by W orld B ank 3.79 Survey respondents also had the option of providing open-ended responses to elucidate any issues related to partnerships. Most comments for this item were isolated remarks but four individuals used this space to explain that problems with GDLN governance served as barriers for partnerships. These comments focused on the lack of by-laws detailing how GDLN is managed and the lack of transparency in core GDLN policies and procedures. Effective Methods for Attracting Partners 3.80 After identifying perceived incentives and barriers for potential partners, survey respondents who had been involved with partnership development were asked to identify which techniques were effective for attracting partners.47 Two-thirds (66%) of respondents indicated that inviting donors to GDLN events was the best method, and nearly the same proportion (65%) indicated that developing a link between DLC managers and local representatives of development organizations was effective. These findings corroborated the stories heard in interviews where individuals spoke about the power of the World Bank for making connections in the development community and 46Q29 47Q30 40 having partners experience GDLN events firsthand. One person who specialized in partnerships emphasized that inviting potential partners to an activity at a distance learning center was much more effective than simply hosting a marketing session and demonstrating videoconference capabilities. 3.81 Survey respondents also identified other practices as being effective for attracting partners, including maintaining contact with alumni (51%) and building developing country case studies showing specific GDLN impact (49%). CRITICAL FACTORS FOR GDLN CENTERS 3.82 A series of factors are important for a distance learning center to be effective, and the most critical of these is the leadership of the distance learning center manager. This general consensus was supported by those interviewed at Bank headquarters regardless of their context or role related to GDLN. Some had extensive experience visiting centers or even serving an interim center manager role while others had only a vicarious understanding of center operations. Conditions for Greater Effectiveness and Efficiency 3.83 GDLN stakeholders described a range of centers, from those with sophisticated facilities and numerous staff to those with outdated equipment and little support. When asked to identify what factors needed to be in place at a center for it to operate efficiently and have the greatest impact, those interviewed nearly universally mentioned the leadership of the center manager. Indeed, this one factor was seen as the critical ingredient to nearly all aspects of a center's existence, from whether it could attract sufficient programming and participants, to whether it could effectively contribute to a country's development dialogue and become financially viable on its own. As one program officer from the field aptly stated, "In the end, the sustainability comes down to whether the manager is a dynamo and the basic economy of the country." 3.84 The survey asked respondents to rate the importance of various factors that contribute to the effectiveness and efficiency of GDLN Centers, and the ten conditions that were deemed most important are presented in Figure 13. Consistent with the interview data, the factor with the highest importance rating was the leadership from the DLC manager, with two-thirds (66%) of respondents labeling this as "very important" and the remaining ones calling it "important" (28%) or "somewhat important" (6%).48 The other conditions that were rated among the most important included capabilities for marking learning events and having a consistent quality assurance system for content. The high ratings of both of these factors were tightly aligned with the interview data and stakeholders nearly always described the capabilities for marketing learning events as directly linked back to the leadership of the DLC manager. 48Q21A-O: Responses were based on a four-point scale--Very important, important, somewhat important, and not important. The frequencies presented here do not include the respondents who answered "don't know." For a complete list of the factors presented in the question and the associated frequencies, see the survey instrument in Annex B. 41 3.85 The condition that was assigned the lowest importance in this survey question was "employment levels/occupations of GDLN participants" with only 5 percent identifying this to be "very important" and an additional 39 percent rating it as "important." Other factors considered to be less critical for effectiveness and efficiency included the quality and size of the facility, a center's ability to develop its own content, the general readiness of the country for reform or innovation, and the political and economic stability of the country. Figure 13: Ten Most Important Conditions for Distance Learning Centers49 Strong leadership from the DLC Manager 3.6 Capabilities for marketing learning events 3.4 Consistent quality assurance system for content 3.3 Geographic location of distance learning centers in 3.2 target countries Engaged local training partners 3.2 Adequate number of DLC support staff 3.2 Scheduling flexibility 3.0 Connection to development dialogue in country 3.0 Ability to make decisions autonomous from host 3.0 institution Engaged local co-financing partners 3.0 1.0 2.0 3.0 4.0 Not Very Important Important 3.86 The question of what conditions allow a GDLN Center to operate effectively and efficiently is explored in more depth in the following chapter focused on the GDLN 49Q21A-O: The items not presented in this graph include employment levels/occupations of GDLN participants--2.5, ability to develop own content--2.8, general readiness of the country for reform or innovation--2.6, quality and size of the facility--2.9, and political/economic stability of country--2.6. 42 Centers. Among the stakeholders at the Bank, little variation existed in perceptions about this topic. An analysis of subgroup responses noted only the following differences: · Those who indicated that they directly provided content (rather than supervising individuals providing content or advising content provider on instructional design) were likely to assign a higher importance level for "scheduling flexibility." · Those who had conducted GDLN-related work in the LAC region were likely to assign a lower importance to "scheduling flexibility" and to "ability to develop own content." Reflections 3.87 At the end of the survey, GDLN stakeholders had the option of providing any further comments about elements that should be explored in this evaluation. Many respondents used this opportunity to offer relatively lengthy suggestions or reflections on critical factors affecting the impact of GDLN. These entries often included recommendations for enhancing DLC operations, usually through changes made by content providers and others at the World Bank in general or improvements by GDLN Services in particular. 3.88 Common themes specifically related to optimizing DLC operations included making GDLN governance more transparent, improving the quality of learning events through better pedagogy, increasing the reliability of technology, and relatively minor logistical changes that would be meaningful for participants. Sample quotes are included below.50 · "I think more work needs to be done to streamline the procedures and systems currently in place to organize and deliver a GDLN activity. Organizational structure of the GDLN unit in WBI seems to be always changing and never made clear to others in WBI, and the systems don't seem to be effectively addressing the needs and concerns of clients. I have had very positive experiences with the local DLCs and their staff, and above all, I find that using GDLN is very rewarding in terms of the numbers of participants we are able to have in our activities at a relatively low cost, and therefore, the impact of the programs increases." · "Activities that are the first and only in a particular topic are very difficult to evaluate. Series that attract participants who know of the success of the previous `chapter' are often driven to register because they understand the procedures to follow and their participation. Moreover, when the videoconferencing is to present cases or open discussions regarding materials that have been distributed in 50 Q32: The quotes presented here are examples of comments related to DLC operations and are not intended to represent the whole range of responses given. The complete set of responses for this open- ended question are provided in Annex B at the end of the survey questionnaire. 43 advance, they tend to be more active and enjoyable than receiving a 30 minute lesson from a speaker who barely moves. Interaction is the key; better interaction yields wider results (apprehension and application of content learned/discussed)." · "Seeing is believing. Technical quality is extremely important, and GDLN sites all suffer equipment and connection problems. Quality is always hampered when PPs are not well prepared, and lack experience in doing DL. This is why GDLN staff must understand network/DLC needs and situations, and must guide new users properly." · "Little things would prove helpful. For instance, courses are offered in many languages, but simple messages flashed across the videoconference network are only in English. Why not make an extra effort and signal respect for the participants and their language by translating basic messages that are subject to repetition, such as 10 Minute Break." 44 4. ORGANIZATION AND OPERATION OF DLCS: THE MANAGERS' PERSPECTIVE 4.1 Distance Learning Centers, the facilities located in each country that form part of the Global Development Learning Network, were first established in June 2000 and have been rapidly growing in number ever since.51 Although there are certain conventions and expectations about the centers, primarily in technical capabilities, there is great diversity among the centers that reflect the country context, date of inauguration, conditions of establishment, and direction provided by the DLC manager and staff. 4.2 Annex F contains a world map that displays the distribution and number of the DLCs by region: Africa, Latin America, Europe and Central Asia, East Asia and Pacific, South Asia, Middle East/North Africa, as well as centers in North America (Canada and Washington, D.C.). In terms of number of centers, the first four regions have the greatest number of facilities--some whose core business is GDLN and some where participation in the network is just part of their activities. THE NATURE OF DIFFERENCES 4.3 "When you get to analyzing details, every country has a totally different story," is how one DLC manager framed the nature of the differences among centers. 4.4 Centers that were first set up had some commonalities in terms of specification and purchase of equipment, advice given about center functionality, guidance provided on staff roles and qualifications, establishment of business policies, setting up financial terms (albeit with some flexibility), and expected learning activities. As the number of centers has grown from 11 to 68 (June 2004), center operations have become increasingly diverse to reflect each unique environment. 4.5 In the web-based survey administered to Bank staff and DLC managers, the responses on several critical questions reflect the diversity and hint at the challenges in terms of issues of quality and impact. Several factors that influence these considerations are described below. 51It is difficult to reliably cite numbers in the development of the network because there are multiple definitions of when a center comes into the network. Some counts rely on when a Memorandum of Understanding is signed. In other instances, a center is counted as part of the network when it is launched--that is, it has participated in a GDLN event. Additionally, dates are sometimes given as calendar year and sometimes as fiscal year. The network began with 11 centers, mostly in Africa. In materials supplied by the Evaluation Group in a pre-proposal conference in July 2002, a graphic showed that by 2001, there were 26; in 2002, 37; in 2003, 61. Of those 61, 45 centers were located in 41 developing countries. Updated sources now concur that as of June 2004, there are 68 centers in the network. 45 Center Location 4.6 Of the managers responding to the survey,52 65 percent were based at a university or institute; 12.5 percent reported that they were based at a World Bank office; 12.5 percent were located independently, with no host institution; and 10 percent were in other settings, such as NGOs, the British Council offices, and the like. 4.7 Based on discussions with managers and other observers during site visits to GDLN centers, location is a crucial factor in a center's operation. It contributes to the local perception of the affiliation, orientation, and quality of the center. In some cases affiliation with an institute or university could confer prestige; in other cases, it would provide a negative connotation. For example, in some countries being associated with training of the public sector would make it difficult to attract the private sector, NGOs, other development organizations. In the initial set-up of centers, those negotiating for construction of the facilities or affiliation with the network may not have realized the implications. 4.8 Linking the DLC to another institution affects the independence that the center and its manager have in pursuing the philosophy and guidance of the GDLN Network. In terms of location within a city, those making the original arrangements often tried to find a central location (generally in the capital city) so that it would be convenient to attend during the workday or at the end of the day. These decisions can influence the type and quantity of participants, based on convenience and other amenities available. It governs the possibility of modifications to the infrastructure as need and experience dictate. Physical location might need to consider security if top-level officials are expected to attend events. All these considerations are linked to a center's effectiveness in reaching target audiences, delivering capacity enhancing activities, and participating in the development dialogue. Business Model 4.9 DLC managers responding to the survey53 were asked to identify the business model that describes their start-up financing (and has an influence on the financial terms that govern their operations and requirements for sustainability). The options offered were three basic types: · Financed out of the World Bank's capital budget as part of a local Bank office · Financed by World Bank loans, credits, or grants from international or bilateral agencies · Financed by an organization, out of its own recurrent income, through upgrading or using existing facilities. 52QDLC1 53QDLC3 46 4.10 Fully 30 percent of the respondents did not find these descriptions adequate to capture the complexity of the arrangements responsible for the center's launch. Those indicating "other" in the survey offered other descriptions of credits and government subsidies, capital budget expenditures coupled with bilateral organization contributions, a mix of international development agencies' financing, and so forth. All these different models provide different contexts for the centers and result in different benefits or constraints to their financial viability. 4.11 Site visits reinforced that various conditions and "rules" stemmed from the centers' establishment. One center had a tight link to government reform based on how the original center loan was set-up, and consequently, was viewed as the organization to rely on for capacity enhancement and reform topics. Another center benefited from the association with the Bank's construction of the facility, and it was seen as a center of development information and activities by the country's development community. In particular, the issue of sustainability--a major issue for DLC managers--appears to be linked to the business model applied. Other Factors 4.12 Finally, there are other measurable and non-measurable factors that contribute to the differences and disparities among centers and affect the quantity and mix of what activities are offered at a center. Based on documentation of activities over the last two years at the four centers visited and discussions with their managers, different patterns were discerned in the type of offerings--courses, seminars, global dialogues, negotiations--and media used--videoconferences, multi-media, e-learning, and a variety of blending. No one mix is better than the others, and DLC directors continue to explore what set of activities provide the best content for the local needs (capacity enhancement and the development dialogue) and meet the practical realities of generating revenue. 4.13 The differences in what is offered at a center are linked to the year the center was established, management and marketing acumen of the DLC manager, receptivity of others in the development community (national and international) to collaborate, financial supports provided for operations, and political and economic conditions in each country. These factors also translate into how the centers are able to address the impact of their knowledge sharing, coordination, and consultation activities. 4.14 A related difference among regions--and centers within these regions--is the wide disparity in utilization rates.54 The importance of these statistics for financial sustainability depends on whether GDLN activities constitute a core business or an add- on, but Table 3 demonstrates that some centers are active (30-60% utilization) and others are really not active partners. There are certainly variations by regions, with numbers of centers and average utilization rates vastly different. Table 3 also provides the individual utilization rates of the four centers visited during this study and reflects their higher level of functioning in the network. 54100% utilization is based upon 1,260 hours per year 47 Table 3: Utilization Rates--FY03 GDLN Center Number of Range FY03 Average Region Centers 6.8% (Mozambique)- Africa (AFR) 8 61.3% (Ghana) 36.1% East Asia and the 5.9% (Singapore)- Pacific (EAP) 10 30.7% (Vietnam) 13.6% Europe and Central 0.5% (Russia-Academy55)- Asia (ECA) 8 8.2% (Ukraine) 3.9% Latin America and 2.1% (Panama56)- the Caribbean (LCR) 14 34.9% (Bolivia) 17.2% Middle East North 0.6% (Saudi Arabia-Riyadh57)- Africa (MNA) 4 21.8% (Jordan) 10.6% South Asia Region 8.3% (India ­ TERI)- (SAR) 3 24.1% (Sri Lanka) 14.0% GDLN Center Location FY03 Average Senegal 32.9% Tanzania 47.0% Vietnam 30.7% Brazil 28.0% COMMON VIEWPOINTS 4.15 Survey results from 43 DLC managers in the Network and discussions and observations during sites visits provide a number of areas where managers have substantial agreement about the role of GDLN in their country, the challenges they face, and the future directions they envision. Importance of DLC Managers 4.16 DLC managers are key to the operation of the centers. This is a view shared not only by the managers but by Bank staff who have worked with GDLN. In all respondents' assessment of the importance of factors in the efficient and effective operation of DLCs,58 the item with the highest mean (indicating highest importance) for the 15 different factors listed, was strong leadership from the DLC manager (mean=3.6, on a 4-point scale). Specifically, 63 percent all survey respondents thought it was "very important," as did 58 percent of the managers themselves. 4.17 In interviews with DLC managers during the site visits, these sentiments were echoed repeatedly. The following quotes demonstrate a sample of their views. 55Many centers were not functioning at the start of the fiscal year. This center only began in March 2003. 56This center only began in May 2003. Of the centers operating virtually all year, the lowest rate was 8.0% in Guatemala. 57During FY03, this center seems to have operated only October-December 2002. 58Q21. See Figure 14 for all responses. 48 · "If you gather all the conditions and you don't have the right manager, it won't work." ..."But if you don't have the right conditions but you have the right manager, he will manage." · "This job is difficult. People really don't understand the efforts we are making." · "The burden for success falls on center director--to find content and recover costs." 4.18 On the other hand, the web-based survey indicated that only 78 percent of the DLC managers had "a clear understanding of what roles you have related to the GDLN."59 The remaining 22 percent responded, "no," they did not. 4.19 The DLC managers' responses in that survey also pointed up some clear disparities in the nature of the roles. About 27 percent of the managers said that, on average, they spent 75% or more of their time working on GDLN-related activities.60 Another 15 percent spent more than half time (but less than 75%). About a third (34%) reported spending less than 25% of their time.61 It might be assumed that these managers are primarily in university-based settings, where the Network is only one of their many responsibilities, even if all their time is devoted to distance learning and/or videoconferencing activities. On the other hand, in a May videoconference with seven LCR region managers who are based at universities, individuals said that, to their surprise and dismay, 50, 60, 70--even 90--percent of their time was occupied with GDLN responsibilities. Other Important Factors in DLC Operations 4.20 In the web-based survey cited above (4.15), all respondents in the Bank and in the field were asked to assess importance of various factors for greater efficiency and effectiveness in DLC operations. The previous chapter summarized the overall assessment of the listed ingredients, but DLC managers have a special perspective. Their views are important as indicators of where they are putting their energies and where they are finding frustrations. The key factors are presented in Figure 14. 4.21 Besides strong leadership from the manager, the critical factors for operating effectively and efficiently include delivering quality content, having marketing 59Q7: The 3 response choices were yes, no, and there is a conflict between my understanding and the expectations of others. 60Q8. 61The remainder (24%) spent 25-49% of their time on GDLN-related activities. 49 capabilities, being supported by adequate staff, building partnerships, and connecting to the development dialogue of the country.62 Figure 14: Factors in Effective and Efficient DLC Operation Strong leadership from 3.5 DLC manager Consistent quality assurance system for 3.5 content Capabilities for marketing 3.5 learning events Adequate number of DLC 3.3 support staff Engaged local co-financing 3.2 partners Connection to development 3.1 dialogue of country Engaged local training 3.1 partners 1.0 2.0 3.0 4.0 Not Very Important Important Bank Priorities 4.22 Another subject where we find value in assessing the views of DLC managers is in ranking of Bank priorities in the use of GDLN. All respondents were asked to rank a series of statements about the network's purposes from 1 to 6 (and to add a seventh if they thought some important function was missing).63 This analysis of the top 2 priority choices in the web-based survey demonstrates that managers concur with some of the priorities listed by the whole group of respondents, but they have their own unique perspective as well. 62Q21: Responses are based on a four-point scale: Very important, important, somewhat important, not important. Respondents had the option of selection "Don't know." The means presented for this question are for those respondents who did not select "don't know." The responses that were less strong are the following (expressed as means): quality and size of facility-3.0, scheduling flexibility-3.0, ability to develop own content-2.9, ability to make decisions autonomous from host institution-2.9, general readiness of the country for reform or innovations-2.9, geographic location of distance learning centers in target countries-2.8, employment levels/occupations of GDLN participants-2.7, political/economic stability of country-2.7. 63Q11 50 4.23 In the DLC managers' view, the top Bank priorities are the following uses: · Disseminating knowledge to as many people as possible (60 percent of managers selected this as either the first or second priority; respondents overall had a similar view) · Providing a unique learning environment to key policy makers and other leaders (55 percent of managers listed this as either the first or second priority; respondents overall had a similar view) 4.24 On the other hand, the next most frequent choices of priorities that DLC managers selected were the following two items: · Maintaining high utilization of distance learning facility to successfully recover costs (35 percent of managers thought this a first or second priority; only 20 percent of respondents overall had this view) · Focusing intensively in a specific content area to address a major national problem (32 percent of managers selected this as first or second; a reduced number--25 percent of respondents overall-- thought this was a first or second priority) 4.25 A high priority for all respondents was bringing training to people who do not have other options (34% made it a first or second choice). This option had a very low rating by managers (7.7%), as it appears they are differentiating about where learning or training opportunities should be targeted. Interview Findings 4.26 To further inform the manager views from the survey presented above, site visits included extensive discussions with DLC managers, interviews with staff, and conversations with other key individuals who provide oversight and/or support to center operations. A videoconference with DLC managers in the Latin American and Caribbean region provided some additional insights. The following three common views emerged from these discussions: the importance of marketing, the high transaction costs associated with setting up and implementing activities, and the interest in establishing subnational networks. Marketing 4.27 All managers visited were very actively and effectively marketing--to gain both partners and participants. Their efforts were related to several important elements that emerged from the surveys--finding financial support for their activities, building interest and commitment from key institutions, and being able to provide a local perspective and local content. 51 4.28 Many marketing techniques were used--some country-specific and others that could be employed in many settings. Managers desire help in marketing and management from GDLN and/or experts in marketing and promotion, but there are numerous home-grown approaches to marketing that have been (or could be) shared. Some examples follow. One site developed and distributed a calendar with the center's logo that could be seen in many offices during the evaluation team's visit. The DLC logo was displayed on other items (pins, flags, etc.) and tied to the national football team when championship games were catching the country's attention. In another country, newspaper articles that describe the learning activities taking place in the DLC were promoted by the DLC and appeared in print regularly by employing the content description of the learning event as the basic text and packaging it for the local media as current event issues. One center manager uses a listserv of development organizations as a communication device with stakeholders and as an information source in implementing new events. Transaction Costs 4.29 Many center managers complained about the paperwork and inadequate systems to handle the many steps of setting up courses or dialogues. "We need support from headquarters to streamline processes. There's a lot of burden from logistics," complained one manager. "We have very high transaction costs--GDLN needs to facilitate transfers," voiced another. Managers at universities in LAC noted that they had not expected the amount of administrative tasks related to being part of GDLN and that the time they spend in management exceeds their expectations. Some in that group are planning to bring in additional staff for relief in the administration of GDLN so they can focus on relations within the university and more academic pursuits. 4.30 Some center staff have developed checklists and locally-based systems to help with the process but they are looking for relief from GDLN Services and their own increased experience in working with the systems to remedy the situation. The problem is exacerbated by limited staff: the manager, activities coordinator, and an IT specialist generally comprise the staff at non-university-based centers. One center tracked that it took 25 worked hours for each event in 2002. They continue to track that measure and aim to improve performance in subsequent years. Interest in Subregional Networks 4.31 The centers visited as part of this study had been generally been operating for two years or more. As they gained more experience in delivering services in the capital city, they have begun to consider the importance and potential of broader dissemination. Many see the value of distribution of knowledge throughout the country as part of the extension to key sectors and regions. 4.32 There are various means of adding one or more national networks. In countries with greater resources an option being explored is to work with universities, training organizations or other private networks to extend one's reach. This approach is particularly appropriate in a country like Brazil, where the GDLN center is in the capital 52 city but a critical focus of development efforts is in the Northeast region. For other countries, when they are addressing capacity enhancement in decentralization efforts or other grassroots projects, the use of existing networks for access to municipalities and provinces will be invaluable. 4.33 Other centers are exploring the costs of constructing one or more additional centers. These locations are often carefully selected to provide a valuable economic or governmental link. The discussions have centered around how a new location will reinforce the development agenda and what kinds of partnerships can be developed as well. The Bank's Information Solutions Group (ISG) has been helpful in looking at technical requirements and how centers can be outfitted with a modest budget. 4.34 An issue that current DLC managers do not appear to have addressed in any detail is how the new centers will be managed and the national network coordinated with budgets that are already constrained. This may be an issue that needs to be addressed in a more coordinated manner. While the purposes are significant and in line with Bank and country goals, what arrangements for administration and monitoring can be made without sacrificing quality? 4.35 In addition to managers' attention to intensifying efforts for knowledge sharing internally, they all expressed interest in knowledge sharing regionally. Some regions are further along in this process than others, but all sites visited mentioned that a critical component to their own professional development and enhancement to the network is through regional associations for mutual support and sharing. Some of the requests that were made for training in marketing and management can be supplied through exchanging effective strategies and peer mentoring for newer managers. CONDITIONS FOR IMPACT 4.36 One question in the web-based survey focused on the conditions for impact. Bank staff and DLC managers were asked what factors were important in influencing whether a GDLN activity is effective in contributing to the development dialogue in a country. The means are presented in Figure 15 below.64 4.37 Similar to the views of staff who are involved with GDLN in general, DLC managers expressed in the web-based survey that they are concerned about quality and relevance; they realize the importance of having appropriate experts and facilitation, which often means seeking out local expertise; they are looking for partnerships that will assist them in their efforts; and they recognize the importance of participant recruitment and follow-through by supervisors of these participants and other high officials. 64Q23: Responses are based on a four-point scale: Very important, important, somewhat important, not important. Respondents had the option of selection "Don't know." The means presented for this question are for those respondents who did not select "don't know." The responses that were less strong are the following (expressed as means): That it links with the Country Assistance Strategy (CAS) or Poverty Reduction Strategy Paper (PRSP)-2.9, location of distance learning center in the capital city-2.7, connectivity with an in-country network for broader distribution-2.7. 53 Figure 15: Most Important Factors for Development Impact (means) Overall quality of activity (content and pedagogy) 3.6 Relevance of the specific thematic area to 3.5 country need Having the appropriate experts and facilitator 3.4 Participant level/occupation 3.4 Involvement of partners who focus on 3.3 development issues Commitment of high level officials to follow 3.3 through 1.0 2.0 3.0 4.0 Not Very Important Important ACCOMPLISHMENTS 4.38 A portion of the web-based Bank survey was directed specifically at DLC managers, and in that section they were asked how their centers contributed to the development dialogue in FY04.65 Their responses (shown in Figure 16) illustrate the areas they are concentrating on to increase the effectiveness of their operations and to make a greater impact. More than half of the respondents were actively engaged in the kinds of activities that increase the relevancy and collaboration in development efforts. 65QDLC2: The other response categories with few responses that are not shown in the figure are the following: linked to sub-national networks to disseminate knowledge more broadly-37%, hired additional staff to increase capacity-35%, located funding to reduce costs for target participants-35%, and other-5%. 54 Figure 16: How DLCs Contributed to Development Dialogue in FY04 100 90 80 70.0 70 62.5 60 52.5 52.5 Respondents of 50 42.5 40 rcentageeP 30 20 10 0 Form ed new Increased ability to O rganized session S upported new E xpanded or prom ising develop local to gain expertise national priority of im proved facilties to partnership in content from other countries building capacity attract decision developm ent addressing sim ilar m akers com m unity problem s KEY ISSUES FOR ENHANCING IMPACT 4.39 DLC managers were also surveyed about impediments to their being effective in their efforts to contribute to the development dialogue in their countries. The listing of potential responses was developed from an earlier survey of managers implemented by GDLN management related to planning for reporting. In that survey, they were asked an open-ended question about challenges. By deriving these categories from the individualized responses, more consistent feedback on the views of all managers could be obtained. 4.40 The top five challenges cited in the survey relate to funding, content, staffing, publicity, and reliability of the technology. (See Figure 17.66) It might be said that funding, mentioned by more than half of the responding DLC managers (58%), was an obvious response in a survey whose results would be communicated to the Bank and 66QDLC4: Other responses with a fewer responses include the following: political climate in country- 25%, cost of maintaining infrastructure-25%, turnover of officials in public offices-18%, target group's reluctance to use new technology-18%, inadequate facilities-15%, lack of autonomy from host institution- 15%, other-13%). 55 figure in policy discussions. However, active centers, like the ones visited for this evaluation, were visibly stretched to meet current demands; conduct marketing to expand the enterprise and its effectiveness; and consider enhancements to improve quality, relevance, and impact. 4.41 Funding aside, almost half of responding managers (48%) selected lack of suitable content to address the country's needs as a challenge for current efforts to be a part of the development dialogue. Between 30 and 40 percent of respondents listed gaps in staffing, publicity, and technology as current hindrances to their engagement in the development dialogue. These same five challenges emerged as issues in on-site interviews and a videoconference discussion with center directors from Latin America. Figure 17: Challenges to Strengthening Development Dialogue in FY04 100 90 80 s 70 nt 60 57.5 spondeeR of 50 47.5 gea 40.0 40 entcreP 32.5 30.0 30 20 10 0 Insufficient funding Lack of suitable Lack of adequate Inadequate publicity Problems with reliable content to address staff for center activities functioning of the country's needs technology 4.42 In addition to these survey results, through interviews DLC managers identified a number of areas that present challenges to their ability to be effective in the short-run and to make a meaningful contribution to development in the long-run. A total of seven themes are presented below with specific quotes from these individuals. Content 4.43 Getting appropriate content is an important focus of a DLC manager's job. They are notified about proposed courses, seminars, and discussions that may be of interest, but after a little experience in operating a center, they begin to look for specific content that is relevant to local interests and which may help build capacity in targeted sectors. All the managers interviewed through the site visits were talking with local partners or 56 potential partners about filling the gap in content material to meet local demand. Such content will allow them to be more effective marketers of the center and of the concept of distance learning in general. This will also assist in the recruitment of participants who can benefit. Based on feedback from participants, focusing on local needs will help strengthen impact. The web-based survey found that about three-quarters of the responding managers (74%) had worked on content directly or through supervision or had advised a content provider on instructional design.67 4.44 Several different concerns and needs are reflected in the following quotes · "When marketing, I'm always asked, what is your curriculum? The vision has always been to have a core set of courses, so people could plan in advance--have 6 or 10 and build on that." · "We need to get content in the pipeline." "Incentives [to develop content] are still unclear." · "Where GDLN needs to really invest is in the coordination unit--to connect supply and demand." · "I went to WBI with a `wish list' of courses related to local needs. WBI had its own method of determining courses. It was not even clear that they were interested in GDLN." Quality 4.45 As noted above (Figure 15), when managers were asked to identify the most important factors influencing whether a GDLN activity is effective in contributing to the development dialogue, the most important item cited by them was the overall quality of the activity (content and pedagogy). Similarly, they also cited a consistent quality assurance system for content as the second most important factor in the effective and efficient operation of a distance learning center (Figure14) (rated almost at the level of strong leadership from the DLC manager). 4.46 Various perspectives on the quality issue and its importance were shared in interviews at the DLC centers. Quality encompasses many dimensions--content, pedagogy, and technical delivery. · "In terms of viability, need to have a certain level of quality--it's important for the center image, the credibility of the DLC." · "GDLN looks very impressive and causes high expectations; participants expect the best image." · "Quality control issue is critical or people won't come back. Need to make sure that it is planned ahead." 67Q12D 57 · "There is a need to invest in GDLN Services in setting up standards, promoting using GDLN as an interactive tool--even best specialists forget the interactivity and that compromises the results." · "With DC origination, monitoring and quality control is more clearly able to be done [there]. Who monitors quality is a complex issue. It needs to be comprehensive--it involves everything from listing the course to giving certificates." 4.47 Although several managers pointed to Bank headquarters' role in assuring quality, DLC managers as a whole are not clear as to whose responsibility it is. When asked in the web-based survey to identify the one entity that should ensure that GDLN content is of high quality,68 19 percent said that this should be the responsibility of GDLN services staff in WBI (which reflects a statistically significant difference from the opinions of other survey respondents). An equal proportion assigned the function to the content provider. Center managers also acknowledged their role--with 14 percent thinking that the distance learning center staff had that responsibility. A number of managers selected the "other" category and made suggestions for a board to set standards and monitor or proposed dual responsibility. Technology 4.48 In center and videoconference interviews managers stressed the importance of technical quality and reliability. During the site visits, the managers noted a perceived reduction in technical failures and glitches and improvements made in the technical quality of transmissions in the past six months or year. However, they all stressed that when a significant technical problem did occur, it may seriously affect the experience for participants and be detrimental to marketing efforts. "You only get one chance to make an impression," observed one. 4.49 Assessment of videoconference activities in 2004 provided a rating of "successful" or "excellent" for 83 percent of the VCs captured in the system (about 60% of them). However, several managers noted that the Network Operations Center (NOC) needs additional resources to be able to monitor and evaluate the technical issues at an appropriate level. This is particularly important for a network where there is uneven allocation of IT specialists among the centers.69 4.50 The focus on technology issues, though, extends beyond transmission. It overlaps with quality and using videoconferencing and blended activities in accordance with best practices for the media. As one DLC manager remarked, "[We have] more technology by far than we are capable of using--at the same time, we need to solve some gaps. People demand better results. This has more to do with bad usage of technology." 68Q10: Other responses not given as frequently were the following: Determined by the consumer (in terms of repeat business)-12%, GDLN regional staff-10%, and instructional designer-5%. The other category had 21% of the responses. 69This assessment system demonstrates interest in tracking technical performance. However, the standards being applied need external validation. 58 GDLN Services and Overall Bank Support 4.51 The GDLN Services teams include activity services, advisory services, business development and marketing, and studio services. In the view from the field these divisions and areas of responsibility are not always clear. Distance learning center staff look for communication and support from GDLN Services to help them with their responsibilities. While acknowledging the support they do receive, they have two criticisms--lack of clarity in whom to contact for questions or help and failing to address some of their pressing needs. 4.52 Lack of clarity is illustrated by the web-based survey which asked respondents if they knew whom to contact for GDLN-related questions on a variety of topics.70 More than one-quarter of DLC managers indicated that "No, it is not clear" in the areas of a proposal for content, content fees, and partnership development. An added complaint voiced in an interview was that when the center is used to working with an individual and that person leaves or changes jobs, the centers are not informed and their requests go unattended. Table 4 summarizes the DLC managers' responses by type of question. Table 4: Clarity in Knowing Whom to Contact (percent of respondents) Topic Yes, always clear Yes, usually clear No, not clear Request for content/ learning activity 34.2 48.8 17.1 Proposals for content 25.6 48.8 25.6 Event scheduling 56.1 34.2 9.8 Technical difficulties 46.3 41.5 12.2 Partnership development 12.5 62.5 25.0 Content fees 20.0 50.0 27.5 Connectivity costs 26.8 53.7 19.5 4.53 The GDLN team has designed various systems to assist in the scheduling and promotion of learning activities. A survey inquiry into how difficult or easy to use DLC managers found them yielded mixed results.71 See Table 5. 70Q6: The response categories are "Yes, it is always clear;" "yes, it is usually clear;" and "no, it is not clear. Respondents were also given the opportunity to note if the question was not applicable (NA). This analysis is for the subset of DLC managers who responded. Depending on the specific item, between 39 and 41 DLC managers provided a response to the topic. 71Q15: The question asked about a comprehensive list of tools. Responses to other tasks and activities may be found in the questionnaire in Annex B. DLC managers do not have access to CRS, so that their assessment of that specific tool cannot be properly interpreted. 59 Table 5: Ease of Use of GDLN Tools (percent of respondents) Entering data into the following tool: Very easy/ Very difficult/ Easy Difficult Learning Catalog 60 40 Activity Management System (AMS) 78 22 4.54 In interviews managers reported that systems were being modified and had been improving. The original participant registration system was termed a disaster. At the time of the survey, 60 percent of the respondents found Learning Catalog easy to use and 40 percent said it was difficult. More than three-quarters (78%) thought that AMS was easy or very easy to use.72 4.55 In interviews DLC managers expressed some frustration and interest in a number of supports from GDLN Services (or headquarters in general--often it was not differentiated), as reflected in the following quotes. · "They could give curriculum, real coordination, all the tools we need, more regular coordination." · "Maybe I don't understand the resources and politics involved but headquarters is not providing tools.... AMS has been evolving--it's pretty good but it could be a lot better." · "GDLN does not necessarily understand realities in the field. I'm sure they are doing their best, but they have a long way to go." 4.56 It was noted that a toolkit had been promised for some time, and centers were eager for this to be delivered. Examples of needs cited were online information that makes it easy for clients to use GDLN and a cost calculator. 4.57 Beyond GDLN Services, DLC managers voiced concerns about the World Bank commitment to the Network and the degree of encouragement that is given to others to use GDLN. Often these expressions were not clearly focused on WBI or World Bank operations or a specific group, but reflected the generalized sentiment that more should be happening from headquarters or other leadership to support center efforts--in terms of funding, content, quality, and other issues that have been detailed in this chapter. 4.58 "Despite huge resources [expended], the Bank hasn't really pulled together on GDLN"--was how one individual identified the issue. Specific instances of the lack of focus and support was the charge that efforts have been too dispersed. It was pointed out that regional vice presidents make different decisions, creating inequities and variations that do not help the network as a whole. 72DLC managers use the Learning Catalog in a read-only capacity, whereas they use AMS for both entering and retrieving data. 60 4.59 Some specific recommendations that center managers offered to demonstrate support and literally aid the centers are presented in the mangers' words. · "The Bank needs to buy in, and we need to sell this to our colleagues in the Bank." · "[The Bank] should have a written rule--using GDLN facilities should be required of task managers." · "Some World Bank staff do not know about the existence of the GDLN. It needs to be marketed within the World Bank. The DLC manager's job is local; the World Bank's job is global." · "WB operational teams have not engaged as much as they could." "The original idea was that they would be supply driven, to demonstrate how to set up events and then people would follow their model and supply their own. That expected supply did not materialize." 4.60 Some aspects of the system or the culture are thought to undermine World Bank or WBI connections with GDLN and the centers. · "Their [World Bank Operations officers'] focus is on projects. It is not clear who owns capacity building. This is a mission culture. They need to do a mission if there is a need to address something. This does not help the distance learning cause. The Bank does that [missions] through travel." · "There are no incentives. Use of GDLN is not in the [task manager's] performance review." · There are gaps in delivery caused by Washington cycles (such as fiscal year, summer vacations) and holidays that do not occur in the field. As one manager noted, "We have staff and bandwidth for all year." Partners 4.61 The importance of partnerships is highlighted in the surveys, interviews and site visits. Staff in many positions emphasize the value of partnerships. In the web-based survey, however, only 38 percent of responding DLC managers replied that they had been involved in developing co-financing or training partnerships for GDLN.73 In three- quarters of those cases, partnerships were developed for both purposes.74 4.62 In the web-based survey, managers reported both approaching organizations to initiate a partnership and being approached by organizations.75 The most frequent types 73Q25. 74Q26. Since these questions are addressed to managers who have developed partnerships, only 16 of the 43 DLC managers answered questions 26 through 31. 75Q27. 61 of organizations were teaching or research institutions, NGOs, donor agencies, national or central government, and the private sector. 4.63 Managers perceived that the greatest incentive for organizations to enter into partnerships was the link they provided to others working in development (cited by 75% of managers who had developed partnerships).76 Other important incentives given were greater visibility or exposure, access to World Bank branding, and the ability to implement development activity on a larger scale--all mentioned by 62 percent of the responding managers. 4.64 From the experience of the managers who had been involved in partnership development, the most effective practices for achieving that goal was inviting donors to GDLN events (mentioned by 88% of responding managers), developing links between themselves and representatives of development organizations (75%), maintaining contact with alumni (62%), and building case studies of developing countries that show the specific impact of GDLN (50%).77 4.65 The main reason that managers found that an organization was not willing to enter into partnerships with the GDLN was lack of experience with such partnerships--noted by 88 percent of managers with partnership development experience.78 Less frequently cited issues were lack of clear communication channels (50%) and the burden of entering data into the Activity Management System (AMS) (44%). 4.66 In interviews with partners and potential partners in the site visit data gathering, the evaluators learned that some established partnerships with international organizations or donors might be short-term, as these partners studied the value and potential of videoconferencing technology. There is a potential for competition by other development organizations with GDLN centers as the use of this technology becomes more accepted and infrastructure costs decrease. 4.67 DLC managers are not clear what their role is regarding partnership development, and where and how they are being supported. They are eager for partners, but the two quotes below summarize their views. · "I don't feel like I am getting any help. We need to get involvement of international organizations but that is not the job of the DLCs." · "At the beginning I was not supposed to spend so much time looking for partnerships. I do not believe that sponsorship is the work of the DLC--that is the work of GDLN services." 4.68 These views surface to some degree because the DLC manager is now looked to for partnership development. Given their other marketing and management assignments, 76 Q28: Other less frequent responses were access to possible bilateral funding-50%, organizations complement each other in a given subject area-44%, and other-6%. 77Q30: The only other response was "other," given by one person. 78Q29: Other less frequent responses were organization's visibility is overshadowed by World Bank-19%, difficulties in negotiating the Memorandum of Understanding-12%, and other-6%. 62 they feel they are being stretched too thin. They are in a good position to identify potential partners, but some managers have not seen the support of the Bank's country office or other exercises of the Bank's clout to effect these relationships. Monitoring and Evaluation 4.69 Although some DLCs offer courses in monitoring and evaluation (M&E), little is practiced at the centers. In the sites visited and through the videoconference with LAC, managers asserted the value of M&E, but they acknowledged that it is not currently being used to inform the centers. When evaluation takes place at the end of a course, the forms are sent to the task manager to tabulate and analyze. Most just forward the raw data; the only center action might be for the coordinator or manager to thumb through the forms to get a rough idea of customer satisfaction. 4.70 Again, the constraints of time and a small staff limit what could be done independently at each center. There is no tool or one source of guidance for centers who do wish to explore this area more. Even researching options for approaches that centers--rather than task managers--might take is time-consuming. Furthermore, at the present time there is no system for communication from the WBI or other content provider who has received and analyzed the data to the center where the event took place. This happens on an individual basis, based on a task manager's initiative, not on a center's interest. 4.71 Several statements from manager interviews reflect the lack of attention given to this issue at present. · "We do not have the staff to look at evaluating what we are doing the way we should." · "No systematic analysis of data, no way to look at trends." · "Would like to get feedback from WBI. It's a black hole. We give forms to course team leaders, but it's not clear if they are compiled centrally." Financial Concerns and Sustainability 4.72 Site visit interviews and the videoconference with DLC managers underscored the point that generating revenue, cost recovery, and the need for sustainability in a finite period of time are critical issues for most managers. One manager termed it: "Very stressful." 4.73 As GDLN centers have varied business models, different arrangements for their construction or affiliation with GLDN, various amounts of time they have been operating, and different country contexts, so they gave different messages about how active they are in pursuing payment for events that take place in their center and to what extent they are looking for additional means to cover the cost of operations. 63 4.74 A survey question addressed the range of options to pay for GDLN-related operating and maintenance expenses.79 Table 6 illustrates that multiple sources were employed by the responding managers. Table 6: Sources of Revenue for DLC Managers Source Percent of Respondents Rented facility for development-related activities 57.5 Collected fees from participants 42.5 Received funding from World Bank 37.5 Received funding from host institution 35.0 Received funding from partner 32.5 Rented facility for non-development-related activities 30.0 Received funding from government 27.5 4.75 The survey responses show that DLCs are supported and sustained by multiple organizations. Renting the facility is the only form of revenue that used by more than half of the respondents. Only three-eights of the respondents received funding in FY04 from the World Bank. Less than half (42.5%) collected fees from participants; in some countries it is not common practice or feasible, and other funders for participants are sought out. Host institutions, partners, and the government also play important roles in the funding equation. 4.76 Sustainability is an issue that figured into many conversations with DLC managers. Some feel the pressure of a defined time frame for reaching financial sustainability. Some are not currently receiving World Bank or host institution funding, so being sustainable is a very real issue they are dealing with. Some occupy a critical position in terms of the Bank's operation in a country that they do not feel that cessation of financial support is likely. 4.77 In an opinion question that was asked of all respondents in the web-based survey,80 83 percent of the managers agreed or strongly agreed that it was not realistic to expect sustainability while focusing on the development agenda. When DLC managers were asked in the survey at what point they expect their center to be fully sustainable without funding from the World Bank,81 about a third (32%) said that they did not currently receive Bank funding. In contrast to their statement that it is not realistic to expect sustainability, 22 percent indicated that they expected to be sustainable in the next 1 to 2 years, and still others (12%) thought it would be 3 to 5 years. A small number (8%) indicated that they did not expect to be sustainable for at least 5 years. In an 79QDLC6 80 Q9i: Response options to the statement "It is not realistic to expect GDLN centers to become fully sustainable while focusing on the development agenda" were strongly disagree, disagree, agree, and strongly agree. 81QDLC7. 64 interview, one manager said, "It is not realistic to have centers develop in 5 years. We need 6 or 7 anyway; we need more support, maybe for 10 years." 4.78 Fully one-quarter reported in the survey that they did not believe that sustainability is a realistic goal. 4.79 The discussion of sustainability raised a number of pragmatic and philosophical issues. The role of the GDLN centers was likened to a university library, in which its learning facilities are open to its students and the university budget covers this important component of knowledge transfer. 4.80 Managers offered several commentaries on this topic. One view of the issue focuses on the incompatibility between generating revenue and the ultimate purpose of the network: "It is not easy to conciliate the requirement of sustainability and the focus on development." Another view challenges the rationale for the requirement: "The sustainability issue is a strange debate. The message from the top is that this is to be self- sustaining. None of the other operations of the World Bank work on a commercial basis." 4.81 In discussions and observations at centers other challenges emerged, such as participant recruitment and dealing with language interpretation issues. They will be addressed in the next section on participant issues. 65 66 5. DELIVERY OF GDLN PROGRAMMING: THE PARTICIPANTS' PERSPECTIVE 5.1 Another facet of the evaluation of GDLN is to look at past participants in DLC activities. By inquiring directly of the intended beneficiaries what the learning experience was like, what further actions ensued, and whether the event made a difference for the individual or an organization or related to a problem, some observations can be made about the conditions that are critical for impact. Combining in- depth interviews and survey data can create an understanding of where GDLN--at least as it is implemented in the site visit countries--has been effective and what is there is need to enhance its effectiveness. 5.2 Many of the former GDLN participants who were interviewed82 through the evaluators' site visits also completed a 22-question closed-ended survey in their own language to gather information systematically over the four different countries. Learning experiences at the Distance Learning Centers are geared to participants' interests, needs, and ability to use the information in the development context of their countries. Their experiences, as reflected in this survey, help illuminate key considerations in designing and implementing the learning experiences and can inform planners of opportunities for enhancing capacity and strengthening impact. WHAT IS KNOWN ABOUT PARTICIPANTS 5.3 Limited statistics are kept by DLCs about the background of their participants. Managers may have a general idea of the background of those who attend events at the center, but they can not statistically describe a "typical" participant or depict with any confidence the various levels of experience of their attendees. The profile of participants interviewed, and specifically the 58 survey respondents, cannot be said to be representative of the group as a whole since it was derived from a convenience sample of individuals contacted by DLC staff and available for an interview, most of whom ultimately completed the written survey. Nonetheless, these data can provide a broad picture of attendees and frame an understanding of their views. Description of Survey Respondents 5.4 The participants who answered the survey and provided a quantitative view of their attitudes and experiences are educated, mature, and experienced.83 About one-third (33%) have at least a Bachelor's degree or equivalent, 43 percent have a Master's degree or equivalent, and other 16 percent have a professional or doctoral degree. The average age of these respondents was 44, and they have been doing professional work for 17 82The questions that formed the semi-structured interviews with participants may be found in the Participant Protocol in Annex C. 83These demographic data are obtained from questions 5, 19, 20, 21, and 22 of the Participant Survey, which may be found in Annex D. 67 years. The largest percentage of them worked for the national or central government (38%), followed by a teaching or research institution (15%), and local government (13%). The next most common work environment for these participants was NGOs and donor agencies (each was associated with 9% of the participants). They also appear to be technology-oriented--with 62 percent indicating that they use e-mail or the Internet every day, and another 24 percent using it at least weekly (10% 3-4 times per week and 14% 1-2 times per week). 5.5 About two-thirds (64%) of these respondents had participated in multiple DLC events--the average (median) number of activities was 2, but ranged from 1 to 1284. In the most recent DLC activity they attended, 88 percent participated with other members of their organization or agency (with 3 colleagues, on average).85 For about half of them, the supervisor attended the learning activity as well. More than three-quarters (79%) of those who went as a group worked together after the event was over in planning or implementing ideas or strategies learned. They also reported that their employer frequently sends staff to DLC activities (65% noted that), and in fact, 29 percent of the respondents said that a motivating factor in attending was their employer's decision to register them. Motivation for Attending 5.6 Describing their reasons for attending, participants identified their expectation that they would learn new techniques and strategies in the field as the prime motivator for attendance, as selected by 69 percent of those surveyed (See Figure 18).86 In other words, they were attracted by the subject but also the thought that new (and presumably valuable) approaches would be learned--that is, there was also an expectation of quality. The importance of the content was underscored as well by the next two most frequent reasons for attendance given: 1) the subject was important for the individual and his/her work and 2) the expectation that it would help in doing one's job better. This is a pragmatic dimension. Getting to know others in the field and an interest in the distance learning experience were also common but less important reasons.87 84Q7 85Data presented in this paragraph come from a set of questions about the relationship of the GDLN activities and the workplace (Questions 12-14). 86Q6 87Reasons selected by less than 30 percent of the respondents were it was employer's decision to register participant-29%, it was reasonable or free price-29%, desire to remain at one's job while learning-22%, recommendations of others-16%, and other-14%. 68 Figure 18: Motivation for Enrollment in Course/Activity 100 90 80 70 ents 60 56.9 Respond 50 of 46.6 46.6 tagen 41.4 40 37.9 rceeP 30 20 10 0 Transmitted new Applied what learned to Continued to work with Formally implemented Conducted additonal knowledge/skills to development plans others met at learning approaches/strategies research on topic others in my event learned organization FEEDBACK ON GDLN ACTIVITIES Important Activity Components 5.7 The survey participants were asked to identify the elements of the most recent learning activity that had the most value for them. Many of the items or processes they named constitute a unique aspect of the pedagogy or the technical capability of videoconferencing. The aspect of the activity that was most frequently cited as valuable was the ideas and discussion from other participants--given by more than 4 in 5 respondents (See Figure 19).88 Through interviews we learned that many of the valued contributions were from participants in other countries. Participants pointed out that the actual experiences of others (such as Uganda's work on AIDS) offered practical and inspiring examples. Some participants gave examples of action taken, based on the advice and successes recounted by other countries' DLC participants. 88Q8 69 Figure 19: Valuable Components in Course/Activity 100 90 82.8 80 75.9 70 stne 60.3 60 pondseR 53.4 50 of geatnecr 40 32.8 Pe 30 20 10 0 Contributions from other Expert presenters Reading materials Networking with other Actions plans or other participants participants materials prepared in course 5.8 In contrast, some participants commented in interviews that if the other countries involved in a videoconference were too different from one's own country, there could be little to take away from the sharing exercise. In virtually every site, people readily gave examples of countries that they thought were "compatible" or had lessons to offer--and countries that had little in common with them. At another level, it was proposed that the cases studies discussed by participants from other countries be chosen and prepared so that there is a benefit to others; if the focus is too narrow there may not be applicability for the other participating groups. More care should go into planning some people recommended. One individual from an international organization provided the example of a session looking at city development strategies. She asserted that cities of a similar scale would be the most appropriate to present case studies and exchange ideas and that sufficient attention should be given to this in the original planning and design of the event. 5.9 Expert presenters were the next most appreciated component in a DLC activity. This facet, a special aspect of videoconferencing, was identified by about three-quarters of the respondents. Highly qualified experts can often be available only because their presentations are able to be transmitted from their home location to multiple sites. Many 70 individuals who were interviewed commented that they would not have had access to such "prominent professionals" through any other means. They also noted that such an event was democratizing--that many individuals in a government department or other organization could hear directly from an expert, not just the one person who was able to travel to Europe or the U.S. 5.10 In interviews participants stressed that they liked the practical approach often present in the GDLN sessions. "People have degrees in the theoretical," one government official remarked. For example, people involved in poverty alleviation projects could present the reality. It was suggested that a mix of experts and those actively working with issues would be the most effective combination for presentations. 5.11 Participants also found the reading materials for the course or activity were valuable; this was named by about 60 percent of them. An important comment that many individuals made about the reading materials was that they needed to be made available in advance so that participants were properly prepared. In interviews, the quality of the written materials was frequently praised but participants felt frustrated that they could not learn some of the basic concepts in advance, reflect on them, and be formulating questions or ideas before the actual session. The sessions are liable to be more enriching and the post-event follow-up stronger if the participants are thoroughly prepared for the activities in the videoconference. 5.12 Two other frequently touted components of DLC learning activities--networking and creating action plans or other practical documents--were also cited as valuable by this group (by 53% and 33%, respectively).89 Several of the participants interviewed described ways in which the networking that occurred within the course had extended to further collaboration. Given the value of networking, participants had several suggestions about structuring the learning event in order to maximize these opportunities. At one DLC visited the lack of space to promote these socializing and networking experiences was commented upon by several participants. Satisfaction Indicators 5.13 The survey contained several indicators of customer satisfaction, by asking whether the participant would attend another event and would recommend to colleagues that they attend a DLC event.90 To both questions participants registered a very positive response, as illustrated below. 89Other components of the DLC activity that were said to have value, but were cited less frequently, were Internet resources-28%, follow-up activities after the event-26%, newsletters-19%, and other-7%. 90Q17 and Q18. 71 Table 7: Agreement with Statements About Future DLC Attendance Percent of Statement Respondents Saying "Yes" Would participate in other DLC learning activities in the future 94.7 Would recommend that a colleague attend a DLC learning activity 96.6 5.14 Participants further responded that the factors that would be important in determining whether or when they participated in the future centered on particular aspects of the course (specifically, the content, cited by 93%, and the length of the course, cited by 55%) as well as the cost, which was named by 57 percent of the respondents.91 5.15 In interviews as well, past participants voiced interest in being "invited" to participate again and provided many examples of topics of interest where the special features of the GDLN would be particular advantageous. The most frequently mentioned were globalization, trade issues, regulatory frameworks, decentralization, government reform and anti-corruption measures, public-private partnerships, and gender issues. Capacity-building was cited in all countries as an important focus on DLC activities. Quality and Relevancy 5.16 In the survey, participants were also asked to rate the quality of the learning event in which they participated and the relevancy of the topics presented to the country's development needs.92 On a 4-point scale, the means of both attributes were high: 3.2 for quality and 3.7 for relevancy. Figure 20 shows the distribution of the respondents' assessments. We also note that these were identified as areas of importance by DLC managers (see Figure 15). 91Q17a: Other factors that were less frequently cited as important were time of year-40%, experts who were participating-34%, partner organizations involved-24%, who the other participants were-21%, and other-7%. 92Q10: A 4-point scale for quality was used--very high quality, high quality, acceptable quality, and low quality. Q11: A 4-point scale for relevance was used--yes, very relevant, yes, relevant, somewhat relevant, and not relevant. 72 Figure 20: Participants' Assessment of Quality and Relevancy (percent of respondents) Quality of the learning event Quality 29.3 60.3 8.6 Very High High Acceptable Relevance of the topics presented at DLC event Relevance 72.4 27.6 Very Relevant Relevant RESULTS OF THE GDLN EXPERIENCE 5.17 Several different measures of the outcomes of the course/activity were provided in the survey. Participants were asked in what aspects of their professional development the event was most effective, how they applied their new knowledge and skills, and to assess the degree and areas of capacity enhancement. 5.18 Figure 21 illustrates the responding participants' views of the ways in which their professional capacity was enhanced through the learning event they considered most valuable.93 93Q9: All responses are presented in the figure--with the exception of "other"--5%. 73 Figure 21: Aspects of Learning Event Where Most Effective for Professional Development 100 90 80 74.1 stnednopseR sntpondeseRfo 70 62.1 60.3 60 56.9 53.4 51.7 of 50 Percentage egatnecreP 40 30 20 10 0 Updating my knowledge Providing new knowledge Providing contact with Enhancing my skills Providing new strategies Facilitating discussion experts or approaches and work between you and colleagues 5.19 The survey results demonstrate that the most significant area where participants thought that the learning event was effective in terms of knowledge gained (either updating knowledge or providing new information). These two options were selected by 74 percent and 62 percent of respondents, respectively. More than half of the responding participants selected the following other outcomes related to their own professional development: providing contact with experts, enhancing skills, providing new strategies or approaches, and facilitating work and discussion with colleagues. 5.20 Perhaps a more interesting assessment that the survey respondents made was in reflecting on the results of their DLC experience. How did they apply what they learned? What tangible results can they cite? Figure 22 depicts the responses in terms of specific applications of the knowledge, skills, strategies, resources, etc. that were garnered from the DLC activity that the participants attended. 74 Figure 22: Work Accomplished as a Result of DLC Learning Event 100 90 80 70 nts onde 60 56.9 spondentseR 50 46.6 46.6 of speRfoegatnecreP 41.4 40 37.9 30 Percentage 20 10 0 Transmitted new Applied what learned Continued to work with Formally implemented Conducted additional knowledge/skills to to development plans others met at learning approaches/strategies research on topic others in my event learned organization 5.21 Only one type of result was reported by more than half of the responding participants. Almost 57 percent said that they transmitted new knowledge and skills to others in their organization. Given the workplace orientation of many of the DLC activities and the fairly common experience of attending with colleagues and continuing to work on the event topic after the session(s) discussed in 5.5, it is not surprising that participants identify this as an outcome. 5.22 Other common manifestations of participants' DLC experience, according to the survey, are applying what was learned to development plans (local, regional, national, or sector); continuing to work with others met at the learning event (actually, more of a process that underscores the networking component rather than a tangible product); formally implementing new approaches or new strategies learned (which was identified as an important motivator to attending and an important result in their experience); and conducting additional research on the topic. These outcomes are all presented in Figure 22. 5.23 Other less frequently reported post-activity applications are changing work practices at one's organization, developing new alliances or partnerships, making presentations on the activity topic to those outside one's organization, producing new 75 written materials on the activity topic, initiating or modifying a project based on new learning, and organizing a new initiative to promote change.94 5.24 GDLN has clearly identified capacity building or capacity enhancement for the individual, the organization, and the country as a desired outcome of the use of this system. Without a more in-depth study of specific courses and specific projects, products, or activities that resulted, it is not possible to independently judge the extent to which capacity enhancement has taken place due to DLC offerings. In this study, a question was developed to learn the participant's perception for his/her own growth and the effects beyond his/her personal development. Respondents were asked to indicate the extent to which they thought the activity they participated in enhanced capacity.95 5.25 The mean score of the assessment of capacity enhancement at three levels is shown in Figure 23. It illustrates that participants were more certain of their own professional development, rating it high (mean = 4). Their assessment of the effect on the organization's or country's enhancement of capacity was not as high (both with a means of 3.6), but still clearly above the mid-point of the scale. The lower rating could also reflect less certainty on the part of the respondent about how to judge these effects at a broader level. Figure 23: Assessment of Capacity Enhancement by Participants (means) Own professional development 4.0 Capacity of organization 3.6 Development of country 3.6 1.0 2.0 3.0 4.0 5.0 Low High 94Q15: The item was phrased, "We are interested in what you did in your work as a result of the DLC learning event you attended. (please check all that apply). The response categories, with less common selection, were developed new alliances or partnerships with your country-34%, changed work practices at your organization-34%, made presentations to others outside your organization -29%, produced new written materials-28%, initiated or modified a project to take new information and approaches into account- 26%, organized a new initiative to promote change-26%, collaborated with new international partners-26%, implemented the planning begun at the learning event-16%, revised policies-14%, revised regulations or legislation-3%, and other-3%. 95Q16: A 5-point scale was used, with the end points marked as "low" (1) and "high" (5). 76 5.26 Through interviews, former participants provided very tangible examples of impacts on their work, their contributions to development in their country, and their serving as a resource for other professionals. Below are a few examples of "success stories." 5.27 Although a new national procurement law was enacted by the government, there had not been any effective capacity enhancement for the mid-level staff who oversee the process. The training offered at first was one that presented theoretical papers and it still was not clear how to administer the law. One procurement officer described the transparency he is now able to introduce into the invitation, evaluation, and awarding processes based on the GDLN course specifically because practical examples were given, relevant case studies were presented, and there was discussion and time to ask questions. 5.28 The director of a municipal council recounted an experience where action planning in an anti-corruption course with a core group of four staff moved into a strategic document that will be presented to the councilors for approval. Their draft paper on anti-corruption is the first of its kind that incorporated staff from the education, health, and finance departments. The course also influenced municipal planning processes to be more participatory. 5.29 A global dialogue on AIDS featured a number of countries describing their efforts to promote education and reduce the infection rate. One participant was motivated to continue exploring strategies of communication to change people's behavior. Now, through her efforts, a "friendly center' has been established for information dissemination on HIV/AIDS to youth. 5.30 After a DLC activity focused on microfinance, the managing director of a youth employment agency developed the proposal to open a savings bank for youth which would operate like a credit bureau. In December 2003 the first branch was launched in the capital and 300 youth opened accounts on the first day. Several other branches are opening in other regions, and other countries are inquiring about the bank. The course helped him link the problem of youth needing self-financing for starting small businesses and the solution of a credit union. 5.31 Staff with little practical experience attended a course on planning and budgeting. As an outgrowth of course activities, they initiated action planning in their ministry. Using participatory processes over the period of a year, they gathered input from youth about what youth needed in a wide range of subjects. This strategic plan is the first of its kind in the Ministry of Youth and is being looked at as a model for others--in terms of product and process. 5.32 An analyst for UNDP had a positive experience in a course on program evaluation working in a subgroup of 12 where they applied World Bank methods to look at social impact of a road project. He now incorporates World Bank techniques as he finds it useful to gather data from the initial stage and look at critical stages in implementation, and not just wait until the end. The event stimulated his discussion with counterparts and gave him new tools and new ideas. 77 5.33 For a special workshop in connection with reforms in the management sector and working with public-private partnerships in transportation, the director wanted to add some international experts to the discussion process. The U.S. Federal Highway Administration (FHA) was thought to be an ideal contributor but could not travel for the event. Through videoconferencing, they participated in a very successful workshop, attended by 80 key individuals, including some from the international development community. The session was organized as a 1/2-hour presentation and an hour for questions. Preparations and planning with the FHA were facilitated by videoconferencing at the DLC as well. KEY ISSUES FOR ENHANCING IMPACT 5.34 Unfortunately, from the scores of participants in DLC events interviewed in person in four countries, there are a limited number of compelling examples of transformations, long-lasting changes, significant reforms--that is, the translation from learning gains to changes in public policy and institutional capacity. From the issues the past participants interviewed candidly raised and the suggestions for improvements they offered, a number of observations can be made about conditions that would promote effectiveness and enhance impact of the DLC centers and their offerings. Strengthening Courses 5.35 Courses and activities offered at DLCs need to be strengthened. In trying to reflect on how they used the content from the course to make changes or apply new techniques, some participants came to the conclusion that the events were too superficial to have a real impact. Although positive about the experience, they would acknowledge that the topic had been just introductory, an overview--but not in sufficient depth to really effect change. Instead of an overview session on urban planning, participants want to focus on specific sectors (such as sanitation or housing or the economy) and are looking for specific tools to use in work that is often specialized. 5.36 Frequently when describing why the activity did not serve as a catalyst for doing things differently, they pointed to the length of the activity. If a topic is presented for only one-half day, it is not likely to be transformative. Strengthening courses is to some extent about content, but it is also about the bringing in the right participants, offering subsequent topics that build on the overview and provide more depth, identifying which participants should be invited to these later sessions, complementing the videoconferencing with center-based facilitation and activities that look at the application and give a more profound experience, and providing follow-up (perhaps through a website) so that when participants return to their work setting there is the expectation of action and the opportunity to continue the dialogue and to continue learning. Another strong recommendation from participants was to disseminate the reading materials early, so that they are adequately prepared to get the most from the center activity. 78 Recruiting Participants 5.37 Targeting certain types of participants also contributes to the likelihood that learning and tools from the DLC activity will be applied. Participants should be selected based on their ability to benefit--that is, having the proper background for the level of the activity and being in a sector or organization where there are chances to employ the new knowledge, skills, or strategies. To some degree selectivity in bringing in participants is at odds with the push for cost recovery and filling all the available seats. However, numerous participants expressed concern about the diversity of the attendees. If a senior official or experienced manager observes a young person with no background and little outlet for working on the subject, it weakens the credibility of the event. This is especially so if, as part of the instructional design, the participants work in groups to address a problem and the inexperienced person cannot contribute. (Obviously, capacity- enhancing events may target exactly that individual--but there must be a match been course expectations and participant capabilities.) On the other hand, it is important to acknowledge that centers have a convening power and can bring together individuals or groups from different arenas who produce a synergy around development issues. 5.38 Another strategy of recruiting of work groups from an organization or government ministry, which frequently does occur, appears to be a particularly good approach for promoting follow-up activities. A critical mass (maybe only three or four from a workplace) can make a difference. Participants cited numerous examples of the importance of having a cadre of professionals to implement what had been learned. Involvement of the supervisor--either in the DLC activity or in communication about the goals and expectations of the DLC offering--helps ensure that the experience carries over to the work setting and is supported and valued. Increasing Marketing and Visibility 5.39 As participants discussed their experiences and provided suggestions for improvement to GDLN and specifically their country's DLC, they often encouraged expansion of promotional efforts by the center. Realizing the potential of the knowledge sharing that the DLC is providing, they urged greater visibility in areas critical to the country's development and where they perceived there would be significant benefits. These marketing efforts work in tandem with the targeting of participants who are in a position to use the global dialogues and courses offered by the centers. Promoting Communities of Practice 5.40 Participants who had been able continue the dialogue begun at the DLC activity and used these connections to further their post-program efforts were particularly advocates of promoting this practice. Participants frequently mentioned their interest in meeting others in their field as a motivation for attending activities and saw it as one of the major benefits of their experience. Networking is an important part of the GDLN philosophy, and conscious efforts on the part of the DLC to foster informal participant interactions during an event and to follow up with communications and actions that reinforce these communities are likely to extend the reach and impact of center activities. 79 Supporting Relevancy 5.41 Although the survey findings demonstrate participants' positive assessment of the relevancy of the activities they have attended, in interviews they repeatedly offered suggestions for helping the centers better focus their efforts on development issues and the needs of the country. Impacts are more likely to be found when participants can make the connection from the global example to conditions in their own setting. The former participants suggested infusing the courses and dialogues with country-specific information and examples. For sessions to be meaningful, interviewees asserted that they needed to relate to the country's experience. The reading materials, the choice of facilitator, the background information given to experts, and the recruitment of well- positioned participants were all methods of strengthening the applicability of the learning activity to the needs of the individuals, their institutions, and the country as a whole. Developing partnerships to bring in local content has been goal and a challenge for the managers who work to ensure the success of the centers. Focusing on Language 5.42 To some extent in Brazil, but most definitely in the site visit to Vietnam, the need to work in the local language was raised as an important subject. It was said by participants and many other informants that most Vietnamese participants do not speak English at a level to be able to benefit from programming that does not have interpretation services provided. On the other hand, we heard repeated critiques about the awkwardness and pace of the interpretation process and the quality of work produced. Numerous participants identified this issue as a barrier to getting maximum benefit from the activities. Several suggestions for improvement were given, including better preparation of the interpreters in the subject and vocabulary of the learning event and better orientation for participants who are using headsets for the translation for the first time. As it is likely to be an issue for a good number of developing countries, this topic needs to be thoughtfully addressed in a more comprehensive manner, with best practices disseminated. Finding Ways to Listen to Participants 5.43 We found that participants had a wealth of ideas about strengthening the impact of GDLN and a belief in the contributions that the DLC could make to their country's development. However, they noted that, with a few exceptions, their input had not been solicited. Practically speaking, it is difficult for DLCs to adopt wide-ranging evaluation strategies. However, feedback from participants can be encouraged and supported in a variety of simple ways that will reinforce communication. Short post-program questionnaires, follow-up activities, and convening focus groups or an advisory panel were means that participants thought they could aid GDLN's efforts in knowledge sharing. Centers may want to consider acquiring some local expertise--from a university, for example, which is the location of a good proportion of DLCs worldwide-- to gain feedback using appropriate methodologies. 80 6. FINDINGS AND RECOMMENDATIONS OVERALL CONCLUSION 6.1 After four years of operation, the GDLN is still in its developmental phase. The network continues to grow rapidly, but there is no widely-held understanding about what support functions are or should be centralized. The centers that comprise the network represent a diverse array of settings, managers, resources, infrastructure, markets, and business models. Those content providers at the World Bank who have used the network tend to report a high level of satisfaction with their experiences and espouse the value of GDLN, particularly for enabling experts in other locations to contribute to a learning event and reducing travel costs. However, there is currently no shared vision among Bank staff about what GDLN is or could become, nor is there evidence of any deliberate effort to leave the vision undefined--a strategy sometimes endorsed for complex adaptive systems. An intentional vision is needed for GDLN to increase its integration into the work of the Bank and sustain a viable, highly utilized network in the future. 6.2 The evolution of GDLN reflects a broader shift within WBI from an emphasis on knowledge transfer--getting the expertise from the Bank out to those who need it--to one on knowledge sharing--convening those with expertise to enhance capacity in development communities. The current strategy for GDLN management is to promote a financially viable network by forging connections with donor agencies and foundations and tapping into the development business. The necessary steps, however, have not been taken to transform GDLN into the knowledge-sharing vehicle of choice for the development community. In fact, the network in its current state may lose its edge as technology for videoconferencing becomes more commonplace in the developing world. 6.3 Efforts underway to decentralize GDLN focus on the end goal of largely autonomous, self-sustaining distance learning centers, but there is not yet a coherent network effectively linking these many locations. GDLN and World Bank management must play a critical central role, drawing on the Bank's stature and convening power without fostering long-term dependence. As outlined in this chapter, the impact and effectiveness of programs facilitated by GDLN could be increased by refining network supports for centers, addressing issues related to content, assuring quality in content and operations, integrating evaluation practices, and actively promoting the network both within the Bank and to external partners in the development community. SUPPORTS FOR GDLN CENTERS 6.4 The impact of GDLN is determined largely by the distance learning centers where learning and knowledge exchange activities are delivered. It is their role to mobilize the appropriate participants, provide a high-quality experience for attendees, and reinforce the content in ways that will foster application, adoption, and adaptation in key sectors. 81 In these settings, the managers must contend with a range of factors to optimize the use of scarce resources. While some aspects of center autonomy are critical for responding to local conditions, there are some functions that need to be refined or more clearly defined centrally by the network in order to support centers better. Findings The Role of the Center Manager 6.5 DLC managers play a critical role in the operation of distance learning centers. GDLN stakeholders at the World Bank and DLC managers themselves agreed that this role is the most important for a center to operate effectively and efficiently. Managers bring an entrepreneurial spirit to centers and often undertake heroic marketing efforts to gain financing, engender commitments from key institutions, and attract participants. At the same time, managers are not clear what is expected of them in their capacity as a center manager. Both survey and interview data revealed that many managers are unsure what responsibilities they are expected to handle and when they can turn to GDLN Services for support. 6.6 DLC managers comprise a diverse pool: some work less than quarter-time on GDLN and some work seemingly around the clock. Some are located at independent centers whose raison d'être is GDLN, whereas others are based in a university setting or preexisting institution where GDLN programming is just part of a larger picture. Some have adequate staff, sufficient equipment, and strong links to a regional association of managers while others may feel they are working in relative isolation with outdated equipment and little staff support. 6.7 Whatever the setting, some managers have noted that important aspects of their responsibilities, such as marketing efforts, risk being compromised by the administrative burden of working with GDLN. They reflect that the excessive paperwork and inadequate systems make setting up each course or dialogue extremely time-consuming. Staffing 6.8 DLC Managers rely on critical support from their staff, which often includes an activities coordinator and an IT specialist. Although the numbers of dedicated staff vary (and there may be none at university-based centers), both GDLN users at the Bank and DLC managers identified having adequate staffing as an important condition for effective operations, and many managers identified gaps in staffing as an impediment to actively contributing to the development dialogue in their countries. Center Facilities 6.9 The understanding of what types of facilities best support GDLN activities has evolved over the life of the network, and the design of some centers is not conducive to optimal GDLN programming. Some earlier centers were established by retrofitting existing facilities, with the essential element being a videoconference room. A separate 82 computer lab was also often prescribed, so that participants would be able to get online without competing for valuable time in the videoconference room. 6.10 Although some participants have reported that such centers have impressive technology, they also note that the layout of these facilities decreases utilization and supports the concept of GDLN as just a technical tool rather than as a blended approach to learning. For example, some centers do not have any space where participants can wait when they first arrive, so it is difficult for the manager to schedule back-to-back sessions. Some centers also lack any sort of cafeteria or social area to engage participants and support networking. There may not be adequate areas for break-out sessions and the interaction of work groups. 6.11 In many cases, newer centers are incorporating lessons gained from the earlier ones, and the computer set-up is more fluid now as designers realize that computers may be used by those in the videoconference session or that a portable lab could be the ideal solution. Managers of other centers who felt limited are actively exploring expansion ideas or developing innovative practices to make do with what they have. Utilization and Sustainability 6.12 Generating revenue, recovering costs, and becoming sustainable are central issues for most DLC managers, but their sense of urgency varies depending on their business models, country contexts, and amount of time they have been in operation. Given the limitations of available content, many managers find that the optimal way to increase utilization is by offering a mix of more traditional training courses and development dialogues, meetings, or negotiations. However, maintaining a high utilization rate is difficult given the labor-intensive process of setting up each session and the limitations of getting programming from Washington given time differences and Washington vacation and fiscal year cycles. 6.13 Managers increase revenue through a variety of practices including renting their facilities, collecting fees from participants, and receiving funding from a local partner or government agency. Even so, many feel great pressure to become financially sustainable and a sizeable number (25%) reported that this is not a realistic goal for a network focused on development. While others indicated that sustainability was possible, many noted that the prescribed timeline was arbitrary and did not take into account local conditions. Costs 6.14 Much confusion exists surrounding cost issues related to GDLN. Both Bank and DLC staff expressed uncertainty about how content and connectivity fees are determined. In fact, negotiating costs for learning events was rated by survey respondents as the most difficult component of using GDLN. Bank staff and DLC managers identified content fees as the issue where it was least clear whom they should contact for information. 6.15 Content providers identified high connectivity fees as a disincentive for using GDLN, but they also asserted that GDLN saved money by eliminating travel costs. 83 Active users described an ideal use of the network where the high overhead costs of running a session were diluted by disseminating the broadcast to more participants at more GDLN sites and tapping into local networks. 6.16 DLC managers and staff were concerned about revenue--especially in light of their charge to be financially sustainable--more than cost issues. For managers, the more significant cost is that of staff time to schedule and deliver an event at the center. Managers shared the confusion of Bank staff about how to determine content fees. Some centers were able to recover content costs by charging participant fees, finding other entities to sponsor participants, or charging a small amount for refreshments. Local custom and acceptance factored into fees charged. Technical Issues 6.17 Both DLC managers and content providers indicated that although the technology has become more reliable over the past year, technical problems continue to pose a barrier to effective use of the network. 6.18 Technical issues reported covered a broad spectrum with audio and visual problems occurring due to signal interruptions, equipment failures, technician errors, and even multiple devices with infrared signals competing for the same frequencies. Some centers struggle to operate smoothly with outdated equipment or no technician on site. Others have full-time technical support and well-functioning equipment but have trouble linking with other centers in the region. 6.19 Those interviewed both at centers and Bank headquarters noted that, in the end, technical quality during a session depends on all participating sites, so having a flawless experience hinges on a vast array of factors. Determinants of quality include not just equipment and transmissions but also the ability of content providers to use appropriate techniques and even, as one stakeholder noted, high enough morale for the studio technician to care. Subregional Networks 6.20 Both DLC managers and content providers are interested in disseminating knowledge beyond the capital city through subregional networks. In countries with adequate resources, GDLN centers sometimes have the option to connect with universities, training organizations, or other private networks to extend their reach. Where such networks do not already exist, GDLN centers are exploring the costs of establishing additional centers in locations that provide a valuable government or economic link to reinforce the development agenda. 6.21 In the absence of clear links between GDLN and existing networks, WBI task managers may view the networks as competing and opt to use the one with the greater potential to disseminate locally, particularly if that network is linked to an education institution with associated subject matter expertise and specialists. In fact, content providers at Bank headquarters noted several examples where they had used videoconference facilities within ministries of education or other institutions rather than 84 those of GDLN. Similarly, some staff chose to use available facilities in World Bank country offices because of lower financial and administrative costs. Systems to Support the Network 6.22 GDLN stakeholders expressed a range of opinions about the various systems provided by the GDLN team for scheduling and capturing data on network activities, but nearly all asserted that further improvements were needed. Many of those interviewed at headquarters were particularly critical of the Activity Management System (AMS), citing that the program was not user-friendly, that it often froze in the middle of an entry, that scheduling an event there did not guarantee that the studio was actually booked, and that requiring its use posed a disincentive to using the network. 6.23 DLC managers were generally more philosophical about AMS, noting that this system in particular needed improvements but that the GDLN team had this process underway. However, many managers also reported that their administrative duties were excessively burdensome and attributed this at least in part to poor system support. They noted that they had been expecting GDLN Services to provide a toolkit and other promised assistance that would include online resources and better advisory services. At the time of this evaluation, such systems were in development. 6.24 Bank task managers also complained about inadequate systems, expressing frustration with using GDLN's AMS for proposing and scheduling events and WBI's Client Registration System (CRS) for entering participant and Level 1 evaluation data. Some content providers also noted difficulties with the Bank's SAP accounting system and the process of getting distance learning centers paid according to the agreed-upon terms. Participants 6.25 Participants are a critical ingredient for enhancing capacity and delivering impact through GDLN. Having the right participant for each offering at the DLC is the aim. Knowing how to accomplish this is the challenge. Marketing the GDLN concept and having access to the appropriate personnel in the relevant organizations and agencies is a basic starting point. At this point this task has been left to the DLC manager--and from their perspective, with little or no support. The need for generating revenue has probably led some managers to open the doors to anyone who wants to attend. Feedback from participants indicates that this undercuts the prestige that is attached to a World Bank initiative and can dilute the effectiveness of the interactions and instructional strategies if some participants are not sufficiently experienced to contribute to higher level topics. Recommendations 6.26 GDLN management needs to clearly define and communicate the network governance structure and responsibilities. In keeping with the network concept, experiences from the field should be systematically collected at headquarters and made available to centers as a benefit of network affiliation. GDLN Services has a critical role in facilitating knowledge management that can build on the practical experiences of 85 successful managers and disseminate best practices in a more structured and systematic way. Suggestions for providing better support to centers include the following: · Clear guidelines for what a center manager's responsibilities are and whom to contact when questions arise regarding a particular subject · Examination of the actual transaction costs of implementing a GDLN event at the center level (This information will inform decisions regarding staffing needs, utilization potential, and steps needed to enhance center capacity.) · Central guidance on best practices for center design (Managers should have easy access to all available information as they contemplate expansion or the setup of another center in the country.96) · Collecting best practices on how to market and structure learning activities for optimal effectiveness and impact (Although the particular types of participants targeted will depend on the context of each country, there are other objectives that virtually all centers share, such as attracting high-level officials to DLC events.) · Examples of programming mix and success stories from the field to illustrate the possibilities in achieving sustainability · Models of center-specific plans with strategies for identifying and recruiting participants in support of their particular country's development agenda so that DLCs might be able to develop a marketing plan suited to their own situation · Input solicited from managers about improving communication and support systems (Many managers have developed checklists and other tools locally that might prove helpful to others in their region.) · Transparency of costs for core GDLN stakeholders (Steps to address this issue include providing a cost calculator (recently developed), clear guidance about whom to contact regarding connectivity and technology-related fees, and standard advice on how to negotiate costs for learning events.) · Efforts to reduce technical difficulties. (This must be a priority for the network, particularly since technical quality is a key factor determining return use among both content providers and participants.) · Assessing what roles and services that headquarters should provide as centers become more active in negotiating connections to existing subregional networks or working to develop their own 96 The GDLN Services Division is addressing this issue. An updated design guide for facilities was released in May 2004. 86 6.27 Several other actions need to be taken in support of GDLN Centers. GDLN Services should continue to assess the reliability and burden of AMS, and perhaps in conjunction with WBI and ISG, examine how systems used by GDLN can be better linked to facilitate transactions, improve inter-operability, and avoid duplication. As part of strategic planning for GDLN, what a realistic timeline is for new centers to become financially self-sustaining should be revisited based on experiences to date with differing local circumstances. 6.28 It is also important to determine within the broader WBI and Bank context what strategy is appropriate for supporting centers in their endeavors to develop partnerships with the local offices of donor agencies. A clear management decision regarding this issue is essential given the evolving vision of having GDLN tap into the business of development. ISSUES RELATED TO CONTENT 6.29 The effectiveness and impact of GDLN as a vehicle for development is determined by the availability and appropriateness of content--largely provided by the World Bank, to date--for network programming. The contribution of GDLN to the development agenda therefore depends on how well activities address local needs, whether activities are designed and delivered for maximum effectiveness, and the degree to which activities are aligned with World Bank and Millennium Development Goals (MDGs) and other assessments of countries' needs. Findings Meeting Local Demand for Content 6.30 Content provided through GDLN is not consistently adapted for local use because the appropriate roles and expectations for doing this are not clearly defined, resources and contacts are not always available to bring in local expertise, and performance measures for WBI task managers may provide disincentives. 6.31 DLC managers receive mixed signals about what their role is regarding adapting and developing content. Some stakeholders argue that the manager's only responsibility is to host activities while running a business; others believe the manager is best suited to plan content development and delivery with local partners. Data from the site visits and web-based survey indicate that DLC managers themselves generally view getting content as an important part of their jobs and are actively looking at how to collaborate with local partners to meet local needs. 6.32 The performance of World Bank task managers is assessed to a sizeable extent by the number of participants they reach and the cost per participant. Interviews revealed that some feel pressured to recycle content (engendering the complaint of its being supply-driven) and reach as many people as possible rather than spending time and resources to customize it for different audiences. 87 Effective Instructional Design 6.33 Reflections from participants illuminated concrete steps for improving the participant experience and increasing impact. Participants are a valuable resource for identifying both the potential and the shortcomings of GDLN activities that they have attended. One common complaint was that participants did not receive materials in advance to prepare for their session. Participants also commented on the too-formulaic nature of some events which had a specified time for a lecture and then a brief rotation to participating centers for questions. This format sometimes included a limited time for small group discussion which seemed inadequate to some for processing information. Some also noted that there were no opportunities to network with their peers since the manager must optimize use of the videoconference rooms and there was nowhere in the center to gather and socialize. In addition, participants noted that sessions had greater impact where follow-up activities were included, either in the form of another session or the establishment of a listserv or other mode to facilitate a community of practice. Alignment with Development Goals 6.34 DLC managers and other GDLN stakeholders understand the network's central mission of furthering the development agenda; however, there appears to be no central push by either GDLN or the World Bank to align the network programming with common objectives such as the World Bank goals or MDGs. To date, GDLN management has not provided guidance to align GDLN activities with any specific development goals and the themes used as an organizing principle for the Activity Management System (AMS) do not correspond to standard World Bank themes or sectors. 6.35 Centers exist in a variety of contexts, with some closely linked to a public reform initiative or to the Country Assistance Strategy and others with a less defined focus as they work to maximize utilization. Content providers at the Bank often use GDLN as one part of their broader work in a country or region to support a formal Bank strategy. However, there appears to have been no formal or centralized effort to associate GDLN with internationally recognized development goals despite the strategic plan for becoming a prominent network of the development community. Recommendations 6.36 The following concrete steps should be taken to maximize the effectiveness and impact of programs delivered through GDLN: · Clearly define the optimal role(s) for a DLC manager regarding adapting content (which may vary by business model or setting). Clearly communicate this role to managers and provide technical assistance where possible to help them meet expectations. Such assistance could include guidance on adapting content and links to local or regional experts or facilitators. A central database with 88 resources, contact information, and possibly even selected case studies could provide valuable support in this process. · Reevaluate the performance measures used to assess WBI task managers. Although there are many incentives for using GDLN, such as eliminating travel and allowing for on-the-job knowledge exchanges, current performance measures provide a disincentive for tailoring programming to local needs or, in some cases, for using the network at all. · Disseminate best practices for instructional design and delivery to distance learning centers in an easy-to-read format, translated into local languages. Such practices could include the following: o Methods for adapting content for local relevance including incorporating local examples or case studies and involving a local facilitator. o Technical tips for a videoconference session including standards for PowerPoint presentations, suggestions for interactivity, and even basic techniques such as how to speak into a microphone and when to zoom the camera. o Suggestions for promoting a community of practice including providing materials in advance, allowing for networking before or after the session, and conducting follow-up activities. · Determine what indicators are appropriate for assessing how well content is aligned with the development agenda. If the network is to be more closely associated with the MDGs or World Bank goals, then standard guidance should be issued to centers and content providers--particularly as the number of external partners increases. Also, systems such as AMS should be reevaluated to determine whether they are capturing the most appropriate information regarding programming and conveying the agreed-upon network philosophy. QUALITY ASSURANCE 6.37 The perception of the overall quality of a GDLN session is a major determinant in whether participants and content providers alike decide to use the network again. All types of GDLN stakeholders agree that content must be of sufficiently high quality, but the issue of quality assurance has no ownership, and this increases reputational risks for the World Bank, GDLN, and associated partners. Findings 6.38 Quality programming is a key ingredient for impact that shapes a center's reputation and attracts participants. All aspects of a GDLN session contribute to its overall quality, including having the right experts, incorporating the appropriate level of 89 interactivity, avoiding technical problems, and even using the optimal size font in a PowerPoint presentation. 6.39 No consensus exists about who should be responsible for quality assurance. Some think this function should be handled in Washington for content that originates there while others think that the users are best able to assess quality from a local need standpoint. Still others have suggested that the regional teams play this role since they are the interface with both the Washington and local perspectives. 6.40 In addition, there is growing discussion about whether quality assurance should even be a network function or how it could be implemented given the emphasis on increasing the content provided by external partners and boosting center utilization for financial sustainability. It remains, however, a major consideration if the World Bank is to confer its reputation on capacity enhancement, learning, and knowledge sharing activities through GDLN. Recommendations 6.41 Clarity is needed about whether or how quality is assured within the network. The following actions could facilitate this process: · Convene a committee representing all GDLN stakeholder groups to address this responsibility. · Collect systematic input to guide decision-making and better understand the implications of decisions about quality. Such input might be gained through focus groups with active content providers, partners, DLC personnel, and participants. · Determine where quality is to be assessed and how it will be measured. · Decide what role for quality assurance resides in Bank headquarters, what role belongs at the regional level, what role belongs at the DLC level, and what role is left to external partners. · Communicate decisions about the quality assurance process to all stakeholders. MONITORING AND EVALUATION 6.42 In general, little emphasis is placed on evaluating or monitoring programs delivered through GDLN, despite the fact that such efforts could provide important information about program outcomes, possible program improvements, and impacts. Findings 6.43 No evidence was found in the records of GDLN centers visited or through interviews with DLC managers that these sites have been able to focus on evaluating the 90 activities that take place in their facilities. This is a necessary step to developing strategies to enhance the vision of the GDLN as a whole and the impact of country- specific efforts by the centers. 6.44 There are several levels at which monitoring and evaluation can take place. Participant feedback questionnaires (Level 1 evaluations) are already a part of courses offered by WBI although these same forms are used for face-to-face courses and are not generally tailored for a distance learning format. Centers usually return these forms to task managers with, at most, a cursory look at the participants' feedback--and often no review at all. Managers expressed interest in getting tabulated results and recognized their value, but had no resources to do this at the local level. 6.45 As GDLN refines its view of how the DLC can be part of the development dialogue of a country and centers offer a broader range of interactions--discussions, dialogues, planning sessions, negotiations, and the like, it would be important to have customer satisfaction data and formative feedback about how to make these sessions more successful. 6.46 In-house data can also supply rich resources for formulating a center's marketing plan and making decisions about learning events to offer and participants to recruit. By being able to track past participants, their employers, and their interests, centers should be able to build on their past experiences to provide the depth and intensity of activities that lead to real impacts. Recommendations 6.47 Monitoring and evaluation (M&E) practices would provide valuable information for GDLN and could be integrated without undue burden on centers through the following steps: · Through collaboration between GDLN and WBI's Evaluation Group, explore ways that meaningful evaluation activities can be infused into center operations. Areas to be considered include how to evaluate the special characteristics of GDLN activities (and which types of activities should be included), how evaluation data analyzed by task managers might be transmitted back to participating centers, whether activities initiated by external partners should be evaluated, what tools would aid managers to systematically collect and analyze their own information sources, and what training would be required to help take advantage of these data. · Consult DLC managers about their needs and center capabilities for implementing evaluation activities. · Reevaluate the items included on WBI's Level 1 participant feedback questionnaires. Standard questions to be used for assessing distance learning activities should focus on the GDLN model that promotes 91 interactivity, participatory experiences, opportunities for application, and opportunities for follow-up. · Make M&E materials and instruments available to centers to facilitate their efforts. PROMOTING THE NETWORK 6.48 Once clear decisions have been made about GDLN governance and critical functions that are to be pursued as a network, the vision of GDLN must be actively marketed both to potential users within the Bank and to external partners. A stronger commitment needs to be made if the network is to achieve its original vision and the goals that management have for it in the development community. Findings Communication and Marketing within the World Bank 6.49 Many potential GDLN users at the Bank are either not familiar with the network or have little concrete guidance about how to use it or even whom to contact to learn more. Although GDLN Services have periodically held an open house for Bank staff, many task managers are unavailable or feel too overloaded to attend. Even among those who have used the network at least some, there is a wide variation in understanding about what constitutes effective use of the network and, again, whom to contact to learn more. This uncertainty persists despite the fact that the GDLN website includes lists of staff (by region and function) and contact information (http://www.gdln.org/contact.html). On a broader level, several stakeholders expressed confusion about the role of GDLN with its focus on programming in thematic areas while WBI is becoming more aligned with Operations with an accompanying shift to emphasizing focus countries. 6.50 Active GDLN users have observed that network use makes sense for some events and not for others, so a clear education process is needed to show where GDLN constitutes value-added. This, combined with revised performance measures for managers, will likely be more effective in promoting World Bank use than just instituting requirements for task managers, as some DLC managers suggested. 6.51 Stakeholders have also highlighted the importance of having the World Bank embrace the use of GDLN in many of its lending and nonlending activities. Clearly modeling use of the network by World Bank personnel will help shape its image to the broader community (e.g., potential users from the UNDP, bi-lateral organizations, NGOs, etc.) as an effective global tool for development. Partnerships 6.52 Although partners play a vital role in the future of GDLN, no coordinated approach appears to currently exist for attracting and solidifying partnerships. 6.53 DLC managers seek partners to gain financial support for activities, build interest and commitment from key institutions, and provide local content. They desire help from 92 the World Bank in connecting them to local representatives of the development community, but some have also worked innovatively to develop effective marketing techniques and build local partnerships. 6.54 GDLN stakeholders at the World Bank recognize the importance of partnerships regardless of whether they have been involved in developing any themselves. Many noted that the process for establishing partnerships is unclear, even when an organization specifically approaches them and expresses interest. Others at the Bank underscored the need to clarify incentives for potential partners both at the global and local levels, since few will have an interest in just investing in a technical infrastructure for abstract purposes or submitting to bureaucratic processes. Recommendations 6.55 A coordinated effort is needed to market the concept of GDLN as a prominent vehicle for development. A focused initiative to sell the network internally and externally suggests the following actions. · Identify which techniques are most effective for marketing GDLN within the Bank. Some users noted that attending a basic session on how to convert content for GDLN is not as compelling as getting to observe or participate in an actual interactive course or event linking different sites. Actively market the network to the Bank. Marketing efforts will need to be persistent since, in the view of some, there is a prevalent mission culture where specialists travel to provide technical assistance and deal with issues in person. · Communicate clear information regarding the GDLN governance structure and roles to stakeholders at the Bank. This is particularly important given the periodic reorganizations and staff turnover in WBI and the uncertainty among many about whom to approach with questions. While the contact lists for GDLN Services, regional management teams, and most DLCs do currently exist, an active communication strategy is needed to disseminate this information. An effective tool could be a clear organization chart with contact information and descriptions of the roles and services available. · Clarify the current Bank philosophy towards partnerships in GDLN. If the Bank is committed to facilitating partnerships and gaining recognition of the network as an effective tool for development, then Bank headquarters will need to play an active role in forging connections between centers and Bank country offices to support marketing efforts. This will involve development of a formal strategy for leveraging the Bank connections at the local level to connect GDLN centers with donor agencies. Bank involvement in this process should be at a facilitation level, using the institution's convening power to bring together potential partners in the 93 development community. Such a strategy would vary depending on how the Bank is perceived in a particular country. ADDRESSING THE RESEARCH QUESTIONS Effectiveness and Impact 6.56 Programs facilitated by GDLN do provide clear impact on development in some cases. Some of the examples cited in this report include a procurement officer using his GDLN experience to implement more transparent practices, a director of a youth employment agency establishing a savings bank for youth, and a Ministry of Youth using a participatory process to develop the first action plan in the agency. However, the instances where program experiences have translated into changes in public policy or institutional capacity have been limited due to a range of factors from having the appropriate content and effective pedagogy to involving the right participants in the right setting. 6.57 Many of the factors considered in examining effectiveness and impact played less of a role than might have been expected. GDLN stakeholders emphasized the importance of minimizing technical issues but also generally felt that the needed improvement process was underway. A range of cost variables enter into using GDLN, including different types of connectivity, equipment costs, and content fees among others. In most cases, stakeholders were less concerned that these costs were prohibitive and more concerned that they were too confusing to figure out. 6.58 Variables related to content and curriculum appeared to directly influence the effectiveness of programs offered through GDLN. The availability of appropriate content was a critical issue, and many stakeholders expressed concern that WBI content was too- supply driven and that there was not sufficient content available to fill network needs. At the same time, participants tended to value their GDLN experiences and offered many concrete suggestions for increasing the potential for impact. Such recommendations included making sessions less superficial (often by increasing the length or offering a series), providing materials in advance, conducting follow-up activities, bringing together the appropriate participants, and supporting communities of practice. 6.59 In the end, a critical factor limiting the impact of GDLN programming appeared to be the questions about roles and governance of the network itself. Without a clear vision for why the network exists and what functions it encompasses, many stakeholders make limited use of GDLN to serve a particular need. DLC managers, facing a mandate for financial sustainability, must sometimes choose between filling their facilities with government computer trainings and reserving space for a critical development dialogue among just a few key individuals. The responsibility for assuring that content is high quality to achieve maximum impact is viewed by some to be a critical network role, but others observe that external partners who pay to use the network can use whatever content they choose. 6.60 A clear network philosophy will need to be established regarding effectiveness and impact as GDLN continues to build partnerships, add centers and subregional 94 networks, and facilitate a growing number of meetings or dialogues with impromptu content. While many factors that lead to better sessions are likely to remain within network control, it is not currently clear that assuring quality and impact have been accepted as a network responsibility. Conditions for Greater DLC Effectiveness and Efficiency 6.61 A DLC's effectiveness and efficiency is influenced by a variety of conditions-- some under the control of DLC management and others a product of center location, start-up terms, quality of the national infrastructure, economic and political circumstances, and the nature of and access to the development community in a particular country. 6.62 First and foremost, everyone agrees that a center's success seems to rely on its leadership. A DLC manager has many responsibilities: for recruitment, obtaining activities that match local needs and coordinating delivery at the local level, generating revenue, and marketing. A great deal rests on the capabilities of the individual and his/her previous affiliations and networks within the sectors of interest in a country. Virtually all managers look to GDLN operations for procedures, systems, and tools. Most did not feel fully supported by these services. 6.63 Many DLCs are struggling with the requirement for financial sustainability but saw it as a function of needing to generate revenue (in different ways, depending on the local context) and not necessarily content and technology costs that were excessive. Many managers feel that they were given too short a time period to achieve sustainability. Centers have not been able to set aside funds for capital and maintenance expenses; their focus has been on meeting operating expenses. For managers, being effective and efficient rests with being assured of quality in the content, pedagogy, and delivery system; having the capability to market specific activities as well as distance learning as a concept; and engaging partners for support. 6.64 Each manager must determine the optimal mix of content, skills, and knowledge exchange activities that will support a financially viable center. Again, this links to national circumstances and the manager's connection to various sectors, initiatives, and institutions. There is no one-size-to-fit-all, but successful managers should have more opportunities to share their strategies with others in the network. Alignment with World Bank Goals and Millennium Development Goals 6.65 While GDLN stakeholders agree that the network should be used for development, there is currently no central guidance that activities or results should be aligned with the World Bank or Millennium Development Goals (MDGs). As an illustration, the Activity Management System catalogs GDLN activity proposals according to a set of themes that show little congruence with the standard themes or sectors of the World Bank. 6.66 The absence of any push to align GDLN activities with standard international development objectives is arguably part of a network philosophy of decentralization, 95 where centers themselves are likely in the best position to assess how to most effectively contribute to the development dialogues of their countries. At the same time, DLC managers often do not have clear incentives to ensure that the results of GDLN programs are aligned with the MDGs or other specific development goals. Managers must achieve financial sustainability for their centers, and in some cases, this mandate may lead to offering an event with the greatest revenue-generating potential rather than the event that will most effectively reach top decision makers may and result in development impact. In countries where there is no collaboration with the Bank country office, there can be little harmonization of country-focused goals. 6.67 Before further in-depth analysis is conducted to assess how well GDLN content is aligned with World Bank goals or MDGs across all regions, the question must be addressed by WBI and GDLN management as to whether this is part of the guiding philosophy of the network. Impact on Capacity Enhancement 6.68 The findings of this evaluation show evidence of capacity enhancement in some instances, where GDLN learning or knowledge exchange activities have resulted in changes in policies or institutions. However, the study also demonstrates that real outcomes related to capacity can only be achieved through a more deliberate strategy that links activities to goals, identifies and targets the right participants, and supports a continuous process rather than an array of isolated events. A single GDLN session is unlikely to yield measurable outcomes, whereas a deliberate series of events exploring a topic in-depth and directly addressing critical local needs is more likely to enhance capacity. 6.69 Participant feedback indicated that positive outcomes from GDLN programs often occur at the individual level through developing or refining skills or fostering confidence in a subject area. Changes at the institutional or country level are more elusive, but a range of variables appear to play critical roles in making these happen. The nature of knowledge transfer and knowledge sharing through GDLN has a strong potential for achieving capacity building for institutions in that a team from an organization may attend together, providing a critical mass. Through the design and pedagogy of well- executed activities, participants engage in active, problem-solving sessions that can be applied in the work of their organization. By participating in GDLN sessions while remaining at their jobs, professionals have opportunities to use the acquired knowledge and skills immediately. Reinforcement of these efforts and promotion of a community of practice further strengthen initial efforts. 6.70 By increased focus on country capacity needs and a coordinated effort with country teams and other Bank specialists on targeted problems, there is also the potential of capacity enhancement at the national level. In the research conducted for this study, we were not able to identify any areas where that has taken place to date. 6.71 In the end, all the variables explored in this study play a role in whether network use results in enhanced capacity. Factors related to overall effectiveness and impact, the 96 alignment of GDLN programming with identifiable and targeted goals, and conditions for DLC effectiveness and efficiency must all be in place for the network to realize its potential for enhancing or transforming institutions. 97 98 ANNEXES 99 100 ANNEX A: METHODOLOGY RESEARCH QUESTIONS 1. As stipulated by the Terms of Reference (TOR), this evaluation of the quality and impact of programs facilitated by GDLN relied on both qualitative and quantitative data to address the following research objectives1: 1. Evaluate effectiveness and impact a) Results. Report on results (outputs, outcomes and impacts) of GDLN programs, including specified objectives as well as unintended consequences. Provide recommendations on defining and measuring results for future GDLN programs and operations. b) Best Practices. Report on best practices and success stories (e.g., characteristics of successful courses, DLCs, and partnerships). c) Curriculum. Report on availability, appropriateness and value of content provided through GDLN. d) Economic costs to offer development content and value-added to WBI programming. Report on cost efficiency of content delivery through DL (e.g., GDLN vs. F2F delivery, to include differences in time required for course participation, travel, availability of content, and opportunity costs) and contribution to WBI programming. e) Organizational Structure. Report on suitability of GDLN organizational structure in obtaining results. f) Pedagogy. Report on role of instructional design services (e.g., standards of instruction and pedagogy) in support of the network. g) Technology. Report on technical quality (e.g., quality, reliability and costs) of the network. 2. Assess congruence between GDLN content and use, and World Bank goals and MDGs. a) Goals and Results. Report on alignment of GDLN results with MDGs, World Bank goals, and WBI goals. Provide recommendations on alignment of GDLN results with MDGs, World Bank goals, WBI goals. b) Content. Report on alignment of content and content objectives with MDGs, World Bank goals and WBI goals. 1The research objectives are listed here exactly as they appear in the TOR. 101 c) Development and Utilization. Report on network development and utilization (e.g., sites, activities, content from WBI and other providers, instructional methods, employment levels/professions of GDLN participants, appropriateness for target countries of DLC locations and distribution, and utilization of GDLN in meeting country development needs). Provide recommendations on expansion of network. d) Participant Networks and Communities of Practice. Report on GDLN ability and success in convening groups to support MDGs, World Bank goals and WBI goals. 3. Identify conditions leading to greater DLC effectiveness and efficiency. a) DLC effectiveness. Compare differences in results achieved by DLCs. b) DLC costs and sustainability. Report on DLC costs (e.g., capital expenses, maintenance expenses, comparison of technology differences such as VSAT vs. ISDN connections). Provide recommendations on financial models for DLC sustainability (especially for capital and maintenance expenses). c) GDLN Operations. Report on relationship between GDLN Support Services operations and procedures (e.g., scope, costs, efficiency, effectiveness, and staffing) and DLC operations (e.g., staffing, reputation of DLC, branding, entrepreneurship). d) Content mix for DLC sustainability. Provide recommendations on optimizing mix of development content and other skills content for DLC sustainability (e.g., offering a combination of development courses and technical/computer training). e) Partnerships. Report on the different partnership models (e.g., successes, difficulties faced by partnerships, appropriateness of types of partnerships to meet needs in respective countries, alternatives, and added value). 4. Assess impact on capacity enhancement. a) Effects of the network. Assess how network membership affects DLCs, in particular, with regard to capacity enhancement in business management skills, technical capabilities, and entrepreneurial and marketing abilities. b) Impact of DL in the host country. Address any capacity enhancement impacts that have resulted from the use of DL in the host country, or in the region. 102 2. The following sections provide detailed descriptions of the data collection activities and analysis used to carry out these research objectives. Findings from this study are presented in this comprehensive report, in a separate executive report focused specifically on DLC operations, and in a separate report on success stories and best practices. DATA COLLECTION ACTIVITIES Review of Extant Data 3. In addition to empirical research, this study was informed by archival data including World Bank reports and strategy papers as well as program and activity data available through systems developed specifically for GDLN or more broadly for WBI. GDLN Documents 4. Especially given the continued evolution of GDLN while this study was underway, we have consulted program materials and internal studies to better understand the program context and track the changes in governance and network development. Key documents developed by the GDLN team that we have examined include the following: · Global Development Learning Network (GDLN): Assessment of Priorities and Strategic Content Development in Europe and Central Asia (June 2002) · Launching a GDLN Center (September 2003) · GDLN-LAC: FY03 Retrospective (September 2003) · GDLN Business Plan (November 2003) · GDLN Questionnaire Initiative (December 2003) · Strategic Plan: Bank Engagement in the Global Development Learning Network (GDLN) FY05-FY07 (May 2004) · GDLN Network Agreement (April 2004) · GDLN Business Policies (April 2004) · GDLN Facilities Design Guide (May 2004) Systems Data 5. We also analyzed activity data related to GDLN maintained in existing databases: · Activity Management System (AMS): This system was developed specifically for GDLN activities and has been upgraded during the period of this evaluation. We were assigned usernames and passwords in order to test the system and extract information for this study. · Rendezvous: This booking system for videoconferencing demonstrates the most complete listings for GDLN sessions to date. However, minimal descriptive 103 detail is provided for a session, and activities are listed by division rather than by task manager. · Client Registration System (CRS): This WBI database provides information about which task managers provide activities through distance learning in which countries. In some cases, participant data and Level 1 evaluation (participants feedback questionnaires) results are included for learning activities. Interviews with World Bank Staff 6. Semi-structured interviews were conducted with 22 World Bank employees at headquarters and 10 more from the field who had a variety of roles related to GDLN. A range of factors were considered in selecting individual stakeholders for interviews. Those selected were identified by others in WBI to be active users, potential users, or knowledgeable about the history and context of GDLN. Potential key informants for interviews were also identified by analyzing activity data available through AMS and CRS. Ultimately, interviews were conducted with every major GDLN stakeholder group at the Bank. 7. These interviews relied on an open-ended protocol but also allowed for stakeholders to raise and explore issues they believed were critical. Where relevant, areas explored included perspectives on GDLN in general, center operations, content development, partnership development, capacity enhancement, regional issues, technology, reporting systems, and recommendations. Most of these sessions lasted 45 minutes to one hour, with telephone follow-up in some cases to ask for clarification or learn about programmatic changes during the study. The interview protocol for these sessions is included in Annex C. Web-based Survey of World Bank Staff and DLC Managers Questionnaire Development 8. The initial set of semi-structured interviews with World Bank staff highlighted key topics related to GDLN and allowed the evaluation team to design a questionnaire with closed-ended response options to systematically explore critical issues. The survey design was also informed by the previous GDLN Questionnaire Initiative (December 2003). In some cases, questions to DLC managers that had been open-ended in that initiative were repeated here with closed-ended response options, so that managers could respond at comparable conceptual levels and the results could be more systematically analyzed. 9. The web-based questionnaire had a total of 208 closed-ended variables with additional options for open-ended remarks; however, respondents had numerous options to skip items that were not applicable given their experiences. In particular, skip patterns were determined based on familiarity with GDLN, whether one had provided content through GDLN, whether one had developed partnerships for GDLN, and whether one was a DLC manager. Selected GDLN stakeholders who had been interviewed were asked to review or pretest the questionnaire. In addition, the web-based survey was tested internally by the Macro team on multiple platforms and browsers for technical 104 reliability. A copy of the questionnaire with response frequencies is included in Annex B. Fielding 10. The data collection phase began on February 18, 2004, and ended on March 29, 2004. 11. There were two waves of respondents for the survey. The first wave included those identified to be GDLN users or stakeholders among Bank employees and a purposive sample of potential users. This wave included a total of 346 possible respondents who were notified of the survey on February 18. After initial contact had been made with the first group, an effort was made to reach a broader sample of those in Bank Operations by using the database of those employees accredited to manage trust funds (mid-to-high level task managers) and selecting every fifth person on the list. The rationale behind including this second wave was to reach more respondents who were not currently active users of GDLN. After eliminating duplicates with the first wave, this second wave included a potential 438 respondents. This second group was notified about the survey on March 12. 12. Potential respondents were first notified of the survey by email. This message sent from the evaluation team explained the purpose of the survey and had instructions for completing it. The instructions included a URL link to the survey, a user ID for each respondent, and a password for each respondent. Once respondents received the survey, they could click on the link, enter their user names and passwords, and start the survey. If they were unable to complete the survey at the first sitting, they could return to it at a later time using the same user names and passwords. This system did not allow respondents to submit more than one completed questionnaire. Follow-up reminders to respondents in the first wave were sent on March 1st and 11th, and for both groups on March 25th. Response Rate 13. Email addresses for a total of 784 people were used for this survey. A number of potential respondents were determined to be ineligible because either they were out of the office until after the time of survey fielding or their email addresses were not functioning. The total of ineligible respondents was 76 (20 from the first wave, 56 for the second wave), resulting in a total of 708 eligible respondents. 14. There were a total of 277 people who responded to the survey (200 from the first wave and 77 from the second wave). Of those who responded, there were 13 who scrolled through the entire survey without answering any questions (2 from the first wave and 11 from the second), reducing the number of completed surveys to 264. 15. In sum, there was a response rate of 60% (198/326) for the first wave and 17% for the second wave (66/382), resulting in a response rate of 37% (264/708) overall. 105 Site Visits to Selected GDLN Centers Selection of Sites 16. The Terms of Reference set forth the requirement to conduct five site visits--2 in Africa, 2 in East Asia/Pacific region, and one in another region of choice. 17. A number of considerations went into determining the appropriate GDLN centers to visit. The goal was to identify sites where we could access a reasonable number of participants for interviews related to the impact of their GDLN activities and where DLC staff would have had sufficient exposure to the network to provide experiences, data, and reflections on issues of quality and impact. 18. In the section below we outline our analysis and thinking. We consulted data on performance in FY01, so that we could learn from the experience of centers that had been doing business for some time, and identify where we might be able to interview participants who had had opportunities to apply their GDLN experiences and where there would be a chance to see impacts. Many of the original possibilities that we listed in the proposal, based on data showing a history of offerings and high utilization, were confirmed in interviews with WBI and GDLN staff who also took into account other less quantifiable considerations, including leadership and engagement in the development dialogue. Africa (AFR region) 19. GDLN centers have an active history in both Anglophone and Francophone Africa, so the site selection included one center in each. Across Africa, the business model has been loan/credit or grant financed. In this region, we used the standard of having conducted 20 substantive learning activities in 2001. · Senegal was chosen for the Francophone country in Africa. Its center had 75 events in FY012, a substantial increase in FY02, and a utilization rate in FY03 of 32.9%. · Originally Ghana was selected for the Anglophone site, as it had held 57 events in FY01, increased FY02 offerings substantially, and has an unofficial percentage utilization rate in FY03 of 61.3%. However, the death of the center manager in December, 2003, prompted a reconsideration of the choice of centers, as we expected that arrangements for a site visit would be difficult right after the loss of their leadership. Also, the institutional memory had been lost. 2According to the RendezVous Videoconferencing System, which may need additional cleaning for complete accuracy but does provide a general pattern of utilization in terms of types of activities, duration, and total volume. 106 · Tanzania was chosen for the Anglophone country in Africa. Its center had 60 events in FY01, double that number in FY02, and a utilization rate in FY03 of 47.0%. East Asia/Pacific (EAP Region) 20. We initially selected two sites in the East Asia and Pacific region. All the different business models have been employed in this region. In EAP we also used the standard of having conducted 20 substantive learning activities in 2001. · The operation of the DLC in Vietnam has been cited by a number of informants, in particular noting the center's connection with government officials and development underway at that country. Although only 14 events were held in FY01, somewhat below our threshold, there has been a dramatic increase in activities in FY02 (to approximately 70). The FY03 utilization rate was 30.7%. We note that although financed through a mix of donor grant and Bank funds, it is co-located with the Bank office in Hanoi. · The second EAP site originally proposed was China, based on the number of learning events in FY01 and the substantial increase in FY02 (to 85). Due to the Shanghai conference, that center could not accommodate a visit. The second choice was Thailand, which operates on Chulalongkorn University and was financed through World Bank loans, credits, or grants, which hosted 24 events in FY01 and about double that the following year. Unfortunately, the DLC in Bangkok was not available for a site visit during the period in which this data collection was being carried out. Latin America (LCR Region) 21. We chose Latin America as the third region in which to carry out a site visit. The DLCs are generally located at an already-existing facility or one which has been upgraded. For medium income countries in EAP, ECA, MNA, and SAR, this has been a more frequent circumstance. Financing is by an agency or a university, out of its own income. Its programming is often much broader than GDLN activities. A number of centers have a level of programming worth considering, although a number of countries were not active until FY03. Our initial choice was Costa Rica, and discussion with numerous staff increased the countries under consideration to include Guatemala, Ecuador, and Peru. However, several key individuals provided thoughtful consultation 107 and recommended that the operations just being launched in Brazil are the most important to be studied to look at the potential for impacts more broadly. The DLC in Brasilia was selected for the site visit in the LAC region. Brazilian operations, which were just begun in FY03, have increased dramatically in the past year. The factors that encourage study of this country's GDLN activities include its strong focus on working on poverty reduction measures in the Northeast, strong local leadership and support, and the tie-ins to other networks. We believe that looking at "network of network" functioning reflects a future model that is likely to be replicated elsewhere. After discussions with the center manager, the decision was to meet with a key partner in Salvador and then to investigate operations and participants in both Brasilia and Fortaleza, which has a network of sites that has been very active. Interviews 22. Site visit activities included interviewing DLC managers, DLC staff, local partners, participants of prior center events, key employers, and other stakeholders. For each visit, the DLC manager made recommendations about which key informants to interview and scheduled appointments. Many of the 108 individuals interviewed had multiple roles.3 An interpreter was relied upon as needed to communicate in the appropriate local language. Telephone interviews were conducted from two sites to reach participants located outside of the capital cities in the respective countries. In some sites, focus groups were held so that participants could actively exchange ideas with each other. In most cases, participants who were interviewed also completed a closed-ended survey questionnaire which is described below. 23. Although the GDLN center at Chulalongkorn University was not available for the proposed site visit during the period of study, a 1-hour telephone call was conducted with the DLC coordinator using key questions from the DLC manager protocol. Activities 24. Site visits also included other data collection activities besides semi-structured interviews and focus groups with stakeholders. To gain a better understanding of DLC operations, the two evaluators requested data and reports from DLC managers and staff and reviewed the statistics captured for GDLN activities on site. This information included GDLN course or event descriptions, participant descriptions and numbers, cost calculations and records, marketing plans, and other business documents related to that center's operation. The evaluators also observed GDLN activities in process where possible to better understand the participant experience and toured the facilities to better understand physical and technical capacity issues. 3Eight of these individuals interviewed during site visits were also Bank employees (excluding DLC managers) and their viewpoints are also included in Chapter 3: Perspectives within the World Bank. 108 Survey of Participants 25. Participants who had attended prior GDLN events were interviewed and then asked to complete a questionnaire in most cases. Participants who had very limited time available were interviewed but not asked to complete the questionnaire. The participant survey was translated into the local language and included a list of items to identify an individual's satisfaction with a GDLN event or series and how he or she used the experience in the weeks, months, or years following it. Completed questionnaires were submitted by 58 past participants. Videoconference Focus Group 26. As none of the site visits involved a DLC in a university setting, a videoconference was convened with a number of Latin American DLCs. Invitations were sent by email to 7 centers located at universities (Mexico, Guatemala, Costa Rica, Columbia, Ecuador, Peru, and Chile) inviting them to join a focus group. All accepted the offer, with 8 DLC managers and coordinators participating from 7 countries. An abbreviated DLC manager protocol was used, highlighting key questions about the influence of the university setting on administration, operations, resources, marketing, and sustainability. The 2-hour session took place on May 7, 2004, and was conducted in Spanish. ANALYSIS 27. Qualitative data from the interviews and from other open-ended survey items were coded to identify dominant themes related to key areas. Quotes included in the report are representative of these themes and intended to illuminate important issues or perspectives among stakeholder groups. 28. The quantitative data from the two surveys were entered into the Statistical Analysis System (SAS) for coding and analysis. For the Web-based Survey of World Bank Staff and DLC Managers, response patterns were analyzed to identify differences in subgroups. Those items commonly used as independent variables included the following. Question numbers are included with the items listed for easy reference to the questionnaire included in Annex B. · The respondent's role (Q1) · The respondent's World Bank division or group if the role was as a content provider (Q1A)4 · The respondent's location in WBI, Bank Operations, or at a DLC · (Q3) The regions in which the respondent had conducted GDLN-related work5 4In other cases, respondents were not asked to identify their divisions since this would have allowed for the identification of individuals. For example, if a WBI unit manager identified his or her division, the accompanying responses could be easily traced to that individual. 109 · The level of awareness about GDLN (Q4) · The level of familiarity with facilities and services available through GDLN (Q5) · The proportion of time spent working on GDLN activities (Q8A) · Whether the respondent had provided content or helped design an activity for GDLN (Q12) · The frequency of using GDLN during FY04 (Q13) · Where a DLC is situated (DLC1) · The DLC business model (DLC3) · Whether respondent was involved in co-financing or content development partnerships (Q26) 29. Unless noted otherwise, differences in subgroup responses have only been included in this report where they were statistically significant at the p<.01 level and where there were sufficient numbers of respondents in each category to support valid tests. 5Respondents had the option to check all regions that apply. DLC managers were assigned an additional variable indicating which region they were actually located in to allow for an accurate subgroup analysis related to manager responses and center operations. 110 ANNEX B: WEB-BASED SURVEY Evaluation of the Global Development Learning Network Web-based Survey of World Bank Personnel and Distance Learning Center Staff N=264 All amounts in % Respondent Information 1. Select the item below that best reflects your role related to the Global Development Learning Network in particular (if applicable) or to the World Bank in general (please choose one): n=264 12.1 GDLN Services Member 29.6 Content Provider (e.g., task team leader, program team member) 4.2 WBI Unit Manager 3.8 Instructional Design Specialist 6.8 GDLN Regional Coordination Team 16.3 Distance Learning Center Manager 6.1 Partnership Development/Capacity 16.3 Other (please specify)______________ Enhancement Specialist 1.9 Consultant (recoded from "Other" responses) 3.0 User (recoded from "Other" responses) (Only for Q1 = Content Provider) 1a. Please indicate your World Bank division or group below: n=70 0.0 WBIVP 0.0 WBICA 5.7 WBIEG 18.6 WBIEN 11.4 WBIFP 0.0 WBIGD 0.0 WBIGG 2.9 WBIGK 12.9 WBIHD 0.0 WBIKL 0.0 WBIKP 17.1 WBIPR 2.9 WBIRC 0.0 WBIST 28.6 Other (please specify) _____________ 2. Have you changed your job title and responsibilities in the past 12 months? n=225 28.0 Yes 72.0 No 3. Please indicate which regions for which you have conducted GDLN-related work (please check all that apply): n=255 40.8 Africa--Sub-Saharan 36.5 East Asia and the Pacific 41.6 Europe and Central Asia 34.5 Latin America and Caribbean 26.7 Middle East and North Africa 16.1 North America 23.1 South Asia 4. What level of awareness do you have about the Global Development Learning Network (GDLN)? n=255 43.1 Have extensive GDLN experience 111 38.0 Have some GDLN experience 7.8 Have heard about GDLN and considered using it 7.8 Have heard about GDLN but never considered using it 3.1 Have never heard of GDLN (Skip to CLOSING) Experience with the Global Development Learning Network 5. How familiar are you with what facilities and services are available through the GDLN? n=247 36.8 Very familiar 35.6 Familiar 19.4 Somewhat familiar 8.1 Not familiar at all (go to 25) 6. Do you know whom to contact for GDLN-related questions regarding the following topics? (Please select NA for those topics that are not applicable for you.) Yes, it is Yes, it is No, it is not always clear usually clear clear NA a. Requests for content / learning activity 22.8 40.6 25.6 11.0 n=219 b. Proposals for content n=213 17.4 33.8 36.2 12.7 c. Event scheduling n=216 39.8 33.8 19.4 6.9 d. Technical difficulties n=217 31.3 36.9 24.0 7.8 e. Partnership development n=213 16.0 35.7 38.5 9.9 f. Content fees n=214 17.8 28.5 40.2 13.6 g. Connectivity costs n=216 24.5 38.0 29.6 7.9 7. Do you have a clear understanding of what role(s) you have related to the GDLN? n=220 66.8 Yes 18.6 No 14.6 There is a conflict between my understanding and the expectations of others 8. a. On average, what proportion of your time is spent working on GDLN-related activities? n=215 51.6 Less than 17.2 25%-49% 9.8 0%-74% 9.3 75%-99% 12.1 100% 25% b. Does this time allotment allow you to effectively complete GDLN-related activities? n=205 112 30.2 Always 45.4 Usually 16.6 Sometimes 6.3 Rarely 1.5 Never 9. In conversations with GDLN stakeholders, we have heard a range of opinions regarding use of the network and on the development of partnerships and program content. Reflect on your experience with the GDLN and indicate your level of agreement with the following statements: Strongly Strongly Agree Agree Disagree Disagree a. The GDLN is an ideal delivery modality to disseminate development knowledge to more 34.3 52.6 12.2 0.9 people. n=213 b. The GDLN is primarily a tool to deliver learning activities through videoconferencing. 20.7 54.0 23.5 1.9 n=213 c. The GDLN offers a comprehensive system to deliver distance learning through a blended approach involving a combination of distance 32.6 51.9 15.1 0.5 learning, face-to-face, e-learning or other pedagogical methods. n=212 d. The World Bank plays a critical role in making connections between distance learning centers and local offices of development agencies. 27.4 49.0 21.6 1.9 n=208 e. Distance learning centers need to be proactive in forming local partnerships with development 52.6 43.5 3.8 0 agencies. n=209 f. Content delivered through the GDLN tends to be supply-driven and does not necessarily 16.8 45.5 32.1 5.7 match local needs. n=209 g. Having the World Bank brand allows distance learning centers to market activities more 24.5 65.2 9.3 1.0 successfully. n=204 h. The GDLN would attract more external partners if it was not so closely affiliated with 4.9 20.5 60.5 14.2 the World Bank. n=205 i. It is not realistic to expect GDLN centers to become fully sustainable while focusing on the 25.6 44.1 25.6 4.7 development agenda. n=211 113 10. Some stakeholders have expressed the need for a standard quality assurance mechanism for GDLN. In your opinion, who should have responsibility for ensuring that GDLN content is of high quality? (Please check one.) n=216 29.6 Content provider 6.9 Instructional designer 13.9 GDLN services staff (based in WBI) 7.4 GDLN regional staff 10.7 Distance learning center staff 14.4 Should be determined by the consumer (in terms of repeat business) 17.1 Other (Please specify)__________________________________ 11. A range of opinions has been expressed about the various uses of the GDLN. Please reflect on these uses and rank the following purposes in terms of their priority for the Bank from 1 to 6, with 1 indicating the highest priority. If you believe an important function has been omitted from this list, please use the other category to specify this priority and increase your numbering to 7. % reflects those who ranked this option as the highest priority (one) 8.8 Maintaining high utilization of distance learning facility to successfully recover costs n=193 38.0 Disseminating knowledge to as many people as possible n=192 22.2 Providing unique learning environment to key policymakers and other leaders n=194 11.9 Bringing training to people who do not have other options n=194 3.6 Introducing modern technology to countries n=196 14.2 Focusing intensively in a specific content area to address a major national problem n=197 21.4 Other (please specify) ____________________________________ n=28 12. Have you ever provided content or helped design an activity for a learning event hosted by the GDLN? (Please check all that apply.) n=211 50.2 Yes, directly provided content or designed an activity 19.9 Yes, supervised individual(s) providing content 25.1 Yes, advised content provider on instructional design 29.4 No (Go to Question 21) Content Development 13. Over the course of this year (FY04), approximately how many activities (individual sessions) did you provide through the GDLN? n=143 24.5 More than 12 14.7 7-12 16.8 4-6 31.5 1-3 12.6 0 114 14. How would you rate your overall experience with the GDLN? n=141 14.2 Excellent 45.4 Very Good 29.8 Good 9.9 Fair 0.7 Poor 15. In designing or hosting GDLN events, how difficult or easy have you found completing the following tasks? (Select NA if the element is not applicable to you.) Very Very Easy Easy Difficult Difficult NA a. Recruiting participants n=147 8.8 41.5 24.5 4.1 21.1 b. Recruiting or identifying appropriate local facilitator n=146 4.8 42.5 28.1 4.1 20.6 c. Reserving appropriate facilities/equipment n=145 12.4 47.6 19.3 6.2 14.5 d. Adapting content to local needs n=145 4.1 42.1 35.2 4.1 14.5 e. Converting face-to-face content for distance learning n=145 4.8 34.5 32.4 6.9 21.4 f. Negotiating costs for learning events n=143 2.1 29.4 27.3 9.8 31.5 g. Entering data into the Client Registration System (CRS) n=144 5.6 16.0 14.6 8.3 55.6 h. Entering data into the Learning Catalog n=143 4.2 20.3 13.3 5.6 56.6 i. Entering data into the Activity Management System (AMS) n=134 6.7 23.9 19.4 9.7 40.3 16. In designing learning events to conduct via the GDLN, how important do you think it is that the learning activity accomplish the following (select NA if the element is not applicable to you): Very Somewhat Not Important Important important important NA a. Provide new knowledge or skills n=142 55.6 33.8 10.6 0.0 0.0 b. Update or refine knowledge or skills n=141 51.1 41.8 7.1 0.0 0.0 c. Raise awareness of specific development issues or trends 50.7 37.1 10.7 0.0 1.4 n=140 d. Provide strategies or approaches to address the development needs of the 50.0 36.4 10.7 0.7 2.1 country n=140 115 e. Provide strategies or approaches to address the problems/issues faced by 36.9 43.3 17.0 1.4 1.4 participants' organization n=141 f. Facilitate networking among people working on the subject matter of the learning event 51.8 35.5 11.4 0.7 0.7 n=141 g. Facilitate the implementation of a reform program n=140 32.1 32.9 23.6 2.9 8.6 17. What factors provide compelling reasons for you to use the GDLN in the future? (Please check up to 3 boxes.) n=146 57.5 Eliminates/reduces travel costs 34.9 Allows participants to learn while on the job 37.7 Enables senior professionals to share knowledge 66.4 Enables experts in other locations to contribute 17.1 Supports active learning approaches 19.2 Provides effective link to in-country network 39.0 Builds communities of practice 4.8 Target audience prefers this delivery 3.4 Other (please specify)_____________________________ 18. What challenges have you encountered that make you less likely to use the GDLN in the future? (Please check up to 3 boxes.) n=146 37.7 Technical difficulties 39.0 Event not being marketed sufficiently 11.0 Technology not attractive to target audience 10.3 Difficult to find appropriate facilitator 29.5 Excessive cost per participant 13.7 Difficult to reserve facilities 16.4 Activity Management System (AMS) is difficult to use 34.3 Distance learning centers not near target audience 24.7 Content not appropriate for distance learning 12.3 Other (please specify)_______________________________ 19. Using the GDLN, how frequently have you been able to convene communities of practice? n=131 5.3 Very Frequently (At least twice a month) 19.9 Frequently (Monthly) 116 35.9 Occasionally (Several times a year) 22.1 Seldom (Quarterly or less) 16.8 Never 20. What changes could be made related to the GDLN that would encourage you to use the network more frequently? Distance Learning Center Operations 21. How important are the following factors for whether a distance learning center operates effectively and efficiently? Very Somewhat Not Don't Important Important important important know a. Strong leadership from the DLC manager 63.5 26.9 5.6 0.0 4.1 n=197 b. Adequate number of DLC support staff 38.8 41.3 14.3 2.0 3.6 n=196 c. Engaged local co- financing partners 27.6 41.8 18.9 3.6 8.1 n=196 d. Engaged local training partners n=193 35.8 44.0 13.0 3.6 3.6 e. Geographic location of distance learning centers in target countries 36.8 42.5 14.5 3.1 3.1 n=193 f. Employment levels / occupations of GDLN 10.9 34.4 31.3 12.5 10.9 participants n=192 g. Consistent quality assurance system for 44.7 35.3 12.6 1.1 6.3 content n=190 h. Capabilities for marketing learning 47.9 34.7 12.6 0.5 4.2 events n=190 i. Ability to develop own content n=192 22.9 37.0 28.1 7.3 4.7 117 Very Somewhat Not Don't Important Important important important know j. Ability to make decisions autonomous from host institution 25.6 41.2 21.9 4.2 7.3 n=192 k. Political/economic stability of country 19.6 31.8 28.0 14.8 5.8 n=189 l. General readiness of the country for reform or 14.3 40.2 27.5 11.1 6.9 innovation n=189 m. Connection to development dialogue in 28.6 43.9 18.0 4.8 4.8 country n=189 n. Quality and size of facility n=193 26.4 40.4 22.8 6.7 3.6 o. Scheduling flexibility n=192 29.2 39.6 25.0 3.1 3.1 (DLC 1 ­ DLC 8 only if answer to Question 1 = Distance Learning Center Manager) [Note: 3 DLC managers selected the "other" category for Question 1 and then identified their role. They were therefore not automatically directed to the DLC questions, so there are 40 rather than 43 respondents for this section.] DLC 1. Please indicate where your distance learning center is situated: n=40 12.5 Based at a World Bank office 65.0 Based at a university/institute 12.5 Located independently (e.g., no host institution) 10.0 Other (please specify)_______________________ DLC 2. How have you strengthened your center's contribution to the development dialogue in your country during this fiscal year (FY04)? (Please check all that apply.) n=40 70.0 Formed new promising partnership in the development community 62.5 Increased ability to develop local content 52.5 Organized session to gain expertise from other countries that have addressed similar problems 42.5 Expanded or improved facilities to attract decision makers 35.0 Hired additional staff to increase capacity 118 37.5 Linked to sub-national networks to disseminate knowledge more broadly 52.5 Supported new national priority of building capacity 35.0 Located funding to reduce costs for target participants 5.0 Other (please specify)___________________________ DLC 3. Please select the business model below that best indicates how the start-up of your center was financed. n=40 10.0 Financed out of the World Bank's capital budget as part of a local Bank office 32.5 Financed by World Bank loans, credits, or grants from international or bilateral agencies 27.5 Financed by an organization, out of its own recurrent income, through upgrading or using existing facilities 30.0 Other (please specify)____________________________ DLC 4. What challenges have impeded your ability to strengthen your center's contribution to the development dialogue in your country during this fiscal year (FY2004)? (Please check all that apply.) n=40 15.0 Inadequate facilities 30.0 Problems with the reliable functioning of the technology 47.5 Lack of suitable content to address country's needs 15.0 Lack of autonomy from host institution 57.5 Insufficient funding 40.0 Lack of adequate staff (in terms of numbers or skills) 32.5 Inadequate publicity for center activities 25.0 Political climate in country 17.5 Turnover of officials in public offices 25.0 Cost of maintaining infrastructure 17.5 Target group's reluctance to use new technology 12.5 Other (please specify)__________________________ DLC 5. a. Approximately what percentage of GDLN activities at your center attract important policymakers or key leaders? n=40 0.0 None 35.0 1-25% 32.5 26-50% 30.0 51-75% 2.5 76-100% b. How difficult or easy has it been to attract important policymakers or key leaders to GDLN activities? 42.5 Very 32.5 Easy 22.5 Difficult 2.5 Very 0 Have not Easy Difficult tried 119 DLC 6. What sources of revenue have you used during this fiscal year to cover GDLN-related operating and maintenance costs? (Please check all that apply.) n=40 57.5 Rented your facility for development-related activities 30.0 Rented your facility for non-development related activities 42.5 Collected fees from participants 35.0 Received funding from host institution (institution where facility is located ­ not the World Bank) 27.5 Received funding from government 32.5 Received funding from partner (not host institution, World Bank, or government) 37.5 Received funding from the World Bank (excluding rentals for facilities for development-related activities) 15.0 Other (please specify)___________________________ DLC 7. At what point do you expect your center to be fully sustainable without funding from the World Bank? (Please select the best option.) n=40 32.5 Do not receive World Bank funding now 22.5 Will be sustainable next 1 to 2 years 12.5 Will be sustainable in 3 to 5 years 7.5 Do not expect to be sustainable for at least 5 years 25.0 Do not believe sustainability is a realistic goal DLC 8. Please describe a GDLN activity or program at your center that you considered to have been most successful. Provide specific details about the topic, the format and duration of the activity, the number and type of participants, the number of sites, and why you consider it successful. Topic: Format (e.g. plenary session, group discussion, etc.) and Duration (e.g. number of days and/or hours): Number and Type of Participants: Number of Sites: Why Successful: 120 Development Impact 22. In your experience, to what extent are GDLN programs or activities aligned with... I don't I am not know enough to familiar answer this Always Usually Sometimes Rarely Never with these question a. World Bank goals n=194 18.0 49.5 15.5 3.6 0.0 3.1 10.3 b. Millennium Development 11.4 33.7 32.1 4.2 0.0 4.7 14.0 goals n=193 c. Country Assistance Strategy(ies) 9.7 25.1 33.3 12.3 1.0 3.1 15.4 n=195 d. Poverty Reduction Strategy 9.7 27.7 29.8 11.3 0.0 4.1 17.4 Papers n=195 23. In general, how important are the following factors in influencing whether a GDLN activity is effective in contributing to the development dialogue in a country? Very Somewhat Not Don't Important Important important important know a. Participant level/occupation n=193 40.4 41.5 13.0 1.6 3.6 b. That it links with the Country Assistance Strategy (CAS) or Poverty Reduction 21.8 36.8 26.4 4.2 10.9 Strategy Paper (PRSP) n=193 c. Relevance of the specific thematic area to country 53.4 35.1 6.8 0.5 4.2 need n=191 d. Connectivity with an in- country network for broader 20.1 42.9 28.0 3.2 5.8 distribution n=189 e. Location of distance learning center in capital 19.3 38.0 27.6 10.4 4.7 city n=192 121 Very Somewhat Not Don't Important Important important important know f. Commitment of high level officials to follow through 43.8 41.2 10.4 0.5 4.2 n=192 g. Involvement of partners who focus on development 40.3 44.0 11.5 0.5 3.7 issues n=191 h. Having the appropriate experts and facilitator 45.8 40.1 10.4 0.0 3.7 n=192 i. Overall quality of the activity (content and 59.7 34.0 3.1 0.0 3.1 pedagogy) n=191 24. What do you think are the substantive areas as listed in the Activity Management System where GDLN can be used effectively to build capacity in a developing country? (Please check up to 5 boxes.) n=204 25.5 Communication 42.2 Education 23.5 Environment 16.7 Gender 50.5 Governance 22.1 Health/Nutrition 14.2 Infrastructure 22.1 Local Development 5.9 Politics 27.5 Private Sector 16.2 Rural Development 22.6 Social Development Development 17.2 Telecommunication 2.0 Transport 10.3 Urban Development and IT 14.2 Finance 18.6 Human Resources 30.4 Poverty 12.8 Social Protection 25. Have you been involved in developing co-financing or training partnerships for the GDLN? n=223 31.8 Yes 68.2 No (Go to 32) Partnership Development 26. For what purpose(s) have you developed Partnerships? n=71 12.7 Co-financing 16.9 Content development 70.4 Both 122 27. Please enter numbers below to indicate what type(s) of organizations you have been in contact with to develop partnerships to date and how often these efforts have been successful: Median Value Shown A. Number of each type of organization that approached you. Type of Organization Number a. Teaching or research 5.0 institution (public or private) b. Non-governmental 3.0 organization (NGO) c. Private sector (for profit) 3.0 d. Donor agency 2.0 e. National or central government 2.0 f. Regional or provincial 2.0 government g. Local government 1.0 h. Other (please specify) 3.0 B. Number of each type of organization that you approached. Type of Organization Number a. Teaching or research 4.0 institution (public or private) b. Non-governmental 3.0 organization (NGO) c. Private sector (for profit) 5.0 d. Donor agency 3.0 e. National or central government 4.0 f. Regional or provincial 3.0 government g. Local government 2.0 h. Other (please specify) 2.5 123 C. The number of partnerships that have been established with each type of organization. Type of Organization Number a. Teaching or research 3.0 institution (public or private) b. Non-governmental 2.0 organization (NGO) c. Private sector (for profit) 2.0 d. Donor agency 2.0 e. National or central government 2.5 f. Regional or provincial 1.0 government g. Local government 0.0 h. Other (please specify) 1.0 28. What incentives are there for organizations to enter into partnerships with the GDLN? (Please check all that apply.) n=71 76.1 Link to others working in development 50.7 Organizations complement each other in a given subject area 57.8 Greater visibility/exposure 64.8 Access to World Bank branding 45.1 Access to possible bilateral funding 69.0 Ability to implement development activity on larger scale 8.5 Other (please specify) ________________________ 29. What are the issues that affect an organization's willingness to enter into partnerships with the GDLN? (Please check all that apply.) n=71 71.8 Lack of experience with such partnerships 33.8 Burdensome to enter data into Activity Management System (AMS) 32.4 Difficulties in negotiating the Memorandum of Understanding (MOU) 52.1 Lack of clear communication channels 21.1 Organization's visibility is overshadowed by World Bank 124 12.7 Other (please specify) ________________________ 30. In your experience, what are effective practices for developing partnerships? (Please check all that apply.) n=71 50.7 Maintain contact with alumni 49.3 Build developing country case studies showing specific impact of GDLN 64.8 Develop link between DLC managers and local representatives of development organizations 66.2 Invite donors to GDLN events (not just marketing sessions) 14.1 Other (please specify)________________________ 31. What critical issues related to partnership development should be examined during the course of this study? Reflection 32. Please provide any other reflections or observations that would be relevant for the Evaluation of the Quality and Impact of Programs Facilitated by the GDLN. Responses for this open-ended question are presented below. The text remains unedited except in instances where passages identify individual respondents: in those cases, sections have been deleted to protect confidentiality. GDLN is a capacity building tool which should be used much more strategically than has been the case to date: - The Bank should provide its borrowers with the assurance --as part of its 'Loan&Credit services--that national staff responsible for implementing investment programs will have the necessary qualifications and skills to carry out their responsibilities according to some agreed basic operational standards". In so doing, the Bank (through WBI) would offer a GDLN-based Development Management Certification program to qualify (and at term 'pre-qualify')public/private sector officials to manage the Bank-financed portfolio. Such a training/capacity enhancement program would aim primarily at World Bank project staff BUT the program content would clearly be adaptable as well to the needs of other donor agencies' development assistance programs. While the Bank would initially finance the effort ( absorb or slightly increase the loan management fee to do so) the ultimate goal would be (i)for other donors to purchase the 'service' for their respective counterpart staff and (ii) for the GDLN program , in enhancing the capacity of Development Managers, to become a common national/ regional/global instrument of the international development community -- and not identified as only a World Bank tool. It is estimated that this program --implemented with the appropriate urgency and dedication could utilize some 40% of the DLCs available time. Combined with partnerships to local and regional training institutions, this program would kick-start national processes of capacity enhancement, mostly at the sub-national levels (rural and urban) where some 90% of the demand for specialized development-related skills and competencies can be found." It is unclear what the top priorities are from GDLN management. This creates confusion at the task manager level. It also seems that there is no communication to top GDLN management. I feel that there is some agenda and that my efforts to continue to use and promote the use of GDLN for training is met with hostility. I am sustained by the competent technical staff. Currently, AMS is not practical for our case. Since our projects are demand driven, we always have a discussion with counterpart first. 125 GDLN is a really good tool to supply basic information to a large group of participants and GDLN is a really good tool for class room style teaching. But GDLN will never be a substitute to face-to-face training activities ; the impact as well as the possibilities for interaction are just better with face-to-face. I think more work needs to be done to streamline the procedures and systems currently in place to organize and deliver a GDLN activity. Organizational structure of the GDLN unit in WBI seems to be always changing and never made clear to others in WBI, and the systems don't seem to be effectively addressing the needs and concerns of clients. I have had very positive experiences with the local DLCs and their staff, and above all, I find that using GDLN is very rewarding in terms of the numbers of participants we are able to have in our activities at a relatively low cost, and therefore, the impact of the programs increases. This is abot the 4th time i eiter get interviewed or participate in a survey on GDLN. Everytime i am promised, i will get to see the outcomes. Not only did this never happened, but worse, i actually have not seen that any results (whichever these may have been) have been implemented. I least, not to the extent that it led to visible improvements. Quite on the contrary. AMS is a waste of time, programming is unreliable, and it's been getting worse as more prople use the network. A worrying trend!!! Activities that are the first and only in a particular topic are very difficult to evaluate. Series that attract participants who know of the success of the previous chapter" are often driven to register because they understand the procedures to follow and their participation. Moreover, when the videoconferencing is to present cases or open discussions regarding materials that have been distributed in advance, they tend to be more active and enjoyable than receiving a 30 minutes lesson from a speaker who barely moves. Interaction is the key; better interaction yields wider results (apprehension and application of content learned/discussed)" The DLCs should be encouraged to develop their own business models independent from the World Bank. In my opinion, the problems of the DLCs stem from the fact that we we have kept the DLCs tied to the Bank. There is a huge market out there which can be tapped by these centers if only we let them do it. A number of technical problems have perhaps reduced recently but they are still there and managing this is a key to the success of GDLN. Rather than making always an excuse of unpredictability of technology, there should be a solid system that minimizes technical problems. This survey was too lengthy (number of questions) and the questions themselves were wordy (cannot scan and respond quickly). There is a need for more focus on provision of suitable content The current management of GDLN is working hard, with good results already, on improving the systems. There is still a disconnect sometimes among the different priorities of the Bank regions and because all the regions have a different budget and source of income (and accountabilities), they make of GDLN whatever they want and it's hard to have a united GDLN. Right now, the Regional Coordinators, ISG and WBI seem to be committed to a common management committee, but still, they are really answerable to different structures and different sources of funds. This definitely affects GDLN. - - Also note the 'good' and the 'bad' that GDLN is a presidential initiative. Many of the categories above are not relevant. The potential of GDLN has not been tapped, especially by development practitioners to help them implement their programs. The survey does not highlight this enough. feel free to contact me for an interview client surveys; impact assessment at participant/client level - how has GDLN activity/training changed the mode of operation/reform process in the country I do not find this questionnaire truely reflects the main issues around GDLN. In particular, it does not reflect the use of GDLN Centers to arrange for meetings and informal workshops/knowledge exchage or dialogue. It also mixes up roles - and questions that only particular people should answer, from their particular perspective. It does not clearly differentiate between World Bank teams" (e.g. instructional designers in WBI) and external clients/partners ... There are a number of further design flaws that I'd be happy to elaborate on given the opportunity This survey doens't ask anything about perceptions of cost, about incentives, about comparisons between different delivery modalities. Would have been very intreresting to have seen this data. It also steadfastly ignores the reality that there is huge variance across the Network in terms of focus, engagement, usage etc., and typically the available answers do not give any option for this variety. GDLN is the only hope that the Bank has if we want to save money and be affective in the long - run. GDLN capacity for dissemination of knowledge is not being used to its fullest,because, in general, there are no linkages with domestic networks in each country. 126 I found the question asking which of the substantive areas GDLN could support but limiting the responder to only 5 was completely ridiculous and therefore didn't answer it. Also, the number of partnerships questions made me want to leap out of my chair. Who has time for this?? And as to the priorities for GDLN, three of the items I didn't prioritize because not only are they of low priority, I believe they shouldn't even be considered on the list. Putting all this money into a system because it will be a unique" way to reach policy makers??? Sending singing telegrams would be more "unique" and considerably less expensive. How about "effective"??" Survey had too many questions GDLN should offer a wider range of services to the DLCs in order to meet their business needs. There should also be a stronger mechanism for attracting high quality content to meet DLC needs. Need to find a better, more flexible business model. It will always require some sort of subsidy, but not necessarily through the Bank. Other distance learning consortia may provide imporant clues, models, etc. - - Also, GDLN should be looking at the wide distance learning community in general and look at standards of best practice (GDLN is not), quality assurance mechanisms, pedagogy, etc. GDLN needs to expand its focus to include other technology including e-learning. the survey didn't ask enough questions on content development -- how GDLN to be sustainable for quality contents? Evaluation of Impact needs, to get started, a good needs assesment and later in time the ability to undertake mid term assesment. You didn't ask any questions on the GDLN studios. I suggest that you make a quality assessment and do it regularly. The quality has decreased radically in the past years. Some of them are still very good when some others have lost their motivation to produce a professional and quality product. GDLN need to find the right incentives to keep this important group motivated and needs to assess the quality of the product delivery more often. Client satisfaction study for instance. The quality of what people see on the screen depends a lot on the technician's committment. So please make sure you evaluate that next time. Thanks. The cost for the participants in the developing countries is too high due to high fees charged by partners. There should be an insentive to promote wider participation at lower fees. We should certify our courses to participants. Diplomas. GDLN activities are becoming too expensive, with connecting fees and other cost --even overtime are going to be charged [under planning]. Little things would prove helpful. For instance, courses are offered in many languages, but simple messages flashed across the video-conference network are only in English. Why not make an extra effort and signal respect for the participants and their language by translating basic messages that are subject to repetition, such as 10 Minute Break." - Suggest to DLC control-room staff at HQ to pay attention and not read newspapers while VCs are in progress." The GDLN can be extremely useful in countries where the lending part of the Bank faces obstacles such as suspension of disbursements or local insecurity GDLN is best seen as a strategic supplement to other forms of learning/knowledge disemmination Evaluation process should be much more transparent, fast, and results to be shared with all partners involved both within and outside the Bank mutual financial benefits of GDLN partners Seeing is believing. Technical quality is extremely important, & GDLN sites all suffer equipment & connection problems. Quality is always hampered when PPs are not well prepared, & lack experience in doing DL. This is why GDLN staff must understand network / DLC needs & situations, & must guide new users properly We need more content and be involved more with the WB activities in our regions and country in order to be their training and stategic comunication tool Explore potentials of GDLN regional coordination offices to minimise central burden at GDLN head office Decentralization of the WB Group has meant decentralization to the regions of some GDLN functions. But this should be extended so that country offices are real partners in the development of and use of GDLN events. Générally all is well but inmy short experience the lack of means and funding is a serious problem for to get more participants for sessions 127 Contents would be determined according to the needs of both public and private sector in the countries of DLCs. And Worldbank may use its connections to find and connect the policy makers, ngos, unions etc. both for private and public sector from different countries with the parallel policies and existing relations. AMS is an important tool but has not been market well and people do not know how to operate it thus it is not usued properly. - The financial arrengements among different partners (in and out of the World Bank) should also be facilitated. If WBI would like to introduces a topic WBI should have in their budget funds to cover the centers' costs. - In my opinion GDLN should treat videoconference activities only as an addition to asynchronic Internet based activities. It's important for the GDLN to evaluate the impact of each program or topic on the country development. - It's also important to establish a good governance among the GDLN and in DLC - After the language problems the GDLN must bilt a community of distance learning in Africa and all over the world. For second question from the end, it is difficult to give an answere because it is highly dependant on the content GDLN can offer areas that are listed. The key factor for GDLN success is to provide relevant content and an appropriate and effective delivery. In order to fulfill these requirements, sometimes the only option is to specifically develop content for a particular country. - - Another way to measure the impact of GDLN activities would be to get information about the job performance of participants after attending to a GDLN activity. Feed back from their institutions would be important for this evaluation. - - GDLN should find program partners that have ready learning programmes relevant to needa and announce this at least six months to one year in advance to enable DLCs market. 1. The level of inclusion of local content to complement the global perspective of the programs - 2. More instructional design support to the speakers to maximize time allotment and keep participants from getting an information overload reflecting on exact numbers of events in a context like this makes for poor answers. Keep working abd believing in the GDLN GDLN is a ice-breaking institution. The combination of Hi-Tech with Education is sound initiative. Nevertheless, GDLN is intended to be an organization apart from WB in such a manner, that staff at HQ usually refer to it as an independent org. - I believe that GDLN and DLCs should be strongly tied to the WB, for WB is a strong source of content, expertise. WB is also brand name" that guarantees support from any organization that ever dealt with the WB. GDLN shouldn't create DLC with the idea of letting them on their own after a period of time in order to have DLCs become "auto sustainable". GDLN, DLCs and the WB are non-for-profit organizations devoted to development of countries. Thus, WB financial support is required in order to keep operating." CDLN has no centralized planning, management, and financing. It prevents the project to develop and work effectively. Courses needs more preparatory study in VLE. Video conferences are necesary only for improvement of human communications and social environment. The network that has been developed in the region have allowed the knowledge sharing. It is important that content providers continue the dialogue with the participants after the event, as just filling in the evaluation sheet is not enough. The real impact of the GDLN product should be the application of the knowledge into the real world, and the impact can me measured only by continuous communication with the participants and their feedback. Thank you. GDLN is quite active on operational side of DLC functioning but it is not showing clear intentions about commitments on content side. If not committing content it is going to be really hard for GDLN to request quality control. Provide assistance for linking with centers or service providers in the countries where there is no DLC or WBCO. - More assistance or approach to experts in sharing knowledge in developed countries. - Provide assistance in searching for courses for professional develelopment. A database of all commercial and institutional training providers and courses may be available on GDLN. While courses are being organised to be run at the DLCs, the PPs should provide for how a DLC can meet all its costs and be able to generate income . This is because DLCs are expected to become sustainable. 128 To allow dlcs to have some time the same courses, It will be relevent to have a simultanus traduction (for example french and english countries in Africa). We are not very far advanced in our relationship with GDLN - apart from use of the technical infrastructure, and initial contacts made with the GDLN facility. However we hope to work together in the future. The main problem we have been dealing with are the costs. It has been very difficult to get finantial support although the prospective donors recognize the utility and importance of this type of activities. Most donors believe that the WB should finance this activities if nothing else by reducing costs. - Initially the participants are a bit suspicious on the technology but this is easily overcomed. GDLN is very important for the countries, and should incourage the connection with in-country networks and interested parties promoting the inclusion and not the exclusivity of GDLN to certain institutions/universities. To making some of our monthly coordination meetings more oriented to discussing GDLN policies, its rol as, for example: a)GDLN one of the many KM/KS/DL initiatives supported by WB; b)itself, a sound option to address MDG,WB goals, CAS,etc. Questions in pages 5,6,7,8,12 of this survey are representative enough on this issues! - Besides, this is a very good survey, congratulations! The question in this questionnaire that asks What do you think are the substantive areas as listed in the AMS where GDLN can be used to build capacity in developing countries?" doesn't make any sense. There is no inherent reason why GDLN cannot help equally in all substantive areas -- why would GDLN be able to help any more or less in 'gender' than in 'finance,' or in 'urban development' thank in 'environment/'?" The process of quality evaluation is important but the content provides should be involved the process of evaluation. But after the several quality evaluation probe the quality of program partners, the quality of programs could be rated by the content provider itself. - Also if the program partner asks a co-certificate for the quality assured programs, the process of implementation of issuing the certificate could be shortened or eliminated. - Strategic Planning and Management System I think is neccesary to improve on Quality System related to contents. Upgrade and renewal of the DLC facilities to effectively meet the ever changing requirements for rich-media communications in the real time. First 3 sections on Partnerships Development" is not particularly useful." Again.. what is the purpose of the network? and which kind of sustainability we are looking for the DLCs if their job suppose to be helping the local development? I believe that we should re-think several concepts used as indicators such as utilization rate and cost per participant.. we cannot enforce the idea of having a room full of people will make a DLC sustainable nor our job with higher impact. I am sorry, but I have litte familiarity with GDLN, and so could not answer most of your questions I don't know anything about it other than its name. I do not know how it works I am only familair with one facility - and that has yet to start though the equipment has been installed. I can say however that I am looking forwqard to working with this facility. selection of appropriate sample for evaluation (subjects with enough exposure) Not that much to say The survey is too long My only experience with GLDN is attending yesterday a VC organized by GLDN GDLN ought to reach out to those who would benefit from its activities, first; there is little awareness of its role and functions. There is a feeling of arrogance from top GDLN management that appears to inhibit working closely with the very committed and competent GDLN technical staff Why am I being forced to complete this questionaire? Content providers that deliver content two multilangual audiences have a difficult time delivering the content via videoconferencing. - - Expansion of network specifically in country to allow for broader community access. Example 6 Community Resource Centers in Nigeria with videoconferencing capability and multimedia rooms. These centers utilized in country networks to extend the GDLN point of presence into the country. - - Online workshop amongst partners, DLC's and WBI to highlight successes and discuss challenges. 129 130 ANNEX C: PROTOCOLS Interview Protocol for World Bank HQ Personnel [Introduction: GDLN evaluation focus and process, confidentiality] A. Perspectives on GDLN 1. What is your role here in _______ ? What responsibilities do you have (or have you had) related to GDLN? 2. What is do you see as the purpose of GDLN? How well has it done to date in fulfilling its mission? What do you think needs to be done to accomplish its goals? B. GDLN Activities and Center Operation 1. How would you define a GDLN activity or event? 2. Are you familiar with the operations of GDLN centers? (if no, skip to next section) 3. What conditions or factors are needed for a GDLN center to operate efficiently and effectively? 4. Is there an established communication process between the distance learning centers and GDLN support services or is this developed on a case-by-case basis? 5. To what degree is GDLN Support Services involved with the operation of GDLN centers? (probe re: staffing, marketing, content decisions, partnership development, and reporting) 6. What incentives do GDLN centers have to report their activities or participants to the World Bank? C. Content Development 1. Are you involved or familiar with content development? (if no, skip section) 2. How are decisions made about what new learning activity content to develop? (probe for different ways content development is initiated and factors guiding decisions) 3. It seems to me that the motivations for developing content include designing learning activities that participants demand and will pay for and providing content aligned with 131 the development agenda. Is this accurate? (Is there any conflict between these two motivations and are there other reasons behind content development?) 4. What is the general process for developing new content for a GDLN activity? 5. How are decisions made about pedagogy or learning approaches in the design of the activity? Are there any standard guidelines for this? 6. Is there an emphasis on converting existing content to be used through GDLN? (probe for related issues such as time, resources, relevance, etc.) 7. How do you become familiar with regional or country needs for content? What is the communication channel for this? 8. To what extent are you able to meet the demand for GDLN content? 9. Are there mechanisms or processes in place to ensure that the content developed is of high quality? (probe for standard criteria, review process) 10. What are the other critical issues related to content development that we should learn about? 11. Are there other external high-quality content developers that GDLN relies on? (probe for how established, process, etc.) D. Partnership Development 1. Have you been involved in developing partnerships to support GDLN centers or activities? (If no, skip section. If yes, probe for type: co-sponsors of training or co- financing partners...) 2. How are these partnerships first established or how are potential new partners identified and approached? 3. Can you provide a specific example of a successful partnership that has been established? 4. Why do organizations, companies, or agencies choose to form these partnerships? What are their incentives? 5. Is there any concern that increasing the reliance on partnerships may weaken the focus on the World Bank's development agenda? (i.e. will content become less aligned with World Bank or Millennium Development Goals?) 6. What are the key areas that we should examine regarding partnership development? 132 E. Capacity Enhancement 1. Do you have a perspective on how well GDLN is utilized to meet country development needs? (if not, skip to next section) 2. What is your view on how well GDLN content is aligned with World Bank or Millennium Development Goals? 3. A recent GDLN survey asked center managers about what it means to be involved in the development dialogue in a country. What types of activities or events qualify as such involvement? 4. Can you provide a specific example of how GDLN has led to capacity enhancement at the individual, institutional, or policy level? 5. What role do within-country networks play in developing capacity? 6. What type of participant should be targeted for GDLN events for effective regional capacity enhancement? (probe for employment levels, professions) 7. What are critical issues that we should examine related to capacity enhancement? F. Regional Issues 1. Does your work with GDLN focus on a particular region? (if no, skip section. If yes, identify region) 2. What are the unique challenges or opportunities facing the ________ region that we should be aware of? 3. What are the logistic and political issues in establishing new GDLN centers in this region? What is the process for this? G. Technology 1. Are you familiar with GDLN technology issues, including compatibility and costs? (if no, skip section) 2. What are the issues related to technology that impact the effectiveness or efficiency of GDLN centers? 3. How do individual DLCs negotiate technology costs with the World Bank? 133 4. What are the challenges with technology that occur as GDLN works to link with within- country networks? How are these challenges addressed? 5. Does membership in the GDLN serve to enhance the technical capabilities of existing distance learning centers? How does this occur? H. Reporting Systems Activity Management System (AMS) 1. Who is eligible to propose activities in AMS? 2. What process is used to match proposed activities with potential users? How are costs and logistics negotiated? 3. Is there any quality assurance process for activities that are entered? (probe for review process or standards) 4. To what degree are GDLN activities captured in this system? What types of activities are less likely to be entered? What incentive is there for entering activities? 5. How was AMS developed? How were the drop-down categories established? (probe: themes) 6. What are the plans for improving AMS? What is the timeline for this? (probe for merge with CRS so activities and participants are linked) 7. Does AMS provide a record of past activities--or only from some starting point? 8. What are other issues related to AMS that we should examine during this study? Client Registration System (CRS) 1. How are participants recorded in CRS? What is process? (ask for demonstration if appropriate) 2. Who is eligible to enter or review data in CRS? 3. What types of information is captured about participants? 4. What incentives do GDLN centers have to provide data about participants? 5. How was CRS developed and what plans are in place for system improvements? Who is giving input to this process? 6. What other issues related to this system should we know about in conducting our study? 134 Recommendations 1. Who are the key people to talk to in headquarters about GDLN and its future? 2. What critical factors or issues would you encourage us to specifically examine as we conduct site visits to GDLN centers in LAC, EAP, and AFR? 3. What do you think is the best way to gauge whether GDLN is fulfilling its mission? 4. One of the challenges with a program such as this is sustainability: are factors in place to allow for sustainability? * As appropriate, ask for recommendations of other key people to interview in related areas. 135 PROTOCOL FOR DLC MANAGER Note: Collect data about all the courses offered and participant data. Review confidentiality of interview with manager. 1. Please describe the courses, dialogues, and seminars that you have offered over the years. (Consider volume, topics, purpose, target audience, partners, changes in emphasis over time.) 2. What role do these elements play in the operation of the DLC: technology, the facilities, content (sources?), participants? How satisfied have you been with each of these elements? 3. Only for Brazil: Brazil began participating in the GDLN relatively late, compared to other South American countries. What was the effect of this? What were the advantages and disadvantages? 4. What are the principal needs that [COUNTRY] has in addressing development in the country? Are some needs national and others regional? a. How do you see GDLN helping to address these needs? Where is this tool particularly effective? b. Are there some areas that GDLN is not a solution? c. As a country, is [COUNTRY] ready for change? If so, in what areas? d. Has the Country Assistance Strategy been a driver for how you use or focus the DLC? 5. What do you look for when you recruit partners? What are the strengths of your current partners? What kinds of partners do you want to involve in the future? For what specific purposes? 6. What are the advantages and challenges of using distance learning? If appropriate: What are the advantages of having the Center inside the Bank? What are the disadvantages? 7. What do you see as the future direction of this country's participation in GDLN? 8. Have there been technology problems? Is technology the weakest link? Please talk about the technology issues you face. What are the difficulties? What is the solution? How likely are these issues to be resolved? 9. What is your target audience? How do you market to them? What incentives are there for people to attend DLC events? a. Have you targeted key employers? b. To what extent is the highest level of government aware of GDLN (such as the president and heads of ministries)? What kind of discussions have you had at that level? 136 10. Have you been able to tap into existing networks in [COUNTRY] for dissemination.? Where has it been possible? Where not? 11. To what extent have you been able to work with other [countries in the region/countries you are interested in working with]? What do you think the benefits are? Are there particular topics that are important? What are impediments? 12. How have you used GDLN services? How have you found their support? 13. Please discuss the cost issues as you perceive them. Have you focused on sustainability? 14. What are the issues you deal with regarding personnel? 15. What has been your experience in working with WB headquarters? 16. What has been your experience in working with WB systems? 17. What does capacity enhancement mean for this DLC? Where has it been/should it be applied? Have you attempted to measure it? 18. What conditions have to be present for DLC events to have an impact? Have you made any effort to measure the impact of what you have been doing? Are you interested in doing so? What are the difficulties, from your perspective? 19. Please describe a DLC event that you thought was particularly effective. 137 PARTNER PROTOCOL Introduction to study and describe confidentiality. 1. Can you tell me something about your organization? How long has it been operating? What is its mission? Who contributes to your work? 2. How long have you been working with the GDLN (the Center for Distance Learning in [CITY])? 3. For what reasons did this partnership begin? 4. What have your organization and the Center worked on to date--both in planning and in implementing courses/dialogues? 5. What are the principal needs that [COUNTRY] has in addressing development in the country? Are some needs national and others regional? a. How do you see GDLN helping to address these needs? Where is this tool particularly effective? b. Are there some areas that GDLN is not a solution? c. As a country, is [COUNTRY] ready for change? If so, in what areas? 6. What was you reaction to working with distance learning? Did you have previous experience using this medium? 7. Have your expectations been met? 8. What do you see as the advantages and disadvantages to distance learning? 9. Have you found the GLDN to be technically reliable? What is your assessment of the quality of the content and the delivery of the material? 10. Have you found any difficulties in scheduling time or in working with GLDN systems, such as AMS? 11. How do costs work when you work with GDLN? 12. In general, who is your audience? Who has been attending your events? a. How do you market to them? Have you have any difficulties in recruiting your desired audience? b. Do you attempt to involve high-level officials? If so, has that been possible to do? If so, what are the most compelling reasons that key officials participate? c. What has been your experience with attracting mid-career and senior staff? What are they looking for when they attend? d. Do you have participants who attend multiple events? Do you have some employers who repeatedly send their staffs? 138 13. Do you have any connections with other networks for broader dissemination (within the country or elsewhere)? 14. What criteria do you use to determine if a course or a dialogue is effective? 15. In the learning events you have been associated with, what impacts have you desired? Do you have any evidence that impacts have been achieved? Can you provide specific examples? 16. Where do you think capacity has been enhanced through your events? At the individual level? The institutional level? Nationally? 17. What are your future plans in working with GDLN? 18. Is there anything else we should know? 139 PARTICIPANT PROTOCOL Describe confidential, purpose of interview or focus group. 1. What learning activity or event have you participated in with GDLN? 2. Could you describe this event? What was its format? Probes: when/where/for how long 3. Who else participated? (number level, occupation) 4. a. What motivated you to attend this event? b. Did it meet your expectations? 5. We are interested in learning what you have done as a result of this GDLN experience. How did this activity benefit you individually (such as knowledge, career development)? How did it benefit your institution? 6. Have you stayed in touch with other participants from this event? Probes: nature/purpose of communication 7. Before you participated in this GDLN event, had you had other experiences with distance learning? If yes, please describe. 8. What are the advantages of distance learning versus all face-to-face? Disadvantages? 9. Based on your experience, do you think GDLN can be used effectively to address country development needs? In what areas? 10. What changes would you suggest for GDLN to better fulfill its mission? 140 ANNEX D: PARTICIPANT SURVEY Questionnaire for Past Participants in GDLN Learning Activities The World Bank has requested an independent evaluation of the impact of the Global Development Learning Network (GDLN). This survey is one part of the evaluation. It provides an opportunity for your input into the assessment of GDLN. Macro International, a U.S. consulting firm, is conducting the study. Your individual response will remain confidential and will be combined with all others as part of a final report. N=58 All amounts in % Name (First and Last): ______________________________ 1. What was the most valuable distance learning activity you participated in that involved the Distance Learning Center (DLC) in [country]? _______________________________ 2a. When did that learning activity take place? Month _________ Year ____________ 2b. Where did you attend the learning event? ___________________________________ 3. What is your current job title? ____________________________________________ 4. Did you have the same job when you participated in the learning activity you listed above? n=52 90.4 Yes (Go to 5) 9.6 No (Go to 4a) 4a. If no, what job title did you have at that time? __________________ 5. At that time, in what type of organization did you work? (please choose one) n=53 15.1 Teaching or research institution 37.7 National or central government (public or private) 1.9 Regional or provincial government 9.4 Non-governmental organization 13.2 Local government (NGO) 9.4 Other. Specify: _________________ 3.8 Private sector (for profit) 9.4 Donor agency 6. What motivated you to enroll in the course/activity? (please check all that apply) n=58 29.3 It was my employer's decision to register me. 29.3 It was free or at a reasonable price. 55.2 I thought it would help me do my job better. 65.5 The subject is very important to me and my work. 69.0 I wanted to learn new techniques and strategies in this field. 141 15.5 Other people recommended the course or the Center to me. 39.7 I was anxious to get to know others who are working in the field. 31.0 I was interested in trying distance learning; it was a new experience. 22.4 I liked that I was able to remain at my home and my job while learning. 13.8 Other. Specify: ___________________________________________________ 7. Have you participated in other DLC events? n=53 64.2 Yes (Go to 7a and 7b) 35.8 No (Go to 8) 7a. If yes, how many? _(mean = 3.2, median = 2, n=28)_ 7b. On what topics? ________________________________________________ 8. What components of that most recent DLC activity you attended had the most value for you? (please check all that apply) n=58 60.3 Reading materials distributed 75.9 Expert presenters 82.8 Contributions (discussion and ideas) from other participants 32.8 Action plans or other materials you prepared during the sessions 53.4 Networking with other participants 27.6 Internet resources (special website for participants or recommended sites) 10.3 Newsletters 25.9 Follow-up activities after the event 6.9 Other. Specify: ______________________ 9. For what aspects of your professional development was the learning event most effective? (please check all that apply) n=58 62.1 Providing new knowledge 74.1 Updating my knowledge 56.9 Enhancing my skills which skills? _______________________________ 53.4 Providing new strategies or approaches related to important issues or problems 60.3 Providing contacts with experts or others actively working on the topic of study 51.7 Facilitating the discussion and work on the subject between you and your colleagues 5.2 Other. Specify: ______________________ 10. Overall, how would you rate the quality of the learning event in which you participated? mean = 3.17, n=58 29.3 Very high 60.3 High quality 8.6 Acceptable 1.7 Low quality quality quality 142 11. Did you think that the topics as presented at that event were relevant to your country's development needs? mean = 3.72, n=58 72.4 Yes, very 27.6 Yes, relevant 0.0 Somewhat 0.0 Not relevant relevant relevant 12. Did you participate in the most recent DLC learning activity with other members of your organization or agency? n=58 87.9 Yes (Go to 12a and 12b) 12.1 No (Go to 13) 12a. If yes, how many others attended with you? (mean = 4.9, median = 3, n=51)_ 12b. If yes, did you work together in planning or implementing ideas or strategies from the DLC learning activity after the session(s) were over? n=47 78.7 Yes 21.3 No 13. Did your supervisor attend the DLC learning activity? n=51 49.0 Yes 51.0 No 14. Does your employer frequently or routinely send staff to DLC courses or other activities? n=46 65.2 Yes 34.8 No 15. We are interested in what you did in your work as a result of the DLC learning event you attended. (please check all that apply) n=58 37.9 Conducted additional research on the topic 56.9 Transmitted your new knowledge or skills to others in your organization 34.5 Changed work practices at your organization 15.5 Implemented the planning begun at the learning event 41.4 Formally implemented one or more of the new approaches or strategies learned 46.6 Applied what was learned to local, regional, national, or sector development plans 29.3 Made presentations to others outside your organization based on what you learned 27.6 Produced new written materials based on what you learned 46.6 Continued to work with others you met at the learning event 13.8 Revised policies 25.9 Initiated or modified a project to take new information and approaches into account 3.4 Revised regulations or legislation 143 25.9 Organized a new initiative to promote change 34.5 Developed new alliances or partnerships within your country 25.9 Collaborated with new international partners 3.4 Other. Specify: ______________________________ 16. On a scale of 1 to 5 (low to high) to what extent do you think the DLC learning activity enhanced: (please circle one number) LOW HIGH mean a. your own professional development n=56 1 2 3 4 5 4.02 b. the capacity of your organization n=54 1 2 3 4 5 3.61 c. the development of your country n=54 1 2 3 4 5 3.59 17. Would you participate in other learning activities hosted by the DLC in the future? n=57 94.7 Yes (Go to 17a) 5.3 No (Go to 17b) 17a. If yes, what factors would be important to determining whether you participated? (please check all that apply) n=56 93.1 Content 24.1 What partner organizations were involved 39.7 Time of year 34.5 The experts who were participating 55.2 Length of course 56.9 Cost 20.7 Who the other participants were 6.9 Other. Specify: __________________ 17b. If no, what factors influence that decision? (please check all that apply) Previous activity was not as I expected Cannot or do not want to pay the cost Quality of previous activity was not high I did not like distance learning approach enough 3.4Other. Specify _ n=2_____________ I found the previous activity too technical 18. Would you recommend to your colleagues that they attend a DLC learning activity? n=58 96.6 Yes 3.4 No 0.0 Not Sure 19. What is the highest education level you have attained? (please choose one) n=58 1.7 Secondary school graduate 6.9 Some post-secondary instruction (university or technical institute) 32.8 Bachelor's degree or equivalent 43.1 Master's degree or equivalent 6.9 Professional degree (such as lawyer or doctor) 8.6 Doctoral degree 144 20. How many years of professional work experience do you have? mean = 18, n=57 years 21. In what year were you born? 19 __60___ (average age = 44, n=58) 22. How frequently do you use e-mail or the Internet? n=58 62.1 Every day 10.3 3-4 times 13.8 1-2 times 12.1 Occasionally 1.7 Never per week per week THANK YOU VERY MUCH FOR COMPLETING THIS SURVEY 145 146 ANNEX E: MEASURING THEMATIC AND SECTORAL ALIGNMENT FD Non-compulsory pensions, insurance and The theme and sector coding Sectors Indicate Which Part of Economy Receives contractual savings system: Support Measuring FE Micro- and SME finance · was released on July 5, 2002 in SAP FG Payment systems, securities clearance AGRICULTURE, FISHING, and FORESTRY and settlement Thematic R/3 and BW; AB Agricultural extension and research FZ General finance sector · provides the basis for analyzing and AJ Animal production and reporting on the content of Bank AH Crops activities (e.g., Bank budget allocations HEALTH and OTHER SOCIAL SERVICES AT Forestry to strategic goals and priority sectors); JA Health Sectoral AI Irrigation and drainage JB Other social services · replaces manually maintained AZ General agriculture, fishing and forestry Alignment databases, resulting in institutional cost savings; INDUSTRY and TRADE LAW and JUSTICE and PUBLIC YA Agricultural marketing and trade of Bank · responds to shareholder ADMINISTRATION YB Agro-industry recommendations for standard BC Central government administration reporting. YC Housing construction BE Compulsory pension and unemployment Activities insurance YY Other domestic and international trade Which activities are coded? BG Law and justice YW Other industry YD Petrochemicals and fertilizers · Theme and sector codes are assigned BH Sub-national government administration YZ General industry and trade to investment and adjustment BK Compulsory health finance operations, economic and sector work, BZ General public administration non-lending technical assistance, ENERGY and MINING research, client training and other LA District heating and energy efficiency activities that directly serve the Bank's INFORMATION and COMMUNICATIONS services October 3, 2003 external clients. CA Information technology LB Mining and other extractive CB Media · Activities that serve the Bank's internal LC Oil and gas CD Postal services needs--e.g., y assurance, operations LD Power CT Telecommunications evaluation, Country Assistance LE Renewable energy Strategies, Sector Strategy Papers, CZ General information and communications LZ General energy knowledge management, training of Bank staff--are not coded. EDUCATION TRANSPORTATION Recent changes to themes and EL Adult literacy/non-formal education TV Aviation sectors: EC Pre-primary education TP Ports, waterways and shipping EP Primary education TW Railways · Effective October 3, 2003, the following ES Secondary education TA Roads and highways changes were introduced: ET Tertiary education TZ General transportation OPCS ADVISORY EV Vocational training o new theme codes 88, "HIV/AIDS," EZ General education SERVICE (X88627) and "Injuries and non- communicable diseases"; WATER, SANITATION, and FLOOD PROTECTION o name change of theme 64 to "Other FINANCE WD Flood protection communicable diseases"; FA Banking WA Sanitation FK Capital markets o new sector code BK, "Compulsory WS Sewerage FB Non-compulsory health finance health finance"; WB Solid waste management FC Housing finance and real estate markets WC Water supply o name change of sector code FB to WZ General water, sanitation and flood "Non-compulsory health finance". protection Themes Correspond to the Goals of Bank Activities Reports TRADE AND INTEGRATION URBAN DEVELOPMENT 45 Export development and 71 Access to urban services for the poor SAP R/3 and Business Warehouse ECONOMIC MANAGEMENT competitiveness provide options for analytical reporting by 72 Municipal finance sector or theme. 20 Analysis of economic growth 46 International financial architecture 73 Municipal governance and institution 21 Debt management and fiscal 47 Regional integration building More Questions? sustainability 48 Technology diffusion 74 Other urban development 22 Economic statistics, modeling and There is additional information available 49 Trade facilitation and market access forecasting on using the new coding schemes. For 50 Other trade and integration example, Guidelines for Using the Theme 23 Macroeconomic management RURAL DEVELOPMENT Codes and Guidelines for Using the 24 Other economic management 75 Rural markets SOCIAL PROTECTION AND RISK 76 Rural non-farm income generation Sector Codes provide greater detail on the MANAGEMENT 77 Rural policies and institutions scope of sectors and themes. The PUBLIC SECTOR GOVERNANCE 52 Natural disaster management 78 Rural services and infrastructure Guidelines include: 25 Administrative and civil service 53 Poverty strategy, analysis, and 79 Other rural development · full name reform monitoring 26 Decentralization 54 Social risk coping · code 27 Public expenditure, financial 87 Social risk mitigation ENVIRONMENT AND NATURAL management, and procurement RESOURCES MANAGEMENT · description of what subjects are 51 Social risk reduction included and excluded 28 Tax policy and administration 80 Biodiversity 55 Vulnerability assessment and 29 Other accountability/anti-corruption monitoring 81 Climate change · responsible Sector Board 30 Other public sector governance 56 Other social protection and risk 82 Environmental policies and institutions management 83 Land management · contact person for inquiries 84 Pollution management and RULE OF LAW Copies of both documents and other environmental health 31 Access to law and justice SOCIAL DEVELOPMENT, GENDER AND useful information are available at the 85 Water resources management OPCS website http://sectors or call OPCS 32 Judicial and other dispute resolution INCLUSION 86 Other environment and natural Advisory Services at x88627. mechanisms 57 Participation and civic engagement resources management 33 Law reform 58 Conflict prevention and post-conflict 34 Legal institutions for a market reconstruction economy 59 Gender 35 Legal services 60 Indigenous peoples 36 Personal and property rights 61 Social analysis and monitoring 37 Other rule of law 62 Other social development FINANCIAL AND PRIVATE SECTOR HUMAN DEVELOPMENT DEVELOPMENT 63 Child health 38 Corporate governance 64 Other communicable diseases 39 Infrastructure services for private 65 Education for all sector development 66 Education for the knowledge economy 40 Regulation and competition policy 67 Health system performance 41 Small and medium enterprise support 68 Nutrition and food security 42 Standards and financial reporting 69 Population and reproductive health 43 State enterprise/bank restructuring 70 Other human development and privatization 88 HIV/AIDS 44 Other financial and private sector 89 Injuries and non-communicable development diseases ANNEX F: GEOGRAPHIC DISTRIBUTION OF GDLN CENTERS Bosnia & Herzegovina Bulgaria Latvia Belgium Lithuania (2) France Poland Germany Russia (3) Canada (2) Netherlands Romania Washington, DC Portugal Slovenia Total=3 Spain Turkey (2)(+1*) Italy Ukraine Bolivia (UK*) (Serbia & Montenegro*) Brazil Total=7 Total=14 Afghanistan China (3) Chile India (+1*) Japan Costa Rica (2) Sri Lanka South Korea Colombia Benin Jordan Total=3 (Malaysia*) Dominican Republic Côte d'Ivoire Egypt Mongolia Ecuador Ethiopia Saudi Arabia Philippines El Salvador Ghana(+1*) Total=3 Singapore Guatemala Mauritania Thailand Honduras Mozambique Timor Leste Mexico (2) Senegal Vietnam Nicaragua Tanzania Australia Panama Uganda Total=12 Peru Total=9 Venezuela Total=68 Total=17 * DLCs that are part of the British Council's Network of Knowledge Learning Centers and therefore connect to GDLN